Provision BDS

From Genesys Documentation
Jump to: navigation, search
This topic is part of the manual Billing Data Service Private Edition Guide for version Current of Billing Data Service.

Learn how to provision Billing Data Service (BDS).

Related documentation:

Tenant Provisioning

Create configmap


  1. Download bds-config archive:
    pureengage-helm-production-local.jfrog.io/bds-config-100.0.003+0029-pe
    The archive contains two tenants examples in the result folder:
    • multiregion configuration file: config-t100-100.json
    • single-region configuration file: config-t101-101.json
    This procedure demonstrates a multi-region scenario.
  2. Extract archive:
    developer@docker:/media/sf_shared/OpenShift$ tar xvzf bds-config-100.0.003+0029-pe.tgz
    bds-config/Chart.yaml
    bds-config/values.yaml
    bds-config/templates/bds-configmap.yml
    bds-config/result/config-t100-100.json
    bds-config/result/config-t101-101.json 
    bds-config/result/gvars.py
    bds-config/result/main.json
  3. Update bds-config/result/config-t100-100.json:
    1. In the "globals" section:
      1. Add "gws" section:
        "gws": {
                    "host": "BDS_CFG_GWS_HOST_PLACEHOLDER",
                    "auth_host": "BDS_CFG_GWS_AUTH_HOST_PLACEHOLDER",
                    "grant_type": "client_credentials",
                    "client_id": "BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER",
                    "client_secret": "BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER"
             }
      2. Add/modify "SourceId": "MultiCloudPE". If the modeValue = MULTICLOUD_PE, the SourceId value is by default set as MultiCloudPE in the globals section.
    2. In tenants/<tenant_name>/, populate contact_center_id with correct value. You can get this value by running a command specified in Get contact center ID from GWS.
      For example: "contact_center_id": "e01f1720-c1ec-11eb-a4b5-53da51249f17"
    3. Configure your Genesys Info Mart endpoint by replacing corresponding placeholders. This step is optional, but if you do not complete it, then you must populate placeholders’ values in bds-manual-secrets (see Create secrets)
      Example:
      "gimdb": {
          "db_type": "postgre",
          "driver_name": "PostgreSQL",
          "server": "BDS_CFG_GLOBALS_GIM_DB_HOST_PLACEHOLDER",
          "port": 5432,
          "database": "BDS_CFG_GLOBALS_GIM_DB_NAME_PLACEHOLDER",
          "username": "BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER",
          "password": "BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER"
      }
      Where,
      • db_type supports sql_server, postgre, and oracle databases
      • driver_name must be FreeTDS, PostgreSQL, and OracleODBC-12.1 correspondingly for sql_server, postgre, and oracle databases.
      • server supports both FQDN and IP values.
    4. Configure your GVP endpoint in each region of your tenant by replacing corresponding placeholders. This step is optional, but if you do not complete it, then you must populate placeholders’ values in bds-manual-secrets (see Create secrets).
      Example:
      "gvp": {
          "gvp_primary_rs_name": "GVP",
          "db_type": "sql_server",
          "driver_name": "FreeTDS",
          "server": "BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_HOST_PLACEHOLDER",
          "port": 1433,
          "database": "BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_NAME_PLACEHOLDER",
          "username": "BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_USR_PLACEHOLDER",
          "password": "BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_PSW_PLACEHOLDER"
      }
    5. Add Consul endpoints in each region of your tenant (For example: /tenants/<tenant_name>/regions/us/ash/):
      "consul": {
                                  "url_api": "BDS_CFG_CONSUL_TOKEN_PLACEHOLDER",
                                  "token": "BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER"
               }
    6. Configure the production_date according to your business needs. Genesys recommends to set the production date as one day prior to the date in which you want to go-live.
      For example, if you want to go-live on January 25, 2022, then you must set the production date as at least one day prior to January 25, that is, January 24, 2022 or earlier dates.
      "production_date": "2022-01-24"
  4. Execute the following command to create the configmap file:
    helm install --debug bds-config bds-config  --version=100.0.003+0029-pe -n bds
  5. Manually create extract and transform folders on shared storage which is mounted by PVC.

Create secrets

This section describes how to create manual secrets.
Important
BDS supports multi-region deployment. If you are configuring using the multi-regional configuration file, you must define the multi-region specific placeholders in the configuration file and secret key-value pairs in the secret file.

OpenShift

  1. Create file with secrets (change all * to your real variables):

    cat <EOF > ./bds-manual-secrets.yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: bds-manual-secrets
      namespace: bds
    type: Opaque
    data:
      BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER:
      BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER:
      BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER:
      BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER:
      BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER:
      BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER:
      BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER:
      BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER:
      BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER:
      BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER:
      BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER:
      BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_USR_PLACEHOLDER:
      BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_PSW_PLACEHOLDER:
      BDS_CFG_CONSUL_TOKEN_PLACEHOLDER:
      BDS_CFG_BDS_DEV_GWS_CLIENTID_PLACEHOLDER:
      BDS_CFG_BDS_DEV_GWS_CLIENT_SECRET_PLACEHOLDER:
    stringData:
      BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER: **
      BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER: **
      BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER: **
      BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_USR_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_PSW_PLACEHOLDER: **
      BDS_CFG_CONSUL_TOKEN_PLACEHOLDER: **
      BDS_CFG_BDS_DEV_GWS_CLIENTID_PLACEHOLDER: **
      BDS_CFG_BDS_DEV_GWS_CLIENT_SECRET_PLACEHOLDER: **
    EOF
    Important
    In the secrets manifest file, specify the base64 encoded secrets values in the data section and the plain text values in the stringData section, which will be converted into base64 string after the file is created. The secrets values are validated either from the data or stringData sections or both. However, in Kubernetes, you cannot leave any values as blank in the secret manifest file.
  2. Install this secret to OpenShift by executing the following command:
    developer@docker:~$ oc create -f bds-manual-secrets.yaml
    secret/bds-manual-secrets created
  3. Since manual secrets are used, you must override bdsApp.secrets.gim.mounts.name, bdsApp.secrets.gvp.mounts.name, bdsApp.secrets.consul.mounts.name to " ". For more details on these parameters, refer Configure BDS.

Kubernetes

  1. Create file with secrets (change all * to your real variables).
    cat <EOF > ./bds-manual-secrets.yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: bds-manual-secrets
      namespace: bds
    type: Opaque
    data:
      BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER:
      BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER:
      BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER:
      BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER:
      BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER:
      BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER:
      BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER:
      BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER:
      BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER:
      BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER:
      BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER:
      BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_USR_PLACEHOLDER:
      BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_PSW_PLACEHOLDER:
      BDS_CFG_CONSUL_TOKEN_PLACEHOLDER:
      BDS_CFG_BDS_DEV_GWS_CLIENTID_PLACEHOLDER:
      BDS_CFG_BDS_DEV_GWS_CLIENT_SECRET_PLACEHOLDER:
    stringData:
      BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER: **
      BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER: **
      BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER: **
      BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_USR_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_PSW_PLACEHOLDER: **
      BDS_CFG_CONSUL_TOKEN_PLACEHOLDER: **
      BDS_CFG_BDS_DEV_GWS_CLIENTID_PLACEHOLDER: **
      BDS_CFG_BDS_DEV_GWS_CLIENT_SECRET_PLACEHOLDER: **
    EOF
    Important
    In the secrets manifest file, specify the base64 encoded secrets values in the data section and the plain text values in the stringData section, which will be converted into base64 string after the file is created. The secrets values are validated either from the data or stringData sections or both. However, in Kubernetes, you cannot leave any values as blank in the secret manifest file.
  2. Install this secret to the Kubernetes cluster by executing the following command:
    kubectl create -f bds-manual-secrets.yaml 
    secret/bds-manual-secrets created
  3. Since manual secrets are used, you must override bdsApp.secrets.gim.mounts.name, bdsApp.secrets.gvp.mounts.name, and bdsApp.secrets.consul.mounts.name to " ". For more details on these parameters, refer Configure BDS.

GKE

  1. Create file with secrets (change all * to your real variables):
    cat <EOF > ./bds-manual-secrets.yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: bds-manual-secrets
      namespace: bds
    type: Opaque
    stringData:
      BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_NAME_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER: **
      BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER: **
      BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GIM_DB_HOST_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GIM_DB_NAME_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER: **
      BDS_CFG_CONTACT_CENTER_ID_PLACEHOLDER: **
      BDS_CFG_SOURCEID_PLACEHOLDER: **
      BDS_CFG_CS_CONSUL_PLACEHOLDER: **
      BDS_CFG_GWS_HOST_PLACEHOLDER: **
      BDS_CFG_GWS_AUTH_HOST_PLACEHOLDER: **
      BDS_CFG_BDS_GWS_CLIENTID_PLACEHOLDER: **
      BDS_CFG_BDS_GWS_CLIENT_SECRET_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_HOST_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_NAME_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_USR_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_PSW_PLACEHOLDER: **
      BDS_CFG_CONSUL_TOKEN_PLACEHOLDER: **
      BDS_CFG_CONSUL_URL_API_PLACEHOLDER: **
      BDS_CFG_GIR_ES_PLACEHOLDER: **
      BDS_CFG_CONSUL_URL_API_REGION_PLACEHOLDER: **
      BDS_CFG_CONSUL_WESTUS2_TOKEN_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_WESTUS2_HOST_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_WESTUS2_NAME_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_WESTUS2_PSW_PLACEHOLDER: **
      BDS_CFG_GLOBALS_GVP_DB_WESTUS2_USR_PLACEHOLDER: **
      BDS_CFG_LEGACY_GLOBALS_SFTP_PSW_PLACEHOLDER: **
      BDS_CFG_LEGACY_GLOBALS_SFTP_USR_PLACEHOLDER: **
      BDS_CFG_SFTP_HOST_PLACEHOLDER: **
      BDS_CFG_SFTP_PATH_PLACEHOLDER: **
  2. Install this secret to your GKE2-2 cluster by executing the following command:
    kubectl create -f bds-manual-secrets.yaml
  3. Since manual secrets are used, you must override bdsApp.secrets.gim.mounts.name, bdsApp.secrets.gvp.mounts.name, bdsApp.secrets.consul.mounts.name to " ". For more details on these parameters, refer Configure BDS.

Create StorageClass, Persistent Volume, and Persistent Volume Claim


OpenShift

  1. Create Storage Class (only if your configuration requires it):
    1. Create file bds-storageclass.yaml by referring the following sample code:
      allowVolumeExpansion: true
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
        name: < storage-class-name >
      parameters:
        kind: Managed
        storageaccounttype: Premium_LRS
      provisioner: < provisioner >
      reclaimPolicy: Retain
      volumeBindingMode: Immediate
    2. Apply this yaml:
      oc create -f bds-storageclass.yaml
  2. Create Persistent Volume (only if your configuration requires it):
    1. Create file bds-pv.yaml by referring the following sample code:
      apiVersion: v1
      kind: PersistentVolume
      metadata:
        name: < pv-name >
      spec:
        accessModes:
        - ReadWriteOnce
        capacity:
          storage: 10Gi
        persistentVolumeReclaimPolicy: Retain
        storageClassName: < storage-class-name >
        volumeMode: Filesystem
    2. Apply this yaml:
      oc create -f bds-pv.yaml
  3. Create Persistent Volume Claim:
    1. Create file bds-pvc.yaml by referring the following sample code:
      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: < pvc-name >
        namespace: < namespace >
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: < storage-class-name >
        volumeMode: Filesystem
    2. Apply this yaml:
      oc create -f bds-pvc.yaml

Kubernetes

  1. Create Persistent Volume (only if your configuration requires it).
    1. Create file pv.yaml by referring the following sample code:
      cat <EOF > ./pv.yaml 
      apiVersion: v1 
      kind: PersistentVolume 
      metadata: 
        name: bds-pv 
        labels: 
          type: local 
      spec: 
        storageClassName: manual 
        capacity: 
          storage: 10Gi 
        accessModes: 
          - ReadWriteOnce 
        hostPath: 
          path: "/genesys/data" 
      EOF
    2. Apply this yaml:
      kubectl create –f pv.yaml
  2. Create Persistent Volume Claim.
    1. Create file pvc.yaml by referring the following sample code:
      cat <EOF > ./pvc.yaml 
      apiVersion: v1 
      kind: PersistentVolumeClaim 
      metadata: 
        name: bds-pvc 
        namespace: bds 
      spec: 
        storageClassName: manual 
        accessModes: 
          - ReadWriteOnce 
        resources: 
          requests: 
            storage: 3Gi 
      EOF
    2. Apply this yaml:
      kubectl create –f pvc.yaml

GKE

  1. Create StorageClass and Persistent Volume, (only if your configuration requires it). If they already exists, then skip to the next step.
  2. Create Persistent Volume Claim.
  3. Create file bds-pvc.yaml by referring the following sample code:
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
    name: bds-pvc
    namespace: bds
    spec:
    accessModes:
    - ReadWriteMany
    resources:
    requests:
    storage: 5Gi
    storageClassName: nfs-client
    volumeMode: Filesystem
  4. Apply bds-pvc.yaml, by executing the following command:.
    kubectl apply -f  bds-pvc.yaml -n bds

Create watermarks on persistent volume


The extract, transform, and load processes in BDS relies on watermark dates. Typically, the watermark date is either the agreed go-live date, or a date that is one day earlier than the date when data exists in all relevant data sources.

Watermark dates are automatically created for BDS persistence volume. You can also manually update the watermark in scenarios where you want to reprocess or catch up the data.

Important
Before creating a watermark file for BDS, a persistence volume must be already provisioned.

To automatically create watermarks, run BDS. Running BDS creates the watermark file and updates all the watermark related fields such as extract_watermark, transform_watermark, and load_watermark with the production date. Note that the production date is configured as part of Tenant provisioning.

Important
Versions prior to 100.0.002.0020 require watermarks file to be created manually. If you are upgrading from previous versions to 100.0.002.0020 or higher, and if the watermark file doesn't exist, then running BDS automatically creates the watermark file and updates the watermark fields with the production date. For BDS to process the data on the go-live date, make sure you set the production_date as at least one day prior to the go-live date.

To manually create watermarks, perform the following steps:

  1. Create the pod yaml by referring the following sample code.
    cat <EOF > ./pv-static.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: static-pv
    spec:
      volumes:
        - name: pv-test
          persistentVolumeClaim:
            claimName: bds-pvc
      containers:
        - name: test
          image: ubuntu
          command: [ "/bin/sh" ]
          args: [ "-c", "while true; do echo hello; sleep 10;done" ]
          volumeMounts:
            - mountPath: "/tmp"
              name: pv-test
    EOF
  2. Execute the following commands to verify the status of the pod created, create the watermarks folder, and create the tenant json file.
    developer@docker:~/OpenShift$ kubectl apply -f pv-static.yaml -n bds
    pod/static-pv created
    developer@docker:~/OpenShift$ kubectl get pod static-pv -n bds
    NAME        READY   STATUS    RESTARTS   AGE
    static-pv   1/1     Running   0          68s
    developer@docker:~/OpenShift$ kubectl exec  -n bds -it static-pv bash
    1000670000@static-pv:/$ cd /tmp
    1000670000@static-pv:/$ mkdir watermarks
    1000670000@static-pv:/tmp$ cd watermarks/
    1000670000@static-pv:/tmp/watermarks$  echo -e "{ \n\t\"extract_watermark\": \"YYYY-MM-DD\",\n\t\"transform_watermark\": \"YYYY-MM-DD\",\n\t\"load_watermark\": \"YYYY-MM-DD\"\n}" > <tenant_name>.json
    Where,
    • extract_watermark - The last successful extraction date in YYYY-MM-DD format, should be greater than or equal to the production date.
    • transform_watermark - The last successful transformation date in YYYY-MM-DD format, should be less than or equal to the extract watermark, and greater than the production date.
    • load_watermark - The last successful load date in YYYY-MM-DD format, should be less than or equal to the transform watermark, and greater than the production date.
    • <tenant_name> - The name of the tenant as it is configured in the Config.json file for BDS. A watermark file will be created in the json format for the tenant with the name specified here, for example, MultiRegionExampleTenant.json.
  3. Execute the following command to verify the watermark values created in the tenant's json file. You will see a result similar to the following example. In this example, MultiRegionExampleTenant.json is the tenant file that contains the extract, transform, and load watermark values for the tenant.
    cd /tmp/watermarks/
    MultiRegionExampleTenant.json
    {
    "extract_watermark": "2021-06-27",
    "transform_watermark": "2021-06-27",
    "load_watermark": "2021-06-27"
    }
  4. Exit and remove pod.


Retrieved from "https://all.docs.genesys.com/BDS/Current/BDSPEGuide/Provision (2022-05-20 21:23:24)"