Provision BDS

From Genesys Documentation
Jump to: navigation, search
This topic is part of the manual Billing Data Service Private Edition Guide for version Current of Billing Data Service.

Learn how to provision Billing Data Service (BDS).

Related documentation:

Tenant Provisioning

This page describes the following provisioning procedures:

Create configmap


  1. Download bds-config archive: 
    pureengage-helm-production-local.jfrog.io/bds-config-100.0.003+0029-pe
    The archive contains two tenants examples in the result folder:
    • multiregion configuration file: config-t100-100.json
    • single-region configuration file: config-t101-101.json
    This procedure demonstrates a multi-region scenario.
  2. Extract archive: 
    developer@docker:/media/sf_shared/OpenShift$ tar xvzf bds-config-100.0.003+0029-pe.tgz
bds-config/Chart.yaml
bds-config/values.yaml
bds-config/templates/bds-configmap.yml
bds-config/result/config-t100-100.json
bds-config/result/config-t101-101.json 
bds-config/result/gvars.py
bds-config/result/main.json
  3. Update bds-config/result/config-t100-100.json:
    1. In the "globals" section:
      1. Add "gws" section: 
        "gws": {
            "host": "BDS_CFG_GWS_HOST_PLACEHOLDER",
            "auth_host": "BDS_CFG_GWS_AUTH_HOST_PLACEHOLDER",
            "grant_type": "client_credentials",
            "client_id": "BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER",
            "client_secret": "BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER"
     }
      2. Add/modify "SourceId": "MultiCloudPE". If the modeValue = MULTICLOUD_PE, the SourceId value is by default set as MultiCloudPE in the globals section.
    2. In tenants/<tenant_name>/, populate contact_center_id with correct value. You can get this value by running a command specified in Get contact center ID from GWS.
      For example: "contact_center_id": "e01f1720-c1ec-11eb-a4b5-53da51249f17"
    3. Configure your Genesys Info Mart endpoint by replacing corresponding placeholders. This step is optional, but if you do not complete it, then you must populate placeholders’ values in bds-manual-secrets (see Create secrets)
      Example:
      "gimdb": {
    "db_type": "postgre",
    "driver_name": "PostgreSQL",
    "server": "BDS_CFG_GLOBALS_GIM_DB_HOST_PLACEHOLDER",
    "port": 5432,
    "database": "BDS_CFG_GLOBALS_GIM_DB_NAME_PLACEHOLDER",
    "username": "BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER",
    "password": "BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER"
}
      Where,
      • db_type supports sql_server, postgre, and oracle databases
      • driver_name must be FreeTDS, PostgreSQL, and OracleODBC-12.1 correspondingly for sql_server, postgre, and oracle databases.
      • server supports both FQDN and IP values.
    4. Configure your GVP endpoint in each region of your tenant by replacing corresponding placeholders. This step is optional, but if you do not complete it, then you must populate placeholders’ values in bds-manual-secrets (see Create secrets).
      Example:
      "gvp": {
    "gvp_primary_rs_name": "GVP",
    "db_type": "sql_server",
    "driver_name": "FreeTDS",
    "server": "BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_HOST_PLACEHOLDER",
    "port": 1433,
    "database": "BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_NAME_PLACEHOLDER",
    "username": "BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_USR_PLACEHOLDER",
    "password": "BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_PSW_PLACEHOLDER"
}
    5. Add Consul endpoints in each region of your tenant (For example: /tenants/<tenant_name>/regions/us/ash/): 
      "consul": {
                            "url_api": "BDS_CFG_CONSUL_TOKEN_PLACEHOLDER",
                            "token": "BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER"
         }
    6. Configure the production_date according to your business needs. Genesys recommends to set the production date as one day prior to the date in which you want to go-live.
      For example, if you want to go-live on January 25, 2022, then you must set the production date as at least one day prior to January 25, that is, January 24, 2022 or earlier dates.
      "production_date": "2022-01-24"
  4. Execute the following command to create the configmap file: 
    helm install --debug bds-config bds-config  --version=100.0.003+0029-pe -n bds
  5. Manually create extract and transform folders on shared storage which is mounted by PVC.

Create secrets

This section describes how to create manual secrets.
Important
BDS supports multi-region deployment. If you are configuring using the multi-regional configuration file, you must define the multi-region specific placeholders in the configuration file and secret key-value pairs in the secret file.

OpenShift

  1. Create file with secrets (change all * to your real variables): 

    cat <EOF > ./bds-manual-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: bds-manual-secrets
  namespace: bds
type: Opaque
data:
  BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER:
  BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER:
  BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER:
  BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER:
  BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER:
  BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER:
  BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER:
  BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER:
  BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER:
  BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER:
  BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER:
  BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_USR_PLACEHOLDER:
  BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_PSW_PLACEHOLDER:
  BDS_CFG_CONSUL_TOKEN_PLACEHOLDER:
  BDS_CFG_BDS_DEV_GWS_CLIENTID_PLACEHOLDER:
  BDS_CFG_BDS_DEV_GWS_CLIENT_SECRET_PLACEHOLDER:
stringData:
  BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER: **
  BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER: **
  BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER: **
  BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_USR_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_PSW_PLACEHOLDER: **
  BDS_CFG_CONSUL_TOKEN_PLACEHOLDER: **
  BDS_CFG_BDS_DEV_GWS_CLIENTID_PLACEHOLDER: **
  BDS_CFG_BDS_DEV_GWS_CLIENT_SECRET_PLACEHOLDER: **
EOF
    Important
    In the secrets manifest file, specify the base64 encoded secrets values in the data section and the plain text values in the stringData section, which will be converted into base64 string after the file is created. The secrets values are validated either from the data or stringData sections or both. However, in Kubernetes, you cannot leave any values as blank in the secret manifest file.
  2. Install this secret to OpenShift by executing the following command: 
    developer@docker:~$ oc create -f bds-manual-secrets.yaml
secret/bds-manual-secrets created
  3. Since manual secrets are used, you must override bdsApp.secrets.gim.mounts.name, bdsApp.secrets.gvp.mounts.name, bdsApp.secrets.consul.mounts.name to " ". For more details on these parameters, refer Configure BDS.

Kubernetes

  1. Create file with secrets (change all * to your real variables). 
    cat <EOF > ./bds-manual-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: bds-manual-secrets
  namespace: bds
type: Opaque
data:
  BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER:
  BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER:
  BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER:
  BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER:
  BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER:
  BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER:
  BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER:
  BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER:
  BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER:
  BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER:
  BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER:
  BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_USR_PLACEHOLDER:
  BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_PSW_PLACEHOLDER:
  BDS_CFG_CONSUL_TOKEN_PLACEHOLDER:
  BDS_CFG_BDS_DEV_GWS_CLIENTID_PLACEHOLDER:
  BDS_CFG_BDS_DEV_GWS_CLIENT_SECRET_PLACEHOLDER:
stringData:
  BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER: **
  BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER: **
  BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER: **
  BDS_CFG_CONSUL_GLOBALS_TOKEN_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_USR_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PL_WESTUS2_PSW_PLACEHOLDER: **
  BDS_CFG_CONSUL_TOKEN_PLACEHOLDER: **
  BDS_CFG_BDS_DEV_GWS_CLIENTID_PLACEHOLDER: **
  BDS_CFG_BDS_DEV_GWS_CLIENT_SECRET_PLACEHOLDER: **
EOF
    Important
    In the secrets manifest file, specify the base64 encoded secrets values in the data section and the plain text values in the stringData section, which will be converted into base64 string after the file is created. The secrets values are validated either from the data or stringData sections or both. However, in Kubernetes, you cannot leave any values as blank in the secret manifest file.
  2. Install this secret to the Kubernetes cluster by executing the following command: 
    kubectl create -f bds-manual-secrets.yaml 
secret/bds-manual-secrets created
  3. Since manual secrets are used, you must override bdsApp.secrets.gim.mounts.name, bdsApp.secrets.gvp.mounts.name, and bdsApp.secrets.consul.mounts.name to " ". For more details on these parameters, refer Configure BDS.

GKE

  1. Create file with secrets (change all * to your real variables): 
    cat <EOF > ./bds-manual-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: bds-manual-secrets
  namespace: bds
type: Opaque
stringData:
  BDS_CFG_GLOBALS_GVP_DB_USER_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_NAME_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_USER_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_GVP_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_CME_GVP_USER_PLACEHOLDER: **
  BDS_CFG_BDS_GLOBALS_GWS_CLIENTID_PLACEHOLDER: **
  BDS_CFG_BDS_GLOBALS_GWS_CLIENT_SECRET_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GIM_DB_HOST_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GIM_DB_NAME_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GIM_DB_USR_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GIM_DB_PSW_PLACEHOLDER: **
  BDS_CFG_CONTACT_CENTER_ID_PLACEHOLDER: **
  BDS_CFG_SOURCEID_PLACEHOLDER: **
  BDS_CFG_CS_CONSUL_PLACEHOLDER: **
  BDS_CFG_GWS_HOST_PLACEHOLDER: **
  BDS_CFG_GWS_AUTH_HOST_PLACEHOLDER: **
  BDS_CFG_BDS_GWS_CLIENTID_PLACEHOLDER: **
  BDS_CFG_BDS_GWS_CLIENT_SECRET_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_HOST_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_NAME_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_USR_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_PL_EASTUS2_PSW_PLACEHOLDER: **
  BDS_CFG_CONSUL_TOKEN_PLACEHOLDER: **
  BDS_CFG_CONSUL_URL_API_PLACEHOLDER: **
  BDS_CFG_GIR_ES_PLACEHOLDER: **
  BDS_CFG_CONSUL_URL_API_REGION_PLACEHOLDER: **
  BDS_CFG_CONSUL_WESTUS2_TOKEN_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_WESTUS2_HOST_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_WESTUS2_NAME_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_WESTUS2_PSW_PLACEHOLDER: **
  BDS_CFG_GLOBALS_GVP_DB_WESTUS2_USR_PLACEHOLDER: **
  BDS_CFG_LEGACY_GLOBALS_SFTP_PSW_PLACEHOLDER: **
  BDS_CFG_LEGACY_GLOBALS_SFTP_USR_PLACEHOLDER: **
  BDS_CFG_SFTP_HOST_PLACEHOLDER: **
  BDS_CFG_SFTP_PATH_PLACEHOLDER: **
  2. Install this secret to your GKE2-2 cluster by executing the following command: 
    kubectl create -f bds-manual-secrets.yaml
  3. Since manual secrets are used, you must override bdsApp.secrets.gim.mounts.name, bdsApp.secrets.gvp.mounts.name, bdsApp.secrets.consul.mounts.name to " ". For more details on these parameters, refer Configure BDS.

Create StorageClass, Persistent Volume, and Persistent Volume Claim


OpenShift

  1. Create Storage Class (only if your configuration requires it):
    1. Create file bds-storageclass.yaml by referring the following sample code: 
      allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: < storage-class-name >
parameters:
  kind: Managed
  storageaccounttype: Premium_LRS
provisioner: < provisioner >
reclaimPolicy: Retain
volumeBindingMode: Immediate
    2. Apply this yaml: 
      oc create -f bds-storageclass.yaml
  2. Create Persistent Volume (only if your configuration requires it):
    1. Create file bds-pv.yaml by referring the following sample code: 
      apiVersion: v1
kind: PersistentVolume
metadata:
  name: < pv-name >
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: < storage-class-name >
  volumeMode: Filesystem
    2. Apply this yaml: 
      oc create -f bds-pv.yaml
  3. Create Persistent Volume Claim:
    1. Create file bds-pvc.yaml by referring the following sample code: 
      apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: < pvc-name >
  namespace: < namespace >
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: < storage-class-name >
  volumeMode: Filesystem
    2. Apply this yaml: 
      oc create -f bds-pvc.yaml

Kubernetes

  1. Create Persistent Volume (only if your configuration requires it).
    1. Create file pv.yaml by referring the following sample code: 
      cat <EOF > ./pv.yaml 
apiVersion: v1 
kind: PersistentVolume 
metadata: 
  name: bds-pv 
  labels: 
    type: local 
spec: 
  storageClassName: manual 
  capacity: 
    storage: 10Gi 
  accessModes: 
    - ReadWriteOnce 
  hostPath: 
    path: "/genesys/data" 
EOF
    2. Apply this yaml: 
      kubectl create –f pv.yaml
  2. Create Persistent Volume Claim.
    1. Create file pvc.yaml by referring the following sample code: 
      cat <EOF > ./pvc.yaml 
apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: bds-pvc 
  namespace: bds 
spec: 
  storageClassName: manual 
  accessModes: 
    - ReadWriteOnce 
  resources: 
    requests: 
      storage: 3Gi 
EOF
    2. Apply this yaml: 
      kubectl create –f pvc.yaml

GKE

  1. Create StorageClass and Persistent Volume, (only if your configuration requires it). If they already exists, then skip to the next step.
  2. Create Persistent Volume Claim.
  3. Create file bds-pvc.yaml by referring the following sample code: 
    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: bds-pvc
namespace: bds
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: nfs-client
volumeMode: Filesystem
  4. Apply bds-pvc.yaml, by executing the following command:. 
    kubectl apply -f  bds-pvc.yaml -n bds

Create watermarks on persistent volume


The extract, transform, and load processes in BDS relies on watermark dates. Typically, the watermark date is either the agreed go-live date, or a date that is one day earlier than the date when data exists in all relevant data sources.

Watermark dates are automatically created for BDS persistence volume. You can also manually update the watermark in scenarios where you want to reprocess or catch up the data.

Important
Before creating a watermark file for BDS, a persistence volume must be already provisioned.

To automatically create watermarks, run BDS. Running BDS creates the watermark file and updates all the watermark related fields such as extract_watermark, transform_watermark, and load_watermark with the production date. Note that the production date is configured as part of Tenant provisioning.

Important
Versions prior to 100.0.002.0020 require watermarks file to be created manually. If you are upgrading from previous versions to 100.0.002.0020 or higher, and if the watermark file doesn't exist, then running BDS automatically creates the watermark file and updates the watermark fields with the production date. For BDS to process the data on the go-live date, make sure you set the production_date as at least one day prior to the go-live date.

To manually create watermarks, perform the following steps:

  1. Create the pod yaml by referring the following sample code. 
    cat <EOF > ./pv-static.yaml
apiVersion: v1
kind: Pod
metadata:
  name: static-pv
spec:
  volumes:
    - name: pv-test
      persistentVolumeClaim:
        claimName: bds-pvc
  containers:
    - name: test
      image: ubuntu
      command: [ "/bin/sh" ]
      args: [ "-c", "while true; do echo hello; sleep 10;done" ]
      volumeMounts:
        - mountPath: "/tmp"
          name: pv-test
EOF
  2. Execute the following commands to verify the status of the pod created, create the watermarks folder, and create the tenant json file. 
    developer@docker:~/OpenShift$ kubectl apply -f pv-static.yaml -n bds
pod/static-pv created
developer@docker:~/OpenShift$ kubectl get pod static-pv -n bds
NAME        READY   STATUS    RESTARTS   AGE
static-pv   1/1     Running   0          68s
developer@docker:~/OpenShift$ kubectl exec  -n bds -it static-pv bash
1000670000@static-pv:/$ cd /tmp
1000670000@static-pv:/$ mkdir watermarks
1000670000@static-pv:/tmp$ cd watermarks/
1000670000@static-pv:/tmp/watermarks$  echo -e "{ \n\t\"extract_watermark\": \"YYYY-MM-DD\",\n\t\"transform_watermark\": \"YYYY-MM-DD\",\n\t\"load_watermark\": \"YYYY-MM-DD\"\n}" > <tenant_name>.json
    Where,
    • extract_watermark - The last successful extraction date in YYYY-MM-DD format, should be greater than or equal to the production date.
    • transform_watermark - The last successful transformation date in YYYY-MM-DD format, should be less than or equal to the extract watermark, and greater than the production date.
    • load_watermark - The last successful load date in YYYY-MM-DD format, should be less than or equal to the transform watermark, and greater than the production date.
    • <tenant_name> - The name of the tenant as it is configured in the Config.json file for BDS. A watermark file will be created in the json format for the tenant with the name specified here, for example, MultiRegionExampleTenant.json.
  3. Execute the following command to verify the watermark values created in the tenant's json file. You will see a result similar to the following example. In this example, MultiRegionExampleTenant.json is the tenant file that contains the extract, transform, and load watermark values for the tenant. 
    cd /tmp/watermarks/
MultiRegionExampleTenant.json
{
"extract_watermark": "2021-06-27",
"transform_watermark": "2021-06-27",
"load_watermark": "2021-06-27"
}
  4. Exit and remove pod.


Retrieved from "https://all.docs.genesys.com/BDS/Current/BDSPEGuide/Provision (2022-05-20 21:23:24)"