Grafana configuration

From Genesys Documentation
Jump to: navigation, search
This topic is part of the manual Operations for version Current of Genesys Multicloud CX Private Edition.

Learn about how to use Grafana to set up a monitoring solution for your services.

Grafana enables you to query, visualize, alert on, and understand your metrics.
Important
Although some services have packaged dashboard configuration within their Helm charts, Genesys Multicloud CX private edition does not currently support monitoring dashboards. The following information is provided purely as guidance based on Genesys experimentation, and does not represent a supported configuration.

Grafana in OpenShift

The community powered Grafana cannot be deployed to the existing OpenShift monitoring namespace. Therefore, a new namespace must be created. Follow the instructions to deploy Grafana in the OpenShift cluster.

Installing Grafana using OpenShift command line interface

Download or clone the Grafana operator version 3.10.1 from https://github.com/integr8ly/grafana-operator, and then change the name of the working directory to grafana-operator-3.10.1. Genesys recommends version 3.10.1. However, you can use any version as per your requirement.

Deploying Grafana operator manually

  1. Create a namespace
    $ oc create namespace <namespace>
  2. Create custom resource definitions that the operator uses.
    $ oc create -f deploy/crds
  3. Create operator roles.
    $ oc create -f deploy/roles

To scrape the ConfigMaps (dashboards) deployed, ConfigMaps must be a part of API groups and modify the cluster role (cluster_role_grafana_operator.yaml) for Grafana.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: grafana-operator
rules:
  - apiGroups:
      - integreatly.org
    resources:
      - grafanadashboards
      - grafanadashboards/status
    verbs: ['get', 'list', 'update', 'watch']
  - apiGroups:
      - ""
    resources:
      - namespaces
      - configmaps
    verbs: ['get', 'list', 'watch']
  - apiGroups:
      - ""
    resources:
      - events
    verbs: ['create', 'patch']

You need cluster roles to scan for dashboards in other namespaces too.

$ oc create -f deploy/cluster_roles

To scan dashboards deployed in all namespaces, add the --scan-all argument to the operator container.

--scan-all: Searches for dashboards in all namespaces. This requires the operator service account to have cluster wide permissions to get, list, update and watch dashboards.

containers:
  - name: grafana-operator
    image: quay.io/integreatly/grafana-operator:v3.10.1
    args:
      - '--scan-all'

To deploy the operator to the namespace, use deploy/operator.yaml:

$ oc create -f deploy/operator.yaml -n <namespace>

To check the status of the operator pod:

$ oc get pods -n <namespace>

Grafana plugins

If a data source or dashboard requires a plugin, it can be added either to a dashboard or any custom environment variables can be added to the Grafana deployment.

Installing plugins using Grafana environment variable through operator

The operator allows you to pass custom environment variables to the Grafana deployment. This means that you can set the GF_INSTALL_PLUGINS flag, as described.

Create and deploy the secret.

oc create -f <secret-name>.yaml -n <namespace>

Create a secret using a plugin.

apiVersion: v1
kind: Secret
metadata:
  name: <secret-name>
type: Opaque
stringData:
  GF_INSTALL_PLUGINS: <plugin-name> <plugin-version>

Add the section to Grafana CR.

deployment:
  envFrom:
    - secretRef:
        name: <secret-name>

Create a new Grafana instance in the namespace.

$ oc create -f deploy/examples/Grafana.yaml -n <namespace>

To acquire Grafana UI login admin credentials:

$ echo "User: admin"
$ echo "Password: $(oc get secret <secret-name> --namespace <namespace> -o jsonpath="{.data.GF_SECURITY_ADMIN_PASSWORD}" | base64 --decode)"

Connecting Prometheus to customized Grafana

The Grafana service account is created alongside the Grafana instance. Genesys provides the cluster-monitoring-view cluster role to the service account.

$ oc adm policy add-cluster-role-to-user cluster-monitoring-view -z <grafana-service-account>

The bearer token for this service account is used to authenticate access to Prometheus in the openshift-monitoring namespace. The following command displays this token:

$ oc serviceaccounts get-token <grafana-service-account> -n <namespace>

In the following YAML, substitute ${BEARER_TOKEN} with the output of the above-mentioned command, save the file,  and deploy oc create -f <filename> -n <namespace>.

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDataSource
metadata:
  name: prometheus-grafanadatasource
  namespace: <namespace>
spec:
  datasources:
    - access: proxy
      editable: true
      isDefault: true
      jsonData:
        httpHeaderName1: 'Authorization'
        timeInterval: 5s
        tlsSkipVerify: true
      name: Prometheus
      secureJsonData:
        httpHeaderValue1: 'Bearer ${BEARER_TOKEN}'
      type: prometheus
      url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
  name: prometheus-grafanadatasource.yaml

For more details, refer to: https://github.com/integr8ly/grafana-operator/tree/master/documentation.

Grafana URL

Exposing Grafana URL

The routes must be exposed explicitly only when you need to create a Grafana instance where ingress is disabled. if ingress is enabled, routes are created implicitly. To get a list of services:

$ oc get service -n <namespace>

To expose the service:

$ oc expose service <grafana service> -n <namespace>

Retrieving Grafana URL

Method 1

From the monitoring namespace, click Networking, click Routes, and then click the Grafana URL under Location to display the custom Grafana user interface.

Method1.pngMethod 2

In Openshift CLI, switch to namespace where Grafana is deployed, and then retrieve the routes.

Method2.png

Grafana in GKE

Google Cloud Monitoring in Grafana

For details about cloud monitoring in Grafana, refer to https://grafana.com/docs/grafana/latest/datasources/google-cloud-monitoring/.

Deploying Prometheus

Prerequisites

Steps to deploy Prometheus

  1. Run the setup from the root of downloaded source. This deploys the Prometheus operator and CRDs.
    kubectl create -f manifests/setup
  2. For Prometheus to scrape the cluster (all namespaces), edit prometheus-clusterRole.yaml.
    metadata:
      labels:
        app.kubernetes.io/component: prometheus
        app.kubernetes.io/name: prometheus
        app.kubernetes.io/part-of: kube-prometheus
        app.kubernetes.io/version: 2.30.0
      name: prometheus-k8s
    rules:
    - apiGroups:
      - ""
      resources:
      - nodes/metrics
      verbs:
      - get
    - nonResourceURLs:
      - /metrics
      verbs:
      - get
    - apiGroups:
      - ""
      resources:
      - services
      - pods
      - endpoints
      verbs:
      - get
      - list
      - watch
  3. After the setup is complete, execute the following command:
    kubectl create -f manifests/
    This deploys the following components.
  4. Deploy required components
    kubectl create -f manifests/

Deploying Grafana

Configuring Grafana

The community-powered Grafana is deployed in a new namespace (ex. monitoring) . Follow the instructions to deploy Grafana in GKE.

Installing using Command Line Interface

Download/clone the Grafana operator rom https://github.com/integr8ly/grafana-operator and change the working directory to grafana-operator-xx.

Steps to deploy Grafana operator manually

  1. Create a new namespace or switch to a namespace (for example: monitoring) where Prometheus is deployed.
    $ kubectl create -f config/crd/bases
  2. Create operator roles.
    $ kubectl create -f deploy/roles
  3. Modify ClusterRoleBinding (cluster_role_binding_grafana_operator.yaml). The namespace must be updated with the current namespace where Grafana is deployed (for example: monitoring).
    apiVersion: rbac.authorization.k8s.io/v1
    
    kind: ClusterRoleBinding
    
    metadata:
    
     name: grafana-operator
    
    roleRef:
    
     name: grafana-operator
    
     kind: ClusterRole
    
     apiGroup: ""
    
    subjects:
    
     - kind: ServiceAccount
    
       name: grafana-operator
    
       namespace: monitoring
  4. Scan for dashboards in other namespaces you also need the cluster roles.
    $ kubectl create -f deploy/cluster_roles
    To scan dashboards deployed in all namespaces, --scan-all should be added to operator container as argument.
    --scan-all: watch for dashboards in all namespaces. This requires the operator service account to have cluster wide permissions to get, list, update and watch dashboards.
  5. Deploy the operator to that namespace you can use deploy/operator.yaml.
    containers:
    
     - name: grafana-operator
    
       image: quay.io/integreatly/grafana-operator:vX.X.X
    
       args:
    
         - '--scan-all'
  6. Deploy the operator to that namespace. You can use deploy/operator.yaml
    $ kubectl create -f deploy/operator.yaml -n <namespace>
  7. Check the status of the operator pod.

Grafana Plugins

If a data source or dashboard requires a plugin, it can be added in the dashboard itself or it can be added as custom environment variable to the Grafana deployment.

Install plugins using Grafana environment variable

The operator allows you to pass custom environment variable to the Grafana deployment. This means that you can set the GF_INSTALL_PLUGINS flag, as described.

  1. Create and deploy the secret kubectl  create -f <secret-name>.yaml -n <namespace>.
    apiVersion: v1
    
    kind: Secret
    
    metadata:
    
     name: <secret-name>
    
    type: Opaque
    
    stringData:
    
     GF_INSTALL_PLUGINS: <plugin-name> <plugin-version>
    
    Add the section to Grafana CR.
    
    deployment:
    
     envFrom:
    
       '''-''' secretRef:
    
           name: <secret-name>

Creating Grafana Instance

  1. Modify Grafana.yaml with the required values before creating Grafana instance. Update name and add hostname if ingress is enabled.
    apiVersion: integreatly.org/v1alpha1
    
    kind: Grafana
    
    metadata:
    
    name: grafana-app
    
    spec:
    
    client:
    
    preferService: true
    
    ingress:
    
    enabled: True
    
    hostname: "grafana.gke1-uswest1.gcpe001.gencpe.com"
    
    pathType: Prefix
    
    path: "/"
    
    config:
    
    log:
    
    mode: "console"
    
    level: "error"
    
    log.frontend:
    
    enabled: true
    
    auth:
    
    disable_login_form: False
    
    disable_signout_menu: True
    
    auth.anonymous:
    
    enabled: True
    
    service:
    
    name: "grafana-service"
    
    labels:
    
    app: "grafana"
    
    type: "grafana-service"
    
    dashboardLabelSelector:
    
    - matchExpressions:
    
    - { key: app, operator: In, values: [grafana] }
    
    resources:
    
    Optionally specify container resources
    
    limits:
    
    cpu: 200m
    
    memory: 200Mi
    
    requests:
    
    cpu: 100m
  2. Create a new Grafana instance in the namespace.
    $ kubectl create -f deploy/examples/Grafana.yaml -n <namespace>
  3. Retrieve the Grafana UI login admin credentials.
    $ echo "User: admin"
    
    $ echo "Password: $(oc get secret <secret-name> --namespace <namespace> -o jsonpath="{.data.GF_SECURITY_ADMIN_PASSWORD}" | base64 --decode)"

Connecting Prometheus to custom Grafana

Deploy Grafana data source kubectl create -f <filename> -n <namespace>. If Grafana instance is deleted and redeployed, you must delete and redeploy Grafana data source as well.

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDataSource
metadata:
  name: grafana-datasource
  namespace: monitoring
spec:
  datasources:
    - access: proxy
      editable: true
      isDefault: true
      name: Prometheus
      type: prometheus
      url: 'http://prometheus-k8s.monitoring.svc:9090'
  name: grafana-datasource.yaml

Grafana dashboards

Importing custom dashboards

To import a custom Grafana dashboard from a JSON file within Grafana, click Import and then click Upload Json file as shown in the following screenshot:

ImportCusDashB.png

Creating Grafana dashboards

To create Grafana dashboard, use the following template:

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: <name>
  namespace: <namespace>
  labels:
    app: grafana --> label should match the dashboardLabelSelector defined in Grafana operator
spec:
  customFolderName: "folder name"
  json:
    ""
  configMapRef:
    name: <Configmap-name>
    key: <key>
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: voice-sips-dashboard-from-cm
data:
  <key>: |-
 <dashboard>
Important
Each product has a set of dashboards that come with the service for you to enable/disable as per your choice.


You can deploy new customized dashboards. You can either deploy them as Grafana dashboard in the namespace or it can be directly loaded on to the Grafana UI. Refer to https://github.com/integr8ly/grafana-operator/tree/master/deploy/examples/dashboards for more details about different ways to deploy a dashboard.