PrivateEdition/Current/Operations/Grafana configuration - Revision history

WikiSysop at 19:31, March 12, 2023

2023-03-12T19:31:58Z

WikiSysop at 17:46, June 30, 2022

2022-06-30T17:46:04Z

Jabashree.amudha@genesys.com: Published

2021-12-20T18:03:28Z

Published

New page

{{Article
|Standalone=No
|DisplayName=Grafana configuration
|TocName=Grafana configuration
|Context=Learn about how to use Grafana to set up a monitoring solution for your services.
|ComingSoon=No
|Section={{Section
|anchor=GrafanaConfig
|alignment=Vertical
|structuredtext=Grafana enables you to query, visualize, alert on, and understand your metrics.{{NoteFormat|Although some services have packaged dashboard configuration within their Helm charts, Genesys Multicloud CX private edition does not currently support monitoring dashboards. The following information is provided purely as guidance based on Genesys experimentation, and does not represent a supported configuration.|}}
==Grafana in OpenShift==
The community powered Grafana cannot be deployed to the existing OpenShift monitoring namespace. Therefore, a new namespace must be created. Follow the instructions to deploy Grafana in the OpenShift cluster.

===Installing Grafana using OpenShift command line interface===
Download or clone the Grafana operator version '''3.10.1''' from https://github.com/integr8ly/grafana-operator, and then change the name of the working directory to '''grafana-operator-3.10.1'''. Genesys recommends version 3.10.1. However, you can use any version as per your requirement.

====Deploying Grafana operator manually====

#Create a namespace
#:<source lang="yaml">$ oc create namespace <namespace></source>
#Create custom resource definitions that the operator uses.
#:<source lang="yaml">$ oc create -f deploy/crds</source>
#Create operator roles.
#:<source lang="yaml">$ oc create -f deploy/roles</source>

To scrape the ConfigMaps (dashboards) deployed, ConfigMaps must be a part of API groups and modify the cluster role ('''cluster_role_grafana_operator.yaml''') for Grafana.
<source lang="yaml">
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: grafana-operator
rules:
- apiGroups:
- integreatly.org
resources:
- grafanadashboards
- grafanadashboards/status
verbs: ['get', 'list', 'update', 'watch']
- apiGroups:
- ""
resources:
- namespaces
- configmaps
verbs: ['get', 'list', 'watch']
- apiGroups:
- ""
resources:
- events
verbs: ['create', 'patch'] </source>
You need cluster roles to scan for dashboards in other namespaces too.
<source lang="yaml">$ oc create -f deploy/cluster_roles</source>

To scan dashboards deployed in all namespaces, add the '''''--scan-all''''' argument to the operator container.

'''''--scan-all''''': Searches for dashboards in all namespaces. This requires the operator service account to have cluster wide permissions to get, list, update and watch dashboards.
<source lang="yaml">
containers:
- name: grafana-operator
image: quay.io/integreatly/grafana-operator:v3.10.1
args:
- '--scan-all' </source>
To deploy the operator to the namespace, use '''deploy/operator.yaml''':
<source lang="yaml">$ oc create -f deploy/operator.yaml -n <namespace> </source>
To check the status of the operator pod:
<source lang="yaml">$ oc get pods -n <namespace></source>

===Grafana plugins===
If a data source or dashboard requires a plugin, it can be added either to a dashboard or any custom environment variables can be added to the Grafana deployment.

====Installing plugins using Grafana environment variable through operator====
The operator allows you to pass custom environment variables to the Grafana deployment. This means that you can set the '''GF_INSTALL_PLUGINS''' flag, as described.

Create and deploy the secret.
<source lang-"yaml"="">oc create -f <secret-name>.yaml -n <namespace></source>
Create a secret using a plugin.
<source lang-"yaml"="">apiVersion: v1
kind: Secret
metadata:
name: <secret-name>
type: Opaque
stringData:
GF_INSTALL_PLUGINS: <plugin-name> <plugin-version>
</source>
Add the section to Grafana CR.
<source lang-"yaml"="">deployment:
envFrom:
- secretRef:
name: <secret-name></source>
Create a new Grafana instance in the namespace.
<source lang-"yaml"="">$ oc create -f deploy/examples/Grafana.yaml -n <namespace></source>
To acquire Grafana UI login admin credentials:
<source lang-"yaml"="">
$ echo "User: admin"
$ echo "Password: $(oc get secret <secret-name> --namespace <namespace> -o jsonpath="{.data.GF_SECURITY_ADMIN_PASSWORD}" | base64 --decode)" </source>

===Connecting Prometheus to customized Grafana===
The Grafana service account is created alongside the Grafana instance. Genesys provides the '''cluster-monitoring-view''' cluster role to the service account.
<source lang="yaml"">
$ oc adm policy add-cluster-role-to-user cluster-monitoring-view -z <grafana-service-account> </source>
The bearer token for this service account is used to authenticate access to Prometheus in the '''openshift-monitoring''' namespace. The following command displays this token:
<source lanf="yaml">
$ oc serviceaccounts get-token <grafana-service-account> -n <namespace></source>
In the following YAML, substitute '''''<code>${BEARER_TOKEN}</code>''''' with the output of the above-mentioned command, save the file, and deploy <tt>oc create -f <filename> -n <namespace></tt>.
<source lang="yaml">
apiVersion: integreatly.org/v1alpha1
kind: GrafanaDataSource
metadata:
name: prometheus-grafanadatasource
namespace: <namespace>
spec:
datasources:
- access: proxy
editable: true
isDefault: true
jsonData:
httpHeaderName1: 'Authorization'
timeInterval: 5s
tlsSkipVerify: true
name: Prometheus
secureJsonData:
httpHeaderValue1: 'Bearer ${BEARER_TOKEN}'
type: prometheus
url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
name: prometheus-grafanadatasource.yaml </source>
For more details, refer to: https://github.com/integr8ly/grafana-operator/tree/master/documentation.

===Grafana URL===

====Exposing Grafana URL====
The routes must be exposed explicitly only when you need to create a Grafana instance where ingress is disabled. if ingress is enabled, routes are created implicitly. To get a list of services:
<source lang="yaml">$ oc get service -n <namespace></source>
To expose the service:
<source lang="yaml">$ oc expose service <grafana service> -n <namespace></source>

====Retrieving Grafana URL====
'''Method 1'''

From the monitoring namespace, click '''Networking''', click '''Routes''', and then click the Grafana URL under '''Location''' to display the custom Grafana user interface.

[[File:Method1.png]]'''Method 2'''

In Openshift CLI, switch to namespace where Grafana is deployed, and then retrieve the routes.

[[File:Method2.png]]<br />
==Grafana in GKE==
===Google Cloud Monitoring in Grafana===
For details about cloud monitoring in Grafana, refer to https://grafana.com/docs/grafana/latest/datasources/google-cloud-monitoring/.
===Deploying Prometheus===
====Prerequisites====

*Create a namespace for deploying Prometheus operator.
*Clone or download source from https://github.com/prometheus-operator/kube-prometheus.
*Make sure you remove the Grafana files. Grafana is deployed using the operator.

====Steps to deploy Prometheus====

#Run the setup from the root of downloaded source. This deploys the Prometheus operator and CRDs.
#:<code class="powershell plain">kubectl create</code> <code class="powershell keyword">-f</code> <code class="powershell plain">manifests/setup</code>
#For Prometheus to scrape the cluster (all namespaces), edit prometheus-clusterRole.yaml.
#:<source lang="yaml">
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.30.0
name: prometheus-k8s
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
- apiGroups:
- ""
resources:
- services
- pods
- endpoints
verbs:
- get
- list
- watch </source>
#After the setup is complete, execute the following command:
#:<code>kubectl create -f manifests/</code>
#:This deploys the following components.
#*[https://prometheus.io/ Prometheus]
#*[https://github.com/prometheus/alertmanager Alertmanager]
#*[https://github.com/prometheus/node_exporter Prometheus node-exporter]
#*[https://github.com/DirectXMan12/k8s-prometheus-adapter Prometheus Adapter for Kubernetes Metrics APIs]
#*[https://github.com/kubernetes/kube-state-metrics kube-state-metrics]
#Deploy required components
#:<code class="powershell plain">kubectl create</code> <code class="powershell keyword">-f</code> <code class="powershell plain">manifests/</code>

===Deploying Grafana===

====Configuring Grafana====
The community-powered Grafana is deployed in a new namespace (ex. monitoring) . Follow the instructions to deploy Grafana in GKE.

====Installing using Command Line Interface====
Download/clone the Grafana operator rom https://github.com/integr8ly/grafana-operator and change the working directory to '''grafana-operator-xx'''.

====Steps to deploy Grafana operator manually====

#Create a new namespace or switch to a namespace (for example: monitoring) where Prometheus is deployed.
#:<source lang="yaml">$ kubectl create -f config/crd/bases </source>
#Create operator roles.
#:<source lang="yaml">$ kubectl create -f deploy/roles</source>
#Modify ClusterRoleBinding (cluster_role_binding_grafana_operator.yaml). The namespace must be updated with the current namespace where Grafana is deployed (for example: monitoring).
#:<source lang="yaml">
apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

name: grafana-operator

roleRef:

name: grafana-operator

kind: ClusterRole

apiGroup: ""

subjects:

- kind: ServiceAccount

name: grafana-operator

namespace: monitoring </source>
#Scan for dashboards in other namespaces you also need the cluster roles.
#:<source lang="yaml">
$ kubectl create -f deploy/cluster_roles </source>
#:To scan dashboards deployed in all namespaces, ''--scan-all'' should be added to operator container as argument.
#:''--scan-all'': watch for dashboards in all namespaces. This requires the operator service account to have cluster wide permissions to get, list, update and watch dashboards.
#Deploy the operator to that namespace you can use deploy/operator.yaml.
#:<source lang="yaml">
containers:

- name: grafana-operator

image: quay.io/integreatly/grafana-operator:vX.X.X

args:

- '--scan-all' </source>
#Deploy the operator to that namespace. You can use deploy/operator.yaml
#:<source lang="yaml">
$ kubectl create -f deploy/operator.yaml -n <namespace></source>
#Check the status of the operator pod.

===Grafana Plugins===
If a data source or dashboard requires a plugin, it can be added in the dashboard itself or it can be added as custom environment variable to the Grafana deployment.

====Install plugins using Grafana environment variable====
The operator allows you to pass custom environment variable to the Grafana deployment. This means that you can set the '''GF_INSTALL_PLUGINS''' flag, as described.

#Create and deploy the secret '''kubectl create -f <secret-name>.yaml -n <namespace>'''.
#:<source lang="yaml">
apiVersion: v1

kind: Secret

metadata:

name: <secret-name>

type: Opaque

stringData:

GF_INSTALL_PLUGINS: <plugin-name> <plugin-version>

Add the section to Grafana CR.

deployment:

envFrom:

'''-''' secretRef:

name: <secret-name>
</source>

===Creating Grafana Instance===

#Modify '''Grafana.yaml''' with the required values before creating Grafana instance. Update name and add hostname if ingress is enabled.
#:<source lang="yaml">apiVersion: integreatly.org/v1alpha1

kind: Grafana

metadata:

name: grafana-app

spec:

client:

preferService: true

ingress:

enabled: True

hostname: "grafana.gke1-uswest1.gcpe001.gencpe.com"

pathType: Prefix

path: "/"

config:

log:

mode: "console"

level: "error"

log.frontend:

enabled: true

auth:

disable_login_form: False

disable_signout_menu: True

auth.anonymous:

enabled: True

service:

name: "grafana-service"

labels:

app: "grafana"

type: "grafana-service"

dashboardLabelSelector:

- matchExpressions:

- { key: app, operator: In, values: [grafana] }

resources:

Optionally specify container resources

limits:

cpu: 200m

memory: 200Mi

requests:

cpu: 100m
</source>
#Create a new Grafana instance in the namespace.
#:<source lang="yaml">$ kubectl create -f deploy/examples/Grafana.yaml -n <namespace></source>
#Retrieve the Grafana UI login admin credentials.
#:<source lang="yaml">
$ echo "User: admin"

$ echo "Password: $(oc get secret <secret-name> --namespace <namespace> -o jsonpath="{.data.GF_SECURITY_ADMIN_PASSWORD}" | base64 --decode)"
</source>

===Connecting Prometheus to custom Grafana===
Deploy Grafana data source kubectl create -f <filename> -n <namespace>. If Grafana instance is deleted and redeployed, you must delete and redeploy Grafana data source as well.
<source lang="yaml">
apiVersion: integreatly.org/v1alpha1
kind: GrafanaDataSource
metadata:
name: grafana-datasource
namespace: monitoring
spec:
datasources:
- access: proxy
editable: true
isDefault: true
name: Prometheus
type: prometheus
url: 'http://prometheus-k8s.monitoring.svc:9090'
name: grafana-datasource.yaml </source>
==Grafana dashboards==

===Importing custom dashboards===
To import a custom Grafana dashboard from a JSON file within Grafana, click '''Import''' and then click '''Upload Json file''' as shown in the following screenshot:

[[File:ImportCusDashB.png]]

===Creating Grafana dashboards===
To create Grafana dashboard, use the following template:
<source lang="yaml">
apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
name: <name>
namespace: <namespace>
labels:
app: grafana --> label should match the dashboardLabelSelector defined in Grafana operator
spec:
customFolderName: "folder name"
json:
""
configMapRef:
name: <Configmap-name>
key: <key>
---
apiVersion: v1
kind: ConfigMap
metadata:
name: voice-sips-dashboard-from-cm
data:
<key>: |-
<dashboard></source>{{NoteFormat|Each product has a set of dashboards that come with the service for you to enable/disable as per your choice.|}}

You can deploy new customized dashboards. You can either deploy them as Grafana dashboard in the namespace or it can be directly loaded on to the Grafana UI. Refer to https://github.com/integr8ly/grafana-operator/tree/master/deploy/examples/dashboards for more details about different ways to deploy a dashboard.
|Status=No
}}
}}

@@ Line 9: / Line 9: @@
 |alignment=Vertical
 |structuredtext=Grafana enables you to query, visualize, alert on, and understand your metrics.{{NoteFormat|Although some services have packaged dashboard configuration within their Helm charts, Genesys Multicloud CX private edition does not currently support monitoring dashboards. The following information is provided purely as guidance based on Genesys experimentation, and does not represent a supported configuration.|}}
 ==Grafana in GKE==
 ===Google Cloud Monitoring in Grafana===

@@ Line 146: / Line 146: @@
 ===Google Cloud Monitoring in Grafana===
 For details about cloud monitoring in Grafana, refer to https://grafana.com/docs/grafana/latest/datasources/google-cloud-monitoring/.
 ===Deploying Prometheus===
 ====Prerequisites====