Corinneh: Published

2022-02-23T20:56:45Z

Published

New page

{{ArticlePEServiceMetrics
|IncludedServiceId=a0519f2a-7110-4037-b752-3393df349c62
|CRD=Supports both CRD and annotations
|Port=9100
|Endpoint=http://<pod-ipaddress>:9100/metrics
|MetricsUpdateInterval=30 seconds
|MetricsDefined=Yes
|MetricsIntro=You can query Prometheus directly to see all the metrics that the Voice Config Service exposes. The following metrics are likely to be particularly useful. Genesys does not commit to maintain other currently available Config Service metrics not documented on this page.
|PEMetric={{PEMetric
|Metric=config_device_response
|Type=counter
|Unit=N/A
|Label=location, tenant, request_type, status
|MetricDescription=Number of device responses for each request.
|SampleValue=2
|UsedFor=Traffic
}}{{PEMetric
|Metric=config_tenant_response
|Type=counter
|Unit=N/A
|Label=location, request_type, status
|MetricDescription=Number of Tenant responses for each request.
|SampleValue=2
|UsedFor=Traffic
}}{{PEMetric
|Metric=config_node_get_response
|Type=counter
|Unit=N/A
|MetricDescription=Number of Get responses for each request.
|UsedFor=Traffic
}}{{PEMetric
|Metric=config_node_agent_response
|Type=counter
|Unit=N/A
|MetricDescription=Number of agent responses for each request.
|UsedFor=Traffic
}}{{PEMetric
|Metric=config_redis_state
|Type=gauge
|Unit=N/A
|Label=location, redis_cluster_name
|MetricDescription=Current Redis connection state:

-1 – error 
0 – disconnected 
1 – connected 
2 – ready
|SampleValue=2
|UsedFor=Errors
}}{{PEMetric
|Metric=service_version_info
|Type=gauge
|Unit=N/A
|Label=version
|MetricDescription=Displays the version of Voice Config Service that is currently running. In the case of this metric, the labels provide the important information. The metric value is always 1 and does not provide any information.
|SampleValue=service_version_info{version="100.0.1000006"} 1
}}{{PEMetric
|Metric=config_health_level
|Type=gauge
|Unit=N/A
|MetricDescription=Health level of the config node:

-1 – error 
0 – fail 
1 – degraded 
2 – pass
|SampleValue=2
|UsedFor=Errors
}}{{PEMetric
|Metric=config_healthcheck_generic_exception
|Type=gauge
|Unit=N/A
|MetricDescription=Generic error during health check.
|SampleValue=0
}}
|AlertsDefined=Yes
|PEAlert={{PEAlert
|Alert=Redis disconnected for 5 minutes
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Redis, then restart Redis.
*If the alarm is triggered only for the pod <nowiki>{{ $labels.pod }}</nowiki>, check to see if there is an issue with the pod.
|BasedOn=redis_state
|Threshold=Redis is not available for pod <nowiki>{{ $labels.pod }}</nowiki> for 5 minutes.
}}{{PEAlert
|Alert=Redis disconnected for 10 minutes
|Severity=Critical
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Redis, then restart Redis.
*If the alarm is triggered only for the pod <nowiki>{{ $labels.pod }}</nowiki>, check to see if there is an issue with the pod.
|BasedOn=redis_state
|Threshold=Redis is not available for the pod <nowiki>{{ $labels.pod }}</nowiki> for 10 minutes.
}}{{PEAlert
|Alert=Pod Failed
|Severity=Warning
|AlertDescription=Actions:

*One of the containers in the pod has entered a failed state. Check the Kibana logs for the reason.
|BasedOn=kube_pod_status_phase
|Threshold=Pod failed <nowiki>{{ $labels.pod }}</nowiki>.
}}{{PEAlert
|Alert=Pod Unknown state
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster.
*If the alarm is triggered only for the pod <nowiki>{{ $labels.pod }}</nowiki>, check to see whether the image is correct and if the container is starting up.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Unknown state for 5 minutes.
}}{{PEAlert
|Alert=Pod Pending state
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster.
*If the alarm is triggered only for the pod <nowiki>{{ $labels.pod }}</nowiki>, check the health of the pod.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Pending state for 5 minutes.
}}{{PEAlert
|Alert=Pod Not ready for 10 minutes
|Severity=Critical
|AlertDescription=Actions:

*If this alarm is triggered, check whether the CPU is available for the pods.
*Check whether the port of the pod is running and serving the request.
|BasedOn=kube_pod_status_ready
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in NotReady state for 10 minutes.
}}{{PEAlert
|Alert=Container restarted repeatedly
|Severity=Critical
|AlertDescription=Actions:

*One of the containers in the pod has entered a failed state. Check the Kibana logs for the reason.
|BasedOn=kube_pod_container_status_restarts_total
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> was restarted 5 or more times within 15 minutes.
}}{{PEAlert
|Alert=Pod memory greater than 65%
|Severity=Warning
|AlertDescription=High memory usage for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_requests_memory_bytes
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Pod memory greater than 80%
|Severity=Critical
|AlertDescription=Critical memory usage for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Restart the service.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_requests_memory_bytes
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 80% for 5 minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 65%
|Severity=Warning
|AlertDescription=High CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_cpu_usage_seconds_total, container_spec_cpu_period
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 80%
|Severity=Critical
|AlertDescription=Critical CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Restart the service.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_cpu_usage_seconds_total, container_spec_cpu_period
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 80% for 5 minutes.
}}
}}

VM/Current/VMPEGuide/VoiceConfigServiceMetrics - Revision history

Corinneh: Published