Corinneh: Published

2022-02-23T20:56:22Z

Published

New page

{{ArticlePEServiceMetrics
|IncludedServiceId=34f32476-4f7c-446d-9224-3a4175f25e24
|CRD=Supports both CRD and annotations
|Port=11500
|Endpoint=http://<pod-ipaddress>:11500/metrics
|MetricsUpdateInterval=30 seconds
|MetricsDefined=Yes
|MetricsIntro=Voice Registrar Service exposes Genesys-defined, Registrar Service–specific metrics as well as some standard Kafka metrics. You can query Prometheus directly to see all the metrics that the Registrar Service exposes. The following metrics are likely to be particularly useful. Genesys does not commit to maintain other currently available Voice Registrar Service metrics not documented on this page.
|PEMetric={{PEMetric
|Metric=registrar_register_count
|Type=counter
|Unit=N/A
|Label=location, tenant
|MetricDescription=Number of registrations.
|UsedFor=Traffic
}}{{PEMetric
|Metric=registrar_health_level
|Type=gauge
|Unit=N/A
|MetricDescription=Health level of the registrar node:

-1 – fail<br />
0 – starting<br />
1 – degraded<br />
2 – pass
|UsedFor=Errors
}}{{PEMetric
|Metric=registrar_request_latency
|Type=histogram
|Unit=milliseconds
|Label=le, location, tenant
|MetricDescription=Time taken to process the request (ms).
|UsedFor=Latency
}}{{PEMetric
|Metric=registrar_active_sip_registrations
|Type=gauge
|Unit=N/A
|Label=tenant
|MetricDescription=Number of active SIP registrations.
|UsedFor=Traffic
}}{{PEMetric
|Metric=kafka_consumer_latency
|Type=histogram
|Label=tenant, topic
|MetricDescription=Consumer latency is the time difference between when the message is produced and when the message is consumed. That is, the time when the consumer received the message minus the time when the producer produced the message.
|UsedFor=Latency
}}{{PEMetric
|Metric=kafka_consumer_state
|Type=gauge
|MetricDescription=Current Kafka consumer connection state:

0 – disconnected<br />
1 – connected
}}
|AlertsDefined=Yes
|PEAlert={{PEAlert
|Alert=Kafka events latency is too high
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple topics, make sure there are no issues with Kafka (CPU, memory, or network overload).
*If the alarm is triggered only for topic <nowiki>{{ $labels.topic }}</nowiki>, check if there is an issue with the service related to the topic (CPU, memory, or network overload).
|BasedOn=kafka_consumer_latency_bucket
|Threshold=Latency for more than 5% of messages is more than 0.5 seconds for topic <nowiki>{{ $labels.topic }}</nowiki>.
}}{{PEAlert
|Alert=Too many Kafka consumer failed health checks
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka.
*If the alarm is triggered only for <nowiki>{{ $labels.container }}</nowiki>, check if there is an issue with the service.
|BasedOn=kafka_consumer_error_total
|Threshold=Health check failed more than 10 times in 5 minutes for Kafka consumer for topic <nowiki>{{$labels.topic}}</nowiki>.
}}{{PEAlert
|Alert=Too many Kafka consumer request timeouts
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka.
*If the alarm is triggered only for <nowiki>{{ $labels.container }}</nowiki>, check if there is an issue with the service.
|BasedOn=kafka_consumer_error_total
|Threshold=There were more than 10 request timeouts within 5 minutes for the Kafka consumer for topic <nowiki>{{$labels.topic}}</nowiki>.
}}{{PEAlert
|Alert=Too many Kafka consumer crashes
|Severity=Critical
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka.
*If the alarm is triggered only for <nowiki>{{ $labels.container }}</nowiki>, check if there is an issue with the service.
|BasedOn=kafka_consumer_error_total
|Threshold=There were more than 3 Kafka consumer crashes within 5 minutes for service <nowiki>{{ $labels.container }}</nowiki>.
}}{{PEAlert
|Alert=Kafka not available
|Severity=Critical
|AlertDescription=Kafka is not available for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka.
*If the alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki>, check if there is an issue with the pod.
|BasedOn=kafka_producer_state, kafka_consumer_state
|Threshold=Kafka is not available for pod <nowiki>{{ $labels.pod }}</nowiki> for 5 consecutive minutes.
}}{{PEAlert
|Alert=Redis disconnected for 5 minutes
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis.
*If the alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki>, check if there is an issue with the pod.
|BasedOn=redis_state
|Threshold=Redis is not available for pod <nowiki>{{ $labels.pod }}</nowiki> for 5 minutes.
}}{{PEAlert
|Alert=Redis disconnected for 10 minutes
|Severity=Critical
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis.
*If the alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki>, check if there is an issue with the pod.
|BasedOn=redis_state
|Threshold=Redis is not available for pod <nowiki>{{ $labels.pod }}</nowiki> for 10 minutes.
}}{{PEAlert
|Alert=Pod Failed
|Severity=Warning
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> failed.

Actions:

*One of the containers in the pod has entered a failed state. Check the Kibana logs for the reason.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Failed state.
}}{{PEAlert
|Alert=Pod Unknown state
|Severity=Warning
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Unknown state.

Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster.
*If the alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki>, check whether the image is correct and if the container is starting up.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Unknown state for 5 minutes.
}}{{PEAlert
|Alert=Pod Pending state
|Severity=Warning
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Pending state.

Actions:

*If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster.
*If the alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki>, check the health of the pod.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Pending state for 5 minutes.
}}{{PEAlert
|Alert=Pod Not ready for 10 minutes
|Severity=Critical
|AlertDescription=Actions:

*If this alarm is triggered, check whether the CPU is available for the pods.
*Check whether the port of the pod is running and serving the request.
|BasedOn=kube_pod_status_ready
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in the NotReady state for 10 minutes.
}}{{PEAlert
|Alert=Container restarted repeatedly
|Severity=Critical
|AlertDescription=Actions:

*One of the container in the pod has entered a Failed state. Check the Kibana logs for the reason.
|BasedOn=kube_pod_container_status_restarts_total
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> was restarted 5 or more times within 15 minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 65%
|Severity=Warning
|AlertDescription=High CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_cpu_usage_seconds_total, kube_pod_container_resource_limits
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Pod memory greater than 65%
|Severity=Warning
|AlertDescription=High memory usage for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_limits
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Pod memory greater than 80%
|Severity=Critical
|AlertDescription=Critical memory usage for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Restart the service.
*Collect the service logs: raise an investigation ticket.
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_limits
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 80% for 5 minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 80%
|Severity=Critical
|AlertDescription=Critical CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
|BasedOn=container_cpu_usage_seconds_total, kube_pod_container_resource_limits
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 80% for 5 minutes.
}}
}}

VM/Current/VMPEGuide/VoiceRegistrarServiceMetrics - Revision history

Corinneh: Published