Corinneh: Published

2022-02-23T22:19:13Z

Published

Corinneh: Published

2022-02-23T22:17:01Z

Published

Corinneh: Published

2022-02-23T20:56:47Z

Published

New page

{{ArticlePEServiceMetrics
|IncludedServiceId=fba64ef9-5625-4497-84ff-f44a2b5bd6d6
|CRD=Supports both CRD and annotations
|Port=8800
|Endpoint=http://<pod-ipaddress>:8800/metrics
|MetricsUpdateInterval=30 seconds
|MetricsDefined=Yes
|MetricsIntro=You can query Prometheus directly to see all the metrics that the Voice Dial Plan Service exposes. The following metrics are likely to be particularly useful. Genesys does not commit to maintain other currently available Dial Plan Service metrics not documented on this page.
|PEMetric={{PEMetric
|Metric=dialplan_health_level
|Type=gauge
|Unit=N/A
|MetricDescription=Aggregated health level of the dialplan node for dependent services such as Redis and Envoy sidecar connection:

-1 – fail 
0 – starting 
1 – degraded 
2 – pass
|SampleValue=2
|UsedFor=Health
}}{{PEMetric
|Metric=dialplan_redis_state
|Type=gauge
|Unit=N/A
|Label=redis_cluster_name
|MetricDescription=Current Redis connection state:

0 – disconnected 
1 – connecting 
2 – connected
|SampleValue=2
|UsedFor=Health
}}{{PEMetric
|Metric=dialplan_total_request
|Type=counter
|Unit=N/A
|Label=tenant, pod, operation_type
|MetricDescription=Number of dialplan requests received.
|UsedFor=Traffic
}}{{PEMetric
|Metric=dialplan_failure_response
|Type=counter
|Unit=N/A
|Label=tenant, pod, operation_type, status, reason
|MetricDescription=The number of Dial Plan failure responses.
|UsedFor=Traffic
}}{{PEMetric
|Metric=dialplan_response_time
|Type=histogram
|Unit=milliseconds
|MetricDescription=Dialplan request processing duration histogram, in ms.
|UsedFor=Latency
}}{{PEMetric
|Metric=dialplan_redis_cache_latency_msec
|Type=histogram
|Unit=milliseconds
|Label=tenant
|MetricDescription=Redis fetch latency, measured in milliseconds.
|UsedFor=Latency
}}
|AlertsDefined=Yes
|PEAlert={{PEAlert
|Alert=DialPlan processing time > 0.5 seconds
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is generated for all dialplan pods, then Redis or network delay might be the most probable cause.
*If the alarm is generated in a single dialplan pod, then it might be due to Envoy or a network issue.
|BasedOn=dialplan_response_time
|Threshold=When the latency for 95% of the dial plan messages is more than 0.5 seconds for a duration of 5 minutes, then this warning alarm is raised for the <nowiki>{{ $labels.container }}</nowiki>.
}}{{PEAlert
|Alert=DialPlan processing time > 2 seconds
|Severity=Critical
|AlertDescription=Actions:

*If the alarm is generated for all dialplan pods, then Redis or network delay might be the most probable cause.
*If the alarm is generated in a single dialplan pod, then it might be due to Envoy or a network issue.
|BasedOn=dialplan_response_time
|Threshold=If the latency for 95% of the dial plan messages is more than 2 seconds for a duration of 5 minutes, then this warning alarm is raised for the <nowiki>{{ $labels.container }}</nowiki>.
}}{{PEAlert
|Alert=Aggregated service health failing for 5 minutes
|Severity=Critical
|AlertDescription=Actions:

*Check the dialplan dashboard for Aggregated Service Health errors and, in case of a Redis error, first check for any issues/crashes in the pod and then restart Redis.
*In the case of an Envoy error, the dialplan container will be restarted by the liveness probe. If the issue still exists after that, restart the pod.
|BasedOn=dialplan_health_level
|Threshold=Dependent services or the Envoy sidecar is not available for 5 minutes in the pod <nowiki>{{ $labels.pod }}</nowiki>.
}}{{PEAlert
|Alert=Redis disconnected for 5 minutes
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Redis and then restart Redis.
*If the alarm is triggered only for the pod <nowiki>{{ $labels.pod }}</nowiki>, check to see if there is an issue with the pod.
|BasedOn=redis_state
|Threshold=Redis is not available for the pod <nowiki>{{ $labels.pod }}</nowiki> for 5 minutes.
}}{{PEAlert
|Alert=Redis disconnected for 10 minutes
|Severity=Critical
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Redis and then restart Redis.
*If the alarm is triggered only for the pod <nowiki>{{ $labels.pod }}</nowiki>, check to see if there is an issue with the pod.
|BasedOn=redis_state
|Threshold=Redis is not available for the pod <nowiki>{{ $labels.pod }}</nowiki> for 10 minutes.
}}{{PEAlert
|Alert=Pod Failed
|Severity=Warning
|AlertDescription=Actions:

*One of the containers in the pod has entered a failed state. Check the Kibana logs for the reason.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> failed.
}}{{PEAlert
|Alert=Pod Unknown state
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster.
*If the alarm is triggered only for the pod <nowiki>{{ $labels.pod }}</nowiki>, check whether the image is correct and if the container is starting up.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Unknown state for 5 minutes.
}}{{PEAlert
|Alert=Pod Pending state
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster.
*If the alarm is triggered only for the pod <nowiki>{{ $labels.pod }}</nowiki>, check the health of the pod.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in the Pending state for 5 minutes.
}}{{PEAlert
|Alert=Pod Not ready for 10 minutes
|Severity=Critical
|AlertDescription=Actions:

*If this alarm is triggered, check whether the CPU is available for the pods.
*Check whether the port of the pod is running and serving the request.
|BasedOn=kube_pod_status_ready
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in the NotReady state for 10 minutes.
}}{{PEAlert
|Alert=Pod memory greater than 65%
|Severity=Warning
|AlertDescription=High memory usage for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs; raise an investigation ticket
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_limits
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Pod memory greater than 80%
|Severity=Critical
|AlertDescription=Critical memory usage for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Restart the service.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_limits
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 80% for 5 minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 65%
|Severity=Warning
|AlertDescription=High CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_cpu_usage_seconds_total, kube_pod_container_resource_limits
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 80%
|Severity=Critical
|AlertDescription=Critical CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Restart the service.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_cpu_usage_seconds_total, kube_pod_container_resource_limits
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 80% for 5 minutes.
}}
}}

@@ Line 1: / Line 1: @@
 {{ArticlePEServiceMetrics
 |IncludedServiceId=fba64ef9-5625-4497-84ff-f44a2b5bd6d6
 |CRD=Supports both CRD and annotations
 |Port=8800

@@ Line 11: / Line 11: @@
 |Type=gauge
 |Unit=N/A
-|MetricDescription=Aggregated health level of the dialplan node for dependent services such as Redis and Envoy sidecar connection:
+|MetricDescription=Aggregated health level of the dialplan node for dependent services such as Redis and the Envoy sidecar connection:
 -1 – fail<br />

VM/Current/VMPEGuide/VoiceDialPlanServiceMetrics - Revision history

Corinneh: Published

Corinneh: Published

Corinneh: Published