Drilldown: PEAlert

HIGH (14)

Triggered when the restart count is beyond the critical threshold. · Triggered when there are no pods ready for CX Contact deployment. · Triggered when the memory usage of a pod is above the critical threshold. · Triggered when the memory utilization of a pod is beyond the threshold. · Triggered when the number of pods ready for a CX Contact deployment is less than or equal to the threshold. · Triggered when the restart count for a pod is beyond the threshold. · Triggered when the CPU utilization of a pod is beyond the threshold · Actions: *Restart the pod. Check if there are any issues with the pod after restart. · Actions: *One of the containers in the pod has entered a failed state. Check the Kibana logs for the reason. · Actions: *If this alarm is triggered, check whether the CPU is available for the pods. *Check whether the port of the pod is running and serving the request. · Triggered when a pod's CPU utilization is beyond the threshold. · The desired number of replicas is higher than the current available replicas for the past 5 minutes. · Triggered when the latency for API responses is beyond the defined threshold. · Triggered when the connection to redis fails for more than 1 minute. · Triggered when a pod's ready count is less than the threshold (1). · Triggered when a pod's restart count is beyond the threshold. · Actions: *Restart the pod. Check if there are any issues with pod after restart. · Triggered when a pod's memory utilization is beyond the threshold. · Triggered when file share usage is greater than the threshold. · Actions: *Check if the new version of the image was deployed. *Check for issues with the Kubernetes cluster.

Other values: ; A GES pod has encountered difficulty resolving DNS requests. A Genesys Authentication pod has CPU usage above 300% during the last 5 minutes. <br /> A Genesys Authentication pod has memory usage above 70% in the last 5 minutes. <br /> A Genesys Authentication pod has memory usage above 90% in the last 5 minutes. A Genesys Authentication pod has restarted 1 or more times during the last 5 minutes. A Genesys Authentication pod has restarted more than 5 times in the last 5 minutes. A nonzero value indicates that errors have been logged during the scrape interval. A noticeable reduction in the number of active calls on a specific SIP Server and no new calls are arriving for processing. Actions: *If a problematic SIP Server is primary, do a switchover, and then restart the former primary server. *If a problematic SIP Server is backup, restart the backup serv A zero value for a recent period (several scrape intervals) indicates that RAA is not operating. Aborted due to initialization error e.g., KAFKA_FQDN is not defined Absence of received requests for pod '"`UNIQ--nowiki-00000060-QINU`"'. Actions: *For pod '"`UNIQ--nowiki-00000061-QINU`"', make sure there are no issues with Orchestration Service and Tenant Service or the network to them. Absence of received requests for pod '"`UNIQ--nowiki-0000007F-QINU`"'. Actions: *For pod '"`UNIQ--nowiki-00000080-QINU`"', make sure there are no issues with Orchestration Service and Tenant Service or the network to them. Actions: *Check Kafka and '"`UNIQ--nowiki-0000004A-QINU`"' service overload, network degradation. Actions: *Check Kafka and '"`UNIQ--nowiki-000000E6-QINU`"' service overload, network degradation. Actions: *Check if there is any problem with pod '"`UNIQ--nowiki-0000005B-QINU`"', then restart the pod. Actions: *Check if there is any problem with pod '"`UNIQ--nowiki-0000005D-QINU`"' and the config node. Actions: *Check if there is any problem with pod '"`UNIQ--nowiki-000000F7-QINU`"', then restart the pod. Actions: *Check if there is any problem with pod '"`UNIQ--nowiki-000000F9-QINU`"' and the config node. Actions: *Check the dialplan dashboard for Aggregated Service Health errors and, in case of a Redis error, first check for any issues/crashes in the pod and then restart Redis. *In the case of an Envoy error, the dialplan container will be restarted by the liveness probe. If the issue still exists Actions: *Collect the service logs for pod '"`UNIQ--nowiki-0000005C-QINU`"'; raise an investigation ticket. *Restart the service. Actions: *Collect the service logs for pod '"`UNIQ--nowiki-0000007B-QINU`"'; raise an investigation ticket. *Restart the service. Actions: *Ensure there are no issues with Kafka or '"`UNIQ--nowiki-00000057-QINU`"' service's CPU and network. Actions: *Ensure there are no issues with Kafka or '"`UNIQ--nowiki-00000066-QINU`"' pod's CPU and network. Actions: *Ensure there are no issues with Kafka or '"`UNIQ--nowiki-0000007C-QINU`"' service's CPU and network. Actions: *Ensure there are no issues with Kafka or '"`UNIQ--nowiki-00000102-QINU`"' pod's CPU and network. Actions: *If the alarm is generated for all dialplan pods, then Redis or network delay might be the most probable cause. *If the alarm is generated in a single dialplan pod, then it might be due to Envoy or a network issue. Actions: *If the alarm is triggered for multiple pods, make sure there are no issues with the SIP Cluster Service (CPU, memory, or network overload). *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000005E-QINU`"', check if there is an issue with the pod (CPU, memory, or network overload Actions: *If the alarm is triggered for multiple pods, make sure there are no issues with the SIP Cluster Service (CPU, memory, or network overload). *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000007D-QINU`"', check if there is an issue with the pod (CPU, memory, or network overload Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for container '"`UNIQ--nowiki-0000004C-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for container '"`UNIQ--nowiki-0000004E-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for container '"`UNIQ--nowiki-00000050-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for container '"`UNIQ--nowiki-000000E8-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for container '"`UNIQ--nowiki-000000EA-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for container '"`UNIQ--nowiki-000000EC-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000057-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-000000F3-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Redis. Restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000059-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Redis. Restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-000000F5-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000035-QINU`"', check the health of the pod. Actions: *If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-0000003A-QINU`"', check the health of the pod. Actions: *If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-0000003D-QINU`"', check the health of the pod. Actions: *If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000044-QINU`"', check the health of the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000004B-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000070-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-00000040-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-00000042-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-00000044-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-0000004A-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-0000004C-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-0000004E-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-00000050-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-00000065-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-00000067-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for '"`UNIQ--nowiki-00000069-QINU`"', check if there is an issue with the service. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis and then restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000033-QINU`"', check to see if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis and then restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000035-QINU`"', check to see if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis and then restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000036-QINU`"', check to see if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis and then restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000038-QINU`"', check to see if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If alarm is triggered only for pod '"`UNIQ--nowiki-00000034-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If alarm is triggered only for pod '"`UNIQ--nowiki-0000003B-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000036-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000003D-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000004D-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000053-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000055-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000057-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000072-QINU`"', check if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, then restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-0000002E-QINU`"', check to see if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, then restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000030-QINU`"', check to see if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, then restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-0000003D-QINU`"', check to see if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, then restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-0000003F-QINU`"', check to see if there is an issue with the pod. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000033-QINU`"', check to see whether the image is correct and if the container is starting up. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000038-QINU`"', check whether the image is correct and if the container is starting up. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-0000003B-QINU`"', check whether the image is correct and if the container is starting up. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000042-QINU`"', check to see whether the image is correct and if the container is starting up. Actions: *If the alarm is triggered for multiple sipproxy-nodes, make sure there are no issues on '"`UNIQ--nowiki-00000057-QINU`"'. *If the alarm is triggered only for sipproxy-node '"`UNIQ--nowiki-00000058-QINU`"', check to see if there is an issue with the service related to the topic (CPU, m Actions: *If the alarm is triggered for multiple sipproxy-nodes, make sure there are no issues on '"`UNIQ--nowiki-00000069-QINU`"'. *If the alarm is triggered only for sipproxy-node '"`UNIQ--nowiki-0000006A-QINU`"', check to see if there is an issue with the service related to the topic (CPU, m Actions: *If the alarm is triggered for multiple topics, ensure there are no issues with Kafka (CPU, memory, or network overload). *If the alarm is triggered only for topic '"`UNIQ--nowiki-0000003E-QINU`"', check if there is an issue with the service related to the topic (CPU, memory, or network Actions: *If the alarm is triggered for multiple topics, ensure there are no issues with Kafka (CPU, memory, or network overload). *If the alarm is triggered only for topic '"`UNIQ--nowiki-00000048-QINU`"', check if there is an issue with the service related to the topic (CPU, memory, or network Actions: *If the alarm is triggered for multiple topics, ensure there are no issues with Kafka (CPU, memory, or network overload). *If the alarm is triggered only for topic '"`UNIQ--nowiki-00000063-QINU`"', check if there is an issue with the service related to the topic (CPU, memory, or network Actions: *If the alarm is triggered for multiple topics, ensure there are no issues with Kafka (CPU, memory, or network overload). *If the alarm is triggered only for topic '"`UNIQ--nowiki-000000E4-QINU`"', check if there is an issue with the service related to the topic (CPU, memory, or network Actions: *If the alarm is triggered for multiple topics, make sure there are no issues with Kafka (CPU, memory, or network overload). *If the alarm is triggered only for topic '"`UNIQ--nowiki-00000048-QINU`"', check if there is an issue with the service related to the topic (CPU, memory, or netwo Actions: *If the alarm is triggered for multiple topics, make sure there are no issues with Kafka (CPU, memory, or network overload). *If the alarm is triggered only for topic '"`UNIQ--nowiki-0000004A-QINU`"', check if there is an issue with the service related to the topic (CPU, memory, or netwo Actions: *Make sure there are no issues with Kafka or '"`UNIQ--nowiki-0000005A-QINU`"' pod's CPU and network. Actions: *Make sure there are no issues with Kafka or '"`UNIQ--nowiki-00000079-QINU`"' pod's CPU and network. Actions: *One of the container in the pod has entered a Failed state. Check the Kibana logs for the reason. Actions: *Restart the pod and check to see if there are any issues with the pod after restart. Actions: *Troubleshoot the SIP Server-to-SBC network connection. Collect network stats and escalate to the Network team to resolve network issues, if necessary. *Troubleshoot the SBC. Actions: *Troubleshoot the SIP Server-to-SIP Proxy nodes network connections. Collect network stats and escalate to the Network team to resolve network issues, if necessary. *Troubleshoot SIP Proxy nodes. All Pulse instances are down. All the MCP ports in MCP LRG are exceeded An individual GES pod is returning excessive HTTP 400 results. An individual GES pod is returning excessive HTTP 401 results. An individual GES pod is returning excessive HTTP 404 results. An individual GES pod is returning excessive HTTP 500 results. Because Dial Plan requests are huge in size or there is a connection issue with the Dial Plan node, the processing queue size increases in size. Actions: *Check SIP Server inbound call rate. *Check the connection between SIP Server and the Dial Plan node. Connection to ORS_REDIS is down. Connection to the GWS Configuration Service is down. Connection to the GWS Environment Service is down. Connection to the Genesys Authentication Service is down. Container '"`UNIQ--nowiki-0000004A-QINU`"' was repeatedly restarted. Actions: *One of the containers in the pod has entered a failed state. Check the Kibana logs for the reason. Container '"`UNIQ--nowiki-00000053-QINU`"' was repeatedly restarted. Actions: *Check if the new version of the image was deployed. *Check for issues with the Kubernetes cluster. Container '"`UNIQ--nowiki-00000062-QINU`"' was repeatedly restarted. Actions: *Check to see if a new version of the image was deployed. Also check for issues with the Kubernetes cluster. Container '"`UNIQ--nowiki-00000065-QINU`"' was repeatedly restarted. Actions: *Check if the new version of the image was deployed. *Check for issues with the Kubernetes cluster. Container '"`UNIQ--nowiki-00000074-QINU`"' was repeatedly restarted. Actions: *Check to see if a new version of the image was deployed. Also check for issues with the Kubernetes cluster. Container '"`UNIQ--nowiki-00000076-QINU`"' was restarted 5 or more times within 15 minutes. Actions: *Check if a new version of the image was deployed. *Check for issues with the Kubernetes cluster. Container '"`UNIQ--nowiki-00000095-QINU`"' was restarted 5 or more times within 15 minutes. Actions: *Check if a new version of the image was deployed. *Check for issues with the Kubernetes cluster. Containers are being throttled more than 1 time per second. Critical CPU load for pod '"`UNIQ--nowiki-00000021-QINU`"'. Critical CPU load for pod '"`UNIQ--nowiki-0000003F-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket. Critical CPU load for pod '"`UNIQ--nowiki-00000043-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket. Critical CPU load for pod '"`UNIQ--nowiki-00000046-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket. Critical CPU load for pod '"`UNIQ--nowiki-00000049-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket. Critical CPU load for pod '"`UNIQ--nowiki-0000004E-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket. Critical CPU load for pod '"`UNIQ--nowiki-00000050-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket. Critical CPU load for pod '"`UNIQ--nowiki-00000051-QINU`"'. Critical CPU load for pod '"`UNIQ--nowiki-00000052-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket. Critical CPU load for pod '"`UNIQ--nowiki-00000057-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000058-QINU`"'; raise an inv Critical CPU load for pod '"`UNIQ--nowiki-00000060-QINU`"'. Critical CPU load for pod '"`UNIQ--nowiki-00000067-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. Critical CPU load for pod '"`UNIQ--nowiki-00000069-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. Critical CPU load for pod '"`UNIQ--nowiki-00000069-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-0000006A-QINU`"'; raise an inv Critical CPU load for pod '"`UNIQ--nowiki-0000006D-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-0000006E-QINU`"' and raise an inv Critical CPU load for pod '"`UNIQ--nowiki-00000076-QINU`"'. Critical CPU load for pod '"`UNIQ--nowiki-0000007B-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. Critical CPU load for pod '"`UNIQ--nowiki-0000007F-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000080-QINU`"' and raise an inv Critical CPU load for pod '"`UNIQ--nowiki-0000009A-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. Critical CPU load for pod '"`UNIQ--nowiki-000000FC-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-00000020-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-0000003B-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket Critical memory usage for pod '"`UNIQ--nowiki-0000003F-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket Critical memory usage for pod '"`UNIQ--nowiki-00000042-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket Critical memory usage for pod '"`UNIQ--nowiki-00000045-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket Critical memory usage for pod '"`UNIQ--nowiki-0000004A-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket Critical memory usage for pod '"`UNIQ--nowiki-0000004C-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket Critical memory usage for pod '"`UNIQ--nowiki-0000004E-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs; raise an investigation ticket Critical memory usage for pod '"`UNIQ--nowiki-00000055-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-0000005D-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service for pod '"`UNIQ--nowiki-0000005E-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-00000064-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-00000065-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs: raise an investigation ticket Critical memory usage for pod '"`UNIQ--nowiki-00000067-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service. *Collect the service logs: raise an investigation ticket Critical memory usage for pod '"`UNIQ--nowiki-0000006F-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service for pod '"`UNIQ--nowiki-00000070-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-00000073-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service for pod '"`UNIQ--nowiki-00000074-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-0000007A-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-00000080-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service for pod '"`UNIQ--nowiki-00000081-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-00000085-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service for pod '"`UNIQ--nowiki-00000086-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-0000009F-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Restart the service for pod '"`UNIQ--nowiki-000000A0-QINU`"'. Critical memory usage for pod '"`UNIQ--nowiki-00000100-QINU`"'. Deadlocked JVM threads exist. Detected critical 5xx errors per second for Pulse container. Detected critical CPU usage by Pulse DCU Pod. Detected critical CPU usage by Pulse LDS Pod. Detected critical CPU usage by Pulse Permissions Pod. Detected critical CPU usage by Pulse Pod. Detected critical DN registration failures on Pulse DCU Stat Server. Detected critical Hikari connections pool usage by Pulse container. Detected critical disk usage by Pulse DCU Pod. Detected critical memory usage by Pulse DCU Pod. Detected critical memory usage by Pulse LDS Pod. Detected critical memory usage by Pulse Permissions Pod. Detected critical memory usage by Pulse Pod. Detected critical number of healthy Pulse containers. Detected too frequent restarts of DCU Pod container. Detected too frequent restarts of LDS Pod container. Detected too frequent restarts of Permissions Pod container. Detected too frequent restarts of Pulse Pod container. Dial Plan node is overloaded as the response latency increases. Actions: *Check that the inbound call rate to SIP Server is not too high. *Check the Dial Plan node CPU and memory usage. *Check the network connection between SIP Server and Dial Plan nodes. Dial Plan node rejects requests with an error or it doesn't respond to requests and requests are timed out. Actions: *Check the network connection between SIP Server and the Dial Plan host. *Check that Dial Plan nodes are running. Fired if the average response time for incoming requests begins to lag. Fires if GES encounters any incoming requests that have exceeded the maximum content length of 10mb on the internal port and 500kb for the external, public-facing port. Fires if there are issues with GES managing VQ Proxy Objects. Fires when fewer than two GES pods have been up for the last 15 minutes. For the past 5 minutes, the desired number of replicas is greater than the number of replicas currently available. Actions: *Check resources available for Kubernetes. Increase resources, if necessary. For the past 5 minutes, the desired number of replicas is higher than the number of replicas currently available. Actions: *Check resources available for Kubernetes. Increase resources, if necessary. GES has been having difficulties contacting Nexus. This alert is only relevant for customers who leverage the Push Notification feature in Genesys Callback. GES has encountered server or connection errors with GWS. GES has failed to submit a callback to ORS. GES has failed to write a message to the log. GES has high CPU usage for 1 minute. GES has high memory usage for a period of 90 seconds. GES is throttling callbacks for a specific tenant. GES is throttling callbacks to a given phone number. GES pods are not in the <tt>Ready</tt> state. Indicative of issues with the Redis connection or other problems with the Helm deployment. GES pods have been excessively crashing and restarting. Genesys Authentication has 0 pods ready in the last 5 minutes. <br /> Genesys Authentication has 1 pod ready in the last 5 minutes. Genesys Authentication has received more than 10 500 responses. Genesys Authentication has received more than 10 5xx responses. Genesys Authentication has received more than 20 login errors for the call center ID in the last 60 seconds. Genesys Authentication received more than 10 errors in the last 30 seconds during context initialization. A spike might indicate a network or configuration problem. Check the logs for details. Genesys Authentication received more than 20 SAML errors for the contact center ID in the last 60 seconds. Genesys Authentication received more than 20 SAML timing errors for the contact center ID in the last 60 seconds. Genesys Authentication received more than 3 errors in PSDK connections in the last 30 seconds. A spike might indicate a problem with the backend or a network issue. Check the logs for details. High CPU load for pod '"`UNIQ--nowiki-0000003D-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High CPU load for pod '"`UNIQ--nowiki-00000041-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High CPU load for pod '"`UNIQ--nowiki-00000044-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High CPU load for pod '"`UNIQ--nowiki-00000047-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High CPU load for pod '"`UNIQ--nowiki-0000004C-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High CPU load for pod '"`UNIQ--nowiki-0000004E-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High CPU load for pod '"`UNIQ--nowiki-0000004F-QINU`"'. High CPU load for pod '"`UNIQ--nowiki-00000050-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket High CPU load for pod '"`UNIQ--nowiki-0000005A-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-0000005B-QINU`"'; raise an investi High CPU load for pod '"`UNIQ--nowiki-0000005E-QINU`"'. High CPU load for pod '"`UNIQ--nowiki-00000061-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High CPU load for pod '"`UNIQ--nowiki-00000063-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High CPU load for pod '"`UNIQ--nowiki-0000006C-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-0000006D-QINU`"'; raise an investi High CPU load for pod '"`UNIQ--nowiki-00000070-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000071-QINU`"' and raise an investi High CPU load for pod '"`UNIQ--nowiki-00000074-QINU`"'. High CPU load for pod '"`UNIQ--nowiki-00000078-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000079-QINU`"'; raise an investi High CPU load for pod '"`UNIQ--nowiki-00000082-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000083-QINU`"' and raise an investi High CPU load for pod '"`UNIQ--nowiki-00000097-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000098-QINU`"'; raise an investi High CPU load for pod '"`UNIQ--nowiki-000000FA-QINU`"'. High container CPU usage. High memory usage for pod '"`UNIQ--nowiki-00000039-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High memory usage for pod '"`UNIQ--nowiki-0000003D-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket High memory usage for pod '"`UNIQ--nowiki-00000040-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket High memory usage for pod '"`UNIQ--nowiki-00000043-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High memory usage for pod '"`UNIQ--nowiki-00000048-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High memory usage for pod '"`UNIQ--nowiki-0000004A-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High memory usage for pod '"`UNIQ--nowiki-0000004C-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High memory usage for pod '"`UNIQ--nowiki-00000053-QINU`"'. High memory usage for pod '"`UNIQ--nowiki-00000060-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000061-QINU`"'; raise an inv High memory usage for pod '"`UNIQ--nowiki-00000062-QINU`"'. High memory usage for pod '"`UNIQ--nowiki-00000063-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High memory usage for pod '"`UNIQ--nowiki-00000065-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs; raise an investigation ticket. High memory usage for pod '"`UNIQ--nowiki-00000072-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000073-QINU`"'; raise an inv High memory usage for pod '"`UNIQ--nowiki-00000078-QINU`"'. High memory usage for pod '"`UNIQ--nowiki-0000007D-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-0000007E-QINU`"'; raise an inv High memory usage for pod '"`UNIQ--nowiki-0000009C-QINU`"'. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-0000009D-QINU`"'; raise an inv High memory usage for pod '"`UNIQ--nowiki-000000FE-QINU`"'. High number of TCP RetransSegs (retransmitted segments). High rate of incoming requests from Workspace Web Edition. If the value of cxi__projects__status is greater than 0, this alarm is set, indicating that reporting is not functioning properly. Indicates that the average duration of aggregation queries specified by the hierarchy, level, and mediaType labels is greater than the deadlock-threshold. Inter-node connectivity between RM nodes is lost for 5mins. JVM garbage collection occurs more than 10 times in the last 30 seconds. JVM garbage collection occurs too often. Kafka is not available for pod '"`UNIQ--nowiki-0000004D-QINU`"'. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000004E-QINU`"', check if there is an issue with the pod. Kafka is not available for pod '"`UNIQ--nowiki-00000050-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000051-QINU`"', check if there is an issue wit Kafka is not available for pod '"`UNIQ--nowiki-00000052-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000053-QINU`"', check if there is an issue wit Kafka is not available for pod '"`UNIQ--nowiki-0000005F-QINU`"'. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Kafka. Restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000060-QINU`"', check if there is an issue with the pod. Kafka is not available for pod '"`UNIQ--nowiki-00000068-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000069-QINU`"', check if there is an issue wit Kafka is not available for pod '"`UNIQ--nowiki-00000087-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Kafka, and then restart Kafka. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000088-QINU`"', check if there is an issue wit Kafka responds with errors at pod '"`UNIQ--nowiki-00000064-QINU`"'. Actions: *For pod '"`UNIQ--nowiki-00000065-QINU`"', make sure there are no issues with Kafka. Kafka responds with errors at pod '"`UNIQ--nowiki-00000066-QINU`"'. Actions: *For pod '"`UNIQ--nowiki-00000067-QINU`"', ensure there are no issues with Kafka. Kafka responds with errors at pod '"`UNIQ--nowiki-00000078-QINU`"'. Actions: *For pod '"`UNIQ--nowiki-00000079-QINU`"', ensure there are no issues with Kafka. Kafka responds with errors at pod '"`UNIQ--nowiki-00000083-QINU`"'. Actions: *For pod '"`UNIQ--nowiki-00000084-QINU`"', make sure there are no issues with Kafka. Main thread consumes too much CPU. Actions: *Collect SIP Server Main thread logs; that is, log files without index in the file name (appname_date.log files). Raise an investigation ticket. Matching IVR profile tenant could not be found for 2mins Media service is out of service. Actions: *Troubleshoot the SIP Server-to-Resource Manager (RM) network connection. Collect network stats and escalate to the Network team to resolve network issues, if necessary. *Troubleshoot RM, consider RM restart. *After 5 minutes, redirect traffic to another s Message handling exceeds 2 secs No DNs are registered on Pulse LDS. No Dial Plan nodes are reachable from SIP Server and all connections to Dial Plan nodes are down. Actions: *Check the network connection between SIP Server and the Dial Plan node host. *Check the Dial Plan node CPU and memory usage. No available SIP Nodes for pod '"`UNIQ--nowiki-0000006B-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with SIP Nodes, and then restart SIP Nodes. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000006C-QINU`"', check if there is an i No available SIP Nodes for pod '"`UNIQ--nowiki-0000008A-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with SIP Nodes, and then restart SIP Nodes. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000008B-QINU`"', check if there is an i No sip-nodes are available for the pod '"`UNIQ--nowiki-00000064-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with sip-nodes. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000065-QINU`"', check to see if there is any issues with t No sip-nodes are available for the pod '"`UNIQ--nowiki-00000076-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with sip-nodes. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000077-QINU`"', check to see if there is any issues with t Number of LMSIP media errors exceeded critical limit Number of SDP parse errors exceeded limit Number of VXMLi 4xx fetch errors exceeded limit Number of VXMLi fetch errors exceeded limit Number of VXMLi fetch timeouts exceeded limit Number of VXMLi parse errors exceeded limit One of the RM container is not in running state for 5mins One or more downstream components (PostGres, Config Server, GWS, ORS) are down. '''Note:''' Because GES goes into a crash loop when Redis is down, this does not fire when Redis is down. Pod '"`UNIQ--nowiki-00000038-QINU`"' failed. Actions: *One of the containers in the pod has entered a Failed state. Check the Kibana logs for the reason. Pod '"`UNIQ--nowiki-0000003A-QINU`"' is in Unknown state. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000003B-QINU`"', check whether the image is correct and if th Pod '"`UNIQ--nowiki-0000003D-QINU`"' is in Pending state. Actions: *If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-0000003E-QINU`"', check the health Pod '"`UNIQ--nowiki-0000003F-QINU`"' failed. Actions: *One of the containers in the pod has entered a Failed state. Check the Kibana logs for the reason. Pod '"`UNIQ--nowiki-00000040-QINU`"' failed. Actions: *One of the containers in the pod has entered a Failed state. Check the Kibana logs for the reason. Pod '"`UNIQ--nowiki-00000040-QINU`"' in NotReady state. Actions: *If this alarm is triggered, check whether the CPU is available for the pods. *Check whether the port of the pod is running and serving the request. Pod '"`UNIQ--nowiki-00000041-QINU`"' is in Unknown state. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000042-QINU`"', check whether the image is correct and if th Pod '"`UNIQ--nowiki-00000042-QINU`"' in Unknown state. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000043-QINU`"', check whether the image is correct and if t Pod '"`UNIQ--nowiki-00000044-QINU`"' is in Pending state. Actions: *If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000045-QINU`"', check the health Pod '"`UNIQ--nowiki-00000045-QINU`"' is in the Pending state. Actions: *If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-00000046-QINU`"', check the hea Pod '"`UNIQ--nowiki-00000047-QINU`"' in NotReady state. Actions: *If this alarm is triggered, check whether the CPU is available for the pods. *Check whether the port of the pod is running and serving the request. Pod '"`UNIQ--nowiki-00000048-QINU`"' in NotReady state. Actions: *If this alarm is triggered, check whether the CPU is available for the pods. *Check whether the port of the pod is running and serving the request. Pod '"`UNIQ--nowiki-00000051-QINU`"' is in NotReady state. Actions: *Restart the pod. Check if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000057-QINU`"' failed. Actions: *One of the containers in the pod has entered a failed state. Check the Kibana logs for the reason. Pod '"`UNIQ--nowiki-00000059-QINU`"' failed. Actions: *One of the containers in the pod has entered a failed state. Check the Kibana logs for the reason. Pod '"`UNIQ--nowiki-00000059-QINU`"' is in Unknown state. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000005A-QINU`"', check whether the image is correct and if th Pod '"`UNIQ--nowiki-0000005B-QINU`"' is in Unknown state. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000005C-QINU`"', check whether the image is correct and if th Pod '"`UNIQ--nowiki-0000005C-QINU`"' is in Pending state. Actions: *If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000005D-QINU`"', check the health of t Pod '"`UNIQ--nowiki-0000005C-QINU`"' is in Unknown state. Actions: *Restart the pod and check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-0000005E-QINU`"' is in Pending state. Actions: *If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster. *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000005F-QINU`"', check the health of t Pod '"`UNIQ--nowiki-0000005E-QINU`"' is in Pending state. Actions: *Restart the pod and check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000060-QINU`"' is in NotReady state. Actions: *Restart the pod and check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000063-QINU`"' is in NotReady state. Actions: *Restart the pod. Check if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-0000006E-QINU`"' is in Failed state. Actions: *Restart the pod. Check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-0000006E-QINU`"' is in Unknown state. Actions: *Restart the pod and check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000070-QINU`"' is in Pending state. Actions: *Restart the pod and check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000070-QINU`"' is in Unknown state for 5 minutes. Actions: *Restart the pod. Check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000072-QINU`"' is in NotReady state. Actions: *Restart the pod and check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000072-QINU`"' is in Pending state for 5 minutes. Actions: *Restart the pod. Check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000074-QINU`"' is in the NotReady state for 5 minutes. Actions: *Restart the pod. Check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000076-QINU`"' has high memory usage. Actions: *Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000077-QINU`"' and raise an inv Pod '"`UNIQ--nowiki-00000088-QINU`"' has high memory usage. Actions: *Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached. *Check Grafana for abnormal load. *Collect the service logs for pod '"`UNIQ--nowiki-00000089-QINU`"' and raise an inv Pod '"`UNIQ--nowiki-0000008D-QINU`"' is in Failed state. Actions: *Restart the pod. Check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-0000008F-QINU`"' is in Unknown state for 5 minutes. Actions: *Restart the pod. Check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000091-QINU`"' is in Pending state for 5 minutes. Actions: *Restart the pod. Check to see if there are any issues with the pod after restart. Pod '"`UNIQ--nowiki-00000093-QINU`"' is in the NotReady state for 5 minutes. Actions: *Restart the pod. Check to see if there are any issues with the pod after restart. Pule LDS is not connected to upstream servers. Pulse DCU Collector does not write snapshots. Pulse DCU Collector is not connected to ConfigServer. Pulse DCU Collector is not connected to DbServer. Pulse DCU Collector is not connected to Stat Server. Pulse DCU Monitor Agents do not provide data. Pulse DCU Stat Server is not connected to ConfigServer. Pulse DCU Stat Server is not connected to IxnServers. Pulse DCU Stat Server is not connected to T-Servers. Pulse LDS Monitor Agents do not provide data. RM Inter node Socket Error for 5mins. RM Resource allocation failed for 1mins RM lost connection to GVP Configuration Server for 5mins. RM pods are not in ready state and RM service is not available Redis is not available for pod '"`UNIQ--nowiki-00000063-QINU`"'. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Redis. Restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000064-QINU`"', check if there is an issue with the pod. Redis is not available for pod '"`UNIQ--nowiki-00000075-QINU`"'. Actions: *If the alarm is triggered for multiple services, ensure there are no issues with Redis. Restart Redis. *If the alarm is triggered only for pod '"`UNIQ--nowiki-00000076-QINU`"', check if there is an issue with the pod. Redis is not available for the pod '"`UNIQ--nowiki-0000003A-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-0000003B-QINU`"', check to see if there is any is Redis is not available for the pod '"`UNIQ--nowiki-0000003D-QINU`"'. Actions: *If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis. *If the alarm is triggered only for the pod '"`UNIQ--nowiki-0000003E-QINU`"', check to see if there SIP Cluster Service responds with errors at pod '"`UNIQ--nowiki-00000066-QINU`"'. Actions: *If the alarm is triggered for multiple pods, make sure there are no issues with the SIP Cluster Service (CPU, memory, or network overload). *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000006 SIP Cluster Service responds with errors at pod '"`UNIQ--nowiki-00000085-QINU`"'. Actions: *If the alarm is triggered for multiple pods, make sure there are no issues with the SIP Cluster Service (CPU, memory, or network overload). *If the alarm is triggered only for pod '"`UNIQ--nowiki-0000008 SIP Node health level fails for pod '"`UNIQ--nowiki-0000004B-QINU`"'. Actions: *Check for failure of dependent services (Redis/Kafka/SIP Proxy/GVP/Dial Plan). *Check for Envoy proxy failure, then restart the pod. SIP Node health level fails for pod '"`UNIQ--nowiki-0000005D-QINU`"'. Actions: *Check for failure of dependent services (Redis/Kafka/SIP Proxy/GVP/Dial Plan). *Check for Envoy proxy failure, then restart the pod. SIP Proxy is overloaded. Actions: *Check SIP Proxy nodes for CPU and memory usage. *If SIP Proxy nodes have acceptable CPU and memory usage, then check for errors or a "hang-up" state which could delay SIP Proxy in forwarding. *Check the SBC side for network delays. SIP trunk is out of service. Actions: *For Primary and Secondary trunks: **Troubleshoot SIP Server-to-SBC network connection. Collect network stats and escalate to the Network team to resolve network issues, if necessary. **Troubleshoot the SBC. For Inter-SIP Server trunks: troubleshoot the SIP Se Specifies that the Gateway Pod has lost connection to Auth service Specifies that the Gateway Pod has lost connection to ElasticSearch Specifies that the Gateway Pod has lost connection to the Environment Service Specifies the number of sign-ins The GWS client credentials provided to GES are incorrect. The Node.js event loop is too slow. The RM mib counter stats will be collected for every 30 seconds and if the mib counter total5xxInviteSent increments from its previous value by 5 within 5 minutes the trigger will flag an alarm. The RM mib counter stats will be collected for every 60 seconds and if the mib counter total4xxInviteSent increments from its previous value by 10 within 60 seconds the trigger will flag an alarm. The Storage account is down and, as a result, the service will not be able to fetch the data. The Voicemail pod is down. The Voicemail pod restarts repeatedly. The Workspace Service deployment doesn't have the desired number of replicas. The Workspace Service has too many 500 responses. The current number of replicas is less than the minimum replicas that should be available. This might be because Kubernetes cannot deploy a new pod or pods are failing to be active/ready. Actions: *If all services have the same issue, then check Kubernetes nodes and Consul health. *If the issue is The current number of replicas is lower than the minimum number of replicas that should be available. Actions: *Check if Kubernetes cannot deploy new pods or if pods are failing in their status to be active/read. The current number of replicas is more than 80% of the maximum number of replicas. Actions: *Check if max replicas must be modified based on load. The number of running instances is 0 The number of statefulset '"`UNIQ--nowiki-00000055-QINU`"' pods in the Ready state has dropped below 60%. Actions: *Check if the new version of the image was deployed. *Check for issues with the Kubernetes cluster. The number of statefulset '"`UNIQ--nowiki-00000067-QINU`"' pods in the Ready state has dropped below 60%. Actions: *Check if the new version of the image was deployed. *Check for issues with the Kubernetes cluster. The request to the config node failed. Action: *Check if there is any problem with pod '"`UNIQ--nowiki-00000079-QINU`"' and config node. The request to the config node failed. Action: *Check if there is any problem with pod '"`UNIQ--nowiki-0000008B-QINU`"' and config node. The sip-node '"`UNIQ--nowiki-00000067-QINU`"' hit capacity limit on '"`UNIQ--nowiki-00000068-QINU`"'. Actions: *If alarm is triggered for multiple services make sure there is no issues with sip-node '"`UNIQ--nowiki-00000069-QINU`"'. *If alarm is triggered only for pod '"`UNIQ--nowiki-000000 The sip-node '"`UNIQ--nowiki-00000079-QINU`"' hit capacity limit on '"`UNIQ--nowiki-0000007A-QINU`"'. Actions: *If alarm is triggered for multiple services make sure there is no issues with sip-node '"`UNIQ--nowiki-0000007B-QINU`"'. *If alarm is triggered only for pod '"`UNIQ--nowiki-000000 The trigger detects that routing timeouts are increasing. Actions: *Check the URS_RESPONSE_MORE5SEC stat value. If it's increasing, then investigate why URS doesn't respond to SIP Server in time. *Check SIPS-to-URS network connectivity. The trigger will flag an alarm when RM pod status is Not ready for 30 mins and this will be controlled by override-value.yaml. The trigger will flag an alarm when RS JMS message queue size goes beyond 15000 (3GB approx. backlog) for 15 mins The trigger will flag an alarm when RS pod status is Not ready for 30 mins and this will be controlled through override-value.yaml file. The trigger will flag an alarm when the Configserver container CPU utilization goes beyond 70% for 15 mins The trigger will flag an alarm when the Configserver container working memory has exceeded 1GB for 15 mins The trigger will flag an alarm when the Configserver container working memory use is over 90% of the limit for 15 mins The trigger will flag an alarm when the MCP container CPU utilization goes beyond 70% for 5 mins The trigger will flag an alarm when the MCP container restarts exceeded 2 for 15 mins The trigger will flag an alarm when the MCP container working memory has exceeded 7GB for 5 mins The trigger will flag an alarm when the MCP container working memory use is over 90% of the limit for 5 mins The trigger will flag an alarm when the RM container CPU utilization goes beyond 80% for 15 mins The trigger will flag an alarm when the RM container Memory utilization goes beyond 80% for 15 mins The trigger will flag an alarm when the RM init container gets failed 5 or more times within 15 mins. The trigger will flag an alarm when the RM or RM SNMP container gets restarted 5 or more times within 15 mins The trigger will flag an alarm when the RS container CPU utilization goes beyond 80% for 15 mins The trigger will flag an alarm when the RS container Memory utilization goes beyond 80% for 15 mins The trigger will flag an alarm when the RS init container gets failed 5 or more times within 15 mins The trigger will flag an alarm when the RS or RS SNMP container gets restarted 5 or more times within 15 mins There are errors creating a JWT token with a websocket client There are errors fetching Auth token with a websocket client There are errors getting information for Auth token with a websocket client There are errors opening a session with a websocket client There are protocol errors with a websocket client There has been an uncaught exception within GES. This alert indicates problems with the cluster states. Applicable only if you have two or more nodes in a cluster. This alert is triggered when the Configserver container has not been running for 15 minutes This alert is triggered when the Configserver container restarts in 15 mins exceeded 4 This alert is triggered when the service-handler container has not been running for 15 minutes This alert is triggered when the service-handler container restarts exceeded 4 for 15 mins This trigger will flag an alarm when the RS PVC size is 50% filled <br /> This trigger will flag an alarm when the RS PVC size is 80% filled <br /> Too many 500 responses. Too many 5xx responses. Too many Kafka producer pending events for pod '"`UNIQ--nowiki-00000048-QINU`"'. Actions: *Ensure there are no issues with Kafka, '"`UNIQ--nowiki-00000049-QINU`"' pod's CPU, and network. Too many Kafka producer pending events for pod '"`UNIQ--nowiki-00000054-QINU`"'. This alert means there are issues with SIP REGISTER processing on this voice-sipproxy. Actions: *Make sure there are no issues with Kafka or with the '"`UNIQ--nowiki-00000055-QINU`"' pod's CPU and network. Too many Kafka producer pending events for pod '"`UNIQ--nowiki-0000005A-QINU`"'. Actions: *Ensure there are no issues with Kafka, '"`UNIQ--nowiki-0000005B-QINU`"' pod's CPU, and network. Too many Kafka producer pending events for pod '"`UNIQ--nowiki-00000066-QINU`"'. This alert means there are issues with SIP REGISTER processing on this voice-sipproxy. Actions: *Make sure there are no issues with Kafka or with the '"`UNIQ--nowiki-00000067-QINU`"' pod's CPU and network. Too many active sessions. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached. *Check the number of voice, digital, and callback calls in the system. Too many active sessions. Actions: *Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has reached. *Check the number of voice, digital, and callback calls in the system. Too many failure responses are sent by the Front End service at pod '"`UNIQ--nowiki-00000062-QINU`"'. Actions: *For pod '"`UNIQ--nowiki-00000063-QINU`"', make sure received requests are valid. Too many failure responses are sent by the Front End service at pod '"`UNIQ--nowiki-00000081-QINU`"'. Actions: *For pod '"`UNIQ--nowiki-00000082-QINU`"', make sure received requests are valid. Total count of errors during context initialization. Total count of errors in PSDK connections. Triggered when CX Contact compliance has no active redis connection for 2 minutes Triggered when DAS exceeds the 4xx error count threshold specified here. Triggered when DAS exceeds the allowed 5xx error count threshold specified here. Triggered when DAS health status is 0. Triggered when DAS is not able to communicate with the workspace. Triggered when DAS pod metrics are unavailable. Triggered when Designer health status is 0. Triggered when Designer is not able to communicate with the workspace. Triggered when Designer pod metrics are unavailable. Triggered when Designer/DAS experiences a PHP Health check failure. Triggered when Designer/DAS experiences a Proxy Health check failure. Triggered when Designer/DAS is not able to reach the Elasticsearch server. Triggered when Designer/DAS is not able to reach the GWS server. Triggered when Elasticsearch internal processing time > 500 ms Triggered when GSP encounters unknown person(s) Triggered when IWD experiences more than 2 errors within 1 minute during operations with database. Triggered when Pulse DCU instances are down. Triggered when Pulse LDS instances are down. Triggered when Pulse Permissions instances are down. Triggered when Pulse instances are down. Triggered when a GCA pod is crash looping. Triggered when a GCA pod is restarted because of OOMKilled. Triggered when a GSP pod is restarted because of OOMKilled Triggered when a the CPU utilization of a pod is beyond the threshold Triggered when average CPU usage is more than 60% Triggered when average CPU usage is more than 80% Triggered when average CPU usage is more than 800 Mb Triggered when average http response time > 500 ms Triggered when average memory usage is more than 60% Triggered when network traffic exceeds 10MB/second for 5 minutes Triggered when overload protection rate is more than 0 Triggered when pod database connections number is above 75. Triggered when request rate is more than 120 requests per seconds on one UCS-X instance Triggered when the CPU utilization of a pod is beyond the threshold. Triggered when the GSP Flink job is not running (number of running jobs equals to 0 or metric is not available) Triggered when the Ingress error rate is above the specified threshold. Triggered when the average time taken by a HTTP request is greater than the threshold (in seconds) specified here. Triggered when the average time taken by a PHP request is greater than the threshold (in seconds) specified here. Triggered when the error rate on this pod is greater than 20% for 15 minutes. Triggered when the latency for dial manager is above the defined threshold. Triggered when the latency for job scheduler is above the defined threshold. Triggered when the latency for list builder is above the defined threshold. Triggered when the latency for list manager is above the defined threshold Triggered when the memory usage on this pod is above 3000 Mb for 15 minutes. Triggered when the number of HTTP errors exceeds 500 responses in 5 minutes Triggered when the number of errors in IWD exceeds the threshold for 15 min period. Triggered when the number of healthy pods drops to critical level Triggered when the pod memory usage is above 3000 MB. Triggered when there are no registered TaskManagers (or metric not available) Triggered when there are too many error responses from the auth service (list manager) for more than the specified time threshold. Triggered when there are too many error responses from the auth service for more than the specified time threshold. Triggered when there is no connection to ElasticSearch Triggered when there is no connection to master DB Triggered when there is no connection to one of the dependent services - GAuth, Config, Prometheus Triggered when there is no connection to tenant DB Triggered when there is no connection to the GAuth service Triggers if the base NodeJS event loop becomes excessive. This indicates significant resource and performance issues with the deployment. Unable to connect to Engage Kafka Unable to connect to GWS auth service Unable to connect to GWS config service Unable to connect to GWS environment service Unable to connect to a customer endpoint Voicemail Service '"`UNIQ--nowiki-0000001E-QINU`"' unable to connect to Config Node. Voicemail Service '"`UNIQ--nowiki-0000001F-QINU`"' unable to connect to the Redis cluster. Voicemail Service '"`UNIQ--nowiki-00000023-QINU`"' Envoy service is not available. Voicemail Service '"`UNIQ--nowiki-00000025-QINU`"' GWS service is not available. Voicemail Service '"`UNIQ--nowiki-00000027-QINU`"' GWS service is not available.

90% for 5m (14)