Drilldown: PEAlert

Critical (7) · HIGH (14)

Deadlocked JVM threads exist. (3) · RM pods are not in ready state and RM service is not available (2) · Triggered when a GSP pod is restarted because of OOMKilled (2) · Triggered when a the CPU utilization of a pod is beyond the threshold (2) · Triggered when the CPU utilization of a pod is beyond the threshold (10) · Triggered when the CPU utilization of a pod is beyond the threshold. (2)

1 for 5m · 70% for 5m · 0 for 1m · 5 for 5m · 300% for 5m · 90% for 5m · 5000ms for 5m · 15mins · 90% · 2 for 1 hour · for 15 minutes · 0 · 1m · 30mins · 80%
Default interval: 180s · 75%
Default interval: 180s · 75% · 5mins · 5
Default interval: 180s · The desired number of replicas is higher than the current available replicas for the past 5 minutes.

Other values: ; (kube_hpa_status_current_replicas{namespace="voice",hpa="sipfe-node-hpa"} * 100) / kube_hpa_spec_max_replicas{namespace="voice",hpa="sipfe-node-hpa"} > 80 for: 5m 0 in 5 minutes 0.2 0.5% 0<br>Default interval: 60s 1 in 5 minutes 1 or more in 5 minutes 10 seconds 100<br>Default interval: 180s 10<br>Default interval: 180s 10s<br>Default interval: 180s 15 mins 15% 1<br>Default interval: 60s 1min 2 minutes 20% for 5m 2500ms for 5m 2m 2mins 30 minutes 30 seconds 300% 5 mins 5 minutes 50% < 0 <2 <80 >0 >10000 >10MBps >500 in 5 minutes >60% Agent health level is Fail for pod '"`UNIQ--nowiki-0000005C-QINU`"' for 5 consecutive minutes. Agent health level is Fail for pod '"`UNIQ--nowiki-000000F8-QINU`"' for 5 consecutive minutes. All connections to Dial Plan nodes are down. At least 6 requests failed per minute for the past 10 minutes. Container '"`UNIQ--nowiki-00000022-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000038-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-0000003A-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000003C-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000003E-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000003E-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000040-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000040-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000041-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000042-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000042-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000043-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000044-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000044-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000045-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000046-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000047-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000047-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000048-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000049-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000049-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-0000004A-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000004A-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-0000004B-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000004B-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000004B-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-0000004D-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000004D-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000004D-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000004F-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000004F-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000004F-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000050-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000051-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000051-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000052-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000053-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000054-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000054-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000056-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000056-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000059-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000005C-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000005F-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000005F-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000060-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000061-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000062-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000062-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000062-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000063-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000063-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000064-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000064-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000065-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000066-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000066-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000066-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000068-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000068-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000006A-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000006B-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000006E-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000006F-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000006F-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000071-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000072-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000074-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000075-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000075-QINU`"' memory usage exceeded 80% for 5 minutes Container '"`UNIQ--nowiki-00000075-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000077-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000077-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000078-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000079-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000007A-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000007B-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000007C-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000007F-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000081-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000082-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-00000084-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000087-QINU`"' memory usage exceeded 80% for 5 minutes Container '"`UNIQ--nowiki-0000008A-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000096-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-00000099-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-0000009B-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-0000009E-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-000000A1-QINU`"' memory usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-000000F2-QINU`"' was restarted 5 or more times within 15 minutes. Container '"`UNIQ--nowiki-000000FB-QINU`"' CPU usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-000000FD-QINU`"' CPU usage exceeded 80% for 5 minutes. Container '"`UNIQ--nowiki-000000FF-QINU`"' memory usage exceeded 65% for 5 minutes. Container '"`UNIQ--nowiki-00000101-QINU`"' memory usage exceeded 80% for 5 minutes. Default memory usage: 3000 MB Default number of connections: 75 Default number of errors: 2 Dependent services or the Envoy sidecar is not available for 5 minutes in the pod '"`UNIQ--nowiki-00000032-QINU`"'. Dependent services or the Envoy sidecar is not available for 5 minutes in the pod '"`UNIQ--nowiki-00000035-QINU`"'. Desired number of replicas is higher than current available replicas for the past 5 minutes. Dial Plan node is overloaded as the response latency increases (more than 1000). During 1 minute, the Dial Plan node rejects more than 5 requests with an error or more than 5 requests time out because the Dial Plan node fails to respond. Fired when number of available replicas does not equal to configured number. For 5 consecutive minutes, the number of replicas is less than the minimum replicas that should be available. For 5 consecutive minutes, the number of replicas is more than 80% of the maximum number of replicas. For 5 minutes For the last 5 minutes, fewer than 60% of the currently available statefulset '"`UNIQ--nowiki-00000056-QINU`"' pods have been in the Ready state. For the last 5 minutes, fewer than 60% of the currently available statefulset '"`UNIQ--nowiki-00000068-QINU`"' pods have been in the Ready state. For the past 5 minutes, the current number of replicas is lower than the minimum number of replicas that should be available. For too many requests, the Front End service at pod '"`UNIQ--nowiki-0000005D-QINU`"' did not send any response (more than 100 requests without a response, measured over 5 minutes). For too many requests, the Front End service at pod '"`UNIQ--nowiki-0000007C-QINU`"' did not send any response (more than 100 requests without a response, measured over 5 minutes). Greater than the value (seconds) of raa.prometheusRule.alerts.longAggregation.thresholdSec in values.yaml. <br/>'''Recommended value''': 300 Health check failed more than 10 times in 5 minutes for Kafka consumer for topic '"`UNIQ--nowiki-00000041-QINU`"'. Health check failed more than 10 times in 5 minutes for Kafka consumer for topic '"`UNIQ--nowiki-0000004D-QINU`"'. Health check failed more than 10 times in 5 minutes for Kafka consumer for topic '"`UNIQ--nowiki-00000066-QINU`"'. Health check failed more than 10 times in 5 minutes for Kafka consumer for topic '"`UNIQ--nowiki-000000E9-QINU`"'. Health check failed more than 10 times in 5 minutes for Kafka consumer for topic  '"`UNIQ--nowiki-0000004B-QINU`"'. Health check failed more than 10 times in 5 minutes for Kafka consumer for topic  '"`UNIQ--nowiki-0000004D-QINU`"'. If the latency for 95% of the dial plan messages is more than 2 seconds for a duration of 5 minutes, then this warning alarm is raised for the '"`UNIQ--nowiki-00000031-QINU`"'. If the latency for 95% of the dial plan messages is more than 2 seconds for a duration of 5 minutes, then this warning alarm is raised for the '"`UNIQ--nowiki-00000034-QINU`"'. Kafka is not available for pod '"`UNIQ--nowiki-0000004C-QINU`"' for 5 consecutive minutes. Kafka is not available for pod '"`UNIQ--nowiki-0000004F-QINU`"' for 5 minutes. Kafka is not available for pod '"`UNIQ--nowiki-00000052-QINU`"' for 5 consecutive minutes. Kafka is not available for pod '"`UNIQ--nowiki-00000054-QINU`"' for 5 consecutive minutes. Kafka is not available for pod '"`UNIQ--nowiki-00000058-QINU`"' for 5 consecutive minutes. Kafka is not available for pod '"`UNIQ--nowiki-00000061-QINU`"' for 5 minutes. Kafka is not available for pod '"`UNIQ--nowiki-0000006A-QINU`"' for 5 consecutive minutes. Kafka is not available for pod '"`UNIQ--nowiki-00000071-QINU`"' for 5 consecutive minutes. Kafka is not available for pod '"`UNIQ--nowiki-00000089-QINU`"' for 5 consecutive minutes. Kafka is not available for pod '"`UNIQ--nowiki-000000F4-QINU`"' for 5 consecutive minutes. Latency for 95% of messages is more than 0.5 seconds for service '"`UNIQ--nowiki-0000005F-QINU`"'. Latency for 95% of messages is more than 0.5 seconds for service '"`UNIQ--nowiki-0000007E-QINU`"'. Latency for more than 5% of messages is more than 0.5 seconds for topic '"`UNIQ--nowiki-0000003F-QINU`"'. Latency for more than 5% of messages is more than 0.5 seconds for topic '"`UNIQ--nowiki-00000049-QINU`"'. Latency for more than 5% of messages is more than 0.5 seconds for topic '"`UNIQ--nowiki-0000004B-QINU`"'. Latency for more than 5% of messages is more than 0.5 seconds for topic '"`UNIQ--nowiki-00000064-QINU`"'. Latency for more than 5% of messages is more than 0.5 seconds for topic '"`UNIQ--nowiki-000000E5-QINU`"'. Main thread consumes too much CPU (more than 65% for 5 consecutive minutes). Media service is out of service for more than 1 minute. More than 10 More than 10 in 30 seconds More than 10 request timeouts appeared in 5 minutes for Kafka consumer for topic '"`UNIQ--nowiki-00000043-QINU`"'. More than 10 request timeouts appeared in 5 minutes for Kafka consumer for topic '"`UNIQ--nowiki-0000004F-QINU`"'. More than 10 request timeouts appeared in 5 minutes for Kafka consumer for topic '"`UNIQ--nowiki-00000068-QINU`"'. More than 10 request timeouts appeared in 5 minutes for Kafka consumer for topic '"`UNIQ--nowiki-000000EB-QINU`"'. More than 100 errors for 5 consecutive minutes. More than 100 errors in 5 consecutive minutes. More than 100 failure responses in 5 consecutive minutes. More than 10000 active streams running. More than 20 in 60 seconds More than 3 Kafka consumer crashes in 5 minutes for service '"`UNIQ--nowiki-00000051-QINU`"'. More than 3 Kafka consumer crashes in 5 minutes for service '"`UNIQ--nowiki-000000ED-QINU`"'. More than 3 Kafka consumer crashes in 5 minutes for topic '"`UNIQ--nowiki-00000045-QINU`"'. More than 3 Kafka consumer crashes in 5 minutes for topic '"`UNIQ--nowiki-0000006A-QINU`"'. More than 3 in 30 seconds More than 300% in 5 minutes More than 400 strategies running in 5 consecutive seconds. More than 5 in 5 minutes More than 600 strategies running in 5 consecutive seconds. More than 70% in 5 minutes More than 90% in 5 minutes N/A Need input No available SIP Nodes for pod '"`UNIQ--nowiki-0000006D-QINU`"' for 5 consecutive minutes. No available SIP Nodes for pod '"`UNIQ--nowiki-0000008C-QINU`"' for 5 consecutive minutes. No sip-nodes are available for the pod '"`UNIQ--nowiki-00000066-QINU`"' for 2 minutes. No sip-nodes are available for the pod '"`UNIQ--nowiki-00000078-QINU`"' for 2 minutes. Number of sent requests is two times higher than received for topic '"`UNIQ--nowiki-0000004B-QINU`"'. Number of sent requests is two times higher than received for topic '"`UNIQ--nowiki-000000E7-QINU`"'. Pod failed '"`UNIQ--nowiki-00000032-QINU`"'. Pod failed '"`UNIQ--nowiki-00000041-QINU`"'. Pod '"`UNIQ--nowiki-00000034-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-00000036-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000037-QINU`"' failed. Pod '"`UNIQ--nowiki-00000037-QINU`"' is in NotReady state for 10 minutes. Pod '"`UNIQ--nowiki-00000039-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-00000039-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-0000003A-QINU`"' failed. Pod '"`UNIQ--nowiki-0000003B-QINU`"' is in the Pending state for 5 minutes. Pod '"`UNIQ--nowiki-0000003C-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-0000003C-QINU`"' is in the NotReady state for 10 minutes. Pod '"`UNIQ--nowiki-0000003E-QINU`"' is in the Pending state for 5 minutes. Pod '"`UNIQ--nowiki-0000003F-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-0000003F-QINU`"' is in the NotReady state for 10 minutes. Pod '"`UNIQ--nowiki-00000040-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-00000041-QINU`"' in NotReady state for 10 minutes. Pod '"`UNIQ--nowiki-00000041-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-00000043-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-00000044-QINU`"' in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-00000045-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000046-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-00000046-QINU`"' is in NotReady state for 10 minutes. Pod '"`UNIQ--nowiki-00000046-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000047-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-00000047-QINU`"' is in the Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000048-QINU`"' in NotReady state for 10 minutes. Pod '"`UNIQ--nowiki-00000048-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000049-QINU`"' in NotReady state for 10 minutes. Pod '"`UNIQ--nowiki-00000049-QINU`"' is in NotReady status for 5 minutes. Pod '"`UNIQ--nowiki-00000050-QINU`"' is in Failed, Unknown, or Pending state. Pod '"`UNIQ--nowiki-00000052-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-00000052-QINU`"' is in NotReady state for 5 minutes. Pod '"`UNIQ--nowiki-00000053-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-00000054-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000055-QINU`"' is in NotReady status for 5 minutes. Pod '"`UNIQ--nowiki-00000058-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-0000005A-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-0000005B-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-0000005B-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-0000005D-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-0000005E-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-0000005F-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-0000005F-QINU`"' is in the NotReady state for 10 minutes. Pod '"`UNIQ--nowiki-00000060-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000061-QINU`"' is in NotReady state for 5 minutes. Pod '"`UNIQ--nowiki-00000061-QINU`"' is in the NotReady state for 10 minutes. Pod '"`UNIQ--nowiki-00000062-QINU`"' is in Failed, Unknown, or Pending state. Pod '"`UNIQ--nowiki-00000064-QINU`"' is in NotReady state for 5 minutes. Pod '"`UNIQ--nowiki-0000006B-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-0000006C-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-0000006D-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-0000006D-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-0000006E-QINU`"' is in NotReady status for 5 minutes. Pod '"`UNIQ--nowiki-0000006F-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-0000006F-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-00000071-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000071-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-00000073-QINU`"' is in NotReady state for 5 minutes. Pod '"`UNIQ--nowiki-00000073-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000075-QINU`"' is in the NotReady state for 5 minutes. Pod '"`UNIQ--nowiki-0000008E-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-00000090-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-00000092-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-00000094-QINU`"' is in the NotReady state for 5 minutes. Pod '"`UNIQ--nowiki-000000EE-QINU`"' is in Failed state. Pod '"`UNIQ--nowiki-000000EF-QINU`"' is in Unknown state for 5 minutes. Pod '"`UNIQ--nowiki-000000F0-QINU`"' is in Pending state for 5 minutes. Pod '"`UNIQ--nowiki-000000F1-QINU`"' is in NotReady status for 5 minutes. Redis is not available for pod '"`UNIQ--nowiki-0000002F-QINU`"' for 5 minutes. Redis is not available for pod '"`UNIQ--nowiki-0000003E-QINU`"' for 5 minutes. Redis is not available for pod '"`UNIQ--nowiki-0000004E-QINU`"' for 5 consecutive minutes. Redis is not available for pod '"`UNIQ--nowiki-00000054-QINU`"' for 5 minutes. Redis is not available for pod '"`UNIQ--nowiki-00000056-QINU`"' for 10 minutes. Redis is not available for pod '"`UNIQ--nowiki-00000056-QINU`"' for 5 minutes. Redis is not available for pod '"`UNIQ--nowiki-00000058-QINU`"' for 10 minutes. Redis is not available for pod '"`UNIQ--nowiki-0000005A-QINU`"' for 5 consecutive minutes. Redis is not available for pod '"`UNIQ--nowiki-00000065-QINU`"' for 5 consecutive minutes. Redis is not available for pod '"`UNIQ--nowiki-00000073-QINU`"' for 5 consecutive minutes. Redis is not available for pod '"`UNIQ--nowiki-00000077-QINU`"' for 5 consecutive minutes. Redis is not available for pod '"`UNIQ--nowiki-000000F6-QINU`"' for 5 consecutive minutes. Redis is not available for the pod '"`UNIQ--nowiki-00000031-QINU`"' for 10 minutes. Redis is not available for the pod '"`UNIQ--nowiki-00000034-QINU`"' for 5 minutes. Redis is not available for the pod '"`UNIQ--nowiki-00000035-QINU`"' for 5 minutes. Redis is not available for the pod '"`UNIQ--nowiki-00000036-QINU`"' for 10 minutes. Redis is not available for the pod '"`UNIQ--nowiki-00000037-QINU`"' for 10 minutes. Redis is not available for the pod '"`UNIQ--nowiki-00000037-QINU`"' for 5 minutes. Redis is not available for the pod '"`UNIQ--nowiki-00000039-QINU`"' for 10 minutes. Redis is not available for the pod '"`UNIQ--nowiki-0000003C-QINU`"' for 5 minutes. Redis is not available for the pod '"`UNIQ--nowiki-0000003E-QINU`"' for 10 minutes. Redis is not available for the pod '"`UNIQ--nowiki-0000003F-QINU`"' for 10 minutes. Redis is not available for the pod '"`UNIQ--nowiki-00000040-QINU`"' for 10 minutes. Requests to the config node fail for 5 consecutive minutes. Response time is greater than 20 milliseconds for 1 minute. SIP Node health level fails for pod '"`UNIQ--nowiki-0000004C-QINU`"' for 5 minutes. SIP Node health level fails for pod '"`UNIQ--nowiki-0000005E-QINU`"' for 5 minutes. SIP Proxy is out of service. SIP response latency for more than 95% of messages forwarded to '"`UNIQ--nowiki-00000059-QINU`"' is more than 1 second for sipproxy-node '"`UNIQ--nowiki-0000005A-QINU`"'. SIP response latency for more than 95% of messages forwarded to '"`UNIQ--nowiki-0000006B-QINU`"' is more than 1 second for sipproxy-node '"`UNIQ--nowiki-0000006C-QINU`"'. SIP softswitch is out of service. SIP trunk is out of service for more than 1 minute. Specified by: raa.<wbr/>prometheusRule.<wbr/>alerts.<wbr/>health.<wbr/>for<br/> '''Recommended value''': 30m The Storage account is down. The Voicemail pod exceeded 80% CPU usage for 5 minutes. The Voicemail pod exceeded 80% memory usage for 5 minutes. The Voicemail pod is down for more than 10 minutes. The absolute value of NROUTINGTIMEOUTS on a specific SIP Server increased by more than 20 in 2 minutes. The absolute value of active calls on a specific SIP Server dropped by more than 30 calls in 2 minutes and no new calls are arriving at the SIP Server for processing. The processing queue size is greater than 10 requests for 1 minute. The restart rate is greater than 0 for 5 minutes The sip-node '"`UNIQ--nowiki-0000006B-QINU`"' hit capacity limit on '"`UNIQ--nowiki-0000006C-QINU`"' for 3 consecutive minutes. The sip-node '"`UNIQ--nowiki-0000007D-QINU`"' hit capacity limit on '"`UNIQ--nowiki-0000007E-QINU`"' for 3 consecutive minutes. There were more than 10 request timeouts within 5 minutes for the Kafka consumer for topic '"`UNIQ--nowiki-0000004D-QINU`"'. There were more than 10 request timeouts within 5 minutes for the Kafka consumer for topic '"`UNIQ--nowiki-0000004F-QINU`"'. There were more than 3 Kafka consumer crashes within 5 minutes for service '"`UNIQ--nowiki-0000004F-QINU`"'. There were more than 3 Kafka consumer crashes within 5 minutes for service '"`UNIQ--nowiki-00000051-QINU`"'. Too many Kafka producer pending events for pod '"`UNIQ--nowiki-0000005B-QINU`"' (more than 100 in 5 minutes). Too many Kafka producer pending events for pod '"`UNIQ--nowiki-00000067-QINU`"' (more than 100 in 5 minutes). Too many Kafka producer pending events for pod '"`UNIQ--nowiki-0000007A-QINU`"' (more than 100 in 5 minutes). Too many Kafka producer pending events for pod '"`UNIQ--nowiki-00000103-QINU`"' (more than 100 in 5 minutes). Too many Kafka producer pending events for service '"`UNIQ--nowiki-0000004A-QINU`"' (more than 100 in 5 minutes). Too many Kafka producer pending events for service '"`UNIQ--nowiki-00000056-QINU`"' (more than 100 in 5 minutes). Too many Kafka producer pending events for service '"`UNIQ--nowiki-00000058-QINU`"' (more than 100 in 5 minutes). Too many Kafka producer pending events for service '"`UNIQ--nowiki-0000005C-QINU`"' (more than 100 in 5 minutes). Too many Kafka producer pending events for service '"`UNIQ--nowiki-00000068-QINU`"' (more than 100 in 5 minutes). Too many Kafka producer pending events for service '"`UNIQ--nowiki-0000007D-QINU`"' (more than 100 in 5 minutes). Triggered when 25% (or more) of GES pods have not been in a Ready state for 10 minutes. Triggered when GES encounters any DNS failures within the last 30 minutes. Triggered when GES encounters any uncaught exceptions. Silenced after 1 minute. Triggered when GES has begun throttling callbacks to a given number within the past 2 minutes. Triggered when GES has failed to connect or communicate with Nexus more than 30 times over the last hour. Triggered when GES has failed to submit a callback to ORS in the past 2 minutes for any reason. Triggered when GES has started throttling callbacks within the past 2 minutes. Triggered when GWS has had any issue with the GES client credentials in the last 5 minutes. Triggered when any component is down for any length of time. Triggered when fewer than two GES pods are up for 15 consecutive minutes. Triggered when memory usage (measured as a ratio of Used Heap Space vs Available Heap Space) is above 80% for a 90-second interval. Triggered when more than 50% of GES pods have not been in a Ready state for 5 minutes. Triggered when one instance of a message with an invalid length is received. Silenced after 2 minutes. Triggered when the average response time for incoming requests is above 1.5 seconds for a sustained period of 15 minutes. Triggered when the connection to the GWS Configuration Service is down. Triggered when the ORS_REDIS connection is down for 5 consecutive minutes. Triggered when the average CPU usage (measured by ges_process_cpu_seconds_total) is greater than 90% for 1 minute. Triggered when the connection to the GWS Environment Service is down. Triggered when the connection to the Genesys Authentication Service is down for 5 minutes. Triggered when the event loop is greater than 5 milliseconds for a period exceeding 5 minutes. Triggered when there are any failures writing to the logs. Silenced after 1 minute. Triggered when there are at least 1000 instances of issues managing VQ Proxy objects within a 10-minute period. Triggered when there has been a GWS server error in the past 5 minutes. Triggered when there have been more than five pod restarts in the past 15 minutes. Triggered when two or more HTTP 400 results are returned from a pod within a 5-minute period. Triggered when two or more HTTP 401 results are returned from a pod within a 5-minute period. Triggered when two or more HTTP 404 results are returned from a pod within a 5-minute period. Triggered when two or more HTTP 500 results are returned from a pod within a 5-minute period. Voicemail Service '"`UNIQ--nowiki-00000024-QINU`"' Envoy service is not available for 10 minutes. Voicemail Service '"`UNIQ--nowiki-00000026-QINU`"' GWS service is not available for 10 minutes. Voicemail Service '"`UNIQ--nowiki-00000028-QINU`"' GWS service is not available for 15 minutes. When the latency for 95% of the dial plan messages is more than 0.5 seconds for a duration of 5 minutes, then this warning alarm is raised for the '"`UNIQ--nowiki-00000030-QINU`"'. When the latency for 95% of the dial plan messages is more than 0.5 seconds for a duration of 5 minutes, then this warning alarm is raised for the '"`UNIQ--nowiki-00000033-QINU`"'. for 30 minutes increase(sipfe_requests_total{pod=~"sipfe-.+"}[5m]) <= 0 and increase(sipfe_requests_total{pod=~"sipfe-.+"}[10m]) > 100 1 2 3 4 5 6