Corinneh: Published

2022-02-23T20:56:29Z

Published

New page

{{ArticlePEServiceMetrics
|IncludedServiceId=17df7b83-4bd2-492a-a705-d03c8cbab63a
|CRD=Supports both CRD and annotations
|Port=11400
|Endpoint=http://<pod-ipaddress>:11400/metrics
|MetricsUpdateInterval=30 seconds
|MetricsDefined=Yes
|MetricsIntro=Voice SIP Proxy Service exposes Genesys-defined, SIP Proxy Service–specific metrics as well as some standard Kafka metrics. You can query Prometheus directly to see all the metrics that the SIP Proxy Service exposes. The following metrics are likely to be particularly useful. Genesys does not commit to maintain other currently available SIP Proxy Service metrics not documented on this page.
|PEMetric={{PEMetric
|Metric=sipproxy_requests_total
|Type=counter
|Unit=N/A
|Label=method
|MetricDescription=Total number of received requests.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sipproxy_rejected_requests_total
|Type=counter
|Unit=N/A
|MetricDescription=The total number of rejected requests.
|UsedFor=Errors
}}{{PEMetric
|Metric=sipproxy_requests_processed_self_total
|Type=counter
|Unit=N/A
|Label=method
|MetricDescription=The total number of received requests that were processed by SIP Proxy itself.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sipproxy_requests_forwarded_total
|Type=counter
|Unit=N/A
|Label=method, request_direction, sip_node_id
|MetricDescription=The total number of forwarded requests.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sipproxy_requests_sip_node_reselected_total
|Type=counter
|Unit=N/A
|MetricDescription=Total count of sip-node reselection.
|UsedFor=Errors
}}{{PEMetric
|Metric=sipproxy_responses_forwarded_total
|Type=counter
|Unit=N/A
|Label=method, sip_node_id, request_direction
|MetricDescription=Total count of forwarded responses.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sipproxy_response_latency
|Type=histogram
|Label=le, sip_node_id, request_direction, target, node_in_cache
|MetricDescription=SIP response latency.
|UsedFor=Latency
}}{{PEMetric
|Metric=sipproxy_register_processed_total
|Type=counter
|Unit=N/A
|MetricDescription=Total number of REGISTER requests that SIP Proxy received for processing.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sipproxy_register_rejected_total
|Type=counter
|Unit=N/A
|MetricDescription=Total number of REGISTER requests for processing that were rejected.
|UsedFor=Errors
}}{{PEMetric
|Metric=sipproxy_calls_per_second_count
|Type=gauge
|Unit=N/A
|MetricDescription=Current calculated calls per second.
|UsedFor=Saturation
}}{{PEMetric
|Metric=sipproxy_active_sip_nodes_count
|Type=gauge
|Unit=N/A
|MetricDescription=Current number of active SIP nodes.
}}{{PEMetric
|Metric=sipproxy_sip_nodes_count
|Type=gauge
|Unit=N/A
|MetricDescription=Current number of discovered SIP nodes.
}}{{PEMetric
|Metric=sipproxy_tenants_count
|Type=gauge
|Unit=N/A
|MetricDescription=Current count of discovered tenants.
}}{{PEMetric
|Metric=sipproxy_consul_record_processing_errors_count
|Type=counter
|Unit=N/A
|MetricDescription=Current number of errors while processing records got from Consul.
}}{{PEMetric
|Metric=sipproxy_consul_errors_count
|Type=counter
|Unit=N/A
|MetricDescription=Current number of Consul errors.
}}{{PEMetric
|Metric=sipproxy_sip_node_is_capacity_available
|Type=gauge
|Label=sip_node_id
|MetricDescription=Indicates whether SIP node has available capacity or not.
}}{{PEMetric
|Metric=service_version_info
|Type=gauge
|Unit=N/A
|Label=version
|MetricDescription=Displays the version of Voice SIP Proxy Service that is currently running. In the case of this metric, the labels provide the important information. The metric value is always 1 and does not provide any information.
|SampleValue=service_version_info{version="100.0.1000006"} 1
}}{{PEMetric
|Metric=sipproxy_health_level
|Type=gauge
|Unit=N/A
|MetricDescription=Health level of the SIP Proxy node:

-1 – fail 
0 – starting 
1 – degraded 
2 – pass
}}{{PEMetric
|Metric=sipproxy_envoy_proxy_status
|Type=gauge
|Unit=N/A
|MetricDescription=Status of the Envoy proxy:

-1 – error 
0 – disconnected 
1 – connected
|SampleValue=1
}}{{PEMetric
|Metric=sipproxy_config_node_status
|Type=gauge
|Unit=N/A
|MetricDescription=Status of the Config node connection:

0 – disconnected 
1 – connected
|SampleValue=1
}}{{PEMetric
|Metric=sip_server_transactions_created_total
|Type=counter
|Unit=N/A
|MetricDescription=Total number of created server transactions.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sip_client_transactions_created_total
|Type=counter
|Unit=N/A
|MetricDescription=Total number of created client transactions.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sip_server_transactions_deleted_total
|Type=counter
|Unit=N/A
|MetricDescription=Total number of deleted server transactions.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sip_client_transactions_deleted_total
|Type=counter
|Unit=N/A
|MetricDescription=Total number of deleted client transactions.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sip_client_transactions_count
|Type=gauge
|Unit=N/A
|MetricDescription=Current number of client transactions.
|UsedFor=Saturation
}}{{PEMetric
|Metric=sip_server_transactions_count
|Type=gauge
|Unit=N/A
|MetricDescription=Current number of server transactions.
|UsedFor=Saturation
}}{{PEMetric
|Metric=sip_server_transactions_rejected_total
|Type=counter
|Unit=N/A
|MetricDescription=Total number of server transactions rejected for internal reasons.
|UsedFor=Errors
}}{{PEMetric
|Metric=sip_proxy_contexts_count
|Type=gauge
|Unit=N/A
|MetricDescription=Current number of active SIP Proxy forwarding contexts.
|UsedFor=Saturation
}}{{PEMetric
|Metric=sip_received_bytes_total
|Type=counter
|Unit=bytes
|Label=transport
|MetricDescription=Total traffic received, measured in bytes.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sip_sent_bytes_total
|Type=counter
|Unit=bytes
|Label=transport
|MetricDescription=Total traffic sent, measured in bytes.
|UsedFor=Traffic
}}{{PEMetric
|Metric=sip_transport_errors_total
|Type=counter
|Unit=N/A
|Label=transport, address
|MetricDescription=Total number of transport errors.
|UsedFor=Errors
}}{{PEMetric
|Metric=sip_stream_transport_wait_drain_total
|Type=counter
|Unit=N/A
|MetricDescription=Total number of requests to wait for drain events on stream transports.
}}{{PEMetric
|Metric=sip_stream_transport_flood_total
|Type=counter
|Unit=N/A
|MetricDescription=Total number of flood events on the stream transports.
}}{{PEMetric
|Metric=http_client_request_duration_seconds
|Type=histogram
|Unit=seconds
|Label=le, target_service_name
|MetricDescription=The time duration between the HTTP client request and the response, measured in seconds.
|UsedFor=Latency
}}{{PEMetric
|Metric=http_client_response_count
|Type=counter
|Unit=N/A
|Label=target_service_name, status
|MetricDescription=The number of HTTP client responses received.
|UsedFor=Traffic
}}{{PEMetric
|Metric=log_output_bytes_total
|Type=counter
|Unit=bytes
|Label=level, format, module
|MetricDescription=The total amount of log output, measured in bytes.
|SampleValue=<tt>log_output_bytes_total{level="info",format="txt",module="sipproxy_node@config-manager"} 3175</tt> <tt>log_output_bytes_total{level="info",format="txt",module="sipproxy_node@sipproxy-node"} 96</tt> <tt>log_output_bytes_total{level="info",format="txt",module="sipproxy_node@sipproxy@sip"} 181</tt> <tt>log_output_bytes_total{level="info",format="json",module="sipproxy_node@config-manager"} 4184</tt> <tt>log_output_bytes_total{level="info",format="json",module="sipproxy_node@sipproxy-node"} 135</tt> <tt>log_output_bytes_total{level="info",format="json",module="sipproxy_node@sipproxy@sip"} 259</tt>
}}{{PEMetric
|Metric=kafka_consumer_recv_messages_total
|Type=counter
|MetricDescription=Number of messages received from Kafka.
|UsedFor=Traffic
}}{{PEMetric
|Metric=kafka_consumer_error_total
|Type=counter
|MetricDescription=Number of Kafka consumer errors.
|UsedFor=Errors
}}{{PEMetric
|Metric=kafka_consumer_latency
|Type=histogram
|MetricDescription=Consumer latency is the time difference between when the message is produced and when the message is consumed. That is, the time when the consumer received the message minus the time when the producer produced the message.
|UsedFor=Latency
}}{{PEMetric
|Metric=kafka_consumer_rebalance_total
|Type=counter
|MetricDescription=Number of Kafka consumer rebalance events.
}}{{PEMetric
|Metric=kafka_consumer_state
|Type=gauge
|MetricDescription=Current state of the Kafka consumer.
}}{{PEMetric
|Metric=kafka_producer__messages_total
|Type=counter
|MetricDescription=Number of messages received from Kafka.
|UsedFor=Traffic
}}{{PEMetric
|Metric=kafka_producer_queue_depth
|Type=gauge
|Label=kafka_location
|MetricDescription=Number of Kafka producer pending events.
|UsedFor=Saturation
}}{{PEMetric
|Metric=kafka_producer_queue_age_seconds
|Type=gauge
|Unit=seconds
|Label=kafka_location
|MetricDescription=Age of the oldest producer pending event in seconds.
}}{{PEMetric
|Metric=kafka_producer_error_total
|Type=counter
|Label=kafka_location
|MetricDescription=Number of Kafka producer errors.
|UsedFor=Errors
}}{{PEMetric
|Metric=kafka_producer_state
|Type=gauge
|Label=kafka_location
|MetricDescription=Current state of the Kafka producer.
}}{{PEMetric
|Metric=kafka_producer_biggest_event_size
|Type=gauge
|Label=kafka_location, topic
|MetricDescription=Biggest event size so far.
|SampleValue=231
}}{{PEMetric
|Metric=kafka_max_request_size
|Type=gauge
|Label=kafka_location
|MetricDescription=Exposed config to compare with biggest event size.
|SampleValue=1000000
}}{{PEMetric
|Metric=kafka_producer_dropped_event_number
|Type=gauge
|MetricDescription=Number of dropped events.
}}
|AlertsDefined=Yes
|PEAlert={{PEAlert
|Alert=Too many Kafka pending events
|Severity=Critical
|AlertDescription=Too many Kafka producer pending events for pod <nowiki>{{ $labels.pod }}</nowiki>. This alert means there are issues with SIP REGISTER processing on this voice-sipproxy.

Actions:

*Make sure there are no issues with Kafka or with the <nowiki>{{ $labels.pod }}</nowiki> pod's CPU and network.
|BasedOn=kafka_producer_queue_depth
|Threshold=Too many Kafka producer pending events for service <nowiki>{{ $labels.container }}</nowiki> (more than 100 in 5 minutes).
}}{{PEAlert
|Alert=SIP server response time too high
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple sipproxy-nodes, make sure there are no issues on <nowiki>{{ $labels.sip_node_id }}</nowiki>.
*If the alarm is triggered only for sipproxy-node <nowiki>{{ $labels.pod }}</nowiki>, check to see if there is an issue with the service related to the topic (CPU, memory, or network overload).
|BasedOn=sipproxy_response_latency_bucket
|Threshold=SIP response latency for more than 95% of messages forwarded to <nowiki>{{ $labels.sip_node_id }}</nowiki> is more than 1 second for sipproxy-node <nowiki>{{ $labels.pod }}</nowiki>.
}}{{PEAlert
|Alert=Pod status failed
|Severity=Warning
|AlertDescription=Actions:

*Restart the pod and check to see if there are any issues with the pod after restart.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Failed state.
}}{{PEAlert
|Alert=Pod status Unknown
|Severity=Warning
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Unknown state.

Actions:

*Restart the pod and check to see if there are any issues with the pod after restart.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Unknown state for 5 minutes.
}}{{PEAlert
|Alert=Pod status Pending
|Severity=Warning
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Pending state.

Actions:

*Restart the pod and check to see if there are any issues with the pod after restart.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Pending state for 5 minutes.
}}{{PEAlert
|Alert=Pod status NotReady
|Severity=Critical
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> is in NotReady state.

Actions:

*Restart the pod and check to see if there are any issues with the pod after restart.
|BasedOn=kube_pod_status_ready
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in NotReady state for 5 minutes.
}}{{PEAlert
|Alert=Container restarted repeatedly
|Severity=Critical
|AlertDescription=Container <nowiki>{{ $labels.container }}</nowiki> was repeatedly restarted.

Actions:

*Check to see if a new version of the image was deployed. Also check for issues with the Kubernetes cluster.
|BasedOn=kube_pod_container_status_restarts_total
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> was restarted 5 or more times within 15 minutes.
}}{{PEAlert
|Alert=No sip-nodes available for 2 minutes
|Severity=Critical
|AlertDescription=No sip-nodes are available for the pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with sip-nodes.
*If the alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki>, check to see if there is any issues with the pod.
|BasedOn=sipproxy_active_sip_nodes_count
|Threshold=No sip-nodes are available for the pod <nowiki>{{ $labels.pod }}</nowiki> for 2 minutes.
}}{{PEAlert
|Alert=sip-node capacity limit reached
|Severity=Warning
|AlertDescription=The sip-node <nowiki>{{ $labels.sip_node_id }}</nowiki> hit capacity limit on <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*If alarm is triggered for multiple services make sure there is no issues with sip-node <nowiki>{{ $labels.sip_node_id }}</nowiki>.
*If alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki> check if there is any issue with the pod
|BasedOn=sipproxy_sip_node_is_capacity_available
|Threshold=The sip-node <nowiki>{{ $labels.sip_node_id }}</nowiki> hit capacity limit on <nowiki>{{ $labels.pod }}</nowiki> for 3 consecutive minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 80%
|Severity=Critical
|AlertDescription=Critical CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs for pod <nowiki>{{ $labels.pod }}</nowiki> and raise an investigation ticket.
|BasedOn=container_cpu_usage_seconds_total, container_spec_cpu_period
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 80% for 5 minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 65%
|Severity=Warning
|AlertDescription=High CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs for pod <nowiki>{{ $labels.pod }}</nowiki> and raise an investigation ticket.
|BasedOn=container_cpu_usage_seconds_total, container_spec_cpu_period
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Pod memory greater than 80%
|Severity=Critical
|AlertDescription=Critical memory usage for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Restart the service for pod <nowiki>{{ $labels.pod }}</nowiki>.
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_requests_memory_bytes
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 80% for 5 minutes
}}{{PEAlert
|Alert=Pod memory greater than 65%
|Severity=Warning
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> has high memory usage.

Actions:

*Check whether the horizontal pod autoscaler has triggered and the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs for pod <nowiki>{{ $labels.pod }}</nowiki> and raise an investigation ticket
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_requests_memory_bytes
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Config node fail
|Severity=Warning
|AlertDescription=The request to the config node failed.

Action:

*Check if there is any problem with pod <nowiki>{{ $labels.pod }}</nowiki> and config node.
|BasedOn=http_client_response_count
|Threshold=Requests to the config node fail for 5 consecutive minutes.
}}
}}

VM/Current/VMPEGuide/VoiceSIPProxyServiceMetrics - Revision history

Corinneh: Published