Corinneh: Published

2022-02-23T20:56:32Z

Published

New page

{{ArticlePEServiceMetrics
|IncludedServiceId=79f6035b-590e-4c34-9432-fcd4b5ff519d
|CRD=Supports both CRD and annotations
|Port=11200
|Endpoint=http://<pod-ipaddress>:11200/metrics
|MetricsUpdateInterval=30 seconds
|MetricsDefined=Yes
|MetricsIntro=You can query Prometheus directly to see all the metrics that the Voice Orchestration Service exposes. The following metrics are likely to be particularly useful. Genesys does not commit to maintain other currently available Orchestration Service metrics not documented on this page.
|PEMetric={{PEMetric
|Metric=orsnode_callevents
|Type=counter
|Unit=N/A
|MetricDescription=Total number of received call events.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_ha_writes
|Type=counter
|Unit=N/A
|MetricDescription=The number of HA writes to Redis.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_ha_reads
|Type=counter
|Unit=N/A
|MetricDescription=The number of HA reads from Redis.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_interactions
|Type=gauge
|Unit=N/A
|MetricDescription=The number of active interactions.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_total_interactions
|Type=counter
|Unit=N/A
|MetricDescription=The total number of interactions that have been created.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_cleared_interactions
|Type=counter
|Unit=N/A
|MetricDescription=The total number of call interactions that have been cleared.
}}{{PEMetric
|Metric=orsnode_strategies
|Type=gauge
|Unit=N/A
|MetricDescription=The number of strategies that are running.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_total_strategies
|Type=counter
|Unit=N/A
|MetricDescription=The total number of strategies that have been created.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_load_errors
|Type=counter
|Unit=N/A
|MetricDescription=The total number of strategy load errors.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_fetch_errors
|Type=counter
|Unit=N/A
|MetricDescription=The total number of errors encountered when a strategy tried to fetch data from a Designer Application Server (DAS).
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_config_errors
|Type=counter
|Unit=N/A
|MetricDescription=The total number of strategy configuration errors.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_invoke_errors
|Type=counter
|Unit=N/A
|MetricDescription=The total number of strategy invoke errors.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_treatments
|Type=counter
|Unit=N/A
|MetricDescription=The total number of strategy treatments.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_failed_treatments
|Type=counter
|Unit=N/A
|MetricDescription=The total number of failed strategy treatments.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_userdata_updates
|Type=counter
|Unit=N/A
|MetricDescription=The total number of times that a strategy updated user data.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_scxml_transitions
|Type=counter
|Unit=N/A
|MetricDescription=The total number of SCXML transitions.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_scxml_events
|Type=counter
|Unit=N/A
|MetricDescription=The total number of SCXML events.
|UsedFor=Traffic
}}{{PEMetric
|Metric=orsnode_scxml_error_events
|Type=counter
|Unit=N/A
|MetricDescription=The total number of SCXML <tt>error.*</tt> events.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_http_fetch_requests
|Type=counter
|Unit=N/A
|MetricDescription=The total number of HTTP <tt>fetch</tt> requests.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_http_fetch_duration
|Type=histogram
|Unit=milliseconds
|MetricDescription=The HTTP fetch time, measured in milliseconds (ms).
|UsedFor=Latency
}}{{PEMetric
|Metric=orsnode_http_fetch_errors
|Type=counter
|Unit=N/A
|MetricDescription=The total number of HTTP <tt>fetch</tt> errors.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_http_fetch_error_status
|Type=histogram
|MetricDescription=Status of the HTTP <tt>fetch</tt> error.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_urs_rlib_latency_msec
|Type=histogram
|Unit=milliseconds
|MetricDescription=The Universal Routing Server (URS) <tt>rlib</tt> latency, measured in milliseconds (ms).
|UsedFor=Latency
}}{{PEMetric
|Metric=orsnode_urs_rlib_errors
|Type=counter
|Unit=N/A
|MetricDescription=The total number of URS <tt>rlib</tt> errors.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_urs_rlib_requests
|Type=counter
|Unit=N/A
|MetricDescription=The total number of URS <tt>rlib</tt> requests.
}}{{PEMetric
|Metric=orsnode_urs_rlib_events
|Type=counter
|Unit=N/A
|MetricDescription=The total number of URS <tt>rlib</tt> events.
}}{{PEMetric
|Metric=orsnode_urs_rlib_timeouts
|Type=counter
|Unit=N/A
|MetricDescription=The total number of URS <tt>rlib</tt> timeouts.
}}{{PEMetric
|Metric=orsnode_redis_state
|Type=gauge
|Unit=N/A
|Label=redis_cluster_name
|MetricDescription=Current Redis connection state.
}}{{PEMetric
|Metric=orsnode_redis_disconnect
|Type=counter
|Unit=N/A
|MetricDescription=The number of times that the ORS node disconnected from Redis.
}}{{PEMetric
|Metric=orsnode_sdr_messages_sent
|Type=counter
|Unit=N/A
|MetricDescription=The number of SDR messages that have been sent.
}}{{PEMetric
|Metric=orsnode_rq_latency_msec
|Type=histogram
|Unit=milliseconds
|Label=le, service
|MetricDescription=Redis queue latency, measured in milliseconds (ms).
|UsedFor=Latency
}}{{PEMetric
|Metric=orsnode_routing_latency_msec
|Type=histogram
|Unit=milliseconds
|MetricDescription=Routing latency, measured in milliseconds (ms).
|UsedFor=Latency
}}{{PEMetric
|Metric=orsnode_rstream_latency_msec
|Type=histogram
|Unit=milliseconds
|Label=le, node
|MetricDescription=Redis stream latency, measured in (ms).
|UsedFor=Latency
}}{{PEMetric
|Metric=orsnode_digital_latency_msec
|Type=histogram
|Unit=milliseconds
|MetricDescription=Digital stream latency, measured in milliseconds (ms).
|UsedFor=Latency
}}{{PEMetric
|Metric=orsnode_sip_health_check
|Type=gauge
|Unit=N/A
|Label=node
|MetricDescription=ORS health check.
}}{{PEMetric
|Metric=orsnode_ixn_health_check
|Type=gauge
|Unit=N/A
|MetricDescription=Interaction health check.
}}{{PEMetric
|Metric=orsnode_rq_state
|Type=gauge
|Unit=N/A
|MetricDescription=Current Redis queue connection state.
}}{{PEMetric
|Metric=orsnode_ixn_events
|Type=counter
|Unit=N/A
|MetricDescription=Total number of interaction stream events received.
}}{{PEMetric
|Metric=orsnode_rq_disconnect
|Type=counter
|Unit=N/A
|MetricDescription=Number of times the ORS node disconnected from the RQ Service.
}}{{PEMetric
|Metric=service_version_info
|Type=gauge
|Unit=N/A
|Label=version
|MetricDescription=Displays the version of Voice Orchestration Service that is currently running. In the case of this metric, the labels provide the important information. The metric value is always 1 and does not provide any information.
|SampleValue=service_version_info{version="100.0.1000006"} 1
}}{{PEMetric
|Metric=orsnode_route_redirected
|Type=counter
|Unit=N/A
|MetricDescription=Total number of EventRouteUsed events without a ReferenceID.
}}{{PEMetric
|Metric=orsnode_balancer_stream_state
|Type=gauge
|Unit=N/A
|Label=balancer_stream_type
|MetricDescription=The state of the voice balancer stream.
}}{{PEMetric
|Metric=orsnode_high_memory
|Type=gauge
|Unit=N/A
|MetricDescription=Indicates when the ORS node is using a lot of memory.
}}{{PEMetric
|Metric=orsnode_urs_rlib_state
|Type=gauge
|Unit=N/A
|MetricDescription=Indicates a Tenant <tt>rlib</tt> request timeout.
}}{{PEMetric
|Metric=orsnode_stuck_interactions
|Type=gauge
|Unit=N/A
|MetricDescription=The number of stuck interactions.
}}{{PEMetric
|Metric=orsnode_urs_scxml_submit_requests
|Type=counter
|Unit=N/A
|MetricDescription=The total number of URS <tt>SCXMLSubmit</tt> requests.
}}{{PEMetric
|Metric=orsnode_urs_scxml_cancel_requests
|Type=counter
|Unit=N/A
|MetricDescription=The total number of URS <tt>SCXMLQueueCancel</tt> requests.
}}{{PEMetric
|Metric=orsnode_urs_queue_submit_done_events
|Type=counter
|Unit=N/A
|MetricDescription=Total number of URS <tt>queue.submit.done</tt> events.
}}{{PEMetric
|Metric=orsnode_health_level
|Type=gauge
|Unit=N/A
|MetricDescription=Summarized health level of the ORS node:

-1 – fail<br />
0 – starting<br />
1 – degraded<br />
2 – pass
}}{{PEMetric
|Metric=orsnode_health_check_error
|Type=gauge
|Unit=N/A
|Label=reason
|MetricDescription=Health check errors for the ORS node:

1 – has error<br />
0 – no error
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_running_applications
|Type=gauge
|Unit=N/A
|MetricDescription=The number of active sessions for each Designer application.
}}{{PEMetric
|Metric=orsnode_failed_applications
|Type=gauge
|Unit=N/A
|MetricDescription=The number of failed sessions for each Designer application.
}}{{PEMetric
|Metric=orsnode_total_applications
|Type=gauge
|Unit=N/A
|MetricDescription=The total number of sessions created for each Designer application.
}}{{PEMetric
|Metric=orsnode_failed_scripts
|Type=gauge
|Unit=N/A
|MetricDescription=The number of scripts that failed to load in the Tenant Service configuration management environment.
}}{{PEMetric
|Metric=orsnode_session_load_time_msec
|Type=histogram
|Unit=milliseconds
|MetricDescription=The time it takes for the strategy to be compiled and go through its initial states.
}}{{PEMetric
|Metric=orsnode_service_started
|Type=gauge
|Unit=N/A
|Label=started
|MetricDescription=Timestamp when the ORS node started.
}}{{PEMetric
|Metric=orsnode_total_terminal_requests
|Type=counter
|Unit=N/A
|MetricDescription=Total number of terminal requests (like Deliver, PlaceInQueue, StopProcessing for Digital and RequestClearCall, RequestRouteCall for Voice).
}}{{PEMetric
|Metric=orsnode_total_non_terminal_requests
|Type=counter
|Unit=N/A
|MetricDescription=Total number of non-terminal requests to the Interaction Server (for Digital) or SIP Server (for Voice).
}}{{PEMetric
|Metric=orsnode_sip_post_errors
|Type=counter
|Unit=N/A
|MetricDescription=Total number of errors encountered in POST requests to the SIP node.
|UsedFor=Errors
}}{{PEMetric
|Metric=orsnode_pending_tlib_requests
|Type=counter
|Unit=N/A
|MetricDescription=Total number of pending TLib requests.
}}{{PEMetric
|Metric=orsnode_sips_rest_connections
|Type=gauge
|Unit=N/A
|MetricDescription=The number of active REST connections with SIP Cluster Service.
}}{{PEMetric
|Metric=orsnode_number_compiled_applications
|Type=counter
|Unit=N/A
|MetricDescription=The number of compiled applications in the cache.
}}{{PEMetric
|Metric=orsnode_cached_applications_size
|Type=gauge
|MetricDescription=The sum of the sizes of the cached applications.
}}{{PEMetric
|Metric=orsnode_tlib_latency_msec
|Type=histogram
|Unit=milliseconds
|Label=le
|MetricDescription=The TLib Rest API request latency, measured in (ms).
|UsedFor=Latency
}}{{PEMetric
|Metric=orsnode_application_size
|Type=gauge
|MetricDescription=The compiled size of the Designer application.
}}{{PEMetric
|Metric=orsnode_application_microstep_count
|Type=gauge
|Unit=N/A
|MetricDescription=The number of microsteps while executing the Designer application.
}}{{PEMetric
|Metric=orsnode_application_run_time_msec
|Type=gauge
|Unit=milliseconds
|MetricDescription=The length of time the Designer application was running, measured in milliseconds (ms).
}}{{PEMetric
|Metric=orsnode_application_compiled_date
|Type=gauge
|Unit=N/A
|MetricDescription=The date on which the Designer application was compiled.
}}{{PEMetric
|Metric=orsnode_application_last_invoked_date
|Type=gauge
|Unit=N/A
|MetricDescription=The date when the Designer application was last invoked.
}}
|AlertsDefined=Yes
|PEAlert={{PEAlert
|Alert=Number of running strategies is too high
|Severity=Warning
|AlertDescription=Too many active sessions.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check the number of voice, digital, and callback calls in the system.
|BasedOn=orsnode_strategies
|Threshold=More than 400 strategies running in 5 consecutive seconds.
}}{{PEAlert
|Alert=Number of running strategies is critical
|Severity=Critical
|AlertDescription=Too many active sessions.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check the number of voice, digital, and callback calls in the system.
|BasedOn=orsnode_strategies
|Threshold=More than 600 strategies running in 5 consecutive seconds.
}}{{PEAlert
|Alert=Redis disconnected for 5 minutes
|Severity=Warning
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis.
*If alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki>, check if there is an issue with the pod.
|BasedOn=redis_state
|Threshold=Redis is not available for the pod <nowiki>{{ $labels.pod }}</nowiki> for 5 minutes.
}}{{PEAlert
|Alert=Redis disconnected for 10 minutes
|Severity=Critical
|AlertDescription=Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with Redis, and then restart Redis.
*If the alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki>, check if there is an issue with the pod.
|BasedOn=redis_state
|Threshold=Redis is not available for the pod <nowiki>{{ $labels.pod }}</nowiki> for 10 minutes.
}}{{PEAlert
|Alert=Pod status Failed
|Severity=Warning
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> failed.

Actions:

*One of the containers in the pod has entered a Failed state. Check the Kibana logs for the reason.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Failed state.
}}{{PEAlert
|Alert=Pod in Unknown state
|Severity=Warning
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Unknown state.

Actions:

*If the alarm is triggered for multiple services, make sure there are no issues with the Kubernetes cluster.
*If the alarm is triggered only for pod <nowiki>{{ $labels.pod }}</nowiki>, check whether the image is correct and if the container is starting up.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Unknown state for 5 minutes.
}}{{PEAlert
|Alert=Pod in Pending state
|Severity=Warning
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Pending state.

Actions:

*If the alarm is triggered for multiple services, make sure the Kubernetes nodes where the pod is running are alive in the cluster.
*If the alarm is triggered only for the pod <nowiki>{{ $labels.pod }}</nowiki>, check the health of the pod.
|BasedOn=kube_pod_status_phase
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> is in Pending state for 5 minutes.
}}{{PEAlert
|Alert=Pod Not ready for 10 minutes
|Severity=Critical
|AlertDescription=Pod <nowiki>{{ $labels.pod }}</nowiki> in NotReady state.

Actions:

*If this alarm is triggered, check whether the CPU is available for the pods.
*Check whether the port of the pod is running and serving the request.
|BasedOn=kube_pod_status_ready
|Threshold=Pod <nowiki>{{ $labels.pod }}</nowiki> in NotReady state for 10 minutes.
}}{{PEAlert
|Alert=Container restored repeatedly
|Severity=Critical
|AlertDescription=Actions:

*One of the containers in the pod has entered a failed state. Check the Kibana logs for the reason.
|BasedOn=kube_pod_container_status_restarts_total
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> was restarted 5 or more times within 15 minutes.
}}{{PEAlert
|Alert=Pod memory greater than 65%
|Severity=Warning
|AlertDescription=High memory usage for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_requests_memory_bytes
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Pod memory greater than 80%
|Severity=Critical
|AlertDescription=Critical memory usage for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Restart the service.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_memory_working_set_bytes, kube_pod_container_resource_requests_memory_bytes
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> memory usage exceeded 80% for 5 minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 65%
|Severity=Warning
|AlertDescription=High CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_cpu_usage_seconds_total, container_spec_cpu_period
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 65% for 5 minutes.
}}{{PEAlert
|Alert=Pod CPU greater than 80%
|Severity=Critical
|AlertDescription=Critical CPU load for pod <nowiki>{{ $labels.pod }}</nowiki>.

Actions:

*Check whether the horizontal pod autoscaler has triggered and if the maximum number of pods has been reached.
*Check Grafana for abnormal load.
*Restart the service.
*Collect the service logs; raise an investigation ticket.
|BasedOn=container_cpu_usage_seconds_total, container_spec_cpu_period
|Threshold=Container <nowiki>{{ $labels.container }}</nowiki> CPU usage exceeded 80% for 5 minutes.
}}
}}

VM/Current/VMPEGuide/VoiceOrchestrationServiceMetrics - Revision history

Corinneh: Published