Karl.Abraham: Published

2022-01-24T06:52:54Z

Published

New page

{{ArticlePEServiceMetrics
|IncludedServiceId=27055834-94d9-43e1-8935-90e64f68b7f5
|CRD=ServiceMonitor / PodMonitor
|Port=9116,

8200
|Endpoint=Metrics endpoints:
<source>
curl -v "http://<RM_POD_IP>:9116/snmp?target=127.0.0.1%3A1161&module=if_mib"
curl -v "http://<RM_POD_IP>:8200/log
</source>

Enable metrics:

Service/Pod Monitoring Settings<source>
prometheus:
enabled: true
metric:
   port: 9116
log:
   port: 8200
</source>Enable for Prometheus operator<source>
podMonitor:
enabled: true
metric:
   path: /snmp
   module: [ if_mib ]
   target: [ 127.0.0.1:1161 ]
log:
   path: /log
</source>monitoring:

*prometheusRulesEnabled: true

*grafanaEnabled: true

monitor:

*monitorName: gvp-monitoring

*logFilePrefixName: RM
|MetricsDefined=Yes
|PEMetric={{PEMetric
|Metric=rmTotal5xxInviteSent
|Type=Gauge
|Unit=Unsigned32
|MetricDescription=Number of 5xx that were received for INVITE sent by RM
|SampleValue=514
|UsedFor=Error
}}{{PEMetric
|Metric=rmTotal4xxInviteSent
|Type=Gauge
|Unit=Unsigned32
|MetricDescription=Number of 4xx that were received for INVITE sent by RM
|SampleValue=10
|UsedFor=Error
}}{{PEMetric
|Metric=rmPRStatus
|Type=Gauge
|Unit=DisplayString
|MetricDescription=GVP_RM_PhysicalResourceTable.Status: Current state of the resource

(rmPRStatus{gvpConfigDBID="174",rmPRName="mcp-10.206.5.89",rmPRStatus="AVAILABLE"})
|SampleValue=1
|UsedFor=Information
}}{{PEMetric
|Metric=rmPRActiveCalls
|Type=Gauge
|Unit=Unsigned32
|MetricDescription=GVP_RM_PhysicalResourceTable.ActiveCalls: Number of calls that currently are handled by the resource.

(rmPRActiveCalls{gvpConfigDBID="174",rmPRName="mcp-10.206.5.89"})
|UsedFor=Traffic
}}{{PEMetric
|Metric=rmPRTotalCalls
|Type=Gauge
|Unit=Unsigned32
|MetricDescription=GVP_RM_PhysicalResourceTable.TotalCalls: Total number of calls that have been handled by this resource since it was connected to RM. (rmPRTotalCalls{gvpConfigDBID="174",rmPRName="mcp-10.206.5.89"})
|SampleValue=150
|UsedFor=Traffic
}}{{PEMetric
|Metric=rmTenantCurrentInboundCalls
|Type=Gauge
|Unit=Unsigned32
|MetricDescription=Number of active inbound calls that use this tenant.
|SampleValue=2
|UsedFor=Traffic
}}{{PEMetric
|Metric=rmTenantPeakCalls
|Type=Gauge
|MetricDescription=Maximum number of concurrent calls to this Tenant since it became active
|SampleValue=10
|UsedFor=Traffic
}}
|AlertsDefined=Yes
|PEAlert={{PEAlert
|Alert=RM Service Down
|Severity=CRITICAL
|AlertDescription=RM pods are not in ready state and RM service is not available
|BasedOn=kube_pod_container_status_running
|Threshold=0
}}{{PEAlert
|Alert=InitContainerFailingRepeatedly
|Severity=CRITICAL
|AlertDescription=The trigger will flag an alarm when the RM init container gets failed 5 or more times within 15 mins.
|BasedOn=kube_pod_init_container_status_restarts_total
|Threshold=15 mins
}}{{PEAlert
|Alert=ContainerRestartedRepeatedly
|Severity=CRITICAL
|AlertDescription=The trigger will flag an alarm when the RM or RM SNMP container gets restarted 5 or more times within 15 mins
|BasedOn=kube_pod_container_status_restarts_total
|Threshold=15 mins
}}{{PEAlert
|Alert=PodStatusNotReady
|Severity=CRITICAL
|AlertDescription=The trigger will flag an alarm when RM pod status is Not ready for 30 mins and this will be controlled by override-value.yaml.
|BasedOn=kube_pod_status_ready
|Threshold=30mins
}}{{PEAlert
|Alert=RMTotal5XXErrorForINVITE
|Severity=HIGH
|AlertDescription=The RM mib counter stats will be collected for every 30 seconds and if the mib counter total5xxInviteSent increments from its previous value by 5 within 5 minutes the trigger will flag an alarm.
|BasedOn=rmTotal5xxInviteSent
|Threshold=5 mins
}}{{PEAlert
|Alert=RMTotal4XXErrorForINVITE
|Severity=MEDIUM
|AlertDescription=The RM mib counter stats will be collected for every 60 seconds and if the mib counter total4xxInviteSent increments from its previous value by 10 within 60 seconds the trigger will flag an alarm.
|BasedOn=rmTotal4xxInviteSent
|Threshold=1min
}}{{PEAlert
|Alert=RMInterNodeConnectivityBroken
|Severity=HIGH
|AlertDescription=Inter-node connectivity between RM nodes is lost for 5mins.
|BasedOn=gvp_rm_log_parser_warn_total
|Threshold=5 mins
}}{{PEAlert
|Alert=RMConfigServerConnectionLost
|Severity=HIGH
|AlertDescription=RM lost connection to GVP Configuration Server for 5mins.
|BasedOn=gvp_rm_log_parser_warn_total
|Threshold=5 mins
}}{{PEAlert
|Alert=RMSocketInterNodeError
|Severity=HIGH
|AlertDescription=RM Inter node Socket Error for 5mins.
|BasedOn=gvp_rm_log_parser_eror_total
|Threshold=5mins
}}{{PEAlert
|Alert=ContainerCPUreached80percentForRM0
|Severity=HIGH
|AlertDescription=The trigger will flag an alarm when the RM container CPU utilization goes beyond 80% for 15 mins
|BasedOn=container_cpu_usage_seconds_total, container_spec_cpu_quota, container_spec_cpu_period
|Threshold=15mins
}}{{PEAlert
|Alert=ContainerCPUreached80percentForRM1
|Severity=HIGH
|AlertDescription=The trigger will flag an alarm when the RM container CPU utilization goes beyond 80% for 15 mins
|BasedOn=container_cpu_usage_seconds_total, container_spec_cpu_quota, container_spec_cpu_period
|Threshold=15mins
}}{{PEAlert
|Alert=ContainerMemoryUsage80percentForRM0
|Severity=HIGH
|AlertDescription=The trigger will flag an alarm when the RM container Memory utilization goes beyond 80% for 15 mins
|BasedOn=container_memory_rss, kube_pod_container_resource_limits_memory_bytes
|Threshold=15mins
}}{{PEAlert
|Alert=ContainerMemoryUsage80percentForRM1
|Severity=HIGH
|AlertDescription=The trigger will flag an alarm when the RM container Memory utilization goes beyond 80% for 15 mins
|BasedOn=container_memory_rss, kube_pod_container_resource_limits_memory_bytes
|Threshold=15mins
}}{{PEAlert
|Alert=MCPPortsExceeded
|Severity=HIGH
|AlertDescription=All the MCP ports in MCP LRG are exceeded
|BasedOn=gvp_rm_log_parser_eror_total
|Threshold=1min
}}{{PEAlert
|Alert=RMServiceDegradedTo50Percentage
|Severity=HIGH
|AlertDescription=One of the RM container is not in running state for 5mins
|BasedOn=kube_pod_container_status_running
|Threshold=5mins
}}{{PEAlert
|Alert=RMMatchingIVRTenantNotFound
|Severity=MEDIUM
|AlertDescription=Matching IVR profile tenant could not be found for 2mins
|BasedOn=gvp_rm_log_parser_eror_total
|Threshold=2mins
}}{{PEAlert
|Alert=RMResourceAllocationFailed
|Severity=MEDIUM
|AlertDescription=RM Resource allocation failed for 1mins
|BasedOn=gvp_rm_log_parser_eror_total
|Threshold=1min
}}
}}

GVP/Current/GVPPEGuide/Resource Manager Metrics - Revision history

Karl.Abraham: Published