WikiSysop at 14:30, June 6, 2022

2022-06-06T14:30:46Z

Jose.druker@genesys.com: Published

2021-12-16T18:12:54Z

Published

New page

{{ArticlePEServiceMetrics
|IncludedServiceId=c39fe496-c79e-4846-b451-1bc8bedb126b
|CRD=PodMonitor
|Port=9249
|Endpoint=Endpoint: /

Selector:
<source lang="bash">matchLabels:
app: {{ template "gsp.fullname" . }}</source>
where the value of <tt>gsp.fullname</tt> depends on deployment parameters such as Helm release name, <tt>.Values.fullnameOverride</tt>, and <tt>.Values.nameOverride</tt>.
|MetricsUpdateInterval=30 seconds
|MetricsDefined=Yes
|MetricsIntro=GSP exposes some standard Apache Flink and Kafka metrics as well as Genesys-defined metrics, which are exposed via the Flink API. Therefore, all GSP metrics start with the prefix '''flink_''' but in some cases the values are calculated by GSP.

You can query Prometheus directly to see all the metrics Flink and the Flink Kafka connector expose through GSP.
*For full information about the standard Flink metrics, see the [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#system-metrics Apache Flink documentation].
*For full information about the Kafka metrics, see the [https://kafka.apache.org/20/documentation.html#monitoring Apache Kafka] or [https://docs.confluent.io/platform/current/kafka/monitoring.html Confluent Kafka] documentation.

The following metrics are likely to be particularly useful. The naming convention is '''<flink_scope_prefix>_<GSP suffix>'''. Genesys does not commit to maintain other currently available GSP metrics not documented on this page.
|PEMetric={{PEMetric
|Metric=flink_taskmanager_job_task_operator_errors_numInvalidRecords
|Type=Gauge
|MetricDescription=Number of invalid input records.
|SampleValue=0
|UsedFor=Error
}}{{PEMetric
|Metric=flink_jobmanager_numRunningJobs
|Type=Gauge
|MetricDescription=Number of running Flink jobs. If less than 1, there is a problem.
|SampleValue=1
|UsedFor=Error
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_user_errors_numOversizedMessages
|Type=Gauge
|Label= 
*operator_name
|MetricDescription=Number of messages exceeding the '''max.request.size''' Kafka option.
|SampleValue=0
|UsedFor=Error
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_tenant_error_total
|Type=Gauge
|Label= 
*operator_name
*tenant
*error
|MetricDescription=Number of issues encountered, such as errors or warnings.
|UsedFor=Error
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_currentInputWatermark
|Type=Gauge
|Unit=milliseconds
|Label= 
*operator_name
|MetricDescription=The last watermark received by this operator/task, in milliseconds since the Unix Epoch (00:00:00 UTC on 1 January 1970). 
'''Note:''' For operators/tasks with two inputs, this is the earlier of the last received watermarks.
|UsedFor=Latency
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_currentOutputWatermark
|Type=Gauge
|Unit=milliseconds
|Label= 
*operator_name:
**Sink:_Agent_State_Facts
**Sink:_Interaction_Facts
|MetricDescription=The last watermark this operator has emitted, in milliseconds since the Unix Epoch.
|UsedFor=Latency
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_records_lag_max
|Type=Gauge
|MetricDescription=The maximum lag in terms of the number of records for any partition in this window. An increasing value over time is your best indication that the consumer group is not keeping up with the producers.
|UsedFor=Latency
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_records_consumed_rate
|Type=Gauge
|MetricDescription=The average number of records consumed per second.
|UsedFor=Traffic
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_numCallsCreated
|Type=Gauge
|MetricDescription=Total number of EventCallCreated events GSP received since it started processing.
|UsedFor=Traffic
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_numCallsCreatedPerSecond
|Type=Gauge
|MetricDescription=Number of EventCallCreated events per second (CPS).
|UsedFor=Traffic
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_numThreadsCreated
|Type=Gauge
|MetricDescription=Total number of CallThreads GSP received since it started processing.
|UsedFor=Traffic
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_numCallThreadsCreatedPerSecond
|Type=Gauge
|MetricDescription=Number of CallThreads per second (CTHPS).
|UsedFor=Traffic
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_numChainsProcessed
|Type=Gauge
|MetricDescription=Total number of EventOCSChainStartProcessing events GSP received since it started processing.
|UsedFor=Traffic
}}{{PEMetric
|Metric=flink_taskmanager_job_task_operator_numChainsProcessedPerSecond
|Type=Gauge
|MetricDescription=Number of EventOCSChainStartProcessing events per second (CPS).
|UsedFor=Traffic
}}{{PEMetric
|Metric=flink_(job|task)manager_Status_JVM_CPU_Load
|Type=Gauge
|Label= 
*pod
|MetricDescription=The recent CPU usage for the JVM process. The value is a double in the <nowiki>[0.0,1.0]</nowiki> interval, where a value of 0.0 means that none of the CPUs were running threads from the JVM process, while a value of 1.0 means that all CPUs were actively running threads from the JVM 100% of the time during the recent period being observed. A negative value means usage data is not available. For more information, see [https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad() {{#replace:https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad()|/|/}}].
|SampleValue=
|UsedFor=Saturation
}}{{PEMetric
|Metric=flink_(job|task)manager_Status_JVM_Memory_Direct_TotalCapacity
|Type=Gauge
|Unit=bytes
|Label= 
*pod
|MetricDescription=The total capacity of all buffers in the direct buffer pool.
|UsedFor=Saturation
}}{{PEMetric
|Metric=flink_(job|task)manager_Status_JVM_Memory_Direct_MemoryUsed
|Type=Gauge
|Unit=bytes
|Label= 
*pod
|MetricDescription=The amount of memory used by the JVM for the direct buffer pool.
|UsedFor=Saturation
}}{{PEMetric
|Metric=flink_(job|task)manager_Status_JVM_Memory_NonHeap_Max
|Type=Gauge
|Unit=bytes
|Label= 
*pod
|MetricDescription=The maximum amount of non-heap memory that can be used for memory management.
|UsedFor=Saturation
}}{{PEMetric
|Metric=flink_(job|task)manager_Status_JVM_Memory_NonHeap_Used
|Type=Gauge
|Unit=bytes
|Label= 
*pod
|MetricDescription=The amount of non-heap memory currently used.
|UsedFor=Saturation
}}{{PEMetric
|Metric=flink_(job|task)manager_Status_JVM_Memory_Heap_Max
|Type=Gauge
|Unit=bytes
|Label= 
*pod
|MetricDescription=The maximum amount of heap memory that can be used for memory management.
|UsedFor=Saturation
}}{{PEMetric
|Metric=flink_(job|task)manager_Status_JVM_Memory_Heap_Used
|Type=Gauge
|Unit=bytes
|Label= 
*pod
|MetricDescription=The amount of heap memory currently used.
|UsedFor=Saturation
}}
|AlertsDefined=Yes
|AlertsIntro=The alerts are based on Flink and Kubernetes cluster metrics.
|PEAlert={{PEAlert
|Alert=GspFlinkJobDown
|Severity=Critical
|AlertDescription=Triggered when the GSP Flink job is not running (number of running jobs equals to 0 or metric is not available)
|BasedOn=flink_jobmanager_numRunningJobs
|Threshold=For 5 minutes
}}{{PEAlert
|Alert=GspOOMKilled
|Severity=Critical
|AlertDescription=Triggered when a GSP pod is restarted because of OOMKilled
|BasedOn=kube_pod_container_status_restarts_total
|Threshold=0
}}{{PEAlert
|Alert=GspNoTmRegistered
|Severity=Critical
|AlertDescription=Triggered when there are no registered TaskManagers (or metric not available)
|BasedOn=flink_jobmanager_numRegisteredTaskManagers
|Threshold=For 5 minutes
}}{{PEAlert
|Alert=GspUnknownPerson
|Severity=High
|AlertDescription=Triggered when GSP encounters unknown person(s)
|BasedOn={{#replace:flink_taskmanager_job_task_operator_tenant_error_total{error="unknown_person",service="gsp"}|_|_}}
|Threshold=For 5 minutes
}}
}}

@@ Line 165: / Line 165: @@
 |AlertsDefined=Yes
 |AlertsIntro=The alerts are based on Flink and Kubernetes cluster metrics.
 |PEAlert={{PEAlert
 |Alert=GspFlinkJobDown

PEC-REP/Current/GIMPEGuide/GSPMetrics - Revision history

WikiSysop at 14:30, June 6, 2022

Jose.druker@genesys.com: Published