Observability in Tenant Service
Contents
Learn about the logs, metrics, and alerts you should monitor for Tenant Service.
Monitoring
Private edition services expose metrics that can be scraped by Prometheus, to support monitoring operations and alerting.
- As described on Monitoring overview and approach, you can use a tool like Grafana to create dashboards that query the Prometheus metrics to visualize operational status.
- As described on Customizing Alertmanager configuration, you can configure Alertmanager to send notifications to notification providers such as PagerDuty, to notify you when an alert is triggered because a metric has exceeded a defined threshold.
The services expose a number of Genesys-defined and third-party metrics. The metrics that are defined in third-party software used by private edition services are available for you to use as long as the third-party provider still supports them. For descriptions of available Tenant Service metrics, see:
See also System metrics.
Tenant Service supports various metrics in Prometheus format, exposed via a dedicated endpoint.
Enable monitoring
Tenant Service deployment supports the creation of a Prometheus monitor resource in order to scrape the monitoring endpoint; this feature is disabled by default. For information about the parameters that enable a Prometheus monitoring resource, see Monitoring and observability.
| Service | CRD or annotations? | Port | Endpoint/Selector | Metrics update interval |
|---|---|---|---|---|
| Tenant Service | PodMonitor | 15000 | /metrics (http://<pod address>:15000/metrics) | 30 seconds (Applicable for any metric(s) that Tenant Service exposes. The update interval is not a property of the metric; it is a property of the optional PodMonitor that you can create.) |
Configure metrics
The metrics that are exposed by the Tenant Service are available by default. No further configuration is required in order to define or expose these metrics. You cannot define your own custom metrics.
The Metrics pages linked to above show some of the metrics the Tenant Service exposes. You can also query Prometheus directly or via a dashboard to see all the metrics available from the Tenant Service.
Alerting
Private edition services define a number of alerts based on Prometheus metrics thresholds.
For descriptions of available Tenant Service alerts, see:
Configure alerts
Private edition services define a number of alerts by default (for Tenant Service, see the pages linked to above). No further configuration is required.
The alerts are defined as PrometheusRule objects in a prometheus-rule.yaml file in the Helm charts. As described above, Tenant Service does not support customizing the alerts or defining additional PrometheusRule objects to create alerts based on the service-provided metrics.
Logging
The Tenant Service Helm Chart values.yaml files include the following configurable log volume options:
- Persistent Volume Claim (PVC) with RWX storage: This creates a shared RWX volume in tenant-monitor; this volume is then claimed across all pods of all tenants. For more information, see tenant-monitor and Samples and references.
- Persistent Volume (PV)/PVC with RWO storage for logging from individual Tenant pods: Mount the PVC from the Tenant StatefulSet volume claim template, instead of using tenant-monitor. For more information about Statefulsets, see Statefulsets in the Kubernetes documentation.
- For more information about the Tenant Service RWO PV/PVC log volume configuration, see tenant-monitor and Samples and references.
- Ephemeral volume (emptyDir) with a Fluent Bit logging sidecar that tails log files and sends them to standard output (stdout). The optional log forwarding to stdout is disabled by default. To enable the log forwarding option, see tenant-monitor.
- For general information about forwarding logs from internal components to stdout, see Sidecar processed logging in the Genesys Multicloud CX Private Edition Operations guide.