PrivateEdition/Current/Operations/Alerting - Revision history

WikiSysop at 19:31, March 12, 2023

2023-03-12T19:31:55Z

WikiSysop at 17:45, June 30, 2022

2022-06-30T17:45:57Z

WikiSysop at 18:01, May 12, 2022

2022-05-12T18:01:09Z

Jabashree.amudha@genesys.com: Published

2021-12-20T18:03:26Z

Published

New page

{{Article
|Standalone=No
|DisplayName=Handling alerts
|Context=Learn about deploying service alerts.<br />
|ComingSoon=No
|Section={{Section
|sectionHeading=Introduction
|anchor=IntroAlerting
|alignment=Vertical
|structuredtext=Alerts notify you when certain metrics exceed specified thresholds. In some services, alerting is enabled by default; in others, you must enable alerting when you deploy the service. See the respective service guides (listed [[Draft:PrivateEdition/Current/PEGuide/GEServices|here]]) for details about service-specific support for alerting.
|Status=No
}}{{Section
|sectionHeading=Alert rules
|anchor=AlertRules
|alignment=Vertical
|structuredtext=By default, most services define alerts for certain key operational parameters. The alerts are '''PrometheusRule''' objects that are defined in a YAML file. The metrics collected from the applicable service are evaluated based on the expression specified in the rule. An alert is triggered if the value of the expression is <tt>true</tt>.

Private edition does not support custom alerts triggered by rules you define yourself. However, some services — for example, [[Draft:DES/Current/DESPEGuide/Observability|Designer]] — enable you to modify certain parameters in the '''values.yaml''' file to customize the predefined alerts by modifying the values that trigger the alert. See the respective service-level guides for information about the limited customization each service might support.
==Prometheus / Alertmanager==
Enable ServiceMonitor or PodMonitor to scrape metrics from the cluster.
To import custom alerts or notification configurations, follow these steps.
===Alerting Rules===
This section describes how to create alert rules and import custom rules.

#Create alert rules. These rules triggers alerts based on the values.
#:<source lang="yaml">
apiVersion: "monitoring.coreos.com/v1"
kind: PrometheusRule
metadata:
name: <name>-alertrules
labels:
genesysengage/monitoring: prometheus
service: <service>
servicename: <servicename>
tenant: <tenant> --> Ex: shared
spec:
groups:
- name: <name>-alert
rules:
- alert: <alert-name>
expr: <expression>
for: <time> For ex: 5m
labels:
severity: <severity> For ex: critical
service: <service>
servicename: <servicename>
annotations:
summary: "<description>"
</source>
#Import the custom rule.
#:<source lang="yaml">
kubectl apply -f <rules.yaml> -n monitoring
</source>
|Status=No
}}{{Section
|sectionHeading=Customizing Alertmanager configuration for notifications
|anchor=Alertmanager
|alignment=Vertical
|structuredtext=Alertmanager sends notifications to the notification provider such as email or Webhook (PagerDuty) when an alert is triggered.

===OpenShift Container Platform===
To add notification configuration, edit the ''alertmanager-main'' secret inside the ''openshift-monitoring'' namespace. You must perform the following steps to receive alert notifications.

#Copy the active '''Alertmanager''' configuration into the '''alertmanager.yaml''' file.
#:<source lang="yaml">
$ oc -n openshift-monitoring get secret alertmanager-main --template='{{index .data "alertmanager.yaml" }}' base64 --decode > alertmanager.yaml
</source>
#Add configuration in the '''alertmanager.yaml''' file.
#:<source lang="yaml">
global:

resolve_timeout: 5m

route:

group_wait: 30s

group_interval: 5m

repeat_interval: 12h

receiver: default

routes:

- match:

alertname: Watchdog

repeat_interval: 5m

receiver: watchdog

- match:

service: <your_service>

routes:

- match:

<your_matching_rules>

receiver: <receiver>

receivers:

- name: default

- name: watchdog

- name: <receiver>

<receiver_configuration>
</source>
#Apply the changes.
#:<source lang="yaml">create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run -o=yaml oc -n openshift-monitoring replace secret --filename=-
</source>

===Alertmanager configuration for Notifications===
Alertmanager sends notifications to the notification provider (such as email or PagerDuty) when an alert is triggered.

To add notification configuration, edit '''alertmanager.yaml''' using the following steps:

#Load the configuration map into a file using the following command.
#:<source lang="yaml">
kubectl get configmap prometheus-alertmanager --namespace=monitoring -o yaml > alertmanager.yaml </source>
#Add the configuration in '''alertmanager.yaml'''.
#:<source lang="yaml">global:
resolve_timeout: 5m
route:
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: default
routes:
- match:
alertname: Watchdog
repeat_interval: 5m
receiver: watchdog
- match:
service: <your_service>
routes:
- match:
<your_matching_rules>
receiver: <receiver>
receivers:
- name: default
- name: watchdog
- name: <receiver>
<receiver_configuration></source>
#Save the changes in the file and replace the configuration map.
#:<source lang="yaml"> kubectl replace configmaps prometheus-alertmanager --namespace=monitoring -f alertmanager.yaml</source>

For more details about configuring receivers for alert notification and how the receiver types are created/configured, refer to [https://docs.openshift.com/container-platform/4.6/post_installation_configuration/configuring-alert-notifications.html Configuring alert notifications].
|Status=No
}}{{Section
|sectionHeading=GKE platform
|anchor=GoogleOpsSuiteAlerting
|alignment=Vertical
|structuredtext====Google Cloud operations suite – Alerting===
Google Cloud operations suite is backed by Stackdriver which ingests and processes alerts based on predefined policy configuration.

Stackdriver utilizes Google Cloud Monitoring API for management of metric and alert policies within the operation suite.

Here are some key features provided by Google Cloud operation suite:

*Google Cloud API supports over 1,500 Cloud Monitoring metrics.
*Alert policies are configured as a resource object in cloud monitoring API.
*Unlike Alert Manager, policies are defined directly through GCP Cloud Monitoring API via REST or GRCP request. There are no custom resource objects in Kubernetes for alert polices in GKE.
*Defining alert policies allows you to define specific conditions and actions to take in reaction to key metrics and other criteria.
*Notification channels are used to specify where alerts should be sent when an incident occurs. For example:
**Webhook
**Email
**PagerDuty

For more details, refer to the following Google document pages:

*[https://cloud.google.com/monitoring/alerts Introduction to alerting]
*[https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies Resource: AlertPolicy]
*[https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.notificationChannels Resource: NotificationChannel]
*[https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.uptimeCheckConfigs Resource: UptimeCheckConfig]

===Google Cloud monitoring API===

====Alert Policy REST API====
All API requests to Google Cloud Monitoring API require proper authentication before you query and apply configuration.

See [https://cloud.google.com/docs/authentication/ Google authentication] for further details.

Here are various functions that are available for creation of custom alert policy.

'''projects.alertPolicies.create'''

POST [https://monitoring.googleapis.com/v3/%7Bname%7D/alertPolicies https://monitoring.googleapis.com/v3/{name}/alertPolicies]

'''projects.alertPolicies.delete'''

DELETE [https://monitoring.googleapis.com/v3/%7Bname https://monitoring.googleapis.com/v3/{name]}

'''projects.alertPolicies.get'''

GET [https://monitoring.googleapis.com/v3/%7Bname https://monitoring.googleapis.com/v3/{name]}

'''projects.alertPolicies.list'''

GET [https://monitoring.googleapis.com/v3/%7Bname%7D/alertPolicies https://monitoring.googleapis.com/v3/{name}/alertPolicies]

'''projects.alertPolicies.patch'''

PATCH [https://monitoring.googleapis.com/v3/%7BalertPolicy.name https://monitoring.googleapis.com/v3/{alertPolicy.name]}

Here is an example:

'''AlertPolicy - NGINX Ingress Uptime Check'''

(This example assumes you have created notification channel and uptime check prior to deployment.)<source lang="yaml">
{
"displayName": "Uptime-Test uptime failure- Ingress",
"documentation": {
"content": "Indicates issue with NGINX Ingress availability. Check ingress-nginx-controller-* in the 'ingress-nginx' namespace",
"mimeType": "text/markdown"
},
"conditions": [
{
"displayName": "Failure of uptime check_id uptime-test",
"conditionThreshold": {
"aggregations": [
{
"alignmentPeriod": "1200s",
"crossSeriesReducer": "REDUCE_COUNT_FALSE",
"groupByFields": [
"resource.label.*"
],
"perSeriesAligner": "ALIGN_NEXT_OLDER"
}
],
"comparison": "COMPARISON_GT",
"duration": "60s",
"filter": "metric.type=\"monitoring.googleapis.com/uptime_check/check_passed\" AND metric.label.check_id=\"<upgtimeCheckConfigs ID> pod " AND resource.type=\"k8s_service\"",
"thresholdValue": 1,
"trigger": {
"count": 1
}
}
}
],
"combiner": "OR",
"enabled": true,
"notificationChannels": [
"projects/gcpe0001/notificationChannels/<notificationChannel ID>"
]
}
</source>
|Status=No
}}
}}

@@ Line 59: / Line 59: @@
 |structuredtext=Alertmanager sends notifications to the notification provider such as email or Webhook (PagerDuty) when an alert is triggered.
-===OpenShift Container Platform===
 ===Alertmanager configuration for Notifications===

@@ Line 182: / Line 182: @@
 *[https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.uptimeCheckConfigs Resource: UptimeCheckConfig]
-===Google Cloud monitoring API===
+===Google Cloud Monitoring API - Alert Policy===
 ====Alert Policy REST API====
@@ Line 211: / Line 211: @@
 PATCH [https://monitoring.googleapis.com/v3/%7BalertPolicy.name https://monitoring.googleapis.com/v3/{alertPolicy.name]}
-Here is an example:
+====Alert Policy example====
-'''AlertPolicy - NGINX Ingress Uptime Check'''
+'''AlertPolicy - NGINX Ingress Uptime Check'''<source lang="yaml">
+{
    "displayName": "Uptime-Test uptime failure- Ingress",

@@ Line 8: / Line 8: @@
 |anchor=IntroAlerting
 |alignment=Vertical
-|structuredtext=Alerts notify you when certain metrics exceed specified thresholds. In some services, alerting is enabled by default; in others, you must enable alerting when you deploy the service. See the respective service guides (listed [[Draft:PrivateEdition/Current/PEGuide/GEServices|here]]) for details about service-specific support for alerting.
+|structuredtext=Alerts notify you when certain metrics exceed specified thresholds. In some services, alerting is enabled by default; in others, you must enable alerting when you deploy the service. See the respective service guides (listed {{Link-AnywhereElse|product=PrivateEdition|version=Current|manual=PEGuide|topic=GEServices|display text=here}}) for details about service-specific support for alerting.
 |Status=No
 }}{{Section
@@ Line 16: / Line 16: @@
 |structuredtext=By default, most services define alerts for certain key operational parameters. The alerts are '''PrometheusRule''' objects that are defined in a YAML file. The metrics collected from the applicable service are evaluated based on the expression specified in the rule. An alert is triggered if the value of the expression is <tt>true</tt>.
-Private edition does not support custom alerts triggered by rules you define yourself. However, some services — for example, [[Draft:DES/Current/DESPEGuide/Observability|Designer]] — enable you to modify certain parameters in the '''values.yaml''' file to customize the predefined alerts by modifying the values that trigger the alert. See the respective service-level guides for information about the limited customization each service might support.
+Private edition does not support custom alerts triggered by rules you define yourself. However, some services — for example, {{Link-AnywhereElse|product=DES|version=Current|manual=DESPEGuide|topic=Observability|display text=Designer}} — enable you to modify certain parameters in the '''values.yaml''' file to customize the predefined alerts by modifying the values that trigger the alert. See the respective service-level guides for information about the limited customization each service might support.
 ==Prometheus / Alertmanager==
 Enable ServiceMonitor or PodMonitor to scrape metrics from the cluster.