PrivateEdition/Current/Operations/Logging approaches - Revision history

WikiSysop at 19:32, March 12, 2023

2023-03-12T19:32:01Z

WikiSysop at 17:46, June 30, 2022

2022-06-30T17:46:06Z

WikiSysop at 18:16, May 12, 2022

2022-05-12T18:16:09Z

Jabashree.amudha@genesys.com: Published

2021-12-20T18:03:57Z

Published

New page

{{Article
|Standalone=No
|DisplayName=Logging overview and approaches
|Context=Learn about the structured, unstructured, and Sidecar logging methods that Genesys Multicloud CX private edition services use.
|ComingSoon=No
|Section={{Section
|sectionHeading=Overview and approaches
|alignment=Vertical
|structuredtext=Application log files contain the important diagnostic information for various issues that may arise. Support of Genesys services rely on access to these application logs. In Genesys Multicloud CX private edition, the Genesys Multicloud CX services write these log files using different methods and formats. Some services write to a standard out/standard error (stdout/stderr) console while others write directly into an RWX shared storage. This data must be accessible outside of the cluster environment for shipping diagnostic logs for further review.

In the OpenShift container platform, the Fluentd data collector collects the logs from multiple nodes and pushes them into the Elasticsearch cluster. You can pull the collective logs from the Elasticsearch cluster into a log viewing tool like Kibana to visualize and track this data.

By default, GKE clusters are natively integrated with Cloud Logging. When you create a GKE cluster, Cloud Logging is enabled by default.
|Status=No
}}{{Section
|sectionHeading=OpenShift cluster logging
|alignment=Vertical
|structuredtext=Private edition services use one of the following approaches:

*'''Kubernetes-supported structured logging''' — The services write structured logs. These logs are written in the standard stdout/stderr console and supported by Kubernetes. Fluentd collects these logs from multiple nodes and formats them by appending Kubernetes pod and project metadata. For more information, see {{Link-SomewhereInThisVersion|manual=Operations|topic=Structured_logging}}.
*'''Sidecar processed logging''' — The services write their logs in a log file. A sidecar container processes these log files and then writes them to the stdout/stderr console. A log aggregator such as Fluentd collects these logs from stdout/stderr and formats them by appending Kubernetes pod and project metadata. For more information, see {{Link-SomewhereInThisVersion|manual=Operations|topic=Sidecar_logging}}.
*'''RWX logging (unstructured)''' — The services write unstructured logs. These unstructured logs can neither be directly processed by a sidecar container nor be collected by Fluentd. These services write their logs in a mounted Persistent Volume Claim (PVC) bound to Persistent Volume (PV) which is backed by an RWX shared storage such as NFS or NAS for ease of access. For more information, see [[Draft:PrivateEdition/Current/Operations/RWXlogging|RWX (unstructured) logging]].
*:{{NoteFormat| A Cluster Administrator must create appropriate PVCs and RWX shared storage path for the services that use the RWX logging method. For more information about creating the log-specific storage, refer to the related {{Link-SomewhereInThisVersion|manual=PEGuide|topic=GEServices}}.<br/><br/>RWX logging is deprecated. It will be phased out with the use of sidecars to facilitate legacy logging behavior.|}}

===OpenShift cluster logging approach===
This diagram illustrates the OpenShift cluster logging approach. Here, the pods write to PVCs that are mounted as persistent volume backed by external NFS shares, and you can access the logs directly. Standard log OpenShift comes with preloaded logging support with Elasticsearch and Fluentd that is contained within the cluster itself. OpenShift uses Fluentd to collect the logs and store them in an external log aggregator such as Elasticsearch or Logstash. You have the option to choose the external log aggregator per your preference.

[[File:OCP logging.png]]Click [[Draft:PrivateEdition/Current/PEGuide/ConfigLog#OpenShift logging|here]] to learn more about logging architecture and its components.
{{NoteFormat|Fluentd, Elasticsearch, and Kibana are the built-in logging collectors for OpenShift clusters. However, you can use an external Elasticsearch or other log aggregators per your requirement.|}}

For details about how OpenShift Container Platform uses Fluentd to collect operations and application logs from your cluster, refer to [https://docs.openshift.com/container-platform/4.6/logging/config/cluster-logging-collector.html Configuring the logging collector].
|Status=No
}}{{Section
|sectionHeading=Sample Kibana queries to find logs
|alignment=Vertical
|structuredtext=Here are some sample queries for you to understand what information could be searched for in Kibana. The search is specific to the values in the query. It returns the log messages that matches the query.

'''Query 1: To return podname , namespace , and container name'''

<tt>kubernetes.pod_name:"t100-0" AND kubernetes.namespace_name:"voice" AND kubernetes.container_name.raw:"tenant"</tt>

'''Output:'''[[File:Query1.png]]

'''Query 2: Any string and logs listed with the string'''

<tt>"Voice-sip"</tt>

'''Output'''[[File:Query2.png]]

'''Query 3: Any combination with service, service name, instance , name , version, any value available in the logs'''

<tt>"service=ixn" AND "servicename=ixn-vqnode"</tt>

'''Output:'''

[[File:Query3.png]]'''Query 4: CallUUID is consistent across interactions'''

<tt>kubernetes.pod_name."voice-sip-0" AND kubernetes.namespace_name."voice"</tt> AND <tt>kubernetes.container_name.raw."voice-sip"</tt> AND "CallUUID.00E5IM3D048KVE8LFG1862LAES0001SP"

'''Output:'''[[File:MicrosoftTeams-image.png]]

<br />
|Status=No
}}{{Section
|sectionHeading=GKE logging
|anchor=GKElogging
|alignment=Vertical
|structuredtext=Google Cloud's operations suite is backed by Google Stackdriver which controls logging, monitoring, and alerting within Google Cloud Platform. System and user workload logs are captured using Google’s own Fluentd DaemonSet called Google-Fluentd that runs on each node in your cluster. The Daemon set parses container logs and pipes them to the stackdriver for processing.

Stackdriver provides built-in log metric capabilities that allows you to monitor specific log events for building dashboards and alert policies.

By default, GKE clusters are natively integrated with cloud logging. When you create a GKE cluster, cloud logging is enabled by default.

You can create a cluster with Logging enabled, or enable Logging in an existing cluster.

[[File:GKE Monitoring.png]]

===Enable cloud logging===
The following table provides the supported values for the <code>--logging</code> flag for the [https://cloud.google.com/sdk/gcloud/reference/container/clusters/create create] and [https://cloud.google.com/sdk/gcloud/reference/container/clusters/update update] commands.
{{{!}} class="wikitable"
{{!}}+
!Source
!Value
!Logs collected
{{!}}-
{{!}}System
{{!}}'''SYSTEM'''
{{!}}Collects logs from:

*Pods running in namespaces <code>kube-system</code>, <code>istio-system</code>, <code>knative-serving</code>, <code>gke-system</code>, and <code>config-management-system</code>.
*Key services that are not containerized including <code>docker</code>/<code>containerd</code> runtime, <code>kubelet</code>, <code>kubelet-monitor</code>, <code>node-problem-detector</code>, and <code>kube-container-runtime-monitor</code>.
*The node's serial ports output, if the VM instance metadata <code>serial-port-logging-enable</code> is set to '''true'''.
{{!}}-
{{!}}Workload
{{!}}'''WORKLOAD'''
{{!}}All logs generated by non-system containers running on user nodes.
{{!}}}

====Console UI====
To enable cloud logging through console UI, follow these steps:

#Navigate to Console UI using: '''https://console.cloud.google.com/kubernetes/list/overview?project=gcpe0001'''
#Select '''Clusters''' and then select the cluster name.
#Under '''Features''', select '''Cloud Logging''', and then click '''Edit'''.
#Select '''Enable Cloud Logging''' and then select '''System and Workflow''' from drop-down.
#Save the changes.

====[[File:ConsoleUI.png]]GCloud CLI====
To enable cloud logging through GCloud CLI, follow these steps:

#Log on to the existing GCloud cluster.
#:<source lang="yaml">gcloud container clusters get-credentials gke1 --zone us-west1-a --project gcpe0001</source>
#Configure the logs to be sent to Cloud Logging by updating a comma-separated list of values to the <code>gcloud container clusters update</code> with <code>--logging</code> flag.
#:<source lang="yaml">gcloud container clusters update gke1 \
--zone=us-west1-a \
--logging=SYSTEM,WORKLOAD </source>

===Accessing logs===

====Log Explorer====
Log explorer is Google's central Logging UI. You can access logs for your Google cloud resources from this console, including GKE, Cloud SQL, VM instances and so on. You can then use logging filters to select the Kubernetes resources, such as cluster, node, namespace, pod, or container logs.

For more details about the console, click [https://console.cloud.google.com/logs/query;cursorTimestamp=2021-09-24T19:02:37.624894Z?project=gcpe0001&folder=true&organizationId=832400987249&query=%0A here].

[[File:Console1.png]]

[[File:Console2.png]]

===Cloud Monitoring Console===
Cloud Monitoring Console allows you to track metrics of resources within your GCP/GKE environment. This console allows you to access your logs from a particular Cluster, Namespace, Node, and Pod.

[[File:CMC1.png]]

===[[File:CMC2.png]]GKE Console===
GKE web console enables you to access to logs on individual pods actively running within a workload.

There is a filter option available to filter specific events, and a drop-down field to target specific severity of log events.

Logs provide a link to access '''Logs Explorer''' from a given pod to access the main logs explorer page for enhanced querying capabilities and other features.

===[[File:GKE Console.png]]Command-Line===
The standard '''kubectl''' logs commands are supported in GKE. They provide actively running stdout logs from containers.

Example: <source lang="yaml">kubebctl logs gvp-mcp-0 -n gvp -c fluentbit | more</source>

[[File:MicrosoftTeams-image (3).png]]<br />
|Status=No
}}
}}

@@ Line 8: / Line 8: @@
 |alignment=Vertical
 |structuredtext=Application log files contain the important diagnostic information for various issues that may arise. Support of Genesys services rely on access to these application logs. In Genesys Multicloud CX private edition, the Genesys Multicloud CX services write these log files using different methods and formats. Some services write to a standard out/standard error (stdout/stderr) console while others write directly into an RWX shared storage. This data must be accessible outside of the cluster environment for shipping diagnostic logs for further review.
 By default, GKE clusters are natively integrated with Cloud Logging. When you create a GKE cluster, Cloud Logging is enabled by default.
@@ Line 24: / Line 22: @@
 *:{{NoteFormat| A Cluster Administrator must create appropriate PVCs and RWX shared storage path for the services that use the RWX logging method. For more information about creating the log-specific storage, refer to the related {{Link-SomewhereInThisVersion|manual=PEGuide|topic=GEServices}}.<br/><br/>RWX logging is deprecated. It will be phased out with the use of sidecars to facilitate legacy logging behavior.|}}
-===OpenShift cluster logging approach===
+===AKS logging approach===
-This diagram illustrates the OpenShift cluster logging approach. Here, the pods write to PVCs that are mounted as persistent volume backed by external NFS shares, and you can access the logs directly. Standard log OpenShift comes with preloaded logging support with Elasticsearch and Fluentd that is contained within the cluster itself. OpenShift uses Fluentd to collect the logs and store them in an external log aggregator such as Elasticsearch or Logstash. You have the option to choose the external log aggregator per your preference.
+In Azure, the [https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-workspace-overview Log Analytics workspace feature] in the Azure Monitor service collects log data from multiple services and system. You can create a single or multiple workspaces and feed the application logs into them.
-For details about how OpenShift Container Platform uses Fluentd to collect operations and application logs from your cluster, refer to [https://docs.openshift.com/container-platform/4.6/logging/config/cluster-logging-collector.html Configuring the logging collector].
+For more detailed instructions, refer [https://github.com/genesys/multicloud-services/tree/main/services/logging Genesys logging github].
 |Status=No
 }}{{Section

@@ Line 20: / Line 20: @@
 *'''Kubernetes-supported structured logging''' — The services write structured logs. These logs are written in the standard stdout/stderr console and supported by Kubernetes. Fluentd collects these logs from multiple nodes and formats them by appending Kubernetes pod and project metadata. For more information, see {{Link-SomewhereInThisVersion|manual=Operations|topic=Structured_logging}}.
 *'''Sidecar processed logging''' — The services write their logs in a log file. A sidecar container processes these log files and then writes them to the stdout/stderr console. A log aggregator such as Fluentd collects these logs from stdout/stderr and formats them by appending Kubernetes pod and project metadata. For more information, see {{Link-SomewhereInThisVersion|manual=Operations|topic=Sidecar_logging}}.
-*'''RWX logging (unstructured)''' — The services write unstructured logs. These unstructured logs can neither be directly processed by a sidecar container nor be collected by Fluentd. These services write their logs in a mounted Persistent Volume Claim (PVC) bound to Persistent Volume (PV) which is backed by an RWX shared storage such as NFS or NAS for ease of access. For more information, see [[Draft:PrivateEdition/Current/Operations/RWXlogging|RWX (unstructured) logging]].
+*'''RWX logging (unstructured)''' — The services write unstructured logs. These unstructured logs can neither be directly processed by a sidecar container nor be collected by Fluentd. These services write their logs in a mounted Persistent Volume Claim (PVC) bound to Persistent Volume (PV) which is backed by an RWX shared storage such as NFS or NAS for ease of access. For more information, see {{Link-SomewhereInThisVersion|manual=Operations|topic=RWXlogging|display text=RWX (unstructured) logging}}.
 *:{{NoteFormat| A Cluster Administrator must create appropriate PVCs and RWX shared storage path for the services that use the RWX logging method. For more information about creating the log-specific storage, refer to the related {{Link-SomewhereInThisVersion|manual=PEGuide|topic=GEServices}}.<br/><br/>RWX logging is deprecated. It will be phased out with the use of sidecars to facilitate legacy logging behavior.|}}
@@ Line 26: / Line 26: @@
 This diagram illustrates the OpenShift cluster logging approach. Here, the pods write to PVCs that are mounted as persistent volume backed by external NFS shares, and you can access the logs directly. Standard log OpenShift comes with preloaded logging support with Elasticsearch and Fluentd that is contained within the cluster itself. OpenShift uses Fluentd to collect the logs and store them in an external log aggregator such as Elasticsearch or Logstash. You have the option to choose the external log aggregator per your preference.
-[[File:OCP logging.png]]Click [[Draft:PrivateEdition/Current/PEGuide/ConfigLog#OpenShift logging|here]] to learn more about logging architecture and its components.
+[[File:OCP logging.png]]Click {{Link-AnywhereElse|product=PrivateEdition|version=Current|manual=PEGuide|topic=ConfigLog|anchor=ConfigLogging|display text=here}} to learn more about logging architecture and its components.
 {{NoteFormat|Fluentd, Elasticsearch, and Kibana are the built-in logging collectors for OpenShift clusters. However, you can use an external Elasticsearch or other log aggregators per your requirement.|}}