PrivateEdition/Current/Operations/Structured logging

A secondary method of logging required for standard stdout/stderr structured logging.

Related documentation:

• Release Notes
• All services

RSS:

• For private edition

This logging method that is required for standard stdout/stderr structured logs that are generated by containers within the Kubernetes environment. Therefore, this method is also called Kubernetes-supported logging. Here, the container is writes stdout/stderr logs to a – var/log/containers directory.

You will be given the option to choose the external log aggregator to implement the aggregation.

Services that use Kubernetes structured logging:

• Genesys Authentication
• Web Services and Applications
• Genesys Engagement Services
• Designer

Important

Some services (such as Genesys Info Mart) use the Kubernetes logging approach with an exception that the logs are written in an unstructured format.

Deploying logging in OpenShift cluster

Prerequisites

• Access to the cluster as a user with the cluster-admin role.
• rsyslogd server daemon installed and listening on TCP port
• rsyslog ruleset for writing container messages to file accessible by NFS/NAS

Deployment process

1. Deploy Elasticsearch and Cluster Logging using their Operators. You can deploy these Operators using any one of the following methods:

   • Installing cluster logging using the web console
   • Installing cluster logging using the CLI

2. If the cluster has taint nodes, Fluentd requires the following toleration levels.

   tolerations:
   - effect: NoSchedule
     operator: Exists
   - key: CriticalAddonsOnly
     operator: Exists
   - effect: NoExecute
     operator: Exists

3. Create a cluster logging instance with Fluentd tolerations mentioned in the previous step. Here's a sample code to build your own YAML file.

   clusterLogging.yaml

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     managementState: "Managed" 
     logStore:
       type: "elasticsearch" 
       retentionPolicy:
         application:
           maxAge: 1d
         infra:
           maxAge: 7d
         audit:
           maxAge: 7d
       elasticsearch:
         nodeCount: 3
         storage:
           storageClassName: "<storage-class-name>"
           size: 200G
         resources:
           requests:
             memory: "8Gi"
         proxy:
           resources:
             limits:
               memory: 256Mi
             requests:
                memory: 256Mi
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana" 
       kibana:
         replicas: 1
     curation:
       type: "curator"
       curator:
         schedule: "30 3 * * *"
     collection:
       logs:
         type: "fluentd" 
         fluentd:
           tolerations:
           - effect: NoSchedule
             operator: Exists
           - key: CriticalAddonsOnly
             operator: Exists
           - effect: NoExecute
             operator: Exists

4. Apply the cluster logging configuration. Use the following command.

   oc apply -f  clusterLogging.yaml

5. Set up log forwarding.

   Syslog forwarding allows you to send a copy of your logs externally, instead of, or in addition to, the default Elasticsearch log store (or log management tool). This allows steaming of structured logs to both be indexed and available through a Kibana (or any data visualization dashboard software) instance, as well as written to flat files to provide to care for troubleshooting.

   Important

   Writing logs to ElasticSearch index or any log management tool is optional. However, storing of structured logs to flat files is required to support Genesys products. Though Rsyslog server is recommended, you can choose your desired method for collection/aggregation of structured syslog log messages.

   Prerequisites

   • Fluentd deployed in cluster for log collection
   • rsyslogd server installed and configured to have listening connection on TCP port
   • rsyslog rules defined to capture syslog messages and write them to a log file
   • rsyslog ruleset defined for writing container messages to a log file which NFS/NAS can access

   Forwarding logs using the syslog protocol

   Create a ClusterLogForwarder custom resource (CR) with one or more outputs to the syslog servers and pipelines that use those outputs. Here is a sample code to build your own YAML file.

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputs:
      - name: rsyslog-east
        type: syslog
        syslog:
          facility: local0
          rfc: RFC3164
          payloadKey: message
          severity: informational
        url: 'tls://rsyslogserver.east.example.com:514'
        secret:
           name: syslog-secret
      - name: rsyslog-west
        type: syslog
        syslog:
         appName: myapp
         facility: user
         msgID: mymsg
         procID: myproc
         rfc: RFC5424
         severity: debug
        url: 'udp://rsyslogserver.west.example.com:514'
     pipelines:
      - name: syslog-east
        inputRefs:
        - audit
        - application
        outputRefs:
        - rsyslog-east
        - default
        labels:
          syslog: east
          secure: true
      - name: syslog-west
        inputRefs:
        - infrastructure
        outputRefs:
        - rsyslog-west
        - default
        labels:
          syslog: west

   • metadata.name: instance
   • metadata.namespace: openshift-logging
   • spec.outputs.name: A unique name name for the output
   • spec.outputs.type: Syslog
   • spec.outputs.syslot: (Optional) specifies the syslog parameters listed below.
   • spec.outputs.url: $protocol://<ip/host>:<port> (protocol = UDP/TCP/TLS, host and port of rsyslogd server)

   Important

   If you are using a TLS prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   • spec.pipelines.inputRefs: log types should be forwarded using that pipeline: application, infrastructure, or audit.
   • spec.pipelines.outputRefs: output to use with that pipeline for forwarding the logs

   For more details on syslog and rsyslog, refer to:

   • https://datatracker.ietf.org/doc/html/rfc5424
   • https://www.rsyslog.com

   Apply the syslog cluster forwarder config using the following command.

   oc apply -f syslog-configmap.yaml

   After creating the cluster log forwarder, Fluentd instances should restart one by one picking up new configuration.

   Important

   If Fluentd pods do not restart on their own, delete them so that they are recreated.

   Forwarding logs using the legacy syslog method

   Create a configuration file called syslog.yaml, with the information needed to forward the logs in the openshift-logging namespace.

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: syslog
     namespace: openshift-logging
   data:
     syslog.conf: |
       <store>
         @type syslog_buffered
         remote_syslog 192.168.45.36
         port 51400
         hostname ${hostname}
         facility local0
         severity info
         use_record true
         payload_key message
       </store>

   Where:

   • @type refers tp the transport type (syslog_buffered is required for rsyslog server to write to file)
   • remote_syslog: host/ip of remote syslog server
   • port: rsyslog server TCP listening port
   • hostname: name of host sending syslog msg
   • facility: machine process that created the syslog event
   • severity: predefined severity level of event

6. Apply syslog configMap.

   oc apply -f syslog.yaml

   After creating the cluster log forwarder, Fluentd instances should restart one by one picking up new configuration.

   Important

   If Fluentd pods do not restart on their own, delete them so that they are recreated.

GKE logging

Click here for details about GKE logging.