Skip to content

Consolidated Logging

What is Consolidated Logging?💣

Breaking down complex systems into smaller parts increases the number of individual components generating log data. These logs are in multiple formats and spread across multiple servers. Centralized logging involves collecting logs from networks, infrastructure, and applications and storing them in one location for analysis. This feature lets administrators quickly identify and troubleshoot network issues by providing a consolidated view of all activity.

Consolidated Logging basics💣

  • A typical logging stack is made of 3 pieces:

    • Collection/Shipper: Consumes log outputs and sends them to the consolidated logging framework.
    • Storage: Stores the shipped logs.
    • Visualization Tool: Allows users to search and view log data.

  • PlatformOne uses:

    • Fluentbit for log collection, transformation, and shipping,
    • ElasticSearch for log storage, and
    • Kibana for visualization.

The diagram below shows how these work together. The diagram shows FluentD (another popular log collector), but the architecture is the same.

EFK Stack

Fluent Bit💣

  • Deployed as a Daemonset in the cluster. Runs on both worker and master nodes.
  • Pushes logs to ES/ECK (Elastic Cloud on Kubernetes).
  • Big Bang configures ElasticSearch with a logstash index that includes all Kubernetes pod logs.

Elastic Cloud on Kubernetes💣

Elastic Cloud on Kubernetes (ECK) helps make Day 2 operations easier with ElasticSearch.

Features include:

  • Deploying and managing multiple Elasticsearch clusters, including Kibana.
  • Simple scaling that allows you to grow with your cluster.
  • Default security on every cluster.

Day 1 Operations 1. Deploy the Elastic-Operator. 2. Deploy an ElasticSearch cluster. 3. Deploy a Kibana instance.

Day 2 Operations * Upgrading to newer versions. * Scaling up to handle increased load.

Installation💣

ECK is deployed using the Operator pattern:

  • ECK simplifies installing ElasticSearch by using custom resource definitions.
  • The example below is very simple, but note that the prodcution operator for Big Bang describes an Elastic Cluster declaratively in 189 lines.
  • You can scale the cluster by simply increasing the count.
kubectl get elasticsearch logging-ek -n=logging -o yaml | wc -l

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.8.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node. ingest: true
      node.store.allow mmap: false

Kibana💣

Kibana

  • UI for searching through indexed logs.
  • Kibana is critical due to the short-lived nature of Kubernetes pods.
    • Historical logs can be difficult to obtain from kubectl.
  • First time login into Kibana will require a configuration of index patterns.
    • An Index is created daily for Kubernetes logs and open policy agent gatekeeper logs. (This helps with rotating, compressing, exporting, and deleting older logs to free up space.)
    • Index Pattern will look like logstash- or violations-

Kibana Index Pattern