EFK Log Collection System in k8s Cluster

The Kubernetes cluster itself does not provide a solution for log collection. Generally speaking, there are three main solutions for log collection:
1. Run an agent on each node to collect logs
Since this agent must run on each node, it is sufficient to run the application directly using the DaemonSet controller.
This method is also only applicable to collecting application logs exported to stdout and stderr
Simply put, this approach is to run a log proxy container on each node.
Collection of logs in this node / var/log and / var/lib/docker/containers / directories
2. Include a sidecar container in each Pod to collect application logs
Running log collection agents in sidecar containers can lead to a lot of resource consumption, because you need to run as many collection agents as you have as many Pod s to collect, and you can't use the kubectl logs command to access these logs.
3. Push the log information directly to the back-end of the collection in the application program.

The popular log collection solutions in Kubernetes are Elastic search, Fluentd and Kibana (EFK) technology stacks, which are now recommended by the authorities.
Elastic search is a real-time, distributed and extensible search engine that allows full-text, structured searches. It is usually used to index and search a large number of log data, as well as to search many different types of documents.

Create Elastic search cluster
Three Elastic search Pods are commonly used to avoid "brain fissure" problems in multi-node clusters with high availability, and StatefulSet controllers are used to create Elastic search Pods.
When creating StatefulSet pod, StorageClass is used directly in its PVC template to automatically generate pv and pvc, which can achieve data persistence. nfs-client-provider is ready in advance.
1. Create a separate namespace

apiVersion: v1
kind: Namespace
metadata:
  name: logging

2. Create Storage Clas, or use existing Storage Clas

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: es-data-db
provisioner: fuseim.pri/ifs      # This value needs to be consistent with the provisioner configuration

3. Before creating StatefulSet pod, you need to create headless services

kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: logging
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

4. Create elastic search stateful set pod
$ docker pull docker.elastic.co/elasticsearch/elasticsearch-oss:6.4.3
$ docker pull busybox

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.io/elasticsearch:latest
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.zen.ping.unicast.hosts
            value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
          - name: discovery.zen.minimum_master_nodes
            value: "2"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: es-data-db
      resources:
        requests:
          storage: 100Gi

$ kubectl get pod -n logging
NAME READY STATUS RESTARTS AGE
es-cluster-0 1/1 Running 0 42s
es-cluster-1 1/1 Running 0 10m
es-cluster-2 1/1 Running 0 9m49s
Three directories are automatically generated on nfs servers for storing data in these three pod s
$ cd /data/k8s
$ ls
logging-data-es-cluster-0-pvc-98c87fc5-c581-11e9-964d-000c29d8512b/
logging-data-es-cluster-1-pvc-07872570-c590-11e9-964d-000c29d8512b/
logging-data-es-cluster-2-pvc-27e15977-c590-11e9-964d-000c29d8512b/
Check es cluster status
$ kubectl port-forward es-cluster-0 9200:9200 --namespace=logging
Execute in another window
$ curl http://localhost:9200/_cluster/state?pretty

Creating kibana with deployment controller

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: logging
  labels:
    app: kibana
spec:
  ports:
  - port: 5601
  type: NodePort
  selector:
    app: kibana

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: logging
  labels:
    app: kibana
spec:
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana-oss:6.4.3
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch:9200
        ports:
        - containerPort: 5601

$ kubectl get svc -n logging |grep kibana
kibana NodePort 10.111.239.0 <none> 5601:32081/TCP 114m
Visit kibana
http://192.168.1.243:32081

Installation Configuration Fluentd
1. Specify the Fluentd configuration file through the ConfigMap object

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluentd-config
  namespace: logging
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  system.conf: |-
    <system>
      root_dir /tmp/fluentd-buffers/
    </system>
  containers.input.conf: |-
    <source>
      @id fluentd-containers.log
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      localtime
      tag raw.kubernetes.*
      format json
      read_from_head true
    </source>
    <match raw.kubernetes.**>
      @id raw.kubernetes
      @type detect_exceptions
      remove_tag_prefix raw
      message log
      stream stream
      multiline_flush_interval 5
      max_bytes 500000
      max_lines 1000
    </match>
  system.input.conf: |-
    <source>
      @id journald-docker
      @type systemd
      filters [{ "_SYSTEMD_UNIT": "docker.service" }]
      <storage>
        @type local
        persistent true
      </storage>
      read_from_head true
      tag docker
    </source>
    <source>
      @id journald-kubelet
      @type systemd
      filters [{ "_SYSTEMD_UNIT": "kubelet.service" }]
      <storage>
        @type local
        persistent true
      </storage>
      read_from_head true
      tag kubelet
    </source>
  forward.input.conf: |-
    <source>
      @type forward
    </source>
  output.conf: |-
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>
    <match **>
      @id elasticsearch
      @type elasticsearch
      @log_level info
      include_tag_key true
      host elasticsearch
      port 9200
      logstash_format true
      request_timeout    30s
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 2M
        queue_limit_length 8
        overflow_action block
      </buffer>
    </match>

In the above configuration file, we configure the docker container log directory and the collection of the docker and kubelet application logs. The collected data is processed and sent to the elastic search:9200 service.
2. Using DaemonSet to create fluentd pod
$ docker pull cnych/fluentd-elasticsearch:v2.0.4
$ docker info
Docker Root Dir: /var/lib/docker

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-es
  namespace: logging
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "namespaces"
  - "pods"
  verbs:
  - "get"
  - "watch"
  - "list"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: fluentd-es
  namespace: logging
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: fluentd-es
  apiGroup: ""
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-es
  namespace: logging
  labels:
    k8s-app: fluentd-es
    version: v2.0.4
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-es
      version: v2.0.4
  template:
    metadata:
      labels:
        k8s-app: fluentd-es
        kubernetes.io/cluster-service: "true"
        version: v2.0.4
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: fluentd-es
      containers:
      - name: fluentd-es
        image: cnych/fluentd-elasticsearch:v2.0.4
        env:
        - name: FLUENTD_ARGS
          value: --no-supervisor -q
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config-volume
          mountPath: /etc/fluent/config.d
      nodeSelector:
        beta.kubernetes.io/fluentd-ds-ready: "true"
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config-volume
        configMap:
          name: fluentd-config

Logs in / var/log and / var/log/containers and / var/lib/docker/containers can be collected
You can also collect logs for docker services and kubelet services
In order to be able to flexibly control which nodes'logs can be collected, we also add a nodSelector attribute here.

nodeSelector:
  beta.kubernetes.io/fluentd-ds-ready: "true"

So label all nodes:
$ kubectl get node
$ kubectl label nodes server243.example.com beta.kubernetes.io/fluentd-ds-ready=true
$ kubectl get nodes --show-labels
Since our cluster is built with kubeadm and master node is stained by default, it is necessary to add tolerance if we want to collect master node's logs as well.

tolerations:
- key: node-role.kubernetes.io/master
  operator: Exists
  effect: NoSchedule

$ kubectl get pod -n logging
NAME READY STATUS RESTARTS AGE
es-cluster-0 1/1 Running 0 10h
es-cluster-1 1/1 Running 0 10h
es-cluster-2 1/1 Running 0 10h
fluentd-es-rf6p6 1/1 Running 0 9h
fluentd-es-s99r2 1/1 Running 0 9h
fluentd-es-snmtt 1/1 Running 0 9h
kibana-bd6f49775-qsxb2 1/1 Running 0 11h
3. Configuration on kibana
http://192.168.1.243:32081
Create index pattern -- First enter logstash - *, and second select @timestamp
4. Create test pod s and view logs on kibana

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args: [/bin/sh, -c,
            'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']

Go back to the Kibana Dashboard page and enter kubernetes.pod_name:counter in the Discover page search bar above.

Tags: Kubernetes ElasticSearch Docker kubelet

Posted on Sat, 24 Aug 2019 08:13:55 -0700 by UnsuitableBadger