Failure of systemctl state probe in Kubernetes pod

In Heketi's glusterd container service, the system CTL probe is used to detect whether the glusterfs service is available. It is found that there is always a failure problem.

It is found that the output information of systemctl status glusterd.service runtime on Ubuntu 18.04 is not what K8s livenessProbe wants, causing the detector to timeout and hang.

  • Using systemctl status glusterd.service can not detect the real state of the service, it will suspend, timeout and return error status code.

Use the following methods to correctly detect the real state of the service:

systemctl is-active --quiet glusterd.service; echo $?; 

Or (similar to):

systemctl is-active sshd >/dev/null 2>&1 && echo 0 || echo 1

Output:

  • Normal 0;
  • It is an error code when it is abnormal.
  • As follows:
        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl is-active --quiet glusterd.service; echo $?;
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl is-active --quiet glusterd.service; echo $?;

The modified k8s yaml file is as follows:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: glusterfs-daemon
  namespace: gluster
  labels:
    k8s-app: glusterfs-node
spec:
  selector:
    matchLabels:
      name: glusterfs-daemon
  template:
    metadata:
      labels:
        name: glusterfs-daemon
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - image: gluster/gluster-centos:latest
        imagePullPolicy: IfNotPresent
        name: glusterfs
        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl is-active --quiet glusterd.service; echo $?;
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl is-active --quiet glusterd.service; echo $?;
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        resources: {}
        securityContext:
          capabilities: {}
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/heketi
          name: glusterfs-heketi
        - mountPath: /run
          name: glusterfs-run
        - mountPath: /run/lvm
          name: glusterfs-lvm
        - mountPath: /etc/glusterfs
          name: glusterfs-etc
        - mountPath: /var/log/glusterfs
          name: glusterfs-logs
        - mountPath: /var/lib/glusterd
          name: glusterfs-config
        - mountPath: /dev
          name: glusterfs-dev
        - mountPath: /sys/fs/cgroup
          name: glusterfs-cgroup
      dnsPolicy: ClusterFirst
      hostNetwork: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/lib/heketi
          type: ""
        name: glusterfs-heketi
      - emptyDir: {}
        name: glusterfs-run
      - hostPath:
          path: /run/lvm
          type: ""
        name: glusterfs-lvm
      - hostPath:
          path: /etc/glusterfs
          type: ""
        name: glusterfs-etc
      - hostPath:
          path: /var/log/glusterfs
          type: ""
        name: glusterfs-logs
      - hostPath:
          path: /var/lib/glusterd
          type: ""
        name: glusterfs-config
      - hostPath:
          path: /dev
          type: ""
        name: glusterfs-dev
      - hostPath:
          path: /sys/fs/cgroup
          type: ""
        name: glusterfs-cgroup

Maybe on different versions of Linux, different versions of systemd, and different parameters. Enter systemctl help to get help from the current version.

Tags: Ubuntu Kubernetes CentOS Linux

Posted on Tue, 07 Jan 2020 07:48:14 -0800 by everknown