Kubernetes pod中systemctl状态探针失败问题

在Heketi的glusterd容器服务，使用systemctl探针来检测glusterfs服务是否可用，发现总是出现失败问题。

经查，在Ubuntu 18.04 上 systemctl status glusterd.service 运行时输出信息不是K8s livenessProbe希望的，导致检测器超时挂起了。

• 使用systemctl status glusterd.service并不能检测到服务的真实状态，会挂起、超时，返回错误状态码。

使用下面的方式，可以正确检测service的真实状态： 

systemctl is-active --quiet glusterd.service; echo $?; 

或者（类似于）：

systemctl is-active sshd >/dev/null 2>&1 && echo 0 || echo 1

输出：

• 正常时 0；
• 非正常时为错误码。

• 如下所示：

        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl is-active --quiet glusterd.service; echo $?;
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl is-active --quiet glusterd.service; echo $?;

修改后的k8s yaml文件如下：

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: glusterfs-daemon
  namespace: gluster
  labels:
    k8s-app: glusterfs-node
spec:
  selector:
    matchLabels:
      name: glusterfs-daemon
  template:
    metadata:
      labels:
        name: glusterfs-daemon
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - image: gluster/gluster-centos:latest
        imagePullPolicy: IfNotPresent
        name: glusterfs
        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl is-active --quiet glusterd.service; echo $?;
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl is-active --quiet glusterd.service; echo $?;
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        resources: {}
        securityContext:
          capabilities: {}
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/heketi
          name: glusterfs-heketi
        - mountPath: /run
          name: glusterfs-run
        - mountPath: /run/lvm
          name: glusterfs-lvm
        - mountPath: /etc/glusterfs
          name: glusterfs-etc
        - mountPath: /var/log/glusterfs
          name: glusterfs-logs
        - mountPath: /var/lib/glusterd
          name: glusterfs-config
        - mountPath: /dev
          name: glusterfs-dev
        - mountPath: /sys/fs/cgroup
          name: glusterfs-cgroup
      dnsPolicy: ClusterFirst
      hostNetwork: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/lib/heketi
          type: ""
        name: glusterfs-heketi
      - emptyDir: {}
        name: glusterfs-run
      - hostPath:
          path: /run/lvm
          type: ""
        name: glusterfs-lvm
      - hostPath:
          path: /etc/glusterfs
          type: ""
        name: glusterfs-etc
      - hostPath:
          path: /var/log/glusterfs
          type: ""
        name: glusterfs-logs
      - hostPath:
          path: /var/lib/glusterd
          type: ""
        name: glusterfs-config
      - hostPath:
          path: /dev
          type: ""
        name: glusterfs-dev
      - hostPath:
          path: /sys/fs/cgroup
          type: ""
        name: glusterfs-cgroup

可能在不同的Linux版本上，systemd的版本不同，参数也可能不一样，输入systemctl help来获取当前版本的帮助。