云原生之K8S中Pod健康检测服务可用性检查详解

　　写在前面本文内容涉及： LivenessProbe  ,ReadinessProbe  两种探针  的一些基本理论ExecAction  ,TCPSocketAction  ,HTTPGetAction  三种检测方式  的Demo
　　中秋明月，豪门有，贫家也有。极慰人心。 ——烽火戏诸侯《剑来》 Pod健康检查和服务可用性检查健康检查的目的
　　探测的目的  : 用来维持 pod的健壮性,当pod挂掉之后，deployment会生成新的pod,但如果pod是正常运行的，但pod里面出了问题，此时deployment是监测不到的。故此需要探测(probe)-pod是不是正常提供服务的探针类似
　　Kubernetes   对 Pod   的健康状态可以通过两类探针来检查:LivenessProbe  和ReadinessProbe  , kubelet定期执行这两类探针来诊断容器的健康状况。都是通过deployment实现的
　　探针类型
　　描述
　　LivenessProbe探针
　　用于判断容器是否存活(Running状态) ,如果LivenessProbe探针探测到容器不健康,则kubelet将杀掉该容器,并根据容器的重启策略做相应的处理。如果一个容器不包含LivenesspProbe探针,那么kubelet认为该容器的LivenessProbe探针返回的值永远是Success。
　　ReadinessProbe探针
　　用于判断容器服务是否可用(Ready状态) ,达到Ready状态的Pod才可以接收请求。对于被Service管理的Pod, Service与Pod Endpoint的关联关系也将基于Pod是否Ready进行设置。如果在运行过程中Ready状态变为False,则系统自动将其从Service的后端Endpoint列表中隔离出去,后续再把恢复到Ready状态的Pod加回后端Endpoint列表。这样就能保证客户端在访问Service时不会被转发到服务不可用的Pod实例上。  检测方式及参数配置
　　LivenessProbe  和ReadinessProbe  均可配置以下三种实现方式。
　　方式
　　描述
　　ExecAction
　　在容器内部执行一个命令,如果该命令的返回码为0,则表明容器健康。
　　TCPSocketAction
　　通过容器的IP地址和端口号执行TC检查,如果能够建立TCP连接,则表明容器健康。
　　HTTPGetAction
　　通过容器的IP地址、端口号及路径调用HTTP Get方法,如果响应的状态码大于等于200且小于400,则认为容器健康。
　　对于每种探测方式，需要设置 initialDelaySeconds  和timeoutSeconds  等参数，它们的含义分别如下。
　　参数
　　描述
　　initialDelaySeconds：
　　启动容器后进行首次健康检查的等待时间，单位为s。
　　timeoutSeconds:
　　健康检查发送请求后等待响应的超时时间,单位为s。当超时发生时, kubelet会认为容器已经无法提供服务,将会重启该容器。
　　periodSeconds
　　执行探测的频率，默认是10秒，最小1秒。
　　successThreshold
　　探测失败后，最少连续探测成功多少次才被认定为成功，默认是1，对于liveness必须是1，最小值是1。
　　failureThreshold
　　当 Pod 启动了并且探测到失败，Kubernetes 的重试次数。存活探测情况下的放弃就意味着重新启动容器。就绪探测情况下的放弃 Pod 会被打上未就绪的标签。默认值是 3。最小值是 1
　　Kubernetes的ReadinessProbe机制可能无法满足某些复杂应用对容器内服务可用状态的判断
　　所以Kubernetes从1.11版本开始,引入PodReady++特性对Readiness探测机制进行扩展,在1.14版本时达到GA稳定版,称其为Pod Readiness Gates。
　　通过Pod Readiness Gates机制,用户可以将自定义的ReadinessProbe探测方式设置在Pod上,辅助Kubernetes设置Pod何时达到服务可用状态(Ready) 。为了使自定义的ReadinessProbe生效,用户需要提供一个外部的控制器(Controller)来设置相应的Condition状态。
　　Pod的Readiness Gates在Pod定义中的ReadinessGate字段进行设置。下面的例子设置了一个类型为www.example.com/feature-1的新ReadinessGate:
　　–
　　新增的自定义Condition的状态(status)将由用户自定义的外部控·制器设置,默认值为False. Kubernetes将在判断全部readinessGates条件都为True时,才设置Pod为服务可用状态(Ready为True) 。
　　这个不是太懂，需要以后再研究下 学习环境准备 [root@vms81.liruilongs.github.io]-[~/ansible]  $mkdir liveness-probe  [root@vms81.liruilongs.github.io]-[~/ansible]  $cd liveness-probe/  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  create ns liveness-probe namespace/liveness-probe created  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl config current-context kubernetes-admin@kubernetes  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl config set-context $(kubectl config current-context) --namespace=liveness-probe Context ＂kubernetes-admin@kubernetes＂ modified. LivenessProbe探针
　　用于判断容器是否存活(Running状态) ,如果LivenessProbe探针探测到容器不健康,则kubelet将杀掉该容器,并根据容器的重启策略做相应的处理 ExecAction方式：command
　　在容器内部执行一个命令,如果该命令的返回码为0,则表明容器健康。
　　资源文件定义  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $cat liveness-probe.yaml apiVersion: v1 kind: Pod metadata:   creationTimestamp: null   labels:     run: pod-liveness   name: pod-liveness spec:   containers:   - args:     - /bin/sh     - -c     - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; slee 10     livenessProbe:       exec:         command:         - cat         - /tmp/healthy       initialDelaySeconds: 5 #容器启动的5s内不监测       periodSeconds: 5 #每5s钟检测一次     image: busybox     imagePullPolicy: IfNotPresent     name: pod-liveness     resources: {}   dnsPolicy: ClusterFirst   restartPolicy: Always  status: {}
　　运行这个deploy。当pod创建成功后，新建文件，并睡眠30s，删掉文件在睡眠。使用liveness检测文件的存在  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  apply  -f liveness-probe.yaml pod/pod-liveness created  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  get pods NAME           READY   STATUS    RESTARTS     AGE pod-liveness   1/1     Running   1 (8s ago)   41s   # 30文件没有重启
　　运行超过30s后。文件被删除，所以被健康检测命中，pod根据重启策略重启  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  get pods NAME           READY   STATUS    RESTARTS      AGE pod-liveness   1/1     Running   2 (34s ago)   99s
　　99s后已经从起了第二次  [root@vms81.liruilongs.github.io]-[~/ansible]  $ansible 192.168.26.83 -m shell -a ＂docker ps | grep pod-liveness＂ 192.168.26.83 | CHANGED | rc=0 >> 00f4182c014e   7138284460ff                                        ＂/bin/sh -c ＂touch /…＂   6 seconds ago   Up 5 seconds             k8s_pod-liveness_pod-liveness_liveness-probe_81b4b086-fb28-4657-93d0-bd23e67f980a_0 01c5cfa02d8c   registry.aliyuncs.com/google_containers/pause:3.5   ＂/pause＂                 7 seconds ago   Up 6 seconds             k8s_POD_pod-liveness_liveness-probe_81b4b086-fb28-4657-93d0-bd23e67f980a_0  [root@vms81.liruilongs.github.io]-[~/ansible]  $kubectl get pods NAME           READY   STATUS    RESTARTS   AGE pod-liveness   1/1     Running   0          25s  [root@vms81.liruilongs.github.io]-[~/ansible]  $kubectl get pods NAME           READY   STATUS    RESTARTS      AGE pod-liveness   1/1     Running   1 (12s ago)   44s  [root@vms81.liruilongs.github.io]-[~/ansible]  $ansible 192.168.26.83 -m shell -a ＂docker ps | grep pod-liveness＂ 192.168.26.83 | CHANGED | rc=0 >> 1eafd7e8a12a   7138284460ff                                        ＂/bin/sh -c ＂touch /…＂   15 seconds ago   Up 14 seconds             k8s_pod-liveness_pod-liveness_liveness-probe_81b4b086-fb28-4657-93d0-bd23e67f980a_1 01c5cfa02d8c   registry.aliyuncs.com/google_containers/pause:3.5   ＂/pause＂                 47 seconds ago   Up 47 seconds             k8s_POD_pod-liveness_liveness-probe_81b4b086-fb28-4657-93d0-bd23e67f980a_0  [root@vms81.liruilongs.github.io]-[~/ansible]  $
　　查看节点机docker中的容器ID，前后不一样，确定是POD被杀掉后重启。 HTTPGetAction的方式
　　通过容器的IP地址、端口号及路径调用HTTP Get方法,如果响应的状态码大于等于200且小于400,则认为容器健康。
　　创建资源文件，即相关参数使用  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $cat liveness-probe-http.yaml apiVersion: v1 kind: Pod metadata:   creationTimestamp: null   labels:     run: pod-livenss-probe   name: pod-livenss-probe spec:   containers:   - image: nginx     imagePullPolicy: IfNotPresent     name: pod-livenss-probe     livenessProbe:       failureThreshold: 3 #当 Pod 启动了并且探测到失败，Kubernetes 的重试次数       httpGet:         path: /index.html         port: 80         scheme: HTTP       initialDelaySeconds: 10  #容器启动后第一次执行探测是需要等待多少秒       periodSeconds: 10   #执行探测的频率，默认是10秒，最小1秒       successThreshold: 1 #探测失败后，最少连续探测成功多少次才被认定为成功       timeoutSeconds: 10 #探测超时时间，默认1秒，最小1秒     resources: {}   dnsPolicy: ClusterFirst   restartPolicy: Always status: {}
　　运行deploy，这个的探测机制访问Ngixn的默认欢迎页  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $vim liveness-probe-http.yaml  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl apply  -f liveness-probe-http.yaml pod/pod-livenss-probe created  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  get pods NAME                READY   STATUS    RESTARTS   AGE pod-livenss-probe   1/1     Running   0          15s  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl exec -it pod-livenss-probe -- rm /usr/share/nginx/html/index.html  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  get pods NAME                READY   STATUS    RESTARTS     AGE pod-livenss-probe   1/1     Running   1 (1s ago)   2m31s
　　当欢迎页被删除时，访问报错，被检测命中,pod重启 TCPSocketAction方式
　　通过容器的IP地址和端口号执行TCP检查,如果能够建立TCP连接,则表明容器健康。
　　资源文件定义 [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $cat liveness-probe-tcp.yaml apiVersion: v1 kind: Pod metadata:   creationTimestamp: null   labels:     run: pod-livenss-probe   name: pod-livenss-probe spec:   containers:   - image: nginx     imagePullPolicy: IfNotPresent     name: pod-livenss-probe     livenessProbe:       failureThreshold: 3       tcpSocket:         port: 8080       initialDelaySeconds: 10       periodSeconds: 10       successThreshold: 1       timeoutSeconds: 10     resources: {}   dnsPolicy: ClusterFirst   restartPolicy: Always status: {}
　　访问8080端口，但是8080端口未开放，所以访问会超时，不能建立连接，命中检测，重启Pod  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  apply  -f liveness-probe-tcp.yaml pod/pod-livenss-probe created  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  get pods NAME                READY   STATUS    RESTARTS   AGE pod-livenss-probe   1/1     Running   0          8s  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  get pods NAME                READY   STATUS    RESTARTS     AGE pod-livenss-probe   1/1     Running   1 (4s ago)   44s  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $ ReadinessProbe探针
　　用于判断容器服务是否可用(Ready状态) ,达到Ready状态的Pod才可以接收请求。负责不能进行访问 ExecAction方式：command
　　资源文件定义，使用钩子建好需要检查的文件  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $cat readiness-probe.yaml apiVersion: v1 kind: Pod metadata:   creationTimestamp: null   labels:     run: pod-liveness   name: pod-liveness spec:   containers:   - readinessProbe:       exec:         command:         - cat         - /tmp/healthy       initialDelaySeconds: 5 #容器启动的5s内不监测       periodSeconds: 5 #每5s钟检测一次     image: nginx     imagePullPolicy: IfNotPresent     name: pod-liveness     resources: {}     lifecycle:       postStart:         exec:           command: [＂/bin/sh＂, ＂-c＂,＂touch /tmp/healthy＂]   dnsPolicy: ClusterFirst   restartPolicy: Always status: {}
　　创建3个有Ngixn的pod,通过POD创建一个SVC做测试用  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $sed ＂s/pod-liveness/pod-liveness-1/＂ readiness-probe.yaml | kubectl apply  -f - pod/pod-liveness-1 created  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $sed ＂s/pod-liveness/pod-liveness-2/＂ readiness-probe.yaml | kubectl apply  -f - pod/pod-liveness-2 created  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  get pods -o wide NAME             READY   STATUS    RESTARTS   AGE    IP             NODE                         NOMINATED NODE   READINESS GATES pod-liveness     1/1     Running   0          3m1s   10.244.70.50   vms83.liruilongs.github.io               pod-liveness-1   1/1     Running   0          2m     10.244.70.51   vms83.liruilongs.github.io               pod-liveness-2   1/1     Running   0          111s   10.244.70.52   vms83.liruilongs.github.io              
　　修改主页文字  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $serve=pod-liveness  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl exec  -it $serve -- sh -c ＂echo $serve > /usr/share/nginx/html/index.html＂  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl exec  -it $serve -- sh -c ＂cat /usr/share/nginx/html/index.html＂ pod-liveness  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $serve=pod-liveness-1  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl exec  -it $serve -- sh -c ＂echo $serve > /usr/share/nginx/html/index.html＂  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $serve=pod-liveness-2  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl exec  -it $serve -- sh -c ＂echo $serve > /usr/share/nginx/html/index.html＂  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $
　　修改标签  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl get pods --show-labels NAME             READY   STATUS    RESTARTS   AGE   LABELS pod-liveness     1/1     Running   0          15m   run=pod-liveness pod-liveness-1   1/1     Running   0          14m   run=pod-liveness-1 pod-liveness-2   1/1     Running   0          14m   run=pod-liveness-2  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl edit pods pod-liveness-1 pod/pod-liveness-1 edited  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl edit pods pod-liveness-2 pod/pod-liveness-2 edited  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl get pods --show-labels NAME             READY   STATUS    RESTARTS   AGE   LABELS pod-liveness     1/1     Running   0          17m   run=pod-liveness pod-liveness-1   1/1     Running   0          16m   run=pod-liveness pod-liveness-2   1/1     Running   0          16m   run=pod-liveness  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $
　　要删除文件检测  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  exec -it pod-liveness -- ls /tmp/ healthy  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  exec -it pod-liveness-1 -- ls /tmp/ healthy
　　使用POD创建SVC  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl expose  --name=svc pod pod-liveness --port=80 service/svc exposed  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl  get ep NAME   ENDPOINTS                                         AGE svc    10.244.70.50:80,10.244.70.51:80,10.244.70.52:80   16s  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl get svc NAME   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE svc    ClusterIP   10.104.246.121           80/TCP    36s  [root@vms81.liruilongs.github.io]-[~/ansible/liveness-probe]  $kubectl get pods -o wide NAME             READY   STATUS    RESTARTS   AGE   IP             NODE                         NOMINATED NODE   READINESS GATES pod-liveness     1/1     Running   0          24m   10.244.70.50   vms83.liruilongs.github.io               pod-liveness-1   1/1     Running   0          23m   10.244.70.51   vms83.liruilongs.github.io               pod-liveness-2   1/1     Running   0          23m   10.244.70.52   vms83.liruilongs.github.io              
　　测试SVC正常，三个POD会正常 负载  [root@vms81.liruilongs.github.io]-[~/ansible]  $while true; do curl 10.104.246.121 ; sleep 1 > done pod-liveness pod-liveness-2 pod-liveness pod-liveness-1 pod-liveness-2 ^C
　　删除文件测试  [root@vms81.liruilongs.github.io]-[~/ansible]  $kubectl exec -it pod-liveness -- rm  -rf /tmp/  [root@vms81.liruilongs.github.io]-[~/ansible]  $kubectl exec -it pod-liveness -- ls /tmp/ ls: cannot access ＂/tmp/＂: No such file or directory command terminated with exit code 2  [root@vms81.liruilongs.github.io]-[~/ansible]  $while true; do curl 10.104.246.121 ; sleep 1; done pod-liveness-2 pod-liveness-2 pod-liveness-2 pod-liveness-1 pod-liveness-2 pod-liveness-2 pod-liveness-1 ^C
　　会发现pod-liveness的pod已经不提供服务了 kubeadm 中的一些健康检测
　　kube-apiserver.yaml中的使用，两种探针同时使用  [root@vms81.liruilongs.github.io]-[~/ansible]  $cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep -A 8 readi     readinessProbe:       failureThreshold: 3       httpGet:         host: 192.168.26.81         path: /readyz         port: 6443         scheme: HTTPS       periodSeconds: 1       timeoutSeconds: 15  [root@vms81.liruilongs.github.io]-[~/ansible]  $cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep -A 9 liveness     livenessProbe:       failureThreshold: 8       httpGet:         host: 192.168.26.81         path: /livez         port: 6443         scheme: HTTPS       initialDelaySeconds: 10       periodSeconds: 10       timeoutSeconds: 15  [root@vms81.liruilongs.github.io]-[~/ansible]  $