All Products
Search
Document Center

Probe examples

Last Updated: Aug 15, 2019

This topic describes how to configure the container liveness and readiness probes.

Kubelets use liveness probes to determine when to restart containers. For example, when an application is running but cannot perform other operations in a container, a deadlock can be detected by liveness probes. In this case, the kubelet restarts the container so that the application can continue to run, even with bugs.

Kubelets use readiness probes to determine whether containers are ready to accept traffic. The kubelet of a pod considers the pod ready only when all containers in the pod are ready. Readiness probes determine which pods run at the service backend. Non-ready pods are deleted from the load balancer of the service.

1. Liveness probe

Many applications enter the broken state after running for a long time, and cannot resume unless they are restarted. Kubernetes provides a liveness probe mechanism to detect and rectify this problem.

The following example uses a BusyBox image to create a container group that contains one container.

  1. # -*- coding: utf-8 -*-
  2. #! /usr/bin/python
  3. import httplib
  4. import json
  5. import urllib
  6. import uuid
  7. import time
  8. import hmac
  9. from hashlib import sha1
  10. import base64
  11. HOST = 'eci.aliyuncs.com'
  12. PORT = 80
  13. AK_ID = 'xxx'
  14. AK_SECRET = 'xxx'
  15. def generateParams(param):
  16. parameters = {}
  17. for k in param:
  18. parameters[k] = param[k]
  19. def call(params):
  20. timestamp = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
  21. parameters = {
  22. 'Format': 'json',
  23. 'Version': '2018-08-08',
  24. 'Product': 'Eci',
  25. 'AccessKeyId': AK_ID,
  26. 'SignatureVersion': '1.0',
  27. 'SignatureMethod': 'HMAC-SHA1',
  28. 'SignatureNonce': str(uuid.uuid1()),
  29. 'Timestamp': timestamp,
  30. }
  31. for k in params:
  32. parameters[k] = params[k]
  33. signature = sign(AK_SECRET, parameters)
  34. parameters['Signature'] = signature
  35. url = "/?" + urllib.urlencode(parameters)
  36. connection = httplib.HTTPConnection(HOST, PORT, 30)
  37. connection.connect()
  38. header = {
  39. 'Content-type': 'application/x-www-form-urlencoded',
  40. "Cache-Control": "no-cache",
  41. "Connection": "Keep-Alive",
  42. }
  43. connection.request('POST', url, body='', headers=header)
  44. response = connection.getresponse()
  45. result = response.read()
  46. if response.status == 200:
  47. return result
  48. else:
  49. print response.status
  50. print result
  51. exit(-1)
  52. def sign(accessKeySecret, parameters):
  53. sortedParameters = sorted(parameters.items(), key=lambda parameters: parameters[0])
  54. canonicalizedQueryString = ''
  55. for (k, v) in sortedParameters:
  56. canonicalizedQueryString += '&' + percent_encode(k) + '=' + percent_encode(v)
  57. stringToSign = 'POST&%2F&' + percent_encode(canonicalizedQueryString[1:])
  58. h = hmac.new(accessKeySecret + "&", stringToSign, sha1)
  59. signature = base64.encodestring(h.digest()).strip()
  60. return signature
  61. def percent_encode(encodeStr):
  62. encodeStr = str(encodeStr)
  63. res = urllib.quote(encodeStr.decode('UTF-8').encode('utf8'), '')
  64. res = res.replace('+', '%20')
  65. res = res.replace('*', '%2A')
  66. res = res.replace('%7E', '~')
  67. return res
  68. def CreateContainerGroup():
  69. params = {
  70. 'Action': 'CreateContainerGroup',
  71. 'RegionId': 'cn-hangzhou',
  72. 'ZoneId': 'cn-hangzhou-g',
  73. 'SecurityGroupId': 'sg-xxx',
  74. 'VSwitchId': 'vsw-xxx',
  75. #'EipInstanceId':'eip-bp150letqu75gqbhy2cq9',
  76. 'ContainerGroupName':'test-container-group',
  77. 'RestartPolicy':'Always',
  78. 'Container.1.Image': 'registry-vpc.cn-hangzhou.aliyuncs.com/google_containers/busybox:latest',
  79. 'Container.1.Name': 'busybox',
  80. 'Container.1.Cpu': 0.25,
  81. 'Container.1.Memory': 0.5,
  82. 'Container.1.ImagePullPolicy': 'Always',
  83. 'Container.1.Command.1': '/bin/sh',
  84. 'Container.1.Arg.1': '-c',
  85. 'Container.1.Arg.2': 'touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600',
  86. 'Container.1.LivenessProbe.Exec.Command.1': 'cat',
  87. 'Container.1.LivenessProbe.Exec.Command.2': '/tmp/healthy',
  88. 'Container.1.LivenessProbe.InitialDelaySeconds': 5,
  89. 'Container.1.LivenessProbe.PeriodSeconds': 5,
  90. #'Container.1.LivenessProbe.FailureThreshold': 3,
  91. #'Container.1.LivenessProbe.SuccessThreshold': 1,
  92. #'Container.1.LivenessProbe.TimeoutSeconds': 5,
  93. }
  94. ret = call(params)
  95. print ret
  96. def DescribeContainerGroup(containerGroupId):
  97. params = {
  98. 'Action': 'DescribeContainerGroups',
  99. 'RegionId': 'cn-hangzhou',
  100. 'ContainerGroupIds': json.dumps(containerGroupId),
  101. 'Status':'running',
  102. }
  103. ret = call(params)
  104. #print ret
  105. return ret
  106. if __name__ == '__main__':
  107. start = time.time()
  108. CreateContainerGroup()
  109. #eci_name = 'eci-bp15rfpp55k1guf7ggjt'
  110. #print json.dumps(json.loads(DescribeContainerGroup([eci_name])), sort_keys=True, indent=4, separators=(', ', ': '))
  111. elapsed = (time.time() - start)
  112. print "Time used: %f" % elapsed

The preceding example configures one container for the container group. LivenessProbe.PeriodSeconds specifies that the kubelet should run a liveness probe every 5 seconds. LivenessProbe.InitialDelaySeconds specifies that the kubelet should wait for 5 seconds before running the probe for the first time. The cat /tmp/healthy command is executed in the container to run probes. If the command is executed successfully, 0 is returned. In this case, the kubelet considers that the container is active and healthy. If a non-0 value is returned, the kubelet kills the container and restarts it.

The following command is executed during the startup of the container:

  1. /bin/sh -c "touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600"

A /tmp/healthy file is generated in the first 30 seconds of the container’s lifecycle. Within the 30 seconds, running the cat /tmp/healthy command returns a success code. After the 30 seconds, running cat /tmp/healthy returns a failure code.

When you view the events of the pod within the first 30 seconds, no failed liveness probe is displayed.

View the events of the pod again 35 seconds later.

Snapshot 2018-12-04 5.36.06 PM.png | center | 827x300

You can also see three warnings, because the default value of Container.1.LivenessProbe. FailureThreshold is 3.

2. Readiness probe

Sometimes, an application cannot process external traffic in a short period. For example, the application needs to load a large amount of data or configurations during startup. In this case, you do not want to kill the application or send requests to it. Kubernetes provides a readiness probe mechanism to detect and alleviate the problem. Containers in a pod can report their non-ready states, in which case they cannot process traffic from Kubernetes.

The readiness probe configuration is the same as the liveness probe configuration, except that the command is readinessProbe instead of livenessProbe.

The HTTP and TCP probe settings for readiness probes are the same as those for liveness probes.

Readiness and liveness probes can be used in the same container. They prevent traffic from reaching non-ready containers and enable containers to restart upon failures.

Reference:

https://k8smeetup.github.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/