Container Compute Service:Use core dumps to analyze program anomalies in an instance

How it works

In Linux, when a program terminates abnormally, the kernel records the state of the random access memory (RAM) allocated to that program and writes it to a file — a process called a core dump. The resulting file is called a core file.

The following figure shows the Linux signals that trigger a core dump. By default, signals whose action is Core generate core files. For details, see Core dump file.

Prerequisites

Before you begin, ensure that you have:

• kubectl installed and a kubeconfig file configured for your ACS cluster. For details, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

• (Optional) A NAS file system with a mount target, if you plan to store core files on NAS

• (Optional) An OSS bucket, if you plan to store core files on OSS (Object Storage Service)

You can also run the following kubectl commands in CloudShell without configuring a local kubeconfig file.

Enable core dumps

Core dumps are disabled by default for ACS pods. To enable them, add the alibabacloud.com/core-pattern annotation to your pod spec and set the path where core files are stored:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    alibabacloud.com/core-pattern: "/data/dump-a/core-%E-%p-%t"
...

The path value also sets the core file naming pattern. The supported format specifiers are:

For all supported specifiers, see Man page of core.

Choose a storage method

The storage method you choose determines how you access core files after a crash:

Mount a remote shared volume (NAS or OSS) to keep core files intact and prevent the pod's rootfs layer from filling up, which would cause CrashLoopBackOff events.

Mount a remotely shared volume

Use an ephemeral container to access core files

Use this approach when excessive core files are generated or when you need to debug a specific pod crash without a remote shared volume. After a core dump event occurs, the core file is saved to the rootfs layer. You can then log in to an injected ephemeral container to analyze the core file. The core file is stored in an emptyDir volume mounted to both the application container and the ephemeral container.

1. Create a Deployment using the following YAML. For details on creating Deployments, see Create a stateless application by using a Deployment.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: coredump-emptydir-volume-test
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
         annotations:
           alibabacloud.com/core-pattern: "/data/dump-a/core-%E-%p-%t" # Core file storage path
       spec:
         containers:
         - name: nginx
           image: registry.cn-hangzhou.aliyuncs.com/acs-sample/nginx:latest
           volumeMounts:
             - name: emptydir-volume
               mountPath: /data/dump-a/
         volumes:
           - name: emptydir-volume
             emptyDir: {}

   This creates a Deployment with an emptydir-volume volume mounted at /data/dump-a/. After a crash, inject an ephemeral container to access core files at that path.

2. In the first terminal, start a local proxy between your client and the cluster:

   Use --port to specify a different port. For details, see kubectl proxy.

   kubectl proxy

   Expected output:

   Starting to serve on 127.0.0.1:8001

3. In the second terminal, inject an ephemeral container into the pod. Replace coredump-emptydir-volume-test-xxxxx with the actual pod name and target-container with the target container name.

   Parameter	Description
   ...namespaces/${NAMESPACE}...	Namespace of the pod
   ...pods/${POD_NAME}...	Name of the pod
   spec: ${SPEC_DETAIL}	Spec of the ephemeral container. Validate the JSON format before submitting.
   Spec.ephemeralContainers.name	Name of the ephemeral container. Must be unique when multiple ephemeral containers are injected.
   Spec.ephemeralContainers.command	Startup command. Optional when using a custom image with a default entrypoint.
   Spec.ephemeralContainers.targetContainerName	Name of the target container in the pod. Required when the pod has multiple containers.
   Spec.ephemeralContainers.volumeMounts	Mount path for the ephemeral container. Must match the core-pattern annotation value.

   curl -k http://127.0.0.1:8001/api/v1/namespaces/default/pods/coredump-emptydir-volume-test-xxxxx/ephemeralcontainers \
     -X PATCH \
     -H 'Content-Type: application/strategic-merge-patch+json' \
     -d '{
     "spec": {
       "ephemeralContainers": [
         {
           "name": "debugger-container-name",
           "command": [
             "/bin/sh",
             "-c",
             "sleep 3600"
           ],
           "image": "registry.cn-hangzhou.aliyuncs.com/acs-sample/nginx:latest",
           "stdin": true,
           "tty": true,
           "targetContainerName": "target-container",
           "volumeMounts": [
             {
               "name": "emptydir-volume",
               "mountPath": "/data/dump-a/"
             }
           ]
         }
       ]
     }
   }'

   The following table describes the key parameters:

4. Log in to the ephemeral container:

   kubectl exec -it -n default coredump-emptydir-volume-test-xxxxx -c debugger-container-name sh

5. Access the core file directory:

   cd /data/dump-a && pwd

   Expected output:

   /data/dump-a

   Core files generated by the crashed container are in this directory. Use gdb to analyze them.

6. After finishing the debugging session, stop the proxy process in the first terminal by pressing Ctrl+C.

What's next

• Create a stateless application by using a Deployment

• Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster