This topic describes how to test, build, and deploy an end-to-end supply chain to secure the lifecycle of a Container Service for Kubernetes (ACK) cluster.

Background information

A software supply chain consists of the build, deploy, and runtime stages. Attackers can launch different types of attacks against your workloads in these stages.
Stage Type of attack
Build
  • Pollution attacks against IDE tools.
  • Exploiting of vulnerabilities and webshells in third-party libraries.
  • Pollution attacks against source code.
Deploy
  • Substitution and tampering attacks against stored data.
  • Hijacking of transmitted data and drive-by download attacks.
Runtime
  • Hijacking of software updates.
  • Exploiting of vulnerabilities and webshells in runtimes.
  • Exploiting of zero-day vulnerabilities in third-party libraries.

Artifact security is the first defense in the build and test stages. An attacker can exploit a malicious container image to escape from the container and gain control over the host. As a result, the attacker can move through the network to access sensitive data or other valued data assets with your Alibaba Cloud account. The admission control mechanism provided by open source Kubernetes can help you verify the security of the provisioned pods in the deploy stage. After you deploy an application, you must monitor the runtime of the application in real time. This way, you can handle security events at the earliest opportunity.

Suggestions on cluster security

We recommend that you follow these suggestions to secure the lifecycle of an ACK cluster.

  • Create smaller container images

    Delete binary files that you do not need from the container image that you want to use. If the container image is downloaded from Docker Hub, you must use Dive to analyze the image. Dive can help you explore the content of each image layer. For more information, see Dive.

    You can also view the content of each image layer in the Container Registry console after you push the image to Alibaba Cloud Container Registry. You must delete all binary files with the SETUID and SETGID bits because these files can be exploited to escalate privileges. You can also delete shells and applications that may be exploited by attackers, such as nc and curl. You can run the following command to search for binary files with the SETUID and SETGID bits:
    find / -perm /6000 -type f -exec ls -ld {} \;
    Add the following command to the container image if you want to remove permissions from the binary files:
    RUN find / -xdev -perm /6000 -type f -exec chmod a-s {} \; || true
  • Use multi-stage builds

    You can use multi-stage builds to create smaller container images. Multi-stage builds can be used to automate continuous integration in CI/CD pipelines. Multi-stage builds allow you to include multiple FROM statements in your Dockerfile. Each FROM statement can use a different base image, and each FROM statement starts a new build stage. Multi-stage builds can help you quickly build trusted container images and keep the size of the container images down. For more information, see Build an image for a Java application by using a Dockerfile with multi-stage builds.

  • Create RAM policies for Container Registry repositories
    Multiple teams in a company may want to use the same Alibaba Cloud account to manage cloud resources. If these teams do not want to share resources, you must create Resource Access Management (RAM) policies to limit the namespaces or repositories that each team can access. For example, cr:ListInstance* specifies all actions that start with cr:ListInstance. You can set acs:cr:*:*:repository/$instanceid/$namespace/* to acs:cr:cn-hangzhou:1234567:repository/cri-123456/ns/* to grant the instance cri-123456 in the cn-hangzhou region the permissions to query repositories in namespaces. The instance belongs to the Alibaba Cloud account 1234567.
    {
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "cr:ListRepository",
                    "cr:GetImageLayer",
                    "cr:GetRepoTag"
                ],
                "Resource": "*"
            },
            {
                "Action": [
                    "cr:List*"
                ],
                "Effect": "Allow",
                "Resource": [
                    "acs:cr:cn-hangzhou:1234567:repository/cri-123456/ns/*",
                ]
            }
        ],
        "Version": "1"
    } 

    For more information about how to configure RAM policies in Container Registry, see Configure policies for RAM users to access Container Registry and Use RAM to grant permissions to access custom OSS buckets.

  • Use Container Registry Enterprise Edition

    Container Registry Enterprise Edition allows you to encrypt cloud-native artifacts that are stored in Container Registry. Container Registry Enterprise Edition also supports image scans and multi-dimensional vulnerability reports to help you secure storage and content. You can enforce access control on container images and Helm charts, and perform fine-grained action auditing to ensure the security of artifacts. We recommend that you perform the following tasks to enhance security: use Container Registry Enterprise Edition in production environments, set repositories to private, use internal endpoints to access Container Registry instances over virtual private clouds (VPCs), disable Internet access, and configure network access control lists (ACLs). For more information, see Create a Container Registry Enterprise Edition instance.

  • Use the cloud-native application delivery chain provided by Container Registry

    Container Registry provides a cloud-native application delivery chain. You can streamline tasks such as image building, image scanning, image global synchronization, and image distribution in the delivery chain. The entire delivery chain is observable, traceable, and secured. For more information, see Create a delivery chain.

    After you push container images to Container Registry, Container Registry automatically scans the images. You can also configure security policies to enable Container Registry to identify security risks in container images and block images whose severity levels are high. Only images that are allowed by security policies are distributed and deployed. The delivery chain ensures the secure delivery and efficient deployment of containerized applications. You can also integrate the API operations for image scanning into your system to schedule image scans.

  • Periodically scan container images for vulnerabilities

    Similar to OS images used by virtual machines, container images may contain binary files or applications in which vulnerabilities exist, or vulnerabilities may be added after the binary files or applications are updated. Therefore, we recommend that you periodically scan your container images for vulnerabilities. Container Registry can automatically scan newly uploaded container images and existing container images every 24 hours. If a container image contains vulnerabilities whose severity levels are HIGH or CRITICAL, you must delete or rebuild the container image. If vulnerabilities are detected in container images that have already been deployed, you must replace the relevant containers at the earliest opportunity.

    You can also invoke the Kubernetes validating webhook to identify critical vulnerabilities in container images. You can invoke the validating webhook before you call the Kubernetes API to reject requests that do not comply with the admission policies defined in the webhook. You can call the CreateRepoTagScanTask operation of the Container Registry API to check whether a container image that is being pulled by a cluster contains critical vulnerabilities. If the container image contains critical vulnerabilities, Container Registry forbid the cluster to deploy pods from the container image and generates an event. The event contains the detected vulnerabilities. For more information, see CreateRepoTagScanTask.

  • Add the USER instruction to your Dockerfiles and run containers as a non-root user

    To ensure pod security, you must avoid running containers as a root user. You can add the USER instruction to the PodSpec. You can use the USER instruction in Dockerfiles as a best practice to ensure pod security. After you add the USER instruction, RUN, ENTRYPOINT, and CMD are executed with the specified user account.

  • Download dependencies from trusted sources

    You must avoid using dependencies that are downloaded from untrusted sources in the software development stage. For more information about container images, artifacts, or dependencies that are trusted by Alibaba Cloud, see Alibaba Cloud images.

    You can use Apsara Devops to build your own repositories. Apsara Devops Artifact Repository Packages is an enterprise-class private repository service provided by Alibaba Cloud. This service allows enterprises to build private repositories to manage Maven, Gradle, and npm packages and repositories. You can also use this service to manage Maven and npm artifacts, build remote repositories, and migrate your repositories with a few clicks. The Apsara Devops Artifact Repository Packages service supports tenant isolation, permission control, and high-availability storage to secure your artifacts.

  • Use image signing and configure signature verification policies

    When you deploy applications in a cluster, you must verify the signatures of container images to ensure that only container images that are signed by trusted authorities are used. This helps you prevent exceptions and malicious code execution.

    Container Registry Enterprise Edition supports image signing. This prevents man-in-the-middle (MITM) attacks and unauthorized image updates or deployments. This way, image consistency and security from distribution to deployment are ensured. Container Registry can automatically sign images in specific namespaces. After an image is pushed to Container Registry, Container Registry automatically signs the image based on the matched signature signing rules. This ensures that your container images are trusted. For more information, see Sign container images.

    You can install the Kritis-validation-hook component in ACK clusters to automatically verify the signatures of container images that are signed by Key Management Service (KMS). This component is developed based on open source Kritis and can be deeply integrated with Container Registry. For more information, see Use kritis-validation-hook to automatically verify the signatures of container images, Kritis, and KMS.

    After you enable image signing, you can configure signature verification policies. Only trusted images that are verified by signature verification policies can be deployed in ACK clusters. You can also configure a signature verification whitelist, and then add sidecar container images injected by third-party components to the whitelist. This way, the system does not verify the signatures of the sidecar container images and pods can be deployed from these images as expected. For more information, see Introduction to kritis-validation-hook.

  • Use Alibaba Cloud Security Center

    You can use Security Center to detect and block threats in runtimes of cloud-native applications. This secures the runtime of each pod. Security Center can automatically obtain information about threats in cloud-native applications and use the information to analyze the threats, identify the sources of the threats, generate suitable responses, and handle the threats. Security Center also associates different types of logs, analyzes contexts, and detects risks in real time, such as malicious code or command execution, SQL injections, and data breaches. This can help prevent intrusions and identify vulnerabilities in your business systems. Security Center can audit actions and identify risks in real time based on Kubernetes logs and operations logs. This helps you mitigate the risks of container escapes, AccessKey breaches, and unauthorized access in ACK and other orchestration platforms. For more information, see What is Security Center?.