Container Service for Kubernetes:Use the API Server audit feature to secure cluster operations

Overview

Displays all events in the cluster and detailed information about high-priority events, including RAM user operations, Internet access, command executions, resource deletions, Secret access, and Kubernetes Common Vulnerabilities and Exposures (CVE) vulnerabilities.

Operations overview

Provides statistics on create, update, delete, and read operations across:

• Computing resources: Deployment, StatefulSet, CronJob, DaemonSet, Job, and Pod

• Network resources: Service and Ingress

• Storage resources: ConfigMap, Secret, and PersistentVolumeClaim (PVC)

• Access control resources: Role, ClusterRole, RoleBinding, and ClusterRoleBinding

Operation details

Shows operation details for a specific resource type. Select or enter a resource type to query in real time. The report displays the total operation count, namespace distribution, success rate, and operation trends over time.

To query CustomResourceDefinition (CRD) resources or resources not listed in the report, enter the plural form of the resource name. For example, to query the AliyunLogConfig CRD, enter AliyunLogConfigs.

CVE vulnerabilities

Displays Kubernetes CVE vulnerabilities in the cluster. Enter a RAM user ID to filter results for that user. For vulnerability details and remediation guidance, see \[CVE Securities\] CVE vulnerability fixes.

Alert example 1: Commands executed in containers

An enterprise wants to forbid users to log on to containers or run commands in containers. When a user runs a command in a container, an alert is immediately generated. The alert message contains information about the container, command, user, event ID, time, and source IP address.

Sample query statement:

verb : create and objectRef.subresource:exec and stage:  ResponseStarted | SELECT auditID as "Event ID", date_format(from_unixtime(__time__), '%Y-%m-%d %T' ) as "Time",  regexp_extract("requestURI", '([^\?]*)/exec\?.*', 1)as "Resource",  regexp_extract("requestURI", '\?(.*)', 1)as "Command" ,"responseStatus.code" as "Status code",
 CASE
 WHEN "user.username" != 'kubernetes-admin' then "user.username"
 WHEN "user.username" = 'kubernetes-admin' and regexp_like("annotations.authorization.k8s.io/reason", 'RoleBinding') then regexp_extract("annotations.authorization.k8s.io/reason", ' to User "(\w+)"', 1)
 ELSE 'kubernetes-admin' END
 as "User account",
CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE  sourceIPs END
as "Source IP address" order by "Time" desc  limit 10000

Condition expression: Event =~ ".*"

Alert example 2: Failed Internet access from the API server

Use this alert to monitor outbound Internet requests from your cluster. It fires when the number of Internet requests reaches the threshold (10) and the failure rate exceeds 50%.

Sample query statement:

* | select ip as "Source IP address", total as "Number of times of Internet access", round(rate * 100, 2) as "Failure rate in percentage", failCount as "Number of times of illegal access", CASE when security_check_ip(ip) = 1 then 'yes' else 'no' end  as "Whether the IP address is risky",  ip_to_country(ip) as "Country", ip_to_province(ip) as "Province", ip_to_city(ip) as "City", ip_to_provider(ip) as "ISP" from (select CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE  sourceIPs END
as ip, count(1) as total,
sum(CASE WHEN "responseStatus.code" < 400 then 0
ELSE 1 END) * 1.0 / count(1) as rate,
count_if("responseStatus.code" = 403) as failCount
from log  group by ip limit 10000) where ip_to_domain(ip) != 'intranet' and ip not LIKE '%,%' and not try(is_subnet_of('7.0.0.0/8', ip)) ORDER by "Number of times of Internet access" desc limit 10000

Condition expression: Source IP address =~ ".*"

Change the Simple Log Service project

To migrate audit logs to a different SLS project:

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the cluster name. In the left-side pane, choose Security > Cluster Auditing.

3. In the upper-right corner, click Change Log Service Project and follow the prompts.

Disable cluster auditing

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the cluster name. In the left-side pane, choose Security > Cluster Auditing.

3. In the upper-right corner of the Cluster Auditing page, click Disable Cluster Auditing.

Use a third-party log service (ACK dedicated clusters only)

Simple Log Service is the recommended storage for audit logs. To use a third-party log service instead, skip Simple Log Service during cluster creation and integrate the third-party service to collect logs. The raw audit log files are available on master nodes at /var/log/kubernetes/kubernetes.audit in JSON format.

Audit policy

The audit policy defines which events are collected and at what level of detail. The four audit levels are:

The audit policy is loaded from /etc/kubernetes/audit-policy.yml on master nodes (set via the --audit-policy-file flag). The following is the default policy used by ACK:

apiVersion: audit.k8s.io/v1 # Required. Set to audit.k8s.io/v1 if the Kubernetes version of the cluster is 1.24 or later and set to audit.k8s.io/v1beta1 if the Kubernetes version of the cluster is earlier than 1.24.
kind: Policy
# No need to generate audit events at the RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # The following types of requests are frequent and the risk of these requests is low. We recommend that you set the rule to None to skip these requests.
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # core
        resources: ["endpoints", "services"]
  - level: None
    users: ["system:unsecured"]
    namespaces: ["kube-system"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["configmaps"]
  - level: None
    users: ["kubelet"] # legacy kubelet identity
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces"]
  # Set the rule to None for read-only URLs, such as /healthz*, /version*, and /swagger*.
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Set the rule to None for events.
  - level: None
    resources:
      - group: "" # core
        resources: ["events"]
  # Set the rule to Metadata for Secrets, ConfigMaps, and TokenReview API requests that may contain sensitive information or binary files.
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
  # Responses may contain large amounts of data. Set the rule to Request so that the response body is not collected.
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # The rule is set to RequestResponse by default for known Kubernetes API requests to collect the request and response bodies.
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # The rule is set to Metadata by default for other requests.
  - level: Metadata

Key behaviors of this policy:

• Logs are generated after response headers are sent, not when requests are received.

• kube-proxy watch requests, kubelet and system:nodes GET requests to nodes, kube-system endpoint operations, and API server GET requests to namespaces are excluded.

• Read operations (get, list, watch) for the authentication, rbac, certificates, autoscaling, and storage APIs are logged at the Request level. Write operations on these APIs are logged at RequestResponse.

• Secrets and ConfigMaps are logged at Metadata only, so their content is never written to audit logs.

Audit backend

Audit events are written as JSON log files to the local file system on master nodes. The API server configuration file at /etc/kubernetes/manifests/kube-apiserver.yaml controls the backend behavior with the following flags: