In a Kubernetes cluster, kube-apiserver generates audit log data that helps administrators track operations performed by different users. Cluster auditing plays an important role in the security and maintenance of the cluster. This topic descries how to use the cluster auditing feature in a registered Kubernetes cluster.

Prerequisites

A cluster registration proxy is created and a self-managed Kubernetes cluster is connected to the cluster registration proxy. For more information, see Create a cluster registration proxy and register an on-premises cluster.

Step 1: Configure an audit policy for master nodes

Log on to a master node and modify the /etc/kubernetes/audit-policy.yaml file based on the following template. You must also perform this step on the other master nodes.
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # The following requests were manually identified as high-volume and low-risk,
  # so drop them.
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # core
        resources: ["endpoints", "services"]
  - level: None
    users: ["system:unsecured"]
    namespaces: ["kube-system"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["configmaps"]
  - level: None
    users: ["kubelet"] # legacy kubelet identity
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces"]
  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Don't log events requests.
  - level: None
    resources:
      - group: "" # core
        resources: ["events"]
  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
  # Get repsonses can be large; skip them.
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Default level for known APIs
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Default level for all other requests.
  - level: Metadata

Step 2: Modify the kube-apiserver configurations on master nodes

Log on to a master node and modify the /etc/kubernetes/manifests/kube-apiserver.yaml file based on the following description. You must also perform this step on the other master nodes.
  • Add --audit-log-* parameters to the command section:
    ...
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-log-path=/var/log/kubernetes/kubernetes.audit
    - --audit-log-maxage=30
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    ...
  • Add aliyun_logs_audit-* parameters to the env section:
    ...
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-log-path=/var/log/kubernetes/kubernetes.audit
    - --audit-log-maxage=30
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    ...
    ...
    env:
    - name: aliyun_logs_audit-${cluster_id}
      value: /var/log/kubernetes/kubernetes.audit
    - name: aliyun_logs_audit-${cluster_id}_tags
      value: audit=apiserver
    - name: aliyun_logs_audit-${cluster_id}_product
      value: k8s-audit
    - name: aliyun_logs_audit-${cluster_id}_jsonfile
      value: "true"
    image: registry-vpc.cn-shenzhen.aliyuncs.com/acs/kube-apiserver:v1.20.4-aliyun.1
    Notice You must replace {cluster_id} with the ID of your cluster. For more information about how to obtain the ID of your cluster, see View basic information.
  • Use the following template to mount /etc/kubernetes/audit-policy.yaml to the pods of kube-apiserver: 
    ...
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-log-path=/var/log/kubernetes/kubernetes.audit
    - --audit-log-maxage=30
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    ...
    ...
    env:
    - name: aliyun_logs_audit-${cluster_id}
      value: /var/log/kubernetes/kubernetes.audit
    - name: aliyun_logs_audit-${cluster_id}_tags
      value: audit=apiserver
    - name: aliyun_logs_audit-${cluster_id}_product
      value: k8s-audit
    - name: aliyun_logs_audit-${cluster_id}_jsonfile
      value: "true"
    image: registry-vpc.cn-shenzhen.aliyuncs.com/acs/kube-apiserver:v1.20.4-aliyun.1
    ...
    ...
    volumeMounts:
    - mountPath: /var/log/kubernetes
      name: k8s-audit
    - mountPath: /etc/kubernetes/audit-policy.yaml
      name: audit-policy
      readOnly: true
    ...
    ...
  volumes:
  - hostPath:
      path: /var/log/kubernetes
      type: DirectoryOrCreate
    name: k8s-audit
  - hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: FileOrCreate
    name: audit-policy
  ...

Step 3: Install the logtail-ds component

For more information, see Step 2: Install the logtail-ds component.

What to do next

For more information about how to use the cluster auditing feature and check the result, see Enable cluster auditing.