kube-apiserver audit logs - User Guide for Kubernetes Clusters| Alibaba Cloud Documentation Center

Parameter settings

If you select the Enable Log Service check box when you create a cluster, kube-apiserver will record audit logs by default. The parameters are listed in the following table.

Note Log on to a master node. You can find the apiserver configuration file in the following path: /etc/kubernetes/manifests/kube-apiserver.yaml.


Parameter	Description
audit-log-maxbackup	The maximum number of audit log files to retain is 10.
audit-log-maxsize	The maximum size of each audit log file is 100 MB.
audit-log-path	The log output path is /var/log/kubernetes/kubernetes.audit.
audit-log-maxage	The maximum number of days to retain audit log files is 7.
audit-policy-file	The path to the audit policy file is /etc/kubernetes/audit-policy.ym.

Log on to a master node. You can find the audit policy file in the following path: /etc/kubernetes/audit-policy.yml. The audit policy file is as follows:

apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # The following requests were manually identified as high-volume and low-risk,
  # so drop them.
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # core
        resources: ["endpoints", "services"]
  - level: None
    users: ["system:unsecured"]
    namespaces: ["kube-system"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["configmaps"]
  - level: None
    users: ["kubelet"] # legacy kubelet identity
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces"]
  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Don't log events requests.
  - level: None
    resources:
      - group: "" # core
        resources: ["events"]
  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
  # Get repsonses can be large; skip them.
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Default level for known APIs
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Default level for all other requests.
  - level: Metadata

Note

Logging does not start immediately after a request is received, but until the response headers are sent.
The following types of requests are not logged: watch requests by kube-proxy, GET requests by kubelet and system:nodes on node resources, operations on an endpoints resource by kube components in the kube-system namespace, and GET requests by apiserver on a namespaces resource.
Requests to URLs that match /healthz*, /version*, or /swagger* are not logged.
Requests on a secrets resource, a configmaps resource, or a tokenreviews resource are logged at the Metadata level because these requests may contain sensitive information or binary files. The Metadata level only records request metadata, such as the requesting user, timestamp, resource, and action, but not the request body or response body.
Requests on sensitive resources such as authenticatioin, rbac, certificates, autoscaling, and storage are logged, including the request body and response body.

View audit log reports

Container Service for Kubernetes provides three audit log reports for each cluster. You can find the following information in these reports:

Important operations performed by users and system components on the target cluster.
The source IP addresses of these operations and the regional distribution of these IP addresses.
The details of operations on different types of resources.
The details of operations by different RAM users.
The details of important operations, such as logging on to containers, accessing secrets, and deleting resources.

Note

If you enabled Log Service when you created the cluster, audit log reports are automatically generated. For more information about the pricing of Log Service, see Billing. If audit log reports are not automatically generated, see Generate audit log reports.
Do not modify audit log reports. If you need custom reports, go to the Log Service console to create new reports.

You can use one of the following methods to view audit log reports.

Log on to the Container Service console. On the Clusters page, find the target cluster and choose More > Cluster Auditing in the Actions column.
Log on to the Container Service console. On the Clusters page, find the target cluster and click the cluster name to go to its details page. In the left-side navigation pane, click Cluster Auditing.

Reports overview

kube-apiserver provides the following three reports: Overview, Operations Overview, and Operation Details.

Overview
This report provides an overview of the events in the cluster and the details of important events, such as access from the Internet, command executions, resource deletion, and access to secrets.

Note By default, the report displays statistics by week. Custom time ranges are supported. You can filter the report based on namespace, RAM user ID, and status code. You can also select one or multiple items to filter the report.
Operations Overview
This report provides statistics on operations that are commonly performed on computing resources, network resources, and storage resources in the cluster. The operations include create, update, delete, and access.
- Computing resources include Deployment, StatefulSet, CronJob, DaemonSet, Job, and Pod.
- Network resources include Service and Ingress.
- Storage resources include ConfigMap, Secret, and PersistentVolumeClaim.
Note
- By default, the report displays statistics by week. Custom time ranges are supported. You can filter the report based on namespace and RAM user ID. You can also select one or multiple items to filter the report.
- To view details of the operations on a resource, go to the Operation Details report.
Operation Details
This report provides operation details by resource type. You can specify a resource type to query operation details in real time. The report contains the total number of operations, distribution of namespaces, operation success rate, temporal order of operations, and other operation details.
Note
- To query operations about a CustomResourceDefinition (CRD) resource registered in Kubernetes or other resources that are not listed in the report, enter the plural form of the target resource name. For example, to query operations about a CRD resource named AliyunLogConfig, enter AliyunLogConfigs.
- By default, the report displays statistics by week. Custom time ranges are supported. You can filter the report based on namespace, RAM user ID, and status code. You can also select one or multiple items to filter the report.

View log records

To customize the queries or analyze audit logs, you can go to the Log Service console to view detailed log records.

Log on to the Container Service console.
Go to the Projects section. Find the target project and click the project name.
The project details page appears. You can find a list of Logstores on the left side of the page. Find the Logstore named audit-${clustered} and click Search and Analysis.
Note
- During the cluster creation process, a Logstore named audit-${clustereid} is automatically created under the target project.
- By default, indexes are already set up in the Logstore. Do not modify the indexes.

You can query audit logs by using the following methods:

To query the operations performed by a RAM user, enter the RAM user ID and click Search & Analytics.
To query the operations performed on a resource, enter the resource name, and click Search & Analytics.
To filter out the operations performed by system components, enter NOT user.username: node NOT user.username: serviceaccount NOT user.username: apiserver NOT user.username: kube-scheduler NOT user.username: kube-controller-manager and click Search & Analytics.

For more information about querying logs, see Query methods.

Set alerts

Log Service allows you to set alerts to monitor the operations that are performed on certain resources in real time. Alerts can be sent through DingTalk Chatbots, custom webhooks, and Message Center. For more information, see Configure an alert.

Note For more information about how to query audit logs, see the query statements in audit reports. You can use the following steps to view query statements: On the project details page, click the Dashboard icon in the left-side navigation pane. Select a dashboard to go to the details page. Select a chart, click the More Vertical icon, and click View Analysis Details.

Example 1: Set an alert on command executions on containers

To monitor command executions on containers, alerts need to be sent immediately when a user logs on to a container and runs commands. The alert notification needs to include the following information: the container that was logged on, executed commands, operator, event ID, operation time, and source IP address.

A sample query statement is as follows:

verb : create and objectRef.subresource:exec and stage:  ResponseStarted | SELECT auditID as "event ID", date_format(from_unixtime(__time__), '%Y-%m-%d %T' ) as "operation time",  regexp_extract("requestURI", '([^\?] *)/exec\?.*', 1)as "resource",  regexp_extract("requestURI", '\?(.*)', 1)as "command" ,"responseStatus.code" as "status code",
 CASE 
 WHEN "user.username" ! = 'kubernetes-admin' then "user.username"
 WHEN "user.username" = 'kubernetes-admin' and regexp_like("annotations.authorization.k8s.io/reason", 'RoleBinding') then regexp_extract("annotations.authorization.k8s.io/reason", ' to User "(\w+)"', 1)
 ELSE 'kubernetes-admin' END  
 as "operating account", 
CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE  sourceIPs END
as "source IP address" limit 100

The conditional expression is event =~ ". *".

Example 2: Set an alert on connection failures to apiserver from the Internet

To prevent attacks on a cluster that allows Internet access, you need to monitor the number of connections from the Internet and the connection failure rate. Alerts need to be sent immediately when the number of connections reaches a threshold and the connection failure rate exceeds a threshold. The alert notification needs to include the following information: the source IP address, the region where the source IP address belongs, and whether the source IP address is suspicious.

A sample query statement is as follows:

* | select ip as "source IP address", total as "number of connections", round(rate * 100, 2) as "connection failure rate%", failCount as "number of illegal connections", CASE when security_check_ip(ip) = 1 then 'yes' else 'no' end  as "whether the IP address is suspicious",  ip_to_country(ip) as "country", ip_to_province(ip) as "province", ip_to_city(ip) as "city", ip_to_provider(ip) as "ISP" from (select CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE  sourceIPs END
as ip, count(1) as total,
sum(CASE WHEN "responseStatus.code" < 400 then 0 
ELSE 1 END) * 1.0 / count(1) as rate,
count_if("responseStatus.code" = 403) as failCount
from log  group by ip limit 10000) where ip_to_domain(ip) ! = 'intranet'  having "number of connections" > 10 and "connection failure rate%" > 50 ORDER by "number of connections" desc limit 100

The conditional expression is source IP address =~ ". *".

Generate audit log reports

If audit log reports are not automatically generated, use the following steps to generate audit log reports.

Note Currently, you can only automate the generation of audit log reports in dedicated Kubernetes clusters.

Enable apiserver audit logs.

Log on to the three master nodes and check the pod configurations on apiserver. Check startup parameters, audit policy file, and mount directories for content that is related to audit logs.

Startup parameters

containers:
  - command:
    - kube-apiserver
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-log-path=/var/log/kubernetes/kubernetes.audit
    - --audit-log-maxage=7
    - --audit-policy-file=/etc/kubernetes/audit-policy.yml

Audit policy file (/etc/kubernetes/audit-policy.yml)

Note If the audit policy file does not exist in the /etc/kubernetes/ path, you can run the vi audit-policy.yml command to create a file and paste the policy file content into the file.

Environment variables

env:
    - name: aliyun_logs_audit-c12ba20***************9f0b
      value: /var/log/kubernetes/kubernetes.audit
    - name: aliyun_logs_audit-c12ba20***************9f0b_tags
      value: audit=apiserver
    - name: aliyun_logs_audit-c12ba20***************9f0b_product
      value: k8s-audit
    - name: aliyun_logs_audit-c12ba20***************9f0b_jsonfile
      value: "true"

Mount directories

volumeMounts:
    - mountPath: /var/log/kubernetes
      name: k8s-audit
    - mountPath: /etc/kubernetes/audit-policy.yml
      name: audit-policy
      readOnly: true
volumes:
    - hostPath:
      path: /var/log/kubernetes
      type: DirectoryOrCreate
      name: k8s-audit
    - hostPath:
      path: /etc/kubernetes/audit-policy.yml
      type: FileOrCreate
      name: audit-policy

Overwrite the /etc/kubernetes/manifests/kube-apiserver.yaml file and restart apiserver. Before you overwrite the existing file, make sure that a copy of this file is created.

If the preceding content does not exist in the pod configurations of apiserver, upgrade your cluster to the latest version. For more information, see Upgrade a cluster.

Upgrade Logtail to the latest version.
- If Log Service components are not installed in your cluster, see Manually install Log Service components.
- If Log Service components are already installed but no audit logs can be found, you need to upgrade Log Service components to the latest version. Make sure that the Logtail version is 0.16.16 or above and it is running on master nodes. For more information, see Upgrade Log Service components.
Change the log parsing configuration.
1. Log on to the Log Service console.
2. Go to the Projects section. Find the target project and click the project name.
3. The project details page appears. You can find a list of Logstores on the left side of the page. Find the Logstore named audit-${clustered} and click the arrow button. In the drop-down list that appears, click Logtail Configurations. On the Logtail Config page, check whether the Mode parameter is set to JSON Mode. If not, set it to JSON Mode.

Billing

To view billing details, go to Bills and click the Overview tab. You can also find fees of audit logs. For more information, see Bill.
For more information about the billing method of audit logs, see Pay-as-you-go.

Support for third-party logging services

You can find the source log file in the /var/log/kubernetes/kubernetes.audit path on master nodes. This file is in standard JSON format. When you create a cluster, you can specify a third party logging service to collect and search audit logs.