Container Service for Kubernetes (ACK) provides capabilities to collect and analyze logs and audit data from various Kubernetes components, including the API server, Ingress, control plane components, and critical Kubernetes events. This helps you perform root cause analysis and trace activities when security or cluster issues arise.
Use cluster auditing
In a Kubernetes cluster, the API server's audit log is a crucial tool for security and operations, helping administrators record and trace user activities. For information about how to collect and analyze audit logs by using Log Service (SLS), configure custom alert rules for audit logs, and disable cluster auditing, see Work with cluster auditing.
The audit policy for Container Service for Kubernetes (ACK) is as follows:
For more information about the audit log fields, see audit.k8s.io/v1 Event.
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't log events for requests in the RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# The following requests are high-volume and low-risk, so they are not logged.
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core
resources: ["endpoints", "services"]
- level: None
users: ["system:unsecured"]
namespaces: ["kube-system"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["configmaps"]
- level: None
users: ["kubelet"] # legacy kubelet identity
verbs: ["get"]
resources:
- group: "" # core
resources: ["nodes"]
- level: None
userGroups: ["system:nodes"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["nodes"]
- level: None
users:
- system:kube-controller-manager
- system:kube-scheduler
- system:serviceaccount:kube-system:endpoint-controller
verbs: ["get", "update"]
namespaces: ["kube-system"]
resources:
- group: "" # core
resources: ["endpoints"]
- level: None
users: ["system:apiserver"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["namespaces"]
# Don't log requests to these read-only URLs.
- level: None
nonResourceURLs:
- /healthz*
- /version
- /swagger*
# Don't log event requests.
- level: None
resources:
- group: "" # core
resources: ["events"]
# Secrets, ConfigMaps, and token reviews can contain sensitive and binary data,
# so only log them at the Metadata level.
- level: Metadata
resources:
- group: "" # core
resources: ["secrets", "configmaps"]
- group: authentication.k8s.io
resources: ["tokenreviews"]
- level: Request
verbs: ["get", "list", "watch"]
resources:
- group: "" # core
- group: "admissionregistration.k8s.io"
- group: "apps"
- group: "authentication.k8s.io"
- group: "authorization.k8s.io"
- group: "autoscaling"
- group: "batch"
- group: "certificates.k8s.io"
- group: "extensions"
- group: "networking.k8s.io"
- group: "policy"
- group: "rbac.authorization.k8s.io"
- group: "settings.k8s.io"
- group: "storage.k8s.io"
# Default level for known APIs.
- level: RequestResponse
resources:
- group: "" # core
- group: "admissionregistration.k8s.io"
- group: "apps"
- group: "authentication.k8s.io"
- group: "authorization.k8s.io"
- group: "autoscaling"
- group: "batch"
- group: "certificates.k8s.io"
- group: "extensions"
- group: "networking.k8s.io"
- group: "policy"
- group: "rbac.authorization.k8s.io"
- group: "settings.k8s.io"
- group: "storage.k8s.io"
- group: "autoscaling.alibabacloud.com"
# Default level for all other requests.
- level: MetadataCustomize cluster audit log rules
ACK allows you to extend the cluster auditing capabilities by adding custom rules. You can configure custom rules in the auditPolicyRules field on the kube-apiserver configuration page.
# The rules must be defined as a list in YAML format.
- level: RequestResponse
resources:
- group: "policy.alibabacloud.com"
resources: ["policies"]
- level: None
userGroups: ["system:nodes"]
verbs: ["update", "patch"]
resources:
- group: ""
resources: ["nodes/status", "pods/status"]The preceding rules achieve two objectives:
-
Enhanced auditing: Logs all operations on Container Service policy templates (
policies.policy.alibabacloud.com) at theRequestResponselevel. -
Reduced noise: Ignores status updates for nodes and Pods from the kubelet component (which belongs to the
system:nodesgroup), such as heartbeats, to reduce unnecessary log entries.
For more information about how to define audit log rules, see Auditing.
When you configure custom rules, note the following:
-
Rule format: You add a list of audit rules (
rules) in the input field, not a completePolicyobject. -
Rule priority: To ensure a baseline level of security auditing for your cluster, you cannot change the log level for requests covered by the default audit rules of Container Service.
-
Rule validation: Container Service validates the added rules based on the definition of the rules field in policies.audit.k8s.io. Ensure you add valid audit rules by consulting the official Kubernetes documentation on Kubernetes Audit. Important: An incorrectly formatted rule can prevent the API server from starting.
Container exec activity auditing
A common attack vector in Kubernetes clusters involves an attacker using the exec command to enter a container and launch lateral attacks. After an attacker gains access to a container, the default API server audit log cannot record the commands they execute. In this scenario, you can use container internal activity auditing. This helps security and operations personnel capture the commands executed by an attacker after entering a container, facilitating root cause analysis and timely mitigation of security events.
Use audit metadata
The Kubernetes audit log includes two annotations that indicate whether a request was authorized, authorization.k8s.io/decision, and the reason for the decision, authorization.k8s.io/reason. You can use these attributes to determine why a specific API call was permitted.
Monitor cluster events with NPD and SLS
node-problem-detector (NPD) is a tool maintained by Alibaba Cloud for diagnosing nodes in Container Service for Kubernetes (ACK). NPD converts node anomalies, such as a hung Docker engine, a hung Linux kernel, outbound network issues, or file descriptor exhaustion, into node events. When used with kube-eventer, it creates a closed-loop system for node event alerting. In addition to real-time node problems detected by NPD, Kubernetes clusters also generate various events as their state changes, such as Pod evictions or image pull failures. For these and other suspicious activities, like a standard user using exec to access a container, the Kubernetes event center in Log Service (SLS) gathers all events from your cluster in real time. It provides storage, querying, analysis, visualization, and alerting capabilities to help security and operations personnel promptly identify risks to cluster stability. For more information, see Event monitoring.
Enable the Ingress dashboard
The Alibaba Cloud Ingress component allows you to log all your HTTP requests to standard output. Alibaba Cloud also integrates the Ingress component with Log Service (SLS), allowing you to quickly create dashboards for log analysis and monitoring. On the Ingress visualization dashboard, you can easily view the overall status of your Ingress, including key metrics like page views (PVs), unique visitors (UVs), traffic, latency, and top URLs. This helps you monitor service traffic in real time and promptly detect malicious access and DoS attacks. For more information, see Ingress Dashboard.
Enable CoreDNS logging
Container Service for Kubernetes (ACK) clusters use CoreDNS as the internal DNS server. You can analyze CoreDNS logs to troubleshoot issues like slow DNS resolution or investigate queries to high-risk domains. The SLS dashboard provides a CoreDNS log analysis report, helping you promptly identify suspicious requests to high-risk domains. For more information, see Analyze and monitor CoreDNS logs.