Kubernetes events indicate cluster state changes such as creating, running, or deleting pods, and component exceptions. The K8s Event Center collects all Kubernetes cluster events in real time and provides storage, query, analysis, visualization, and alerting capabilities.
Background information
The Kubernetes architecture is based on a state machine. When the system transitions between states, it generates events. Transitions between normal states produce Normal events, while transitions between normal and abnormal states produce Warning events.
ACK (Container Service for Kubernetes) provides an out-of-the-box event monitoring solution for containerized environments. It uses the ACK-maintained node-problem-detector (NPD) and its included component, kube-eventer, to provide event monitoring.
-
node-problem-detector (NPD) is a tool for diagnosing Kubernetes nodes. It converts node anomalies, such as Docker Engine hangs, Linux kernel hangs, public network access exceptions, and file descriptor issues, into node events. When combined with kube-eventer, it enables a closed-loop alerting system for node events. For more information, see NPD.
-
kube-eventer is an open source tool maintained by ACK for archiving Kubernetes events. It can push cluster events to external systems such as DingTalk, Simple Log Service (SLS), and EventBridge. It also provides filtering capabilities based on event levels. This enables real-time collection, targeted alerting, and asynchronous archiving of events. For more information, see kube-eventer.
Prerequisites
A Kubernetes cluster is created, such as a Container Service for Kubernetes (ACK) cluster or an ACK Serverless cluster.
Billing
The K8s Event Center is free when both conditions are met:
-
The associated Logstore uses the default 90-day data retention period.
-
Daily data volume is less than 256 MB (approximately 250,000 events).
Examples:
-
With the default 90-day retention and 1,000 events per day, the K8s Event Center is free.
-
With a 105-day retention and 1,000 events per day, storage fees apply after the 90-day free period. The billable item is Storage Space - Log Storage. Billable items of the pay-by-feature billing model.
Step 1: Deploy kube-eventer and node-problem-detector
Alibaba Cloud Kubernetes
For an ACK cluster, the ack-node-problem-detector component integrates kube-eventer and Node Problem Detector (NPD). Deploy only this component as described in Event monitoring. For an ACK Serverless cluster, deploy the kube-eventer component separately.
NPD uses configurations and third-party plug-ins to detect node issues and generate corresponding cluster events. A Kubernetes cluster also generates its own events during state transitions, such as pod evictions or image pull failures. The Kubernetes Event Center of Simple Log Service (SLS) centralizes all Kubernetes events in real time and provides storage, query, analysis, visualization, and alerting. To integrate your cluster events with the Kubernetes Event Center, follow these steps:
-
If you selected Install node-problem-detector and Create Event Center during cluster creation, you can skip to Step 2 to view the Kubernetes Event Center. For information about how to install the NPD component during cluster creation, see Create an ACK managed cluster.
-
If you did not select Install node-problem-detector and Create Event Center during cluster creation, follow these steps to install it manually.
Log on to the ACK console. In the left navigation pane, click Clusters.
-
On the Clusters page, click the name of the target cluster. In the left-side navigation pane, choose .
-
On the Logs and Monitoring tab, find and install ack-node-problem-detector.
Other Kubernetes clusters
-
Deploy kube-eventer.
-
Install kubectl by following Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
-
Create a configuration file named eventer.yaml by using the following sample:
apiVersion: apps/v1 kind: Deployment metadata: labels: name: kube-eventer name: kube-eventer namespace: kube-system spec: replicas: 1 selector: matchLabels: app: kube-eventer template: metadata: labels: app: kube-eventer annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: dnsPolicy: ClusterFirstWithHostNet serviceAccount: kube-eventer containers: - image: registry.cn-hangzhou.aliyuncs.com/acs/kube-eventer:v1.2.5-cc7ec54-aliyun name: kube-eventer command: - "/kube-eventer" - "--source=kubernetes:https://kubernetes.default" ## .send to sls ## --sink=sls:https://{endpoint}?project={project}&logStore=k8s-event®ionId={region-id}&internal=false&accessKeyId={accessKeyId}&accessKeySecret={accessKeySecret} - --sink=sls:https://cn-beijing.log.aliyuncs.com?project=k8s-xxxx&logStore=k8s-event®ionId=cn-beijing&internal=false&accessKeyId=xxx&accessKeySecret=xxx env: # If TZ is assigned, set the TZ value as the time zone - name: TZ value: "Asia/Shanghai" volumeMounts: - name: localtime mountPath: /etc/localtime readOnly: true - name: zoneinfo mountPath: /usr/share/zoneinfo readOnly: true resources: requests: cpu: 10m memory: 50Mi limits: cpu: 500m memory: 250Mi volumes: - name: localtime hostPath: path: /etc/localtime - name: zoneinfo hostPath: path: /usr/share/zoneinfo --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kube-eventer rules: - apiGroups: - "" resources: - events verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kube-eventer roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-eventer subjects: - kind: ServiceAccount name: kube-eventer namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: kube-eventer namespace: kube-systemParameter
Type
Required
Description
endpoint
string
Yes
The SLS endpoint. Endpoints.
Project
string
Yes
The SLS Project.
Logstore
string
Yes
The SLS Logstore.
internal
string
This parameter is required for self-managed Kubernetes clusters.
Set to false for self-managed Kubernetes clusters.
region ID
string
This parameter is required for self-managed Kubernetes clusters.
The region ID of the SLS Project. Endpoints.
AccessKey ID
string
This parameter is required for self-managed Kubernetes clusters.
The AccessKey ID. We recommend that you use a RAM user's AccessKey pair. AccessKey pair.
AccessKey Secret
string
This parameter is required for self-managed Kubernetes clusters.
The AccessKey Secret. We recommend that you use a RAM user's AccessKey pair. AccessKey pair.
-
Run the following command to apply the configurations in eventer.yaml to the cluster:
kubectl apply -f eventer.yamlExpected output:
deployment.apps/kube-eventer created clusterrole.rbac.authorization.k8s.io/kube-eventer created clusterrolebinding.rbac.authorization.k8s.io/kube-eventer created serviceaccount/kube-eventer created
-
-
Deploy Node Problem Detector.
Step 2: Create a K8s Event Center instance
Creating a K8s Event Center instance automatically creates a Logstore named k8s-event and associated dashboards in the target Project.
-
Log on to the Simple Log Service console.
-
In the Log Application section, on the Intelligent O&M tab, click K8s Event Center.
-
On the Event Center Management page, click Add in the upper-right corner.
-
In the Create Event Center panel, configure the parameters and click Next.
-
If you select Select Existing Project, select an existing Project from the Project drop-down list. The Project is used to manage K8s Event Center resources, such as the Logstore and dashboards.
-
If you select Select Kubernetes Cluster from Container Service for Kubernetes, select an existing cluster from the K8s Cluster drop-down list. Simple Log Service then creates a Project named
k8s-log-{cluster-id}to manage K8s Event Center resources, such as the Logstore and dashboards.
-
Step 3: Use the K8s Event Center instance
After creating a K8s Event Center instance and deploying kube-eventer and NPD, you can access the following features.
On the K8s Event Center page, find the target K8s Event Center instance and click the
icon to perform the following operations.

|
Actions |
Description |
|
View event overview |
The Event Overview page shows event totals, day-over-day error comparisons, alert statistics, error trends, and pod OOM details. Note
Pod OOM events identify the node, process name, and process ID, but not the specific pod. Run a custom query to find pod restart events near the OOM time to identify the pod. |
|
Query event details |
The Event Detail Query page shows events filtered by type, object, host, namespace, and name. |
|
View pod lifecycle |
The Pod Lifecycle page visualizes events throughout a pod lifecycle, with filtering by event level. |
|
View node events |
The Node Events page shows node lifecycle events and event lists. |
|
View core component events |
The Core Component Events page shows core component events such as ECS restart failures or unimplemented URL patterns. |
|
Configure alerts |
The Alert Configuration page lets you set up alerts for the K8s Event Center. Configure alerts. |
|
Run custom queries |
The Custom Query page lets you run custom query and analysis statements. All events are stored in a Logstore, giving you access to custom queries, event consumption, custom reports, and custom alerts. Quick start for search and analysis. You can find the Project name in the following ways:
|
|
Update version |
The Version Update page lets you upgrade the K8s Event Center. |
Delete a K8s Event Center instance
On the page, find the target instance, and in its Actions column, click the
icon.
Log sample
Sample log entry:
hostname: cn-hangzhou.i-***********"
level: Normal
pod_id: 2a360760-****
pod_name: logtail-ds-blkkr
event_id: {
"metadata":{
"name":"logtail-ds-blkkr.157b7cc90de7e192",
"namespace":"kube-system",
"selfLink":"/api/v1/namespaces/kube-system/events/logtail-ds-blkkr.157b7cc90de7e192",
"uid":"2aaf75ab-****",
"resourceVersion":"6129169",
"creationTimestamp":"2019-01-20T07:08:19Z"
},
"involvedObject":{
"kind":"Pod",
"namespace":"kube-system",
"name":"logtail-ds-blkkr",
"uid":"2a360760-****",
"apiVersion":"v1",
"resourceVersion":"6129161",
"fieldPath":"spec.containers{logtail}"
},
"reason":"Started",
"message":"Started container",
"source":{
"component":"kubelet",
"host":"cn-hangzhou.i-***********"
},
"firstTimestamp":"2019-01-20T07:08:19Z",
"lastTimestamp":"2019-01-20T07:08:19Z",
"count":1,
"type":"Normal",
"eventTime":null,
"reportingComponent":"",
"reportingInstance":""
}
|
Field |
Type |
Description |
|
hostname |
string |
The hostname where the event occurred. |
|
level |
string |
The log level. Valid values: Normal and Warning. |
|
pod_id |
string |
The pod ID. Included only for pod-related events. |
|
pod_name |
string |
The pod name. Included only for pod-related events. |
|
event_id |
json |
The event details in JSON format. |
FAQ
No data in K8s Event Center
After deployment, new events are automatically collected. Search for them on the Custom Query page with the time range set to 1 day. If no data appears, check the following:
-
No new events were generated since deployment.
You can run the
kubectl get events --all-namespacescommand to check for new events in the cluster. -
The kube-eventer or node-problem-detector parameters are incorrect.
-
If you use an ACK cluster, perform the following steps:
-
Log on to the ACK console.
-
On the Clusters page, click the name of your target cluster.
-
In the navigation pane on the left, choose .
-
On the Helm page, click Update for Update.
-
Check and modify the parameters as described in Step 1: Deploy kube-eventer and node-problem-detector.
-
-
For a self-managed Kubernetes cluster, follow Collect Kubernetes events for parameter configuration.
-
View container logs for an event
-
If you use an ACK cluster, perform the following steps:
-
Log on to the ACK console.
-
On the Clusters page, click the name of your target cluster.
-
In the navigation pane on the left, choose .
-
Select Namespace from the Namespace drop-down list.
-
On the Pods page, click Log in the row of the target pod.
-
-
For a self-managed Kubernetes cluster, check the logs of pods prefixed with
eventer-slsin the kube-system namespace.