Event monitoring

Background

ACK Serverless provides an out-of-the-box event monitoring solution for containerized environments. The kube-eventer component, maintained by the ACK team, provides this capability.

kube-eventer is an open source tool maintained by the ACK team for exporting Kubernetes events. It can send cluster events to systems such as DingTalk, SLS, and EventBridge, and provides multi-level filtering for real-time collection, targeted alerting, and asynchronous archiving. For more information, see kube-eventer.

This topic describes event monitoring through the following three use cases:

• Use case 1: Use Log Service to store Kubernetes events

• Use case 2: Use EventBridge to store Kubernetes events

• Use case 3: Use a DingTalk chatbot for Kubernetes monitoring and alerting

Use case 1: Store Kubernetes events with SLS

Alibaba Cloud Log Service (SLS) allows you to persistently store Kubernetes events for enhanced archiving and auditing. For more information, see Create and use the K8s Event Center.

1. Create a project and a Logstore.

   1. Log on to the Simple Log Service console.

   2. In the Projects section, click Create Project. Fill in the basic project information and click Create.

      This example creates a project named k8s-log4j in the same region as the Kubernetes cluster, China (Hangzhou).

      Note

      To reduce costs and improve efficiency, we recommend that you create your SLS project in the same region as your Kubernetes cluster. This allows log data to be transmitted over the internal network. This avoids data transfer fees for public network traffic and reduces latency for real-time log collection and querying.

   3. After creation, the k8s-log4j project appears in the project list. Click the project name to open the project details page.

   4. By default, you are on the Logstore page. In the left-side pane, click the plus sign (+) to open the Create Logstore dialog box.

   5. Enter the Logstore configuration information and click OK.

      This example creates a Logstore named k8s-logstore.

      

   6. After the Logstore is created, a prompt appears to guide you through data import. Click Data Import Wizard to open the Import Data dialog box.

   7. Select Log4j 1/2 and follow the on-screen instructions to configure it.

      This example uses the default settings. You can customize the configuration for your specific log data use case.

      

2. Configure log4j in the Kubernetes cluster.

   1. On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.

   2. On the Helm page, find the kube-eventer component. In the Actions column, click Update. Modify the following parameters and click OK.

      • Under npd, set enabled to false.

      • Set eventer.sinks.sls.enabled to true.

      • Set eventer.sinks.sls.logstore to the name of the Logstore that you created in Step 1.

      • Set eventer.sinks.sls.project to the name of the Project that you created in Step 1.

3. After you perform an operation on the cluster that generates an event, such as deleting a pod or creating an application, log on to the SLS console to view the collected data. For more information, see Consume log data by using an API.

   

4. Configure indexing and archiving. For more information, see Create an index.

   1. In the SLS console, go to the Projects section and click the name of your project.

   2. Click the icon next to the Logstore name and choose Search & Analysis.

   3. In the upper-right corner, click Enable Index.

   4. In the Search & Analysis dialog box, configure the index and click OK.

      The log query and analysis page appears.

      Note

      • The index configuration takes effect within one minute.

      • After you enable or modify an index, the new configuration applies only to data written after the change.

   5. (Optional) For offline archiving and computing, you can ship data from the Logstore to MaxCompute or OSS. For more information, see Create a MaxCompute shipping task (new) and Create an OSS shipping task (new).

Use case 2: Store Kubernetes events with EventBridge

EventBridge is a serverless event bus service provided by Alibaba Cloud. It allows Alibaba Cloud services, custom applications, and SaaS applications to connect through a standard, centralized hub. You can send ACK events to EventBridge to build loosely coupled, distributed, event-driven architectures. For more information about EventBridge, see What is EventBridge?.

1. Activate EventBridge. For more information, see Activate EventBridge and grant permissions.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.

3. On the Helm page, find the kube-eventer component. In the Actions column, click Update. Set eventer.sinks.eventbridge.enable to true to enable the EventBridge sink, and then click OK.

   eventer:
     accessKeyId: ""
     accessKeySecret: ""
     enabled: true
     env:
       - TZ: Asia/Shanghai
     image:
       pullPolicy: Always
       repository: registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/kube-eventer
       tag: v1.2.11-0620284-aliyun
     othersinks: null
     resources:
       limits:
         cpu: "1.0"
         memory: 1024Mi
       requests:
         cpu: 100m
         memory: 200Mi
     sinks:
       dingtalk:
         enabled: false
         label: ""
         level: Warning
         monitorkinds: Node,Pod
         monitornamespaces: ""
         token: ""
       eventbridge:
         enabled: false

4. After the data sink is enabled, view the container events in the EventBridge console.

   1. Log on to the EventBridge console.

   2. In the left-side navigation pane, click Event Buses.

   3. On the Event Buses page, click the name of the target event bus.

   4. In the left-side navigation pane, click Event Trace.

   5. Select an event query method, set the query conditions, and click Search.

   6. In the event list, find the event you want to view and click Event Details in the Actions column.

      For more information, see Query events.