An Out-of-the-Box: Centralized Audit Upgrade of Kubernetes Logs in Multiple Clusters

By Yili (Xiao Yi) from Alibaba Cloud Storage

Background

It has been a common practice for many users to use Log Service to collect Kubernetes container logs. Users can enable Log Service when creating a cluster to quickly collect container logs of a Kubernetes cluster, including the standard output of containers and text files in containers.

Please see Collect log data from containers by using Log Service for more information about how to use Kubernetes audit logs
Please see Create and use a K8s event center for more information about how to use the Kubernetes event center
Please see Analyze and monitor the access log of nginx-ingress for more information about how to use Ingress access logs

Limits

In scenarios where a user has multiple ACK clusters in multiple regions, if the users want to centrally audit and query the Kubernetes logs of multiple ACK clusters in each region, they need to perform additional operations.

Raw Mode

For example, users can manually create a data transformation job to deliver logs from each Logstore to the same destination database. However, this operation has the following limits:

Limit 1: The operation is cumbersome. There must be corresponding jobs established for each type of log of each ACK cluster in each region, such as event log, audit log, and Ingress log.
Limit 2: Automated updates cannot be implemented. When a new ACK cluster is created, the preceding operations have to be repeated, which is not real-time and automated.

Log Audit (before Upgrade)

Log Audit has introduced the Kubernetes log centralized audit function to solve the problems, which means that log audit will automatically perform a centralized audit and query on Kubernetes event logs, audit logs, and Ingress logs that meet the automation conditions.

However, the use of Log Audit (before upgrade) (even if the central account has upgraded the service-linked role AliyunServiceRoleForSLSAudit) to perform a centralized audit on Kubernetes logs still requires complicated separated authentication. The following three operations are required to perform a centralized audit and centralized query of Kubernetes logs:

Operation 1: The user needs to create a sls-audit-service-monitor role for the central account.
Operation 2: Authorize sls-audit-service-monitor role with AliyunLogAuditServiceMonitorAccess policy.
Operation 3: In addition, the user needs to authorize the role to perform operations on the Kubernetes project under ACK (see the following).

{
   "Version": "1",
   "Statement": [
       {
           "Action": "log:*",
           "Resource": [
               "acs:log:*:*:project/k8s-log-*"
           ],
           "Effect": "Allow"
       }
   ]
}

Log audit (before upgrade) solves the problems of limits 1 and 2, but due to the complexity of its separated custom authentication operations, it brings some inconvenience to users. Users cannot truly enjoy the out-of-the-box feature, centralized audit and centralized query. In addition, custom roles can be deleted and tampered with by users easily, which will affect the user experience.

The Real Out-of-the-Box

After Log Audit Upgrade

With the upgrade of Log Audit Central account authentication from the custom role sls-audit-service-monitor required by users to the one-click authorization of service-linked role AliyunServiceRoleForSLSAudit, Kubernetes log collection separated authentication under Log Audit is also on the agenda.

After the log audit has carried out in-depth cooperation with SLS data processing, it can automatically create and run data to process jobs and consume and write logs based on the role through the service-linked role AliyunServiceRoleForSLSAudit. This enables centralized audit, query, and storage of Kubernetes logs in multiple out-of-the-box clusters. Now, the user can enjoy such conveniences by clicking Authorize to authorize the service-linked role AliyunServiceRoleForSLSAudit at the first time of use.

Centralized Audit of Kubernetes Logs	Raw Mode	Log Audit Enabled Kubernetes (Before Upgrade)	Log Audit Enabled Kubernetes (After Upgrade)
Manually Create Related Jobs	Yes	None	None
Real-Time Automated Updates	No	Yes	Yes
Complex Custom Authentication	None	Yes	None

Example

The following example shows how to enable a centralized audit of Kubernetes logs in a multi-ACK cluster. Please see Cloud service resource coverage for more information.

An Example of Centralized Audit of Kubernetes Logs in Multiple Clusters

An Example of Centralized Query of Kubernetes Logs in Multiple Regions

Community

An Out-of-the-Box: Centralized Audit Upgrade of Kubernetes Logs in Multiple Clusters

Background

Limits

Raw Mode

Log Audit (before Upgrade)

The Real Out-of-the-Box

After Log Audit Upgrade

Example

Read previous post:

Read next post:

Alibaba Cloud Community

You may also like

Comments

Alibaba Cloud Community

Related Products

ACK One

Container Service for Kubernetes

Log Service

Log Management for AIOps Solution