Alibaba Cloud Service Mesh:KubeAPI operation audit

Step 1: Enable KubeAPI operation audit

Step 2: View KubeAPI audit reports

1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Mesh Security Center > KubeAPI Operation Audit.

3. On the Mesh Audit page, click the API Access Overview or Operation Details tab to view report details.

   • API Access Overview: View detailed information about the KubeAPI access behavior of the mesh instance.

     The API Access Overview page includes the Access source distribution pie chart, the Unauthorized access attempts panel, and the Public network access list table. The Public network access list includes columns such as Source IP address, Access count, Failure rate %, Unauthorized access attempts, High-risk IP, Country, Province, City, and ISP.

   • Operation Details: View a detailed list of operations for a specified resource in the mesh instance.

     Select or enter a resource type to perform a real-time query. You can view the total count of various events, namespace distribution, time-series trends, and a detailed operation list. The top of the Resource operations tab provides filters such as Resource type, Namespace, and Status code. The overview area displays the number of operators, namespaces, operation methods, and the success rate. Below this area, you can find panels such as the Operation trace timeline, Create event namespace distribution, Update event namespace distribution, Access event namespace distribution, and Delete event namespace distribution.

Related operations

Detailed log records

To run custom queries and analyze audit logs, go to the Simple Log Service console to view detailed log records.

1. Log on to the Simple Log Service console.

2. In the Projects section, click the Project named mesh-log-${Mesh-ID}.

3. Select the Logstore named audit-${Mesh-ID} and click Search & Analyze to view the corresponding audit logs.

   Note

   • When you enable mesh audit, a Logstore named audit-${Mesh-ID} is automatically added to the generated Project.

   • The Logstore for audit logs has an index configured by default. Do not modify the index. Otherwise, reports may not work correctly.

   • By default, Simple Log Service stores API Server audit logs from ASM for 30 days. To change the default retention period, see Manage a Logstore.

   Common ways to search audit logs include the following:

   • To query the operations performed by a specific RAM user, enter the RAM user ID in the search box and click Search & Analyze.

   • To query the operations on a specific resource, enter the resource name in the search box and click Search & Analyze.

   • To filter out operations performed by system components, enter NOT user.username: node NOT user.username: serviceaccount NOT user.username: apiserver NOT user.username: kube-scheduler NOT user.username: kube-controller-manager in the search box and click Search & Analyze.

   For more information about query and analysis methods, see Query overview.

   Alerts

   To receive real-time alerts for operations on specific resources, use the alert feature of Simple Log Service. Supported notification channels include SMS, DingTalk chatbots, email, custom webhooks, and Notification Center. For more information, see Alerts.

   You can also use the query statements from the audit reports to query audit logs:

   • Example 1: Alert on container command execution

     A company has strict restrictions on the use of its mesh instance and does not allow users to log on to containers or execute commands. If a user executes a command, an alert must be triggered immediately. The alert should include details such as the specific container, the executed command, the operator, the event ID, the time, and the source IP address.

     • Query statement:

       verb : create and objectRef.subresource:exec and stage:  ResponseStarted | SELECT
         auditID as "event_id",
         date_format(from_unixtime(__time__), '%Y-%m-%d %T') as "event_time",
         regexp_extract("requestURI", '([^\?]*)/exec\?.*', 1) as "resource",
         regexp_extract("requestURI", '\?(.*)', 1) as "command",
         "responseStatus.code" as "status_code",
         CASE
           WHEN "user.username" != 'kubernetes-admin' THEN "user.username"
           WHEN "user.username" = 'kubernetes-admin' AND regexp_like("annotations.authorization.k8s.io/reason", 'RoleBinding')
             THEN regexp_extract("annotations.authorization.k8s.io/reason", ' to User "(\w+)"', 1)
           ELSE 'kubernetes-admin'
         END as "operator",
         CASE
           WHEN json_array_length(sourceIPs) = 1 THEN json_format(json_array_get(sourceIPs, 0))
           ELSE sourceIPs
         END as "source_ip"
       LIMIT 100

     • Conditional expression:

       operation_event =~ ".*"

   • Example 2: Alert on public access failures

     A mesh instance has public access enabled. To prevent malicious attacks, you need to monitor the number of public access attempts and the failure rate. If the number of access attempts from an IP address reaches a threshold (for example, 10) and the failure rate exceeds a threshold (for example, 50%), an alert must be triggered immediately. The alert should include the geographic region of the user's IP address, the source IP address, and whether it is a high-risk IP address.

     • Query statement:

       * | SELECT
           ip                                           AS "source_ip",
           total                                        AS "request_count",
           round(rate * 100, 2)                         AS "failure_rate_pct",
           failCount                                    AS "forbidden_count",
           CASE WHEN security_check_ip(ip) = 1 THEN 'yes' ELSE 'no' END
                                                        AS "is_high_risk_ip",
           ip_to_country(ip)                            AS "country",
           ip_to_province(ip)                           AS "province",
           ip_to_city(ip)                               AS "city",
           ip_to_provider(ip)                           AS "isp"
       FROM (
           SELECT
               CASE
                   WHEN json_array_length(sourceIPs) = 1
                       THEN json_format(json_array_get(sourceIPs, 0))
                   ELSE sourceIPs
               END                                       AS ip,
               count(1)                                  AS total,
               sum(CASE WHEN "responseStatus.code" < 400 THEN 0 ELSE 1 END) * 1.0
                   / count(1)                            AS rate,
               count_if("responseStatus.code" = 403)     AS failCount
           FROM log
           GROUP BY ip
           LIMIT 10000
       )
       WHERE ip_to_domain(ip) != 'intranet'
       HAVING "request_count" > 10
          AND "failure_rate_pct" > 50
       ORDER BY "request_count" DESC
       LIMIT 100

     • Conditional expression:

       source_ip =~ ".*"

   Recreating mesh audit

   If you accidentally delete the SLS Project for mesh audit, you must recreate it to continue using the feature.

   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Mesh Security Center > KubeAPI Operation Audit.

   3. On the Mesh Audit page, in the Rebuild Mesh Audit dialog box, click Recreate.

      The name of the recreated Project is the same as the previous Project name with a timestamp appended.

Related documents

• To send timely alert notifications to contacts when important resources change, see Configure audit alerts for operations on mesh resources.

• You can configure workload identity, peer authentication, request authentication, and authorization policies in the mesh to manage mesh resources with more granularity and improve mesh security. For more information, see Overview of zero trust security.