All Products
Search
Document Center

Simple Log Service:Configure log collection policies

Last Updated:Jul 08, 2024

The Log Audit Service application allows you to collect logs from Alibaba Cloud services across multiple accounts and store the logs in a centralized manner. If the log audit feature is enabled for an Alibaba Cloud service, Simple Log Service collects all logs that meet specified conditions from the service by default. You can configure log collection policies to specify the accounts, regions, and instances from which logs are collected. This way, you can collect logs at a fine-grained level. This topic describes how to configure log collection policies.

Supported Alibaba Cloud services

You can configure log collection policies for ApsaraDB RDS, PolarDB-X 1.0, PolarDB, Server Load Balancer (SLB), Application Load Balancer (ALB), Virtual Private Cloud (VPC), Alibaba Cloud DNS (DNS), and Container Service for Kubernetes (ACK).

Alibaba Cloud service

Log source

Property

Description

ApsaraDB RDS

ApsaraDB RDS instance

account.id

The ID of the Alibaba Cloud account to which the ApsaraDB RDS instance belongs.

region

The ID of the region where the ApsaraDB RDS instance resides. Example: cn-shanghai.

instance.id

The ID of the ApsaraDB RDS instance.

instance.name

The name of the ApsaraDB RDS instance.

instance.db_type

The type of the databases that are created on the ApsaraDB RDS instance. Valid values: mysql, pgsql, and mssql.

instance.db_version

The version of the database engine. Example: 8.0.

tag.*

The custom tag.

You can replace the asterisk (*) in the tag.* property with a custom tag name.

PolarDB

PolarDB cluster

account.id

The ID of the Alibaba Cloud account to which the PolarDB cluster belongs.

region

The ID of the region where the PolarDB cluster resides. Example: cn-shanghai.

cluster.id

The ID of the PolarDB cluster.

cluster.name

The name of the PolarDB cluster.

cluster.db_type

The database type that is supported by the PolarDB cluster. Valid value: MySQL.

cluster.db_version

The version of the database engine. Valid values: 5.6, 5.7, and 8.0.

tag.*

The custom tag.

You can replace the asterisk (*) in the tag.* property with a custom tag name.

PolarDB-X 1.0

PolarDB-X 1.0 instance

account.id

The ID of the Alibaba Cloud account to which the PolarDB-X 1.0 instance belongs.

region

The ID of the region where the PolarDB-X 1.0 instance resides. Example: cn-shanghai.

instance.id

The ID of the PolarDB-X 1.0 instance.

instance.name

The name of the PolarDB-X 1.0 instance.

SLB

SLB instance

account.id

The ID of the Alibaba Cloud account to which the SLB instance belongs.

region

The ID of the region where the SLB instance resides. Example: cn-shanghai.

instance.id

The ID of the SLB instance.

instance.name

The name of the SLB instance.

instance.network_type

The network type of the SLB instance. Valid values: vpc and classic.

instance.vpc_id

The ID of the VPC where the SLB instance resides.

instance.address_type

The address type of the SLB instance. Valid values: intranet and internet.

tag.*

The custom tag.

You can replace the asterisk (*) in the tag.* property with a custom tag name.

ALB

ALB instance

account.id

The ID of the Alibaba Cloud account to which the ALB instance belongs.

region

The ID of the region where the ALB instance resides. Example: cn-shanghai.

instance.id

The ID of the ALB instance.

instance.name

The name of the ALB instance.

instance.vpc_id

The ID of the VPC where the ALB instance resides.

instance.address_type

The address type of the ALB instance. Valid values: Intranet and Internet.

tag.*

The custom tag.

You can replace the asterisk (*) in the tag.* property with a custom tag name.

Private DNS

VPC

account.id

The ID of the Alibaba Cloud account to which the VPC belongs.

region

The ID of the region where the VPC resides.

instance.id

The ID of the VPC.

instance.name

The name of the VPC.

tag.*

The custom tag.

You can replace the asterisk (*) in the tag.* property with a custom tag name.

Public DNS and Global Traffic Manager

Domain name

account.id

The Alibaba Cloud account ID of the domain name owner.

domain

The domain name.

ACK (Kubernetes audit log)

Kubernetes cluster

region

The ID of the region where the Kubernetes cluster resides. Example: cn-shanghai.

cluster.id

The ID of the Kubernetes cluster.

cluster.name

The name of the Kubernetes cluster.

cluster.type

The type of the Kubernetes cluster. Valid values: Kubernetes, ManagedKubernetes, and ASK.

cluster.network_mode

The network type of the Kubernetes cluster. Valid values: vpc and classic.

tag.*

The custom tag.

You can replace the asterisk (*) in the tag.* property with a custom tag name.

ACK (Kubernetes event center)

Kubernetes cluster

region

The ID of the region where the Kubernetes cluster resides. Example: cn-shanghai.

cluster.id

The ID of the Kubernetes cluster.

cluster.name

The name of the Kubernetes cluster.

cluster.type

The type of the Kubernetes cluster. Valid values: Kubernetes, ManagedKubernetes, and ASK.

cluster.network_mode

The network type of the Kubernetes cluster. Valid values: vpc and classic.

tag.*

The custom tag.

You can replace the asterisk (*) in the tag.* property with a custom tag name.

ACK (Ingress access log)

Kubernetes cluster

region

The ID of the region where the Kubernetes cluster resides. Example: cn-shanghai.

cluster.id

The ID of the Kubernetes cluster.

cluster.name

The name of the Kubernetes cluster.

cluster.type

The type of the Kubernetes cluster. Valid values: Kubernetes, ManagedKubernetes, and ASK.

cluster.network_mode

The network type of the Kubernetes cluster. Valid values: vpc and classic.

tag.*

The custom tag.

You can replace the asterisk (*) in the tag.* property with a custom tag name.

log.*

The content of the log.

Configure a log collection policy

  1. Log on to the Simple Log Service console.

  2. In the Log Application section, click the Audit & Security tab. Then, click Log Audit Service.

  3. Choose Access to Cloud Products > Global Configurations. In the upper-right corner of the page that appears, click Modify.

  4. Find the Alibaba Cloud service for which you want to configure a log collection policy and click Collection Policy.

  5. Configure a log collection policy.

    You can configure a log collection policy in basic edit mode or advanced edit mode. You can use the basic edit mode to configure a simple log collection policy. If the basic edit mode does not meet your business requirements, you can enable the advanced edit mode. In advanced edit mode, you can flexibly configure a complex log collection policy.

    Note
    • You can configure multiple policies based on your business scenario.

    • In advanced edit mode, you can edit policy statements. After you edit a policy statement, you cannot directly return to the basic edit mode.

    • To return to the basic edit mode, you must delete all policy statements and save the changes. Then, click Collection Policy.

    • Configure a log collection policy in basic edit mode.

      1. In the Add Policy section, configure the parameters and click Add Policy. The following table describes the parameters.

        Note

        If you turn on Default Collection Policy, the last line of the collection policy is accept "*"(Default Policy - Accept). If you turn off Default Collection Policy, the last line of the collection policy is drop "*"(Default Policy - Discard).

        采集策略-002

        Parameter

        Description

        Action

        The action that is performed when Simple Log Service collects logs based on the log collection policy. For more information, see Policy syntax.

        Properties

        The property of the log source. The available properties vary based on the log source that you use. For more information, see Supported Alibaba Cloud services.

        Operator

        The match mode that corresponds to an operator. If you select Exact Match, the operator is ==. For more information, see Policy syntax.

        Property value

        The value of the property. You can specify multiple values for a property.

      2. In the Added Policies section, confirm the details of the log collection policy that you configured.

        You can modify the policy and change the order of the policy.

        • To modify the policy, click Edit on the right side of the policy.

        • To change the order of the policy, click the upward or downward arrow on the right side of the policy.

        采集策略-003

      3. Confirm the settings and click OK.

    • Configure a log collection policy in advanced edit mode.

      1. Turn on Advanced Edit Mode.

      2. In the Rule field, configure a log collection policy and click OK.

        For information about the policy syntax, see Policy syntax.

        采集策略-001

  6. On the Global Configurations page, click OK.

Policy syntax

  • Actions

    • Keep: If the log source matches a policy, Simple Log Service attempts to match the log source against the next policy and determines whether to collect logs based on subsequent policies. If the log source does not match the policy, Simple Log Service does not collect logs and no longer attempts to match the log source against subsequent policies.

    • Drop: If the log source matches a policy, Simple Log Service does not collect logs and no longer attempts to match the log source against subsequent policies. If the log source does not match the policy, Simple Log Service attempts to match the log source against the next policy and determines whether to collect logs based on subsequent policies.

    • Accept: If the log source matches a policy, Simple Log Service collects logs and no longer attempts to match the log source against subsequent policies. If the log source does not match the policy, Simple Log Service attempts to match the log source against the next policy and determines whether to collect logs based on subsequent policies.

  • Matching modes

    Matching mode

    Description

    Exact match

    Exact match is performed based on strings.

    • Operator: ==.

    • Example: keep instance.db_type == "mysql". This policy evaluates to true for an ApsaraDB RDS for MySQL instance.

    Wildcard match

    Data is matched based on wildcard characters. The wildcard characters include asterisks (*) and question marks (?). An asterisk (*) specifies zero or multiple characters. A question mark (?) specifies one character.

    • Operator: ==.

    • Examples:

      • keep instance.name == "backend*". This policy evaluates to true for an instance whose name starts with backend.

      • keep instance.name == "active?". This policy evaluates to true for an instance whose name starts with active and a random character.

    Regex match

    Data is matched based on regular expressions.

    • Operator: ~=.

    • Example: keep instance.name ~= "^\d+$". This policy evaluates to true for an instance whose name contains only digits.

    Note

    By default, Simple Log Service performs partial match. To enable exact match, you must prefix a regular expression with a caret (^) and suffix the regular expression with a dollar sign ($).

    Numeric value comparison

    The comparison of numeric values.

    • Operators:

      • Operators for direct comparison: greater-than (>), greater-than-or-equal to (>=), equal-to (=), less-than-or-equal-to (<=), and less-than (<).

      • Operators used to compare numeric values within a closed interval. Example: : [*, 100]. You can use an asterisk (*) to specify an infinite interval.

    • Examples:

      • keep tag.level >= 2. This policy evaluates to true for an instance whose value of the tag.level property is greater than or equal to 2.

      • keep tag.level : [*, 10]. This policy evaluates to true for an instance whose value of the tag.level property is less than or equal to 10.

      • keep tag.level : [1, 10]. This policy evaluates to true for an instance whose value of the tag.level property is within the closed interval [1, 10].

    Logical operator

    • Keywords:

      • and, AND, and &&: The keywords are not case-sensitive.

      • or and OR: The keywords are not case-sensitive.

      • not, NOT, and exclamation point (!): The keywords are not case-sensitive.

    • Examples:

      • keep (tag.level > 10) and (region == "cn-shanghai"). This policy evaluates to true for an instance whose value of the tag.level property is greater than 10 and that resides in the China (Shanghai) region.

      • keep (tag.level > 10) or (region == "cn-shanghai"). This policy evaluates to true for an instance whose value of the tag.level property is greater than 10 or that resides in the China (Shanghai) region.

      • keep not region == "cn-shanghai". This policy evaluates to true for an instance that does not reside in the China (Shanghai) region.

    Global match

    If no property is specified in a log collection policy, the system matches log sources against all available properties for the policy. Examples:

    • keep "abc". This policy evaluates to true for logs that contain the abc string.

    • accept "*". This policy evaluates to true for all log sources.

    Note
    • If you use global match, you must enclose specified characters in double quotation marks ("").

    • Global match is available only in advanced edit mode.

  • Character escape

    If a log collection policy contains special characters such as asterisks (*) and backslashes (\), you must escape the special characters. Example: keep instance.name == "abc\*". This policy evaluates to true for an instance whose name is abc*.

Common scenarios

  • Collect the logs of instances that reside in specific regions

    In this example, only the logs of instances that reside in regions within the Chinese mainland are collected based on the configured collection policies.

    # only scan cn region
    keep region == "cn-*"
    
    # accept by default
    accept "*"
  • Collect the logs of instances that have specified tags

    In this example, only the logs of instances whose value of the type tag is production are collected based on the configured collection policies. The value production is not case-sensitive.

    # only scan "production" instances
    keep tag.type ~= "(?i)^production$"
    
    # accept by default
    accept "*"
  • Complex scenarios

    If the level: high tag is used in log collection policies, the logs of ApsaraDB RDS for MySQL instances, ApsaraDB RDS for SQL Server instances, and ApsaraDB RDS for PostgreSQL instances are collected. If the level: high tag is not used, only the logs of ApsaraDB RDS for MySQL instances are collected. The following code shows the log collection policies that are involved:

    # accept all high level instances
    accept tag.level == "high"
    
    # only scan mysql
    keep instance.db_type == "mysql"
    
    # accept by default
    accept "*"