Simple Log Service:Host text log collection

If you log on with an Alibaba Cloud account, you have all operation permissions by default and can directly perform related operations.

If you log on with a RAM user, contact the Alibaba Cloud account owner to grant permissions by using system policies or custom policies:

• System policies: These are predefined policies created and maintained by Simple Log Service. You can directly use them but cannot modify their content.

  • AliyunLogFullAccess: This policy grants the permissions to manage Simple Log Service resources.

  • AliyunLogReadOnlyAccess: This policy grants the read-only permissions on all Simple Log Service resources.

• Custom policies: If you have high data security requirements, you can create custom policies to implement fine-grained permission management, such as:

  Replace ${regionName}, ${uid}, ${projectName}, and ${logstoreName} in the Resource field with your region name, Alibaba Cloud account ID, project name to access, and Logstore name to query.

  • To manage Logstores and collection configurations in existing projects of the Alibaba Cloud account and view collected logs, the Alibaba Cloud account needs to create and bind a custom policy with the least privilege as follows:

    {
      "Version": "1",
      "Statement": [
        {
        # Read-only access to Project
          "Effect": "Allow",
          "Action": [
            "log:ListProject",
            "log:GetAcceleration",
            "log:ListDomains",
            "log:GetLogging",
            "log:ListTagResources"
          ],
          "Resource": "acs:log:${regionName}:${uid}:project/*"
        },
        {
          "Effect": "Allow",
          "Action": "log:GetProject",
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}"
        },
        {
        # Manage Logstore
          "Effect": "Allow",
          "Action": [
            "log:ListLogStores",
            "log:*LogStore",
            "log:*Index",
            "log:ListShards",
            "log:GetCursorOrData",
            "log:GetLogStoreHistogram",
            "log:GetLogStoreContextLogs"
          ],
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}/*"
        },
        {
        # Manage LoongCollector (Logtail) data ingestion
          "Effect": "Allow",
          "Action": "log:*",
          "Resource": [
            "acs:log:${regionName}:${uid}:project/${projectName}/logtailconfig/*",
            "acs:log:${regionName}:${uid}:project/${projectName}/machinegroup/*"
          ]
        },
        {
        # Query saved searches
          "Effect": "Allow",
          "Action": "log:ListSavedSearch",
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}/savedsearch/*"
        },
        {
        # Query dashboards
          "Effect": "Allow",
          "Action": "log:ListDashboard",
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}/dashboard/*"
        }
        # Query logs from a specific Logstore
        {
          "Effect": "Allow",
          "Action": "log:GetLogStoreLogs",
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}/logstore/${logstoreName}"
        }
      ]
    }

  • To configure collection for existing Logstores of the Alibaba Cloud account, the Alibaba Cloud account needs to create and bind a custom policy with the least privilege as follows:

    {
      "Version": "1",
      "Statement": [
        {
        # Read-only access to Project
          "Effect": "Allow",
          "Action": [
            "log:ListProject",
            "log:GetAcceleration",
            "log:ListDomains",
            "log:GetLogging",
            "log:ListTagResources"
          ],
          "Resource": "acs:log:${regionName}:${uid}:project/*"
        },
        {
          "Effect": "Allow",
          "Action": "log:GetProject",
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}"
        },
        {
        # Read-only access to Logstore
          "Effect": "Allow",
          "Action": [
            "log:ListLogStores",
            "log:GetLogStore",
            "log:GetLogStoreHistogram",
            "log:GetIndex",
            "log:CreateIndex",
            "log:UpdateIndex",
            "log:ListShards",
            "log:GetCursorOrData",
            "log:GetLogStoreContextLogs"
          ],
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}/*"
        },
        {
        # Manage LoongCollector (Logtail) data ingestion
          "Effect": "Allow",
          "Action": "log:*",
          "Resource": [
            "acs:log:${regionName}:${uid}:project/${projectName}/logtailconfig/*",
            "acs:log:${regionName}:${uid}:project/${projectName}/machinegroup/*"
          ]
        },
        {
        # Query saved searches
          "Effect": "Allow",
          "Action": "log:ListSavedSearch",
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}/savedsearch/*"
        },
        {
        # Query dashboards
          "Effect": "Allow",
          "Action": "log:ListDashboard",
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}/dashboard/*"
        }
         # Query logs from a specific Logstore
        {
          "Effect": "Allow",
          "Action": "log:GetLogStoreLogs",
          "Resource": "acs:log:${regionName}:${uid}:project/${projectName}/logstore/${logstoreName}"
        }
      ]
    }

If you do not have an available project, follow these steps to create a basic project. For more information, see Manage projects.

Log on to the Simple Log Service console, click Create Project, and complete the following basic configuration. You can keep other settings at their default values:

• Region: Select an appropriate Alibaba Cloud region based on the log source and other information. After a project is created, its region cannot be changed.

• Project Name: Set a name that is unique within the Alibaba Cloud region. After a project is created, its name cannot be changed.

If you do not have an available Logstore, follow these steps to create a basic Logstore. For more information, see Manage Logstores.

1. Log on to the Simple Log Service console, and click the project that manages your log resources in the Project List.

2. On the Storage > Logstores tab, click the + icon.

   

3. Enter a Logstore Name and keep other settings at their default values.

LoongCollector is a new generation log collection agent launched by Alibaba Cloud Simple Log Service (SLS), serving as an upgraded version of Logtail.

When your ECS instance and the Simple Log Service project are in the same account and same region, choose Automatic Installation.

Choose Manual Installation in any of the following situations:

• The ECS instance and the Simple Log Service project are in the same account but different regions.

• The ECS instance and the Simple Log Service project belong to different accounts.

• The server is from another cloud provider or is self-managed.

Simple Log Service defaults to single-line mode, which splits and stores logs by line. This causes multi-line logs containing stack information to be split line by line, with each line stored and displayed as an independent log, which is not conducive to analysis.

To address this issue, you can enable multi-line mode to change how Simple Log Service splits logs, and configure a regular expression to match the starting line of logs. This allows the original logs to be split and stored according to the starting line rules. Example:

Original log:

[2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened
    at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
    at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
    at TestPrintStackTrace.main(TestPrintStackTrace.java:16)

Comparison between single-line mode and multi-line mode:

content:[2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened
    at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
    at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
    at TestPrintStackTrace.main(TestPrintStackTrace.java:16)

Configure multi-line mode on the Logtail Configuration page. After configuration, click Save:

• Processing Configuration: Enable Multi-line Mode.

• Type: Select Custom or Multi-line JSON.

  • Custom: The format of the original logs is not fixed, and you need to configure a First Line Regular Expression to identify the starting line of each log.

    • First Line Regular Expression: Supports automatic generation or manual input. The regular expression needs to match a complete line of data. For the example above, the matching regular expression is \[\d+-\d+-\w+:\d+:\d+,\d+]\s\[\w+]\s.*.

      • Click Auto Generate Regular Expression, then in the Sample Log text box, select the log content you want to extract, and click Generate.

      • Click Manually Enter Regular Expression, enter the regular expression. After configuration, click Validate.

  • Multi-line JSON: Select this when the original logs are all in standard JSON format. LoongCollector (Logtail) will automatically handle line breaks within a single JSON log.

• Processing Method If Splitting Fails:

  • Discard: Directly discard this log segment.

  • Retain Single Line: Retain each line of log text as a separate log.

Configure the following settings on the Logtail Configuration page. After configuration, click Save:

• Global Configuration > Other Global Configurations: Configure the log topic type.

  • Log Topic Type: Select the method to generate log topics.

    • Machine Group Topic: Simple Log Service supports applying one LoongCollector (Logtail) configuration to multiple machine groups. You can use Machine Group Topic to distinguish logs from different machine groups. When LoongCollector (Logtail) reports data, it uploads the Machine Group Topic of the server's machine group as the log topic to the project. You need to specify the log topic as a query condition when querying logs.

    • File Path Extraction: If different users or applications store logs in different top-level directories, but the subdirectories and log file names are the same, Simple Log Service cannot clearly distinguish which user or application generated the logs during log collection. In this case, the File Path Extraction method can be used to distinguish logs generated by different users or applications. You can use a regular expression to fully match the file path and upload the result of the expression match (user name or application name) as the log topic to Simple Log Service.

      Use regular expressions to extract topics from file paths

      Note

      You must escape the forward slash (/) in a regular expression that is used to match a log file path.

      Scenario 1: Different users record logs in different directories, but the log file names are the same. The directory paths are as follows.

      /data/logs
      ├── userA
      │   └── serviceA
      │       └── service.log
      ├── userB
      │   └── serviceA
      │       └── service.log
      └── userC
          └── serviceA
              └── service.log

      If you only configure the file path as /data/logs and the file name as service.log in the Logtail Configuration, LoongCollector (Logtail) will collect the content of all three service.log files to the same Logstore, making it impossible to distinguish which user generated the logs. You can use a regular expression to extract values from the file path to generate different log topics.

      Regular expression	Extraction result
      \/data\/logs\/(.*)\/serviceA\/.*	__topic__: userA __topic__: userB __topic__: userC

      Scenario 2: If a single log topic is not sufficient to distinguish the source of logs, you can configure multiple regular capturing groups in the log file path to extract key information. Capturing groups include named capturing groups (?P<name>) and unnamed capturing groups.

      • If you use named capturing groups, the generated tag field is __tag__:{name};

      • If you use unnamed capturing groups, the generated tag field is __tag__:__topic_{i}__, where {i} is the sequence number of the capturing group.

      Note

      When there are multiple capturing groups in the regular expression, the __topic__ field will not be generated.

      For example, if the file path is /data/logs/userA/serviceA/service.log, you can extract multiple values from the file path using the following methods:

      Example	Regular expression	Extraction result
      Use unnamed capturing groups for regular extraction.	\/data\/logs\/(.*?)\/(.*?)\/service.log	__tag__:__topic_1__: userA __tag__:__topic_2__: serviceA
      Use named capturing groups for regular extraction.	\/data\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/service.log	__tag__:user: userA __tag__:service: serviceA

      Verification: After configuration, query logs by log topic: On the log query and analysis page, enter the corresponding generated log topic, such as __topic__: userA or __tag__:__topic_1__: userA, to query logs with the respective topic. For more information, see Query syntax and features.

      

    • Custom: Enter customized:// + custom topic name to use a custom static log topic.

Configure the following settings on the Logtail Configuration page:

• Input Configuration: Enable Allow File To Be Collected For Multiple Times. By default, a log file can only match one LoongCollector (Logtail) configuration. When enabled, the same file can be collected by multiple LoongCollector (Logtail) configurations.

Configure the following settings on the Logtail Configuration page:

When a blacklist is in use, computational overhead is generated. We recommend that you add up to 10 entries to the blacklist.

• Input Configuration > Other Input Configurations: Enable Collection Blacklist and select the appropriate blacklist method for configuration.

  You can specify exact directories and file names. You can also use wildcard characters to specify directories and file names. When you configure this parameter, you can use only asterisks (*) or question marks (?) as wildcard characters.

  • File Path Blacklist: Configure file paths to be ignored during collection. Examples:

    • /home/admin/private*.log: Ignore all files in the /home/admin/ directory that start with private and end with .log.

    • /home/admin/private*/*_inner.log: Ignore all files in directories under /home/admin/ that start with private and end with _inner.log.

  • File Blacklist: Configure file names to be ignored during collection. Example:

    • app_inner.log: Ignore all files named app_inner.log during collection.

  • Directory Blacklist: Directory paths cannot end with a forward slash (/). Examples of directory paths:

    • /home/admin/dir1/: The directory blacklist will not take effect.

    • /home/admin/dir*: Ignore all files in subdirectories that start with dir under the /home/admin/ directory during collection.

    • /home/admin/*/dir: Ignore all files in second-level subdirectories named dir under the /home/admin/ directory during collection. For example, files in the /home/admin/a/dir directory are ignored, while files in the /home/admin/a/b/dir directory are collected.

Example:

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

On the Logtail Configuration page, click Add Processing Plugin in the Processing Configuration tab, select Native Processing Plugin > Data Masking, and configure the following settings:

• Original Field: The original field that stores the log content before parsing.

• Masking Method: Supports const and md5

  • const: Replace sensitive content with the string you specify.

  • md5: Replace sensitive content with its corresponding MD5 value.

• Replacement String: When Masking Method is set to const, you need to enter a string to replace the sensitive content.

• Content Expression Before Replacement: Used to find sensitive content. Configure using RE2 syntax.

• Content Expression To Be Replaced: The expression for sensitive content. Configure using RE2 syntax.

On the Logtail Configuration page, click Add Processing Plugin in the Processing Configuration tab, select Native Processing Plugin > Native Plugin: Data Filtering, and configure the whitelist: After configuration, only logs that meet the whitelist conditions will be collected.

• Field Name: The log field to be filtered.

• Field Value: The regular expression used for filtering. Only full-text matching is supported, not partial keyword matching.

On the Logtail Configuration page, click Add Processing Plugin in the Processing Configuration tab, select Native Processing Plugin > Time Parsing, and configure the following settings:

The Time Parsing plugin is typically used in combination with other plugins to parse time fields extracted from the original log and write the parsing results to the _time_ field.

• Original Field: The original field that stores the log content before parsing.

• Time Format: Set the corresponding time format based on the time content in the log.

• Time Zone: Select the time zone of the log time field. If you do not select this parameter, the time zone of the environment where the Logtail process resides is used by default.

Configure the following settings on the Logtail Configuration page:

• Output Configuration: Specify the Compression Method for data transmission.

  Note

  When configuring Logtail through the SDK, if this field is not specified, the default compression method depends on the Logtail version:

  • For Logtail 1.3.4 and earlier versions, the default is lz4.

  • For versions after Logtail 1.3.4, the default is zstd.

  When configuring LoongCollector through the SDK, if this field is not specified, the default compression method is zstd.

  • lz4: Fast compression speed, relatively low compression ratio.

  • zstd: High compression ratio, slightly lower speed, high memory usage.