Simple Log Service:Batch-collect server text logs

Procedure:

1. On the Logstores page, click next to the target logstore name to expand it.

2. Click Data Collection > Logtail Configurations. On the One-time Logtail Configuration tab, click Add Logtail Configuration.

3. In the Quick Data Import dialog box, click Integrate Now on the One-Time File Collection - Host card.

4. On the Machine Group Configurations page, configure the following parameters:

   • Scenario: Servers

   • Installation Environment: ECS

   • Select Machine Group: Select an action based on the LoongCollector and machine group status on the target server:

     • If LoongCollector is installed and has been added to a machine group, select it from the Source Machine Group list and add it to the Applied Machine Group list. You do not need to create a new one.

     • If LoongCollector is not installed, click Create Machine Group:

       The following steps describe the one-click process to install LoongCollector and create a machine group.

       1. The system automatically lists the ECS instances that are in the same region as the Project. Select one or more instances from which you want to collect logs.

       2. Click Install and Create Machine Group. The system automatically installs LoongCollector on the selected ECS instances.

       3. Enter a Name for the machine group and click OK.

       Note

       If the installation fails or remains in a pending state, verify that the ECS instance is in the same region as the Project.

     • To add a server with LoongCollector already installed to an existing machine group, see How do I add a server to an existing machine group?

5. Check heartbeat status: Click Next. When the Heartbeat Status of Machine Group section appears, check the Heartbeat. If the status is OK, the machine group is connected properly. Click Next to proceed to the Logtail configuration page.

   If the status is FAIL, establishing the initial heartbeat may take some time. Wait about two minutes and then refresh the heartbeat status. If the status is still FAIL after refreshing, see Machine group heartbeat connection fails for further troubleshooting.

Global Configurations:

• Configuration Name: Specify a custom name for the collection configuration. The name must be unique within the Project and cannot be modified after creation. The naming rules are as follows:

  • Can only contain lowercase letters, digits, hyphens (-), and underscores (_).

  • Must start and end with a lowercase letter or a digit.

• Execution Timeout: The default is 600s (10 minutes), and the timeout can be set from 600 to 604,800 seconds (10 minutes to 7 days). If a collection task exceeds this timeout, it is forcibly stopped, and any remaining data is not collected.

  Important

  Note on configuration updates: When you update a configuration, its validity period is reset. To prevent duplicate task execution or unexpected data reporting, ensure that the machine group scope is correct and that the previous task execution time did not exceed the Execution Timeout.

• Force rerun on update: Off by default.

  • Off:

    • When you update collection configuration parameters other than Execution Timeout or Input Configurations, the system resumes the current collection progress instead of restarting the task. This ensures uninterrupted collection.

    • If you modify the Execution Timeout or Input Configurations, the system reruns the collection task.

  • On: When you update the collection configuration, the task forcibly restarts. This ensures all data is processed and reported according to the latest settings. Note: Data collected before the update is not deleted. To clear previously collected data, see Soft deletion in Simple Log Service.

Input Configurations:

• Type: One-time File Collection (LoongCollector 3.3+ required).

• File Path: The path from which logs are collected.

  Note

  LoongCollector determines the list of files to collect and their sizes when it retrieves the configuration. It does not collect new files or content appended to existing files after this point.

  • Linux: The path must start with a forward slash (/). For example, /data/mylogs/**/*.log collects all files with the .log extension in all subdirectories of /data/mylogs.

  • Windows: The path must start with a drive letter. For example, C:\Program Files\Intel\**\*.Log.

• Maximum Directory Monitoring Depth: The maximum directory depth that the wildcard ** in the File Path can match. The default value is 0, which means that only the current directory is monitored.

Scenario 2: Processing structured logs

192.168.*.* - - [15/Apr/2025:16:40:00 +0800] "GET /nginx-logo.png HTTP/1.1" 0.000 514 200 368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.*.* Safari/537.36"

body_bytes_sent: 368
http_referer: -
http_user_agent : Mozi11a/5.0 (Nindows NT 10.0; Win64; x64) AppleMebKit/537.36 (KHTML, like Gecko) Chrome/131.0.x.x Safari/537.36
remote_addr:192.168.*.*
remote_user: -
request_length: 514
request_method: GET
request_time: 0.000
request_uri: /nginx-logo.png
status: 200
time_local: 15/Apr/2025:16:40:00

Procedure: In the Processor Configurations area of the Logtail Configurations page:

1. Add a processor plugin: Click Add Processor and configure a plugin such as regex, delimiter, or JSON parsing based on your log format. This example uses an NGINX log and selects Native Processor > Data Parsing (NGINX Mode).

2. NGINX Log Configuration: Copy the entire log_format definition from your NGINX server's configuration file (nginx.conf) and paste it into this text box.

   Example:

   log_format main  '$remote_addr - $remote_user [$time_local] "$request" ''$request_time $request_length ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent"';

   Important

   The format definition here must exactly match the format that is used to generate the log on your server. Otherwise, parsing fails.

3. Common parameters: The following parameters are common across multiple data processor plugins and serve the same purpose.

   • Original field: Specifies the source field to parse. The default is content, which is the entire collected log entry.

   • Retain original field if parsing fails: Recommended. If the plugin fails to parse a log, for example, due to a format mismatch, this option retains the original log content in the specified original field.

   • Retain original field if parsing succeeds: This option retains the original log content even after successful parsing.

Procedure: In the Global Configurations > Other Global Configurations > Log Topic Type area, select a topic generation method. The following types are supported:

• Machine group topic: When you apply a collection configuration to multiple machine groups, LoongCollector automatically uses the server's machine group name as the value for the __topic__ field. This is useful for categorizing logs by host.

• Custom: Use the format customized://<custom-topic-name>, for example, customized://app-login. This method is suitable for static topics with fixed business identifiers.

• File path extraction: Extracts key information from the full path of a log file to dynamically tag the log source. This is useful when multiple users or applications share the same log filename but reside in different paths. For example, when multiple users or services write logs to different top-level directories but use the same subdirectory path and filename, you cannot distinguish the source by filename alone:

  /data/logs
  ├── userA
  │   └── serviceA
  │       └── service.log
  ├── userB
  │   └── serviceA
  │       └── service.log
  └── userC
      └── serviceA
          └── service.log

  In this case, you can configure file path extraction and use a regular expression to extract key information from the full path. The matched result is then uploaded to the Logstore as the log topic.

  File path extraction rules

  When you configure the regular expression, the system automatically determines the output field format based on the number and names of the capture groups. The rules are as follows:

  In a regular expression for a file path, you must escape forward slashes (/).

  Capture group type	Use case	Generated field	Regex example	Sample matched path	Sample generated field
  Single capture group (only one (.*?))	When you need only one dimension to distinguish sources, such as by username or environment.	Generates the __topic__ field.	\/logs\/(.*?)\/app\.log	/logs/userA/app.log	__topic__: userA
  Multiple Capture Groups - Unnamed (multiple (.*?))	When you need multiple dimensions to distinguish sources but do not require semantic labels.	Generates the tag field __tag__:__topic_{i}__, where {i} is the index of the capture group.	\/logs\/(.*?)\/(.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:__topic_1__:userA; __tag__:__topic_2__:svcA
  Named multiple capture groups (using (?P<name>.*？)	When you need multiple dimensions to distinguish sources and want clear, meaningful field names for easy querying and analysis.	Generate the tag field __tag__:{name}	\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:user:userA; __tag__:service:svcA

Adding a server to a machine group

To add a new server, such as a newly deployed ECS instance or a self-managed server, to an existing machine group, follow these steps to associate it with the group and apply its collection configuration.

Prerequisites

• An existing machine group.

• LoongCollector (Logtail) is installed on the new server.

Procedure

1. View the target machine group's identifier.

   1. In the target Project, click Resources > Machine Groups in the left-side navigation pane.

   2. On the Machine Groups page, click the name of the target machine group.

   3. On the machine group configuration page, view the machine group identifier.

2. Perform one of the following actions based on the identifier type.

   Note

   A single machine group cannot contain both Linux and Windows servers. Do not configure the same user-defined identifier on both Linux and Windows servers. To configure multiple user-defined identifiers on a single server, separate them with a line break.

   • Type 1: The machine group identifier is an IP address

     1. On the server, run the following command to open the app_info.json file and view the ip value.

        cat /usr/local/ilogtail/app_info.json

     2. On the configuration page of the target machine group, click Modify and enter the IP address of the server. If you have multiple IP addresses, separate them with line breaks.

     3. After you complete the configuration, click Save and confirm the heartbeat status. If the status is OK, the server automatically applies the machine group's collection configuration.

        If the heartbeat status is FAIL, see Machine group heartbeat is FAIL for troubleshooting instructions.

   • Type 2: The machine group identifier is a user-defined identifier

     Based on the operating system, write the user-defined identifier string that matches the target machine group to the specified file:

     If the directory does not exist, create it manually. The file path and name are fixed by Log Service and cannot be customized.

     • Linux: Write a custom string to the /etc/ilogtail/user_defined_id file.

     • Windows: Write a custom string to C:\LogtailData\user_defined_id.

Extract log fields using a regular expression and parse the log into key-value pairs. Each field can be independently queried and analyzed.

127.0.0.1 - - [16/Aug/2024:14:37:52 +0800] "GET /wp-admin/admin-ajax.php?action=rest-nonce HTTP/1.1" 200 41 "http://www.example.com/wp-admin/post-new.php?post_type=page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0"

body_bytes_sent: 41
http_referer: http://www.example.com/wp-admin/post-new.php?post_type=page
http_user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; ×64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0
remote_addr: 127.0.0.1
remote_user: -
request_method: GET
request_protocol: HTTP/1.1
request_uri: /wp-admin/admin-ajax.php?action=rest-nonce
status: 200
time_local: 16/Aug/2024:14:37:52 +0800

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (Regex Mode):

• Regular Expression: The expression used to match logs. Generate it automatically or enter it manually:

  • Automatic generation:

    • Click Generate.

    • In the Log Sample, select the log content to extract.

    • Click Generate Regular Expression.

      

  • Manual entry: Manually Enter Regular Expression based on the log format.

  After configuration, click Validate to test whether the regular expression can correctly parse the log content.

• Extracted Field: The field name (Key) that corresponds to the extracted log content (Value).

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Structure log content using a separator to parse it into multiple key-value pairs. Both single-character and multi-character separators are supported.

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (Delimiter Mode):

• Delimiter: Specifies the character used to split log content.

  Example: For a CSV file, select Custom and enter a comma (,).

• Quote: If a field value contains the separator, you must enclose the field value in quotes to prevent incorrect splitting.

• Extracted Field: Specify the field name (Key) for each column in the order that they appear. The rules are as follows:

  • Field names can contain only letters, digits, and underscores (_).

  • Must start with a letter or an underscore (_).

  • Maximum length: 128 bytes.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Structure an Object-type JSON log by parsing it into key-value pairs.

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (JSON Mode):

• Original Field: The field that contains the raw log to be parsed. The default value is content.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Parse a nested JSON log into key-value pairs by specifying the expansion depth.

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Extended Processor > Expand JSON Field:

• Original Field: Specifies the name of the source field to expand, such as content.

• JSON Expansion Depth: The expansion depth of the JSON object, where 0 (the default) indicates full expansion, 1 indicates expansion of the current level, and so on.

• Character to Concatenate Expanded Keys: The separator for field names when a JSON object is expanded. The default value is an underscore (_).

• Name Prefix of Expanded Keys: The prefix for field names after JSON expansion.

• Expand Array: Expands an array into key-value pairs with indexes.

  Example: {"k":["a","b"]} is expanded to {"k[0]":"a","k[1]":"b"}.

  To rename the expanded fields (for example, from prefix_s_key_k1 to new_field_name), add a rename fields plugin afterward to complete the mapping.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Use the json_extract function to extract JSON objects from a JSON array.

Example:

[{"key1":"value1"},{"key2":"value2"}]

json1:{"key1":"value1"}
json2:{"key2":"value2"}

Procedure: In the Processor Configurations section of the Logtail Configuration page, switch the Processing Mode to SPL, configure the SPL Statement, and use the json_extract function to extract JSON objects from the JSON array.

Example: Extract elements from the JSON array in the log field content and store the results in new fields json1 and json2.

* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

c-ip: cs-username
cs-bytes: sc-substatus
cs-method: cs-method
cs-uri-query: cs-uri-query
cs-uri-stem: cs-uri-stem
cs-username: s-port
date: #Fields:
s-computername: s-sitename
s-ip: s-ip
s-sitename: time
sc-bytes: sc-status
sc-status: c-ip
sc-win32-status: cs (User-Agent)
time: date
time-taken: sc-win32-status

• For information about other parameters, see the general parameter descriptions in Scenario 2: Structured Logs.

Data masking

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Parse the time field in the log and set the parsing result as the log's __time__ field.

Example:

{"level":"INFO","timestamp":"2025-09-23T19:11:47+0800","cluster":"yilu-cluster-0728","message":"User logged in successfully","userId":"user-123"}

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Time Parsing:

• Original Field: The field that contains the log content before parsing.

• Time Format: Set the time format that corresponds to the timestamps in the log.

• Time Zone: Select the time zone for the log time field. By default, this is the time zone of the environment where the LoongCollector (Logtail) process is running.