Simple Log Service:Collect host text logs in a single operation

Procedure

1. On the Logstores page, click next to the target Logstore name to expand it.

2. Click Data Import > Logtail Configurations. On the One-time Logtail Configuration tab, click Add Logtail Configuration.

3. In the Quick Data Import dialog box, click Import Now on the One-time File Collection - Host card.

4. On the Machine Group Configuration page, configure the following parameters:

   • Use Case: Host Scenario

   • Installation Environment: ECS

   • Configure Machine Group: The next action depends on whether LoongCollector is installed and a machine group exists.

     • If LoongCollector is installed and added to a machine group, select the machine group from the Source Machine Groups list and add it to the Applied Machine Groups list. A new machine group is not required.

     • If LoongCollector is not installed, click Create Machine Group.

       These steps guide you through automatically installing LoongCollector and creating a machine group.

       1. The system automatically lists ECS instances that are in the same region as your Project. Select one or more instances from which you want to collect logs.

       2. Click Install and Create as Machine Group. The system automatically installs LoongCollector on the selected ECS instances.

       3. Enter a Name for the machine group and click OK.

       Note

       If the installation fails or remains in a pending state, verify that the ECS instance is in the same region as the Project.

     • To add a server with LoongCollector already installed to an existing machine group, see the FAQ topic How do I add a server to an existing machine group?

5. Check heartbeat status: Click Next. The Machine Group Heartbeat Status section appears. Verify that the Heartbeat status is OK, which indicates a successful connection. Then, click Next to continue to the Logtail configuration.

   If the status is FAIL, the initial heartbeat may take a moment to establish. Wait about two minutes, then refresh the status. If the status remains FAIL, see Machine group heartbeat connection fails for troubleshooting.

global configuration:

• Configuration name: Enter a custom name for the collection configuration. The name must be unique within the Project and cannot be modified after creation. The naming rules are as follows:

  • Can contain only lowercase letters, digits, hyphens (-), and underscores (_).

  • Must start and end with a lowercase letter or a digit.

• Execution timeout: The default is 600 seconds (10 minutes), and the valid range is from 600 to 604,800 seconds (10 minutes to 7 days). If a task exceeds this timeout, the system stops it and does not collect any remaining data.

  Important

  Important: When you update a configuration, its effective period is reset. To avoid duplicate tasks or unexpected data reporting, ensure that the machine group scope is correct and that the previous task execution did not exceed the specified execution timeout.

• Force task rerun on update : This option is turned off by default.

  • Off:

    • When you update any collection parameters other than execution timeout or the input configuration, the system resumes the current collection progress instead of restarting the task. This ensures collection continuity.

    • If you modify the execution timeout or the input configuration, the system reruns the collection task.

  • On: When you update the collection configuration, the collection task reruns from the beginning. This ensures all data is processed and reported using the new settings. Note: Previously collected data is not deleted. To clear previously collected data, see Log Service soft delete.

input configuration:

• Type: one-time file collection (available in LoongCollector 3.3 and later).

• File path: The path from which logs are collected.

  Note

  LoongCollector determines the list of files and their sizes for collection when it retrieves the configuration. It does not collect new files or content appended after this point.

  • Linux: The path must start with a forward slash (/). For example, /data/mylogs/**/*.log collects all files with the .log extension in all subdirectories of /data/mylogs.

  • Windows: The path must start with a drive letter. For example, C:\Program Files\Intel\**\*.Log.

• Maximum directory monitoring depth: The maximum directory depth that the ** wildcard in the file path can match. The default is 0, which means it monitors only the current directory.

Scenario 2: Processing structured logs

192.168.*.* - - [15/Apr/2025:16:40:00 +0800] "GET /nginx-logo.png HTTP/1.1" 0.000 514 200 368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.*.* Safari/537.36"

body_bytes_sent: 368
http_referer: -
http_user_agent : Mozi11a/5.0 (Nindows NT 10.0; Win64; x64) AppleMebKit/537.36 (KHTML, like Gecko) Chrome/131.0.x.x Safari/537.36
remote_addr:192.168.*.*
remote_user: -
request_length: 514
request_method: GET
request_time: 0.000
request_uri: /nginx-logo.png
status: 200
time_local: 15/Apr/2025:16:40:00

Procedure: In the processor configuration section of the Logtail configuration page:

1. Add a processor: Click Add Processing Plugin and configure a regex, delimiter, or JSON processor based on your log format. This example uses NGINX logs and selects native processor > NGINX-mode processor.

2. NGINX log configuration: Copy the complete log_format definition from your NGINX server's configuration file (nginx.conf) and paste it into the text box.

   Example:

   log_format main  '$remote_addr - $remote_user [$time_local] "$request" ''$request_time $request_length ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent"';

   Important

   The format definition here must exactly match the format used to generate the logs on your server. Otherwise, parsing will fail.

3. Common parameters: The following parameters are common across multiple data processors and serve the same purpose.

   • Source field: Specifies the source field to be parsed. The default is content, which refers to the entire collected log entry.

   • Keep source field on parse failure: Recommended. If the processor fails to parse a log (for example, due to a format mismatch), this option retains the original log content in the specified source field.

   • Keep source field on parse success: If selected, this option retains the original log content after successful parsing.

Procedure: In the Global Configuration > Other Global Configurations > Log Topic Type section, select how to generate the topic. You can use one of the following three methods:

• Machine group topic: When you apply a collection configuration to multiple machine groups, LoongCollector automatically uses the name of the server's machine group as the value for the __topic__ field. This is useful for scenarios where you want to categorize logs by host.

• Custom: Use the format customized://<your-topic-name>, for example, customized://app-login. This method is suitable for static topics with fixed business identifiers.

• File path-based extraction: Extracts key information from the full path of a log file to dynamically tag the log source. This is useful when multiple users or applications share the same log filename but reside in different paths. For example, when multiple users or services write logs to different top-level directories but use the same subdirectory and filename, you cannot distinguish the source by filename alone.

  /data/logs
  ├── userA
  │   └── serviceA
  │       └── service.log
  ├── userB
  │   └── serviceA
  │       └── service.log
  └── userC
      └── serviceA
          └── service.log

  In this case, you can configure file path-based extraction and use a regular expression to extract key information from the full path. Log Service then uploads the matched result to the Logstore as the log topic.

  File path-based extraction rules: Based on regular expression capture groups

  When you configure the regular expression, the system automatically determines the output field format based on the number and names of the capture groups. The rules are as follows:

  In a regular expression for a file path, you must escape forward slashes (/).

  Capture group type	Use case	Generated field	Regex example	Sample matched path	Sample generated field
  Single capture group (one (.*?))	When you need only one dimension to distinguish sources (for example, username or environment).	Generates the __topic__ field.	\/logs\/(.*?)\/app\.log	/logs/userA/app.log	__topic__:userA
  Multiple unnamed capture groups (multiple (.*?))	When you need multiple dimensions to distinguish sources but do not require semantic labels.	Generates tag fields where the key is formatted as __topic_{i}__, where {i} is the capture group index.	\/logs\/(.*?)\/(.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:__topic_1__:userA; __tag__:__topic_2__:svcA
  Multiple named capture groups (using (?P<name>.*?))	When you need multiple dimensions to distinguish sources and want clear, meaningful field names for easy querying and analysis.	Generates tag fields where the key is the specified capture group name.	\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:user:userA; __tag__:service:svcA

Add a server to a machine group

To add a new server to an existing machine group, such as a newly deployed ECS instance or a self-managed server, follow these steps to associate it with the group and apply its collection configuration.

Prerequisites

• An existing machine group.

• LoongCollector is installed on the new server.

Procedure

1. View the machine group identifier of the target machine group.

   1. In the target Project, click Resource > Machine Groups in the left-side navigation pane.

   2. On the Machine Groups page, click the name of the target machine group.

   3. On the machine group configuration page, view the machine group identifier.

2. Perform one of the following actions based on the identifier type.

   Note

   A single machine group cannot contain both Linux and Windows servers. Do not configure the same user-defined identifier on both Linux and Windows servers. You can configure multiple user-defined identifiers on a single server by separating them with a line break.

   • Type 1: The machine group identifier is an IP address.

     1. On the server, run the following command to open the app_info.json file and view the value of the ip parameter.

        cat /usr/local/ilogtail/app_info.json

     2. On the configuration page of the target machine group, click Modify and enter the IP address of the server. If you have multiple IP addresses, separate them with line breaks.

     3. After you complete the configuration, click Save and check the heartbeat status. If the status is OK, the system automatically applies the machine group's collection configuration to the server.

        If the heartbeat status is FAIL, see The machine group heartbeat status is FAIL for further troubleshooting.

   • Type 2: The machine group identifier is a user-defined identifier.

     Based on the operating system, write the user-defined identifier string that matches the target machine group to the specified file.

     If the directory does not exist, create it manually. The file path and name are fixed and cannot be customized.

     • Linux: Write the user-defined identifier to the /etc/ilogtail/user_defined_id file.

     • Windows: Write the user-defined identifier to the C:\LogtailData\user_defined_id file.

Extract log fields using a regular expression and parse the log into key-value pairs. Each field can be independently queried and analyzed.

127.0.0.1 - - [16/Aug/2024:14:37:52 +0800] "GET /wp-admin/admin-ajax.php?action=rest-nonce HTTP/1.1" 200 41 "http://www.example.com/wp-admin/post-new.php?post_type=page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0"

body_bytes_sent: 41
http_referer: http://www.example.com/wp-admin/post-new.php?post_type=page
http_user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; ×64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0
remote_addr: 127.0.0.1
remote_user: -
request_method: GET
request_protocol: HTTP/1.1
request_uri: /wp-admin/admin-ajax.php?action=rest-nonce
status: 200
time_local: 16/Aug/2024:14:37:52 +0800

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (Regex Mode):

• Regular Expression: The expression used to match logs. Generate it automatically or enter it manually:

  • Automatic generation:

    • Click Generate.

    • In the Log Sample, select the log content to extract.

    • Click Generate Regular Expression.

      

  • Manual entry: Manually Enter Regular Expression based on the log format.

  After configuration, click Validate to test whether the regular expression can correctly parse the log content.

• Extracted Field: The field name (Key) that corresponds to the extracted log content (Value).

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Structure log content using a separator to parse it into multiple key-value pairs. Both single-character and multi-character separators are supported.

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (Delimiter Mode):

• Delimiter: Specifies the character used to split log content.

  Example: For a CSV file, select Custom and enter a comma (,).

• Quote: If a field value contains the separator, you must enclose the field value in quotes to prevent incorrect splitting.

• Extracted Field: Specify the field name (Key) for each column in the order that they appear. The rules are as follows:

  • Field names can contain only letters, digits, and underscores (_).

  • Must start with a letter or an underscore (_).

  • Maximum length: 128 bytes.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Structure an Object-type JSON log by parsing it into key-value pairs.

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (JSON Mode):

• Original Field: The field that contains the raw log to be parsed. The default value is content.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Parse a nested JSON log into key-value pairs by specifying the expansion depth.

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Extended Processor > Expand JSON Field:

• Original Field: Specifies the name of the source field to expand, such as content.

• JSON Expansion Depth: The expansion depth of the JSON object, where 0 (the default) indicates full expansion, 1 indicates expansion of the current level, and so on.

• Character to Concatenate Expanded Keys: The separator for field names when a JSON object is expanded. The default value is an underscore (_).

• Name Prefix of Expanded Keys: The prefix for field names after JSON expansion.

• Expand Array: Expands an array into key-value pairs with indexes.

  Example: {"k":["a","b"]} is expanded to {"k[0]":"a","k[1]":"b"}.

  To rename the expanded fields (for example, from prefix_s_key_k1 to new_field_name), add a rename fields plugin afterward to complete the mapping.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Use the json_extract function to extract JSON objects from a JSON array.

Example:

[{"key1":"value1"},{"key2":"value2"}]

json1:{"key1":"value1"}
json2:{"key2":"value2"}

Procedure: In the Processor Configurations section of the Logtail Configuration page, switch the Processing Mode to SPL, configure the SPL Statement, and use the json_extract function to extract JSON objects from the JSON array.

Example: Extract elements from the JSON array in the log field content and store the results in new fields json1 and json2.

* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

c-ip: cs-username
cs-bytes: sc-substatus
cs-method: cs-method
cs-uri-query: cs-uri-query
cs-uri-stem: cs-uri-stem
cs-username: s-port
date: #Fields:
s-computername: s-sitename
s-ip: s-ip
s-sitename: time
sc-bytes: sc-status
sc-status: c-ip
sc-win32-status: cs (User-Agent)
time: date
time-taken: sc-win32-status

• For details on other parameters, see Scenario 2: Structured logs.

Data masking

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Parse the time field in the log and set the parsing result as the log's __time__ field.

Example:

{"level":"INFO","timestamp":"2025-09-23T19:11:47+0800","cluster":"yilu-cluster-0728","message":"User logged in successfully","userId":"user-123"}

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Time Parsing:

• Original Field: The field that contains the log content before parsing.

• Time Format: Set the time format that corresponds to the timestamps in the log.

• Time Zone: Select the time zone for the log time field. By default, this is the time zone of the environment where the LoongCollector (Logtail) process is running.