Simple Log Service:Collect host text logs in a one-time operation

Procedure:

1. On the Logstores page, click before the name of the target Logstore to expand it.

2. Click Data Ingestion > Logtail Configuration. On the One-time Logtail Configurations tab, click Add Logtail Configuration.

3. In the Quick Data Import dialog box, click Import Data on the One-time File Collection - Host card.

4. On the Machine Group Configuration page, configure the following parameters:

   • Scenario: Host

   • Installation Environment: ECS

   • Configure Machine Group: Based on the LoongCollector installation status and machine group configuration of the target server, perform one of the following operations:

     • If LoongCollector is installed and has been added to a machine group, select it from the Source Machine Group list and add it to the Applied Machine Groups list. You do not need to create it again.

     • If LoongCollector is not installed, click Create Machine Group:

       The following steps guide you through automatically installing LoongCollector and creating a machine group.

       1. The system automatically lists ECS instances in the same region as the Project. Select one or more instances from which you want to collect logs.

       2. Click Install and Create as Machine Group. The system automatically installs LoongCollector on the selected ECS instances.

       3. Configure a Name for the machine group and click OK.

       Note

       If the installation fails or remains in a waiting state, check whether the ECS region is the same as the Project region.

     • To add a server with LoongCollector already installed to an existing machine group, see How do I add a server to an existing machine group?

5. Check Heartbeat Status: Click Next. The Machine Group Heartbeat Status section appears. Check the Heartbeat status. If the status is OK, the machine group connection is normal. Click Next to go to the Logtail configuration page.

   If the status is FAIL, it may take some time for the initial heartbeat to be established. Wait for about two minutes and then refresh the heartbeat status. If the status is still FAIL after you refresh the page, see The heartbeat of a machine group is FAIL for troubleshooting information.

Global Configurations:

• Configuration Name: A custom name for the collection configuration. It must be unique within its Project. It cannot be modified after creation. Naming conventions:

  • Can contain only lowercase letters, digits, hyphens (-), and underscores (_).

  • Must start and end with a lowercase letter or a digit.

• Execution Timeout: The default is 600 s (10 minutes). The range is from 600 to 604,800 seconds (10 minutes to 7 days). If the collection is not complete within this time, the task is automatically stopped and does not collect the remaining parts.

  Important

  Note on configuration delivery: The validity period of the configuration is reset when you update it. To avoid duplicate delivery or unexpected data reporting, confirm that the machine group scope is accurate and that the last task execution time has not exceeded the Execution Timeout.

• Force Rerun on Update: Disabled by default.

  • Shutdown:

    • When you update collection configuration parameters other than Execution Timeout or Input Configurations, the system continues the current collection progress. It does not restart the collection task, which helps maintain collection continuity.

    • If you modify Execution Timeout or Input Configurations, the system restarts the collection task.

  • Enabled: Forces the collection task to restart when you update the collection configuration. This ensures that all data processing and reporting align with the latest changes. Note: Data collected before the update is not deleted. To clear it, see Simple Log Service soft delete.

Input Configurations:

• Type: One-time File Collection (available for LoongCollector 3.3 and later).

• File Path: The path for log collection.

  Note

  The list of files for one-time collection and the size of each file are determined at the time that LoongCollector pulls the configuration. New files or content appended to existing files are not collected.

  • Linux: Starts with "/", such as /data/mylogs/**/*.log, which indicates all files with the .log extension in the /data/mylogs directory and its subdirectories.

  • Windows: Starts with a drive letter, such as C:\Program Files\Intel\**\*.Log.

• Maximum Directory Monitoring Depth: The maximum directory depth that the wildcard character ** in the File Path can match. The default is 0, which means only the current directory is monitored.

Scenario 2: Structured logs

192.168.*.* - - [15/Apr/2025:16:40:00 +0800] "GET /nginx-logo.png HTTP/1.1" 0.000 514 200 368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.*.* Safari/537.36"

body_bytes_sent: 368
http_referer: -
http_user_agent : Mozi11a/5.0 (Nindows NT 10.0; Win64; x64) AppleMebKit/537.36 (KHTML, like Gecko) Chrome/131.0.x.x Safari/537.36
remote_addr:192.168.*.*
remote_user: -
request_length: 514
request_method: GET
request_time: 0.000
request_uri: /nginx-logo.png
status: 200
time_local: 15/Apr/2025:16:40:00

Procedure: In the Processing Configurations area of the Logtail Configuration page:

1. Add Parsing Plugin: Click Add Processing Plugin and configure a plugin such as regex parsing, delimiter parsing, or JSON parsing based on the actual format. This example uses NGINX log collection. Select Native Processing Plugins > NGINX Mode Parsing.

2. NGINX Log Configuration: Completely copy the log_format definition from the Nginx server configuration file (nginx.conf) and paste it into this text box.

   Example:

   log_format main  '$remote_addr - $remote_user [$time_local] "$request" ''$request_time $request_length ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent"';

   Important

   The format definition here must be exactly the same as the format that generates the logs on the server. Otherwise, log parsing will fail.

3. General configuration parameters: The following parameters appear in multiple data parsing plugins, and their functions and usage are consistent.

   • Source Field: Specifies the source field name to be parsed. The default is content, which is the entire collected log content.

   • Keep Source Field on Failure: Recommended. If a log cannot be successfully parsed by the plugin (for example, because of a format mismatch), this option ensures that the original log content is not lost but is fully retained in the specified source field.

   • Keep Source Field on Success: If selected, the original log content will be retained even if the log is parsed successfully.

Procedure: In Global Configurations > Other Global Configurations > Topic Generation Mode, select a topic generation method. The following three types are supported:

• Machine Group Topic: When a collection configuration is applied to multiple machine groups, LoongCollector automatically uses the name of the machine group to which the server belongs as the value for the __topic__ field. This is suitable for scenarios where logs are divided by host.

• Custom: The format is customized://<custom_topic_name>, for example, customized://app-login. This is suitable for static topic scenarios with fixed business identifiers.

• Extract from File Path: Extracts key information from the full path of the log file to dynamically mark the log source. This is suitable for situations where multiple users or applications share the same log filename but have different paths. When multiple users or services write logs to different top-level directories, but the sub-paths and filenames are the same, the source cannot be distinguished by filename alone. For example:

  /data/logs
  ├── userA
  │   └── serviceA
  │       └── service.log
  ├── userB
  │   └── serviceA
  │       └── service.log
  └── userC
      └── serviceA
          └── service.log

  In this case, you can configure Extract from File Path and use a regular expression to extract key information from the full path. The matched result is then used as the topic and uploaded to the Logstore.

  File path extraction rule: Based on regular expression capturing groups

  When configuring a regular expression, the system automatically determines the output field format based on the number and naming of the capturing groups. The rules are as follows:

  In the regular expression for the file path, you need to escape the forward slash (/).

  Capturing Group Type	Scenario	Generated Field	Regex Example	Sample Matched Path	Sample Generated Field
  Single capturing group (only one (.*?))	Only one dimension is needed to distinguish the source (such as username or environment)	Generates the __topic__ field	\/logs\/(.*?)\/app\.log	/logs/userA/app.log	__topic__:userA
  Multiple non-named capturing groups (multiple (.*?))	Multiple dimensions are needed to distinguish the source, but no semantic labels are required	Generates a tag field __tag__:__topic_{i}__, where {i} is the ordinal number of the capturing group	\/logs\/(.*?)\/(.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:__topic_1__userA; __tag__:__topic_2__svcA
  Multiple named capturing groups (using (?P<name>.*?))	Multiple dimensions are needed to distinguish the source, and you want the field meanings to be clear for easy query and analysis	Generates a tag field __tag__:{name}	\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:user:userA; __tag__:service:svcA

How do I add a server to an existing machine group?

When you have an existing machine group and want to add a new server, such as a newly deployed ECS instance or a self-managed server, to it to inherit its collection configuration, you can follow these steps.

Prerequisites:

• A configured machine group already exists.

• LoongCollector is installed on the new server.

Procedure:

1. View the target machine group ID:

   1. On the target Project page, click Resource > Machine Groups in the navigation pane on the left.

   2. On the Machine Groups page, click the name of the target machine group.

   3. On the machine group configuration page, view the machine group ID.

2. Perform the corresponding operation based on the ID type:

   Note

   A single machine group cannot contain both Linux and Windows servers. Do not configure the same custom ID on both Linux and Windows servers. A server can be configured with multiple custom IDs, separated by line feeds.

   • Type 1: Machine group ID is an IP address

     1. On the server, run the following command to open the app_info.json file and view the ip value.

        cat /usr/local/ilogtail/app_info.json

     2. On the target machine group configuration page, click Modify and enter the server's IP address. Separate multiple IP addresses with line feeds.

     3. After the configuration is complete, click Save and confirm the heartbeat status. After the heartbeat status is OK, the server automatically applies the machine group's collection configuration.

        If the heartbeat status is FAIL, see The heartbeat of a machine group is FAIL for further troubleshooting.

   • Type 2: Machine group ID is a custom ID

     Depending on the operating system, write a custom ID string that is consistent with the target machine group to the specified file:

     If the directory does not exist, create it manually. The file path and name are fixed by Simple Log Service and cannot be customized.

     • Linux: Write the custom string to the /etc/ilogtail/user_defined_id file.

     • Windows: Write the custom string to C:\LogtailData\user_defined_id.

Extract log fields using a regular expression and parse the log into key-value pairs. Each field can be independently queried and analyzed.

127.0.0.1 - - [16/Aug/2024:14:37:52 +0800] "GET /wp-admin/admin-ajax.php?action=rest-nonce HTTP/1.1" 200 41 "http://www.example.com/wp-admin/post-new.php?post_type=page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0"

body_bytes_sent: 41
http_referer: http://www.example.com/wp-admin/post-new.php?post_type=page
http_user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; ×64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0
remote_addr: 127.0.0.1
remote_user: -
request_method: GET
request_protocol: HTTP/1.1
request_uri: /wp-admin/admin-ajax.php?action=rest-nonce
status: 200
time_local: 16/Aug/2024:14:37:52 +0800

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (Regex Mode):

• Regular Expression: The expression used to match logs. Generate it automatically or enter it manually:

  • Automatic generation:

    • Click Generate.

    • In the Log Sample, select the log content to extract.

    • Click Generate Regular Expression.

      

  • Manual entry: Manually Enter Regular Expression based on the log format.

  After configuration, click Validate to test whether the regular expression can correctly parse the log content.

• Extracted Field: The field name (Key) that corresponds to the extracted log content (Value).

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Structure log content using a separator to parse it into multiple key-value pairs. Both single-character and multi-character separators are supported.

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (Delimiter Mode):

• Delimiter: Specifies the character used to split log content.

  Example: For a CSV file, select Custom and enter a comma (,).

• Quote: If a field value contains the separator, you must enclose the field value in quotes to prevent incorrect splitting.

• Extracted Field: Specify the field name (Key) for each column in the order that they appear. The rules are as follows:

  • Field names can contain only letters, digits, and underscores (_).

  • Must start with a letter or an underscore (_).

  • Maximum length: 128 bytes.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Structure an Object-type JSON log by parsing it into key-value pairs.

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (JSON Mode):

• Original Field: The field that contains the raw log to be parsed. The default value is content.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Parse a nested JSON log into key-value pairs by specifying the expansion depth.

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Extended Processor > Expand JSON Field:

• Original Field: Specifies the name of the source field to expand, such as content.

• JSON Expansion Depth: The expansion depth of the JSON object, where 0 (the default) indicates full expansion, 1 indicates expansion of the current level, and so on.

• Character to Concatenate Expanded Keys: The separator for field names when a JSON object is expanded. The default value is an underscore (_).

• Name Prefix of Expanded Keys: The prefix for field names after JSON expansion.

• Expand Array: Expands an array into key-value pairs with indexes.

  Example: {"k":["a","b"]} is expanded to {"k[0]":"a","k[1]":"b"}.

  To rename the expanded fields (for example, from prefix_s_key_k1 to new_field_name), add a rename fields plugin afterward to complete the mapping.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Use the json_extract function to extract JSON objects from a JSON array.

Example:

[{"key1":"value1"},{"key2":"value2"}]

json1:{"key1":"value1"}
json2:{"key2":"value2"}

Procedure: In the Processor Configurations section of the Logtail Configuration page, switch the Processing Mode to SPL, configure the SPL Statement, and use the json_extract function to extract JSON objects from the JSON array.

Example: Extract elements from the JSON array in the log field content and store the results in new fields json1 and json2.

* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

c-ip: cs-username
cs-bytes: sc-substatus
cs-method: cs-method
cs-uri-query: cs-uri-query
cs-uri-stem: cs-uri-stem
cs-username: s-port
date: #Fields:
s-computername: s-sitename
s-ip: s-ip
s-sitename: time
sc-bytes: sc-status
sc-status: c-ip
sc-win32-status: cs (User-Agent)
time: date
time-taken: sc-win32-status

• For other parameters, see the general configuration parameter description in Scenario 2: Structured logs.

Data masking

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Parse the time field in the log and set the parsing result as the log's __time__ field.

Example:

{"level":"INFO","timestamp":"2025-09-23T19:11:47+0800","cluster":"yilu-cluster-0728","message":"User logged in successfully","userId":"user-123"}

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Time Parsing:

• Original Field: The field that contains the log content before parsing.

• Time Format: Set the time format that corresponds to the timestamps in the log.

• Time Zone: Select the time zone for the log time field. By default, this is the time zone of the environment where the LoongCollector (Logtail) process is running.