Simple Log Service:Collect logs whose timestamps are accurate to the nanosecond

Configure in the console

After you install LoongCollector and configure the machine group, define log collection and processing rules on the Logtail Configuration page.

1. Enable nanosecond precision support

Define the log collection source and collection rules, and enable nanosecond precision support.

Global Configurations:

• Configuration Name: A unique name for the configuration within the project. The name cannot be changed after creation.

• Other Global Configurations: To enable support for nanosecond precision, turn on the Advanced Parameters switch and enter the following JSON content:

  {
    "EnableTimestampNanosecond": true
  }

Input Configurations:

• Set Type to Text Log Collection.

• File Path: The path from which logs are collected.

  • Linux: Must start with a forward slash (/). For example, /data/mylogs/**/*.log indicates all files with the .log extension in the /data/mylogs directory and its subdirectories.

  • Windows: Must start with a drive letter. For example, C:\Program Files\Intel\**\*.Log.

• Maximum Directory Monitoring Depth: The maximum depth of directories to monitor when you use the ** wildcard character in the File Path. The default value is 0, which monitors only the current directory.

2. Configure processing plugins

Because the source log is in JSON format, add the Data Parsing (JSON Mode) plugin in the Processor Configurations section to extract the string that contains the nanosecond timestamp from the raw log and store it as a separate field.

1. Add a log sample

   Assume the log format in the log file is as follows, where the asctime field contains a nanosecond-precision timestamp.

   {
     "asctime": "2023-10-25 23:51:10,199999999",
     "filename": "generate_data.py",
     "levelname": "INFO",
     "lineno": 51,
     "module": "generate_data",
     "message": "{\"no\": 14, \"inner_loop\": 166, \"loop\": 27451, \"uuid\": \"9be98c29-22c7-40a1-b7ed-29ae6c8367af\"}",
     "threadName": "MainThread"
   }

2. Add the JSON Parsing plugin

   Click Add Processor, select Native Processor > Data Parsing (JSON Mode), and click OK.

3. Add a time parsing plugin

   Convert the time string extracted in the previous step (the asctime field) into a standard nanosecond timestamp and use it as the event time for the log entry.

   Plugin name	Core feature	Use case
   Time Parsing	Basic time parsing	Simple use cases with a fixed format.
   Extract Log Time (strptime Time Format)	Flexible, supports a rich set of strptime formats	Recommended. Comprehensive features and compatible with de facto standards.
   Extract Log Time (Go Time Format)	Uses Go standard library formats	use cases where you are familiar with Go or the log format matches the Go standard library.

   Time Parsing

   Click Add Processor, select Native Processor > Time Parsing, and configure the following parameters:

   • Original Field: The field that contains the time value before parsing. For example, asctime.

   • Time Format: Set the time format to match the time field in the log. For example, in the format %Y-%m-%d %H:%M:%S,%f, %f represents the fractional part of a second with nanosecond precision.

     The time format string must exactly match the time format in the raw log, including the separator between seconds and nanoseconds, such as a comma (,) or a period (.). Otherwise, parsing will fail.

   • Time Zone: The time zone for the log time field. The machine's time zone is used by default.

   Extract Log Time (strptime Time Format)

   Click Add Processor, select Extended Processor > Extract Log Time (strptime Time Format), and then configure the following parameters:

   • Original Field: The field that contains the time value to be parsed. For example, asctime.

   • Time Format: Set the time format based on the values of the time field in the log. For example: %Y-%m-%d %H:%M:%S,%f. In this format, %f represents the fractional part of a second with nanosecond precision.

     The time format string must exactly match the time format in the raw log, including the separator between seconds and nanoseconds, such as a comma (,) or a period (.). Otherwise, parsing will fail.

   Extract Log Time (Go Time Format)

   Click Add Processor, select Extended Processor > Extract Log Time (Go Time Format), and configure the following parameters:

   • Original Time Field: The Original Field that contains the time value before parsing. For example, asctime.

   • Time Format: You can set the format for the time field in the raw log. The format must follow the Go time format specification. The format template is based on Go's reference time: 2006-01-02 15:04:05 -0700 MST. The time format for this example is 2006-01-02 15:04:05,999999999.

     The time format string must exactly match the time format in the raw log, including the separator between seconds and nanoseconds, such as a comma (,) or a period (.). Otherwise, parsing will fail.

   • New Time Field: The destination field for the parsed time, such as result_asctime.

   • New Time Format: The format of the time value after parsing, which must follow the Go time format specification, such as 2006-01-02 15:04:05,999999999Z07:00.

Configure using a CRD (Kubernetes)

In an Alibaba Cloud Container Service for Kubernetes (ACK) cluster or a self-managed Kubernetes cluster, configure the collection of nanosecond-precision timestamps using an AliyunLog Custom Resource Definition (CRD). The following sections provide configuration examples for the three different plugins.

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: ${your-config-name}
spec:
  config:
    aggregators: []
    global:
      EnableTimestampNanosecond: true
    inputs:
      - Type: input_file
        FilePaths:
          - /test/sls/json_nano.log
        MaxDirSearchDepth: 0
        FileEncoding: utf8
        EnableContainerDiscovery: true
    processors:
      - Type: processor_parse_json_native
        SourceKey: content
      - Type: processor_parse_timestamp_native
        SourceKey: asctime
        SourceFormat: '%Y-%m-%d %H:%M:%S,%f'
    flushers:
      - Type: flusher_sls
        Logstore: ${your-logstore-name}
    sample: |-
      {
        "asctime": "2025-11-03 15:39:14,229939478",
        "filename": "log_generator.sh",
        "levelname": "INFO",
        "lineno": 204,
        "module": "log_generator",
        "message": "{\"no\": 45, \"inner_loop\": 15, \"loop\": 1697, \"uuid\": \"80366fca-a57d-b65a-be07-2ac1173505d9\"}",
        "threadName": "MainThread"
      }
  project:
    name: ${your-project-name}
  logstores:
    - name: ${your-logstore-name}

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: ${your-config-name}
spec:
  config:
    aggregators: []
    global:
      EnableTimestampNanosecond: true
    inputs:
      - Type: input_file
        FilePaths:
          - /test/sls/json_nano.log
        MaxDirSearchDepth: 0
        FileEncoding: utf8
        EnableContainerDiscovery: true
    processors:
      - Type: processor_parse_json_native
        SourceKey: content
      - Type: processor_strptime
        SourceKey: asctime
        Format: '%Y-%m-%d %H:%M:%S,%f'
        KeepSource: true
        AlarmIfFail: true
        AdjustUTCOffset: false
    flushers:
      - Type: flusher_sls
        Logstore: ${your-logstore-name}
    sample: |-
      {
        "asctime": "2025-11-03 15:39:14,229939478",
        "filename": "log_generator.sh",
        "levelname": "INFO",
        "lineno": 204,
        "module": "log_generator",
        "message": "{"no": 45, "inner_loop": 15, "loop": 1697, "uuid": "80366fca-a57d-b65a-be07-2ac1173505d9"}",
        "threadName": "MainThread"
      }
  project:
    name: ${your-project-name}
  logstores:
    - name: ${your-logstore-name}

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: ${your-config-name}
spec:
  config:
    aggregators: []
    global:
      EnableTimestampNanosecond: true
    inputs:
      - Type: input_file
        FilePaths:
          - /test/sls/json_nano.log
        MaxDirSearchDepth: 0
        FileEncoding: utf8
        EnableContainerDiscovery: true
    processors:
      - Type: processor_parse_json_native
        SourceKey: content
      - Type: processor_gotime
        SourceKey: asctime
        SourceFormat: '2006-01-02 15:04:05,999999999'
        DestKey: result_asctime
        DestFormat: '2006-01-02 15:04:05,999999999Z07:00'
        SetTime: true
        KeepSource: true
        NoKeyError: true
        AlarmIfFail: true
    flushers:
      - Type: flusher_sls
        Logstore: ${your-logstore-name}
    sample: |-
      {
        "asctime": "2025-11-03 15:39:14,229939478",
        "filename": "log_generator.sh",
        "levelname": "INFO",
        "lineno": 204,
        "module": "log_generator",
        "message": "{"no": 45, "inner_loop": 15, "loop": 1697, "uuid": "80366fca-a57d-b65a-be07-2ac1173505d9"}",
        "threadName": "MainThread"
      }
  project:
    name: ${your-project-name}
  logstores:
    - name: ${your-logstore-name}