Simple Log Service:Collect logs with nanosecond-precision timestamps

Step 1: Create a project and a LogStore

If you already have a project and a LogStore, skip to Step 2: Configure a machine group and install LoongCollector.

Otherwise, create the resources that manage and store your logs:

1. Log on to the Simple Log Service console.

2. Click Create Project and configure the following parameters:

   • Region: Select the region based on where your logs originate. The region cannot be changed after the project is created.

   • Project Name: Enter a name that is globally unique within Alibaba Cloud. The name cannot be changed after the project is created.

3. Keep the default settings for the other parameters and click Create. For more information, see Manage projects.

4. Click the project name to go to the project.

5. In the left-side navigation pane, choose Log Storage and click +.

6. On the Create LogStore page, configure the following parameters:

   • Logstore Name: Set a name that is unique within the project. The name cannot be changed after the LogStore is created.

   • Logstore Type: Select Standard or Query based on the specification comparison.

   • Billing Mode:

     • Pay-by-feature: You are billed separately for each resource, such as storage, indexes, and read and write operations. This mode suits small-scale scenarios or scenarios in which your feature usage is not yet certain.

     • Pay-by-ingested-data: You are billed only for the raw data ingested. This mode includes a 30-day free storage period and free features such as data transformation and delivery. The cost model is simple, which suits scenarios in which the storage period is close to 30 days or the data processing pipeline is complex.

   • Data Retention Period: Set the number of days to retain logs. Valid values: 1 to 3,650. A value of 3,650 indicates permanent retention. Default value: 30.

7. Keep the default settings for the other parameters and click OK. For more information, see Manage LogStores.

Step 2: Configure a machine group and install LoongCollector

install LoongCollector on your server and add the server to a machine group. This topic uses an ECS instance on which LoongCollector is installed as an example, where the ECS instance and the SLS project belong to the same Alibaba Cloud account and region. If the ECS instance and the project do not belong to the same account or region, or if you use a self-managed server, see Install LoongCollector to install the collector manually.

| Click the target project. On the LogStores page:

• Click in front of the target LogStore name to expand it.

• Click next to Data Collection.

• In the dialog box that appears, select a text log template. This topic uses the Single Line - Text Logs template as an example. Click Integrate Now. All text log templates differ only in the parsing processor. The rest of the configuration process is the same, and you can modify all settings later.

Configuration steps:

1. On the Machine Group Configurations page, configure the following parameters:

   • Scenario: Servers

   • Installation Environment: ECS

2. Configure the machine group: Select the action that matches the LoongCollector installation status and the machine group status of the target server:

   • If LoongCollector is already installed and the server already belongs to a machine group, select the group from the Source Machine Group list and add it to the Applied Machine Group list. You do not need to create the group again.

   • If LoongCollector is not installed, click Create Machine Group: The following steps guide you through the automatic installation of LoongCollector and the creation of a new machine group.

     1. The system automatically lists the ECS instances that reside in the same region as the project. Select one or more instances from which you want to collect logs.

     2. Click Install and Create Machine Group. The system automatically installs LoongCollector on the selected ECS instances.

     3. Configure the Name of the machine group and click OK. If the installation fails or stays in the waiting state, check whether the ECS instance resides in the same region as the project.

3. To add a server on which LoongCollector is already installed to an existing machine group, see the FAQ topic How do I add a server to an existing machine group?

Step 3: Create a collection configuration

Configure by using the console， go to the Logtail Configurations page to define the log collection and processing rules.

1. Enable nanosecond precision support Define the log source and the collection rules, and enable nanosecond precision support.

Global Configurations:

• Configuration Name: Set a name that is unique within the project. The name cannot be changed after the configuration is created.

• Other Global Configurations: Turn on the Advanced Parameters switch and enter the following JSON content to enable nanosecond precision support:

{
  "EnableTimestampNanosecond": true
}

Input Configurations:

• Type: Text Log Collection.

• File Path: The path from which logs are collected.

  • Linux: Start with a forward slash (/). For example, /data/mylogs/**/*.log specifies all files with the .log extension in the /data/mylogs directory.

  • Windows: Start with a drive letter. For example: C:\Program Files\Intel\**\*.Log.

• Maximum Directory Monitoring Depth: The maximum directory depth that the ** wildcard in File Path can match. Default value: 0, which monitors only the current directory.

  2. Configure processors Because the source log is in JSON format, add the Data Parsing (JSON Mode) processor in the Processing Steps section. This processor separates the string that contains the nanosecond timestamp from the raw log and stores it as a standalone field.

• Add a log sample

  Assume that the log file contains logs in the following format, where the asctime field contains a timestamp with nanosecond precision.

{
  "asctime": "2023-10-25 23:51:10,199999999",
  "filename": "generate_data.py",
  "levelname": "INFO",
  "lineno": 51,
  "module": "generate_data",
  "message": "{\"no\": 14, \"inner_loop\": 166, \"loop\": 27451, \"uuid\": \"9be98c29-22c7-40a1-b7ed-29ae6c8367af\"}",
  "threadName": "MainThread"
}

• Add a JSON parsing processor

  Click Add Processor, select Native Processor > Data Parsing (JSON Mode), and then click Confirm.

• Add a time parsing processor

  Convert the time string extracted in the previous step (the asctime field) into a standard nanosecond timestamp, and use it as the event time of the log.

3. Configure indexes After you complete the Logtail configuration, click Next. The Query and Analysis Configurations page appears:

• The system enables full-text indexing by default, which supports keyword search in the raw log content.

• To run exact queries by field, wait until Preview Data loads on the page, and then click Automatic Index Generation. SLS generates a field index based on the first entry in the preview data.

  After you complete the configuration, click Next to complete the setup of the entire collection process.

Configure by using a CRD (Kubernetes scenarios) In an ACK cluster or a self-managed Kubernetes cluster, you can configure the collection of nanosecond-precision timestamps by using an AliyunLog CRD. Save the following YAML content to a file and run kubectl apply -f <filename> to apply the configuration. The following samples show the configurations for the three processors.

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: ${your-config-name}
spec:
  config:
    aggregators: []
    global:
      EnableTimestampNanosecond: true
    inputs:
      - Type: input_file
        FilePaths:
          - /test/sls/json_nano.log
        MaxDirSearchDepth: 0
        FileEncoding: utf8
        EnableContainerDiscovery: true
    processors:
      - Type: processor_parse_json_native
        SourceKey: content
      - Type: processor_parse_timestamp_native
        SourceKey: asctime
        SourceFormat: '%Y-%m-%d %H:%M:%S,%f'
    flushers:
      - Type: flusher_sls
        Logstore: ${your-logstore-name}
    sample: |-
      {
        "asctime": "2025-11-03 15:39:14,229939478",
        "filename": "log_generator.sh",
        "levelname": "INFO",
        "lineno": 204,
        "module": "log_generator",
        "message": "{\"no\": 45, \"inner_loop\": 15, \"loop\": 1697, \"uuid\": \"80366fca-a57d-b65a-be07-2ac1173505d9\"}",
        "threadName": "MainThread"
      }
  project:
    name: ${your-project-name}
  logstores:
    - name: ${your-logstore-name}

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: ${your-config-name}
spec:
  config:
    aggregators: []
    global:
      EnableTimestampNanosecond: true
    inputs:
      - Type: input_file
        FilePaths:
          - /test/sls/json_nano.log
        MaxDirSearchDepth: 0
        FileEncoding: utf8
        EnableContainerDiscovery: true
    processors:
      - Type: processor_parse_json_native
        SourceKey: content
      - Type: processor_strptime
        SourceKey: asctime
        Format: '%Y-%m-%d %H:%M:%S,%f'
        KeepSource: true
        AlarmIfFail: true
        AdjustUTCOffset: false
    flushers:
      - Type: flusher_sls
        Logstore: ${your-logstore-name}
    sample: |-
      {
        "asctime": "2025-11-03 15:39:14,229939478",
        "filename": "log_generator.sh",
        "levelname": "INFO",
        "lineno": 204,
        "module": "log_generator",
        "message": "{\"no\": 45, \"inner_loop\": 15, \"loop\": 1697, \"uuid\": \"80366fca-a57d-b65a-be07-2ac1173505d9\"}",
        "threadName": "MainThread"
      }
  project:
    name: ${your-project-name}
  logstores:
    - name: ${your-logstore-name}

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: ${your-config-name}
spec:
  config:
    aggregators: []
    global:
      EnableTimestampNanosecond: true
    inputs:
      - Type: input_file
        FilePaths:
          - /test/sls/json_nano.log
        MaxDirSearchDepth: 0
        FileEncoding: utf8
        EnableContainerDiscovery: true
    processors:
      - Type: processor_parse_json_native
        SourceKey: content
      - Type: processor_gotime
        SourceKey: asctime
        SourceFormat: '2006-01-02 15:04:05,999999999'
        DestKey: result_asctime
        DestFormat: '2006-01-02 15:04:05,999999999Z07:00'
        SetTime: true
        KeepSource: true
        NoKeyError: true
        AlarmIfFail: true
    flushers:
      - Type: flusher_sls
        Logstore: ${your-logstore-name}
    sample: |-
      {
        "asctime": "2025-11-03 15:39:14,229939478",
        "filename": "log_generator.sh",
        "levelname": "INFO",
        "lineno": 204,
        "module": "log_generator",
        "message": "{\"no\": 45, \"inner_loop\": 15, \"loop\": 1697, \"uuid\": \"80366fca-a57d-b65a-be07-2ac1173505d9\"}",
        "threadName": "MainThread"
      }
  project:
    name: ${your-project-name}
  logstores:
    - name: ${your-logstore-name}

After you apply the CRD configuration, verify that the collection configuration is effective by checking the LogStore for incoming logs with nanosecond-precision timestamps.

Step 4: Verify the results

After you complete the configuration, wait a few moments for new log data to be collected into the LogStore.

On the query and analysis page of SLS, view the collected logs. The console automatically optimizes the display based on the high-precision time information and shows the time in millisecond, microsecond, or nanosecond format.

In the log query results, timestamps appear in a format such as 2025-11-03 17:34:40.747598929.

To verify that nanosecond precision is working correctly, run a query to check the __time_ns_part__ field:

* | SELECT __time__, __time_ns_part__, * FROM log WHERE __time_ns_part__ > 0 ORDER BY __time__, __time_ns_part__ LIMIT 10

If the query returns results with __time_ns_part__ values in the range of 0 to 999,999,999, nanosecond precision is enabled and functioning correctly. If the __time_ns_part__ field is empty or contains only zeros, review your collection configuration to verify that:

• The Advanced Parameters switch is turned on and contains { "EnableTimestampNanosecond": true }.

• The time parsing processor is configured with the correct time format that matches your raw log timestamps.

• The Logtail version on your server is 1.8.0 or later.

Why are the nanosecond timestamps in collected logs not parsed correctly?

After you configure collection, the high-precision time is not extracted as expected.

The index time in the log viewer is 10-26 00:30:39, but the asctime of the log record is 2023-10-26 00:30:10,199999999. The two values do not match, which indicates that the high-precision time is not extracted correctly. The log record details are as follows:

__file_offset__  xxx
__tag__  xxx
asctime: 2023-10-26 00:30:10,199999999
filename: xxx
levelname: INFO
lineno: 51
message: {"no": 14, "inner_loop": 166, "loop": 27451, "uuid": "9be98c29-22c7-40a1-b7ed-29ae6c8367af"}
module: generate_data
threadName: MainThread

Cause

The processor mode supports %f, but the time format must be identical to the content of the source time.

Solution

1. Log on to the LoongCollector (Logtail) server and check the logs. You find a large number of STRPTIME_PARSE_ALARM exception logs.

   tail -f /usr/local/ilogtail/logtail_plugin.LOG
   2023-10-26 00:30:39 [WRN] [strptime.go:164] [processLog] [##1.0##xxxx,xxx]    AlarmType:STRPTIME_PARSE_ALARM    strptime(2023-10-26 00:30:10,199999999, %Y-%m-%d %H:%M:%S %f) failed: 0001-01-01 00:00:00 +0000 UTC, <nil>

2. Modify the log parsing format of the processor.

   In the raw log, the time is 2023-10-26 00:30:10,199999999, and the separator between the seconds and the high-precision time (milliseconds in this case) is a comma (,). In the parsing format %Y-%m-%d %H:%M:%S %f, the separator between the seconds and the high-precision time is a space. To fix the issue, change the time conversion format in the collection configuration to %Y-%m-%d %H:%M:%S,%f.