Container Service for Kubernetes:Collect Kubernetes pod text logs (sidecar mode)

1. Modify the pod YAML configuration

1. Define shared volumes

   In spec.template.spec.volumes, add three shared volumes at the same level as containers:

   volumes:
     # Shared log directory (written by the application container, read by the sidecar)
     - name: ${shared_volume_name} # <-- The name must match the name in volumeMounts
       emptyDir: {}
     # Signal directory for inter-container communication (for graceful start/stop)
     - name: tasksite
       emptyDir:
         medium: Memory  # Use memory as the medium for better performance
         sizeLimit: "50Mi"
     # Shared host time zone configuration: Synchronizes the time zone for all containers in the pod
     - name: tz-config # <-- The name must match the name in volumeMounts
       hostPath:
         path: /usr/share/zoneinfo/Asia/Shanghai  # Modify the time zone as needed
   

2. Configure application container mounts

   In the volumeMounts section of your application container, such as your-business-app-container, add the following volume mounts:

   Ensure that the application container writes logs to the ${shared_volume_path} directory so that LoongCollector can collect them.

   volumeMounts:
     # Mount the shared log volume to the application log output directory
     - name: ${shared_volume_name}
       mountPath: ${shared_volume_path}  # Example: /var/log/app
     # Mount the communication directory
     - name: tasksite
       mountPath: /tasksite  # Shared directory for communication with the LoongCollector container
     # Mount the timezone file
     - name: tz-config
       mountPath: /etc/localtime
       readOnly: true
   

3. Inject the LoongCollector sidecar container

   In the spec.template.spec.containers array, append the following sidecar container definition:

   - name: loongcollector
     image: aliyun-observability-release-registry.cn-shenzhen.cr.aliyuncs.com/loongcollector/loongcollector:v3.1.1.0-20fa5eb-aliyun
     command: ["/bin/bash", "-c"]
     args:
       - |
         echo "[$(date)] LoongCollector: Starting initialization"
         # Start the LoongCollector service
         /etc/init.d/loongcollectord start
         # Wait for the configuration to download and the service to be ready
         sleep 15
         # Verify the service status
         if /etc/init.d/loongcollectord status; then
           echo "[$(date)] LoongCollector: Service started successfully"
           touch /tasksite/cornerstone
         else
           echo "[$(date)] LoongCollector: Failed to start service"
           exit 1
         fi
         # Wait for the application container to complete (via the tombstone file signal)
         echo "[$(date)] LoongCollector: Waiting for application container to complete"
         until [[ -f /tasksite/tombstone ]]; do
           sleep 2
         done
         # Allow time to upload remaining logs
         echo "[$(date)] LoongCollector: Business completed, waiting for log transmission"
         sleep 30
         # Stop the service
         echo "[$(date)] LoongCollector: Stopping service"
         /etc/init.d/loongcollectord stop
         echo "[$(date)] LoongCollector: Shutdown complete"
     # health check
     livenessProbe:
       exec:
         command: ["/etc/init.d/loongcollectord", "status"]
       initialDelaySeconds: 30
       periodSeconds: 10
       timeoutSeconds: 5
       failureThreshold: 3
     # resource configuration
     resources:
       requests:
         cpu: "100m"
         memory: "128Mi"
       limits:
         cpu: "2000m"
         memory: "2048Mi"
     # environment variable configuration
     env:
       - name: ALIYUN_LOGTAIL_USER_ID
         value: "${your_aliyun_user_id}"
       - name: ALIYUN_LOGTAIL_USER_DEFINED_ID
         value: "${your_machine_group_user_defined_id}"
       - name: ALIYUN_LOGTAIL_CONFIG
         value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
       # Enable full drain mode to ensure all logs are sent before the pod terminates
       - name: enable_full_drain_mode
         value: "true"  
       # Append pod environment information as log tags
       - name: ALIYUN_LOG_ENV_TAGS
         value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
       # Automatically inject pod and node metadata as log tags
       - name: "_pod_name_"
         valueFrom:
           fieldRef:
             fieldPath: metadata.name
       - name: "_pod_ip_"
         valueFrom:
           fieldRef:
             fieldPath: status.podIP
       - name: "_namespace_"
         valueFrom:
           fieldRef:
             fieldPath: metadata.namespace
       - name: "_node_name_"
         valueFrom:
           fieldRef:
             fieldPath: spec.nodeName
       - name: "_node_ip_"
         valueFrom:
           fieldRef:
             fieldPath: status.hostIP
     # Volume mounts (shared with the application container)
     volumeMounts:
       # Read-only mount for the application log directory
       - name: ${shared_volume_name} # <-- Shared log directory name
         mountPath: ${dir_containing_your_files} # <-- Path to the shared directory in the sidecar
         readOnly: true
       # Mount the communication directory
       - name: tasksite
         mountPath: /tasksite
       # Mount the timezone
       - name: tz-config
         mountPath: /etc/localtime
         readOnly: true
   

2. Adapt the application container lifecycle logic

Depending on the workload type, modify the application container to support a coordinated exit with the sidecar:

# 1. Wait for LoongCollector to be ready
echo "[$(date)] Application: Waiting for LoongCollector to be ready..."
until [[ -f /tasksite/cornerstone ]]; do
  sleep 1
done
echo "[$(date)] Application: LoongCollector is ready, starting application logic"
# 2. Execute core application logic (ensure logs are written to the shared directory)
echo "Hello, World!" >> /app/logs/business.log
# 3. Save the exit code
retcode=$?
echo "[$(date)] Application: Task completed with exit code: $retcode"
# 4. Notify LoongCollector that the application task is complete
touch /tasksite/tombstone
echo "[$(date)] Application: Tombstone created, exiting"
exit $retcode

# Define the signal handler function
_term_handler() {
    echo "[$(date)] [nginx-demo] Caught SIGTERM, starting graceful shutdown..."
    # Send a QUIT signal to Nginx for a graceful stop
    if [ -n "$NGINX_PID" ]; then
        kill -QUIT "$NGINX_PID" 2>/dev/null || true
        echo "[$(date)] [nginx-demo] Sent SIGQUIT to Nginx PID: $NGINX_PID"
        # Wait for Nginx to stop gracefully
        wait "$NGINX_PID"
        EXIT_CODE=$?
        echo "[$(date)] [nginx-demo] Nginx stopped with exit code: $EXIT_CODE"
    fi
    # Notify LoongCollector that the application container has stopped
    echo "[$(date)] [nginx-demo] Writing tombstone file"
    touch /tasksite/tombstone
    exit $EXIT_CODE
}
# Register the signal handler
trap _term_handler SIGTERM SIGINT SIGQUIT
# Wait for LoongCollector to be ready
echo "[$(date)] [nginx-demo]: Waiting for LoongCollector to be ready..."
until [[ -f /tasksite/cornerstone ]]; do 
    sleep 1
done
echo "[$(date)] [nginx-demo]: LoongCollector is ready, starting application logic"
# Start Nginx
echo "[$(date)] [nginx-demo] Starting Nginx..."
nginx -g 'daemon off;' &
NGINX_PID=$!
echo "[$(date)] [nginx-demo] Nginx started with PID: $NGINX_PID"
# Wait for the Nginx process
wait $NGINX_PID
EXIT_CODE=$?
# Also notify LoongCollector if the exit was not caused by a signal
if [ ! -f /tasksite/tombstone ]; then
    echo "[$(date)] [nginx-demo] Unexpected exit, writing tombstone"
    touch /tasksite/tombstone
fi
exit $EXIT_CODE

3. Set the termination grace period

In spec.template.spec, set a termination grace period that is long enough to allow LoongCollector to upload all remaining logs.

spec:
  # ... Your other existing spec configurations ...
  template:
    spec:
      terminationGracePeriodSeconds: 600  # 10-minute graceful stop period

4. Variables

5. Apply configuration and verify

1. Run the following command to deploy the changes:

   kubectl apply -f <YOUR-YAML>

2. Check the pod status to confirm that the LoongCollector container was injected successfully:

   kubectl describe pod <YOUR-POD-NAME>

   If you see two containers (the application container and loongcollector) and their status is Running, the injection is successful.

Global Configurations:

• Configuration Name: A custom name for the collection configuration. This name must be unique within the project and cannot be changed after it is created. Naming conventions:

  • Can contain only lowercase letters, digits, hyphens (-), and underscores (_).

  • Must start and end with a lowercase letter or a digit.

Input Configurations:

• Type: Select Text Log Collection.

• Logtail Deployment Mode: Select Sidecar.

• File Path Type:

  • Path in Container: Collect log files from within a container.

  • Host Path: Collect local service logs from the host.

• File Path: The path for log collection.

  • Linux: The path must start with a forward slash (/). For example, /data/mylogs/**/*.log specifies all files that have the .log extension in the /data/mylogs directory and its subdirectories.

  • Windows: The path must start with a drive letter, such as C:\Program Files\Intel\**\*.Log.

• Maximum Directory Monitoring Depth: Specifies the maximum directory depth for the ** wildcard in the File Path. The default value of 0 monitors only the current directory.

Configure processing rules to structure raw logs into searchable key-value pairs. First, add a log sample:

In the Processor Configurations section of the Logtail Configuration page, click Add Sample Log and enter a sample log. The system identifies the format and generates parsing rules automatically.

Scenario 2: Structured logs

192.168.*.* - - [15/Apr/2025:16:40:00 +0800] "GET /nginx-logo.png HTTP/1.1" 0.000 514 200 368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.*.* Safari/537.36"

body_bytes_sent: 368
http_referer: -
http_user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.x.x Safari/537.36
remote_addr:192.168.*.*
remote_user: -
request_length: 514
request_method: GET
request_time: 0.000
request_uri: /nginx-logo.png
status: 200
time_local: 15/Apr/2025:16:40:00

Configuration steps: In the Processor Configurations area of the Logtail Configurations page:

1. Add a parsing plugin: Click Add Processor, and configure a plugin such as a regular expression, delimiter, or JSON parsing plugin based on your log format. For example, to collect NGINX logs, select Native Processor > Data Parsing (NGINX Mode).

2. NGINX Log Configuration: Copy the entire log_format definition from your Nginx server's configuration file (nginx.conf) and paste it into this text box.

   Example:

   log_format main  '$remote_addr - $remote_user [$time_local] "$request" ''$request_time $request_length ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent"';

   Important

   The format definition must exactly match the log_format on your server. Otherwise, parsing will fail.

3. General configuration parameter descriptions: The following parameters are common to many parsing plugins and have consistent functions.

   • Original Field: Specifies the source field to be parsed. Defaults to content, which is the entire collected log entry.

   • Retain Original Field if Parsing Fails: Recommended. If the plugin fails to parse a log, this option retains the raw log content in the original field.

   • Retain Original Field if Parsing Succeeds: If selected, this option retains the raw log content even after successful parsing.

By default, all logs are sent to the current Logstore with lz4 compression. To distribute logs to multiple Logstores, configure additional output destinations.

Use regular expressions to extract fields from logs into key-value pairs for independent querying and analysis.

127.0.0.1 - - [16/Aug/2024:14:37:52 +0800] "GET /wp-admin/admin-ajax.php?action=rest-nonce HTTP/1.1" 200 41 "http://www.example.com/wp-admin/post-new.php?post_type=page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0"

body_bytes_sent: 41
http_referer: http://www.example.com/wp-admin/post-new.php?post_type=page
http_user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; ×64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0
remote_addr: 127.0.0.1
remote_user: -
request_method: GET
request_protocol: HTTP/1.1
request_uri: /wp-admin/admin-ajax.php?action=rest-nonce
status: 200
time_local: 16/Aug/2024:14:37:52 +0800

Procedure: In the Processor Configurations section of the Logtail Configurations page, click Add Processor and select Native Processor > Data Parsing (Regex Mode).

• Regular Expression: A regular expression to match log content. You can generate it automatically or enter it manually.

  • Auto-generate:

    • Click Generate.

    • In the Log Sample box, select the log content to extract.

    • Click Generate Regular Expression.

      For example, if you paste a log in Apache Combined format into the Log Sample box, it will contain fields such as client IP, timestamp, request method and path, status code, Referer, and User-Agent.

  • Manual entry: Manually Enter Regular Expression based on the log format.

  After entering the expression, click Validate to test if it correctly parses the log content.

• Extracted Field: Set a field name (key) for each piece of extracted log content (value).

• For information about other parameters, see the general parameter descriptions in Use case 2: Structured logs.

Use delimiter parsing to split log content into key-value pairs. Supports single-character and multi-character delimiters.

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configurations page, click Add Processor and select Native Processor > Data Parsing (Delimiter Mode).

• Delimiter: The character used to split the log content.

  Example: For a CSV file, select Custom and enter a comma (,).

• Quote: When a field value contains the delimiter, you must specify a quote character to wrap the field to prevent incorrect splitting.

• Extracted Field: Assign a field name (key) to each column in sequential order. The field names must meet the following rules:

  • Can contain only letters, digits, and underscores (_).

  • Must start with a letter or an underscore (_).

  • Can be up to 128 bytes in length.

• For information about other parameters, see the general parameter descriptions in Use case 2: Structured logs.

Parses logs formatted as JSON objects into key-value pairs.

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configurations page, click Add Processor and select Native Processor > Data Parsing (JSON Mode).

• Original Field: The default value is content. This field contains the raw log to be parsed.

• For information about other parameters, see the general parameter descriptions in Use case 2: Structured logs.

Expands a field containing a nested JSON object into multiple key-value pairs.

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

Procedure: In the Processor Configurations section of the Logtail Configurations page, click Add Processor and select Extended Processor > Expand JSON Field.

• Original Field: The name of the original field to expand, for example, content.

• JSON Expansion Depth: The expansion level of the JSON object. A value of 0 (the default) fully expands the object, 1 expands only the current level, and so on.

• Character to Concatenate Expanded Keys: The character used to join field names during JSON expansion. The default is an underscore (_).

• Name Prefix of Expanded Keys: A prefix to add to the field names after JSON expansion.

• Expand Array: Turn on this switch to expand arrays into key-value pairs with indexes.

  Example: {"k":["a","b"]} is expanded to  k[0]: "a", k[1]: "b".

  To rename an expanded field (for example, from prefix_s_key_k1 to new_field_name), add a Rename Fields processor to complete the mapping.

• For information about other parameters, see the general parameter descriptions in Use case 2: Structured logs.

Structure log content based on the definitions in your Apache log configuration file, parsing it into multiple key-value pairs.

1 192.168.1.10 - - [08/May/2024:15:30:28 +0800] "GET /index.html HTTP/1.1" 200 1234 "https://www.example.com/referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36"

http_referer:https://www.example.com/referrer
http_user_agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36
remote_addr:192.168.1.10
remote_ident:-
remote_user:-
request_method:GET
request_protocol:HTTP/1.1
request_uri:/index.html
response_size_bytes:1234
status:200
time_local:[08/May/2024:15:30:28 +0800]

Procedure: In the Processor Configurations section of the Logtail Configurations page, click Add Processor and select Native Processor > Data Parsing (Apache Mode).

• Log Format: combined

• APACHE LogFormat Configuration: The system automatically populates this field based on the selected Log Format.

  Important

  Ensure the auto-populated content exactly matches the LogFormat definition in your server's Apache configuration file (typically /etc/apache2/apache2.conf).

• For information about other parameters, see the general parameter descriptions in Use case 2: Structured logs.