Simple Log Service:Collect text logs in NGINX configuration mode

Solution overview

In NGINX configuration mode, Logtail structures log content based on the log_format directive. Before collecting logs, configure the log_format and access_log directives in /etc/nginx/nginx.conf. log_format defines the log structure, and access_log specifies the log file path.

• Log format and storage path

  The following example uses the default log_format and access_log values:

  log_format main  '$remote_addr - $remote_user [$time_local] "$request" '
                   '$request_time $request_length '
                   '$status $body_bytes_sent "$http_referer" '
                   '"$http_user_agent"';
  access_log  /var/log/nginx/access.log  main;

  Log fields:

  Field	Description
  remote_addr	Client IP address.
  remote_user	Client username.
  time_local	Server local time, enclosed in brackets [].
  request	Request URI and HTTP protocol.
  request_time	Request processing time, in seconds.
  request_length	Total request length, including the request line, headers, and body.
  status	HTTP response status code.
  body_bytes_sent	Response body size in bytes, excluding headers.
  http_referer	Referrer URL.
  http_user_agent	Client browser user-agent string.

• Raw log

  Sample log entry generated using the log_format above:

  192.168.1.1 - - [11/Dec/2024:11:21:03 +0800] "GET /nginx-logo.png HTTP/1.1" 0.000 514 200 368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

• Collected logs in a Logstore: On the query and analysis page, each log entry is parsed into key-value pairs and displayed with metadata such as the instance name, log path, and Unix timestamp.

Procedure

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project you want.

3. On the Log Storage > Logstores tab, click the logstore you want.

4. In the left-side navigation pane, click Logtail Configurations, and then click Add Logtail Configuration.

5. In the Quick Data Import page, select Nginx - Text Logs > Integrate Now.

6. In the Machine Group Settings step, set Use Case to Host Scenario and Environment to ECS. Move your machine group (for example, group-ip) from Source Machine Group to Applied Machine Group, then click Next.

7. In the Logtail Configuration step, configure the following parameters:

   1. For Configuration Name, enter a configuration name, such as nginx-logs.

   2. For File Path, enter the log file path. For example, /var/log/nginx/**/access* matches all files that start with access in the /var/log/nginx directory and its subdirectories.

   3. For Processor Configurations, click Data Parsing (NGINX Mode). On the Processor tab, enter the log_format configuration from your NGINX config file. SLS auto-extracts the fields. Example:

      log_format main  '$remote_addr - $remote_user [$time_local] "$request" '
                       '$request_time $request_length '
                       '$status $body_bytes_sent "$http_referer" '
                       '"$http_user_agent"';

      For Source Field, enter content. SLS auto-generates a regular expression from the configuration and extracts the fields.

   4. Keep the defaults for other parameters. Collect text logs from servers.

8. In the Query, Analysis and Visualization step, click Refresh to preview collected data.

9. Click Next to complete the configuration. Click Query Log to open the Logstore query page. Index creation takes about one minute. You can then view logs on the Raw Log tab. Quick start for query and analysis.

References

• Troubleshoot Logtail collection issues on Linux servers using the automatic diagnostic tool. How do I use the automatic diagnostic tool of Logtail?

• If the preview or query page shows no data, see What do I do if errors occur when I use Logtail to collect logs?

• For collection errors such as regex parsing failures, invalid file paths, or shard overload, see How do I view Logtail collection errors? and How do I troubleshoot the common errors that may occur when Simple Log Service collects logs?

• A file can have only one Logtail configuration by default, but multiple configurations are supported. How do I collect multiple copies of logs in a file?

• Servers on a corporate intranet can also send logs to Simple Log Service. Collect logs from servers in a corporate intranet.

• To distinguish logs from servers with identical paths or file names, use Machine Group Topic. To distinguish logs by user or instance, use File Path Extraction.

• For website access analysis, diagnostics, and alert configuration, see Collect and analyze NGINX access logs.