This topic describes how to use Logtail to collect the NGINX access logs of an Alibaba Cloud Elastic Compute Service (ECS) instance. This topic also describes how to query and analyze the collected logs.
An ECS instance is available. For more information, see ECS quick start.
The ECS instance continuously generates logs.Important Logtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.
In this example, the logs are stored in the /var/log/nginx/access.log file, and the sample log is
127.0.0.1 - - [10/Jun/2022:12:36:49 +0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36". In this example, the full regex mode is used to collect logs based on the sample log. For more information, see Collect logs in full regex mode.
Step 1: Activate Simple Log Service
Step 2: Create a project and a Logstore
Create a project.
In the Projects section, click Create Project.
In the Create Project panel, configure the following parameters. For other parameters, retain the default settings. For more information, see Create a project.
The name of the project. The name must be unique within your Alibaba Cloud account. After the project is created, you cannot change the name of the project.
The region where the data center of the project resides. We recommend that you select the region where the ECS instance resides. Then, you can use an internal network of Alibaba Cloud to accelerate log collection.
After the project is created, you cannot change the region or migrate the project to another region.
Create a Logstore.
After the project is created, you are prompted to create a Logstore.
In the Create Logstore panel, configure the following parameters. For other parameters, retain the default settings. For more information, see Create a Logstore.
The billing mode of the Logstore. Valid values: Pay-by-ingested-data and Pay-by-feature. For more information, see Billable items.
The name of the Logstore. The name must be unique in the project to which the Logstore belongs.
After the Logstore is created, you cannot change the name of the Logstore.
The number of shards. Simple Log Service provides shards that allow you to read and write data.
Each shard supports a write capacity of 5 MB/s and 500 writes/s and a read capacity of 10 MB/s and 100 reads/s. If one shard can meet your business requirements, you can set Shards to 1.
Specifies whether to enable the automatic sharding feature. If you turn on Automatic Sharding, Simple Log Service increases the number of shards when the existing shards cannot accommodate the data that is written.
If the specified number of shards can meet your business requirements, you can turn off Automatic Sharding.
Step 3: Collect logs
After the Logstore is created, you are prompted to collect data.
By default, you can use only one Logtail configuration to collect logs from a log file. For more information about how to use multiple Logtail configurations to collect logs from a log file, see What do I do if I want to use multiple Logtail configurations to collect logs from a log file?
In the Created dialog box, click OK.
In the left-side navigation section of the Import Data dialog box, click On-premises Open Source/Commercial Software. Then, find RegEx - Text Log in the left-side section and click Integrate Now.
Create a machine group.
On the ECS Instances tab, select the ECS instance and click Create.
For more information, see Install Logtail on ECS instances.
In the Parameter Confirmation dialog box, click OK.
Confirm that the value of Execution Status is Success. Then, click Complete Installation.
In the Create Machine Group step, enter a machine group name and retain the default settings for other parameters. Then, click Next.
For more information, see Create an IP address-based machine group.
- Select the new machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next. Important If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
Create a Logtail configuration and click Next.
Configure the following parameters and retain the default settings for other parameters. For more information, see Collect logs in full regex mode.
The name of the Logtail configuration. The name must be unique in the project.
After the Logtail configuration is created, you cannot change the name of the Logtail configuration.
The directory and name of log files. The value varies based on the location of the logs on your server. In this example, specify /var/log/nginx/access.log.
A valid sample log that is collected from an actual scenario. In this example, enter the following sample log:
127.0.0.1 - - [10/Jun/2022:12:36:49 +0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
If you turn on Extract Field, Simple Log Service can extract log content in key-value pairs by using a regular expression.
The regular expression.
In the Log Sample field, select the content that you want to extract and click Generate Regular Expression. A regular expression is automatically generated. In this example, the following regular expression is generated:
Click Manual to specify a regular expression. Then, click Validate to check whether the regular expression can be used to parse the sample log and extract content from the sample log. For more information, see How do I test a regular expression?
After the content of the sample log is extracted as values by using the regular expression, you must specify a key for each value. For example, if Value is
127.0.0.1, you can set Key to
remote_addr.After you configure the parameters, click Next. Then, Log Service starts to collect logs.Note
- A Logtail configuration requires up to 3 minutes to take effect.
- If an error occurs when you use Logtail to collect logs, see How do I view Logtail collection errors? and How do I troubleshoot the common errors that may occur when Simple Log Service collects logs?
- Preview data, configure indexes, and then click Next. By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
Step 4: Query and analyze logs
After you configure indexes, you can query and analyze logs.
In the End step of the wizard, click Log Query.
You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.
On the query and analysis page of the Logstore that you specify, enter a query statement and select a time range.
For example, you can execute the following query statement to count the number of requests that correspond to each status code. The query and analysis results are displayed in a table.
* | SELECT status, COUNT(status) AS total GROUP BY status
Query and analysis results
Simple Log Service can display query and analysis results in charts. For more information, see Overview of visualization.
Am I charged if I only create projects and Logstores?
By default, shard resources are reserved when you create a Logstore. You are charged for active shards. For more information, see Why am I charged for active shards?
What do I do if logs fail to be collected?
When you use Logtail to collect logs, a failure may occur due to Logtail heartbeat failures, collection errors, or invalid Logtail configurations. For more information, see What do I do if errors occur when I use Logtail to collect logs?
What do I do if I can query logs but cannot analyze logs on the query and analysis page of a Logstore?
If you want to analyze logs, you must configure indexes for log fields and turn on Enable Analytics for the fields. For more information, see Create indexes.