This topic describes how to collect logs of Alibaba Cloud Elastic Compute Service (ECS) instances in the Log Service console. This topic also describes how to query and analyze the collected logs.
- An ECS instance is available. For more information, see ECS quick start.
- Logs are available on the ECS instance.
127.0.0.1|#|-|#|13/Apr/2020:09:44:41 +0800|#|GET /1 HTTP/1.1|#|0.000|#|74|#|404|#|3650|#|-|#|curl/7.29.0. Delimiter mode is used in this example to collect the sample log.
Step 1: Activate Log Service
- Log on to the Log Service console.
- Activate Log Service as prompted.
Step 2: Create a project and a Logstore
- Log on to the Log Service console.
- Create a project.
- In the Projects section, click Create Project.
- In the Create Project panel, set the required parameters. The following table describes the parameters.
You can use the default values for other parameters.
Parameter Description Project Name The name of the project. The name must be unique in a region and cannot be modified after the project is created. Region The data center of the project. We recommend that you select the region where the ECS instance resides. Then, you can use the internal network of Alibaba Cloud to accelerate log collection.
After you create a project, you cannot migrate the project to another region or modify the region to which the project belongs.
- Click OK.
- Create a Logstore. After you create a project, you are prompted to create a Logstore.
In the Create Logstore panel, set the required parameters. The following table describes the parameters. You can use the default values for other parameters.
Parameter Description Logstore Name The name of the Logstore. The name must be unique in the project to which the Logstore belongs. After the Logstore is created, you cannot modify the name. Shards Log Service provides shards to read and write data.
Each shard supports a write speed of 5 MB/s and 500 write operations per second and a read speed of 10 MB/s and 100 read operations per second. If one shard can meet your business requirements, you can set Shards to 1.
Automatic Sharding If you turn on the Automatic Sharding switch, and the read or write speed does not meet your requirements, Log Service increases the number of shards.
If the existing number of shards can meet your business requirements, you can turn off the Automatic Sharding switch.
Step 3: Collect logs
- In the Import Data section, select Delimiter Mode - Text Log.
- Select the created project and Logstore and click Next.
- Install Logtail.
- On the ECS Instances tab, select the destination ECS instance and click Execute Now.
- Confirm that the Execution Status is Success and click Complete Installation.
- Create an IP address-based machine group and click Next. The following table describes how to set the required parameters. You can use the default values for other parameters.
Parameter Description Parameter The name of the machine group. The name must be unique in a project and cannot be modified after the machine group is created. IP addresses The IP address of the ECS instance. Separate multiple IP addresses with line feeds.Notice Windows and Linux servers cannot be added to the same machine group.
- Select and move the destination machine group from Source Server Groups to Applied Server Groups, and then click Next. Notice If you want to apply a machine group immediately after it is created, the heartbeat status of the machine group may be FAIL. This issue occurs because no servers in the machine group have not been connected to Log Service. In this case, you can click Automatic Retry. If the issue persists, see What can I do if the Logtail client has no heartbeat?
- Create a Logtail configuration file and click Next. The following table describes how to set the required parameters. You can use the default values for other parameters.
Parameter Description Config Name The name of the Logtail configuration file. The name must be unique in a project and cannot be modified after the file is created. Log Path The directory and name of the log file.The file names can be complete names or names that contain wildcards. For more information, see Wildcard matching. If the log files match the specified pattern, the log files in all levels of subdirectories under a specified directory are monitored. Examples:
- /apsara/nuwa/…/*.log indicates that the files whose extension is .log in the /apsara/nuwa directory and its subdirectories are monitored.
- /var/logs/app_*/*.log indicates that each file that meets the following conditions is monitored: The file name contains .log. The file is stored in a subdirectory (at all levels) of the /var/logs directory. The name of the subdirectory matches the app_* pattern.
- By default, each log file can be collected by using only one Logtail configuration file.
- You can use only asterisks (*) and question marks (?) as wildcard characters in the log path.
Log Sample Enter a valid sample log entry that is collected from an actual scenario. Example:
127.0.0.1|#|-|#|13/Apr/2020:09:44:41 +0800|#|GET /1 HTTP/1.1|#|0.000|#|74|#|404|#|3650|#|-|#|curl/7.29.0
Delimiter Select a delimiter based on the log format. Example:
|#|.Note If you select Hidden Characters as the quote, you must enter a character in the following format:
0xthe hexadecimal ASCII code of the non-printable character. For example, to use the non-printable character whose hexadecimal ASCII code is 01, you must enter 0x01.
Extracted Content Log Service extracts the log content based on the specified sample log and delimiter. The extracted log content is delimited into values. You must specify a key for each value.Click Next to complete the configuration and use Logtail to collect logs.Note
- The Logtail configuration file requires at most 3 minutes to take effect.
- If an error occurs when you use Logtail to collect logs, see Diagnose collection errors.
- Configure indexes. Note
- Indexes are applicable only to the log data that is newly written to Log Service.
- To query and analyze logs, you must configure field indexes for log fields and turn on the Enable Analytics switch for the fields. For more information, see Configure indexes.
- After the collected logs are displayed in the Preview Data section, click Automatic Index Generation.
- In the Automatically Generate Index Attributes dialog box, confirm the index configurations, and then click OK.
- Click Next.
Step 4: Query and analyze logs
- In the Projects section, click the project in which you want to query and analyze logs.
- On the tab, click the Logstore where logs are stored.
- Enter a query statement in the search box, select a time range, and then click Search & Analyze. Example: You can execute the following query statement to view the location of IP addresses of the previous day. You can also display the query result on a chart.
- Query statement
* | select count(1) as c, ip_to_province(remote_addr) as address group by address limit 100
- Query result
The following figure shows that 329 IP addresses were located in Guangdong province and 313 IP addresses were located in Beijing on the previous day. Log service allows you to display the query result on a chart. For more information, see Chart overview.
- Query statement
- Am I charged if I only create projects and Logstores?
Log Service provides shards to read and write data. By default, shard resources are reserved when you create a Logstore. Therefore, you are charged for active shards. For more information, see Why am I charged for active shards?
- What can I do if logs fail to be collected?
When you use Logtail to collect logs, a collection failure may occur due to unexpected Logtail heartbeats, collection errors, or invalid Logtail configurations. For more information, see Troubleshoot collection errors
- What can I do if I can query logs but cannot analyze logs on the Query & Analysis
page of a Logstore?
If you want to analyze logs, you must configure field indexes for log fields and turn on the Enable Analytics switch for the fields. For more information, see Configure indexes.