Log Service is a platform service provided by Alibaba Cloud to handle massive log collection, storage, and query. You can use Log Service to collect logs from the service cluster. Log Service also supports real-time consumption, real-time query and shipping logs to OSS+Spark or MaxCompute (coming soon) for further analysis.
The following sample introduces how to collect text logs by using Logtail, preview logs, query the logs collected and ship logs to OSS.
The basic workflow of using Log Service is as follows.
For details, refer to FAQ about account management.
Use a registered Alibaba Cloud account to log on to the Log Service product page and click Get it Free.
Note: Logtail requires an access key to collect log data.
On the Log Service console, move the cursor above your username in the upper-right corner and click accesskeys in the displayed menu.
In the pop-up confirmation box, click Continue to manage AccessKey.
On the Access Key Management page, click Create Access Key in the upper-right corner.
In the pop-up box, click Agree and create.
When you first log on to the Log Service console, the system will prompt you to create a project. To create projects on subsequent logins, click Create Project in the upper-right corner.
Note: For details about creating a project, refer to Create a project.
When creating a project, you must specify the Project Name and Region.
After creating a project, you will be prompted to create a Logstore. You can also go to the project and click Create in the upper-right corner.
Note: For details about creating a Logstore, refer to Create a Logstore.
When creating a Logstore, you must specify how you are going to use these logs.
Log Service supports various log sources and collection modes; for details, refer to Collection modes.
Download the installation package.
Download the Logtail installation package onto the ECS instance. The download address of Windows installation package is http://logtail-release.oss-cn-hangzhou.aliyuncs.com/win/logtail_installer.zip.
Unzip the installation package into the current directory and enter the
logtail_installerdirectory. Run cmd as the administrator and run the installation command
.\logtail_installer.exe install cn_hangzhou.
Note: You must run different installation commands according to the network environment and the region of Log Service. This quick start uses China East 1 (Hangzhou) and classic network as an example. For the installation commands of other regions, refer to Install Logtail on Windows.
Log on to the Log Service Console.
Select the desired project, and click the project name or click Manage on the right.
Select the desired Logstore and click Logtail Config Manage.
Click Create in the upper-right corner.
Select a data source.
Specify the collection mode.
This example uses full mode. For details about Logtail configuration, refer to Use logtail to collect text files.
Apply the configuration to a machine group.
Note: If no machine group is available, you must first create a machine group.
- It can take up to 3 minutes for the Logtail configuration to take effect.
- If you need to collect IIS access logs, you must first refer to the IIS Log collection best practices to configure IIS.
Use API to write logs
Use SDKs to write logs
In addition to APIs, Log Service also provides SDKs in multiple languages (Java, .NET, PHP, and Python) to facilitate log writes. For a complete SDK reference, refer to SDK Reference.
Log service provides various ways to consume your collected logs, including previewing, querying, shipping logs to other products.
After you collect log data through Log Service, you can preview the collected logs. By specifying the Shard ID and time, you can preview the first 10 packets.
In the following example, the preview Shard ID is 0 and the time range is the first 10 packets from the previous 15 minutes.
API log consumption
SDK log consumption
In addition to APIs, Log Service also provides SDKs in multiple languages (Java, .NET, PHP, and Python) that allow you to easily consume logs. For a complete SDK reference, refer to Log Service SDKs.
After you collect log data in Log Service, you can click Search in the Log Consumption Mode column to go to the query interface.
Queries are performed by the specified log topic (at present, only logs written using API have topics), keyword, or time. In the histogram, green indicates that the data are precise during this time period, and yellow indicates the data are imprecise. You can click on or drag the yellow portion to perform further queries. Imprecise log data will not be returned in “Match Logs”.
In the following example, the query conditions are blank and the time range is set to the previous 15 minutes.
Use APIs to query logs
Use SDKs to query logs
In addition to APIs, Log Service also provides SDKs in multiple languages (Java, .NET, PHP, and Python) that allow for easy log querying. For a complete SDK reference, refer to Log Service SDKs.
Prepare the OSS environment.
In order to ship logs in Log Service to OSS, you must first prepare an OSS environment as explained in the following process.
Activate the OSS service.
Create an OSS bucket and perform the relevant authorization operations.
Note: The OSS bucket should be in the same region as the Log Service Project. Data can only be shipped to OSS buckets in the same region as the Log Service project.
Create OSS shipping rules.
Log Service provides the function to ship log data to OSS. In the Logstore list, click OSS in the Log Consumption Mode column, and then click Enable to set the OSS shipping rules.
You must specify the OSS bucket to post to, the permission console role name (ARN), whether or not the data are compressed, and other attributes.
View OSS shipping tasks.
In the OSS shipping task management on the console, you can view the shipping task statuses. After a log has been imported, you can view its data on the OSS console. In addition, the Ship logs to OSS explains how to use the imported data.