This topic describes commonly used terms in Log Service to help you better understand and use Log Service.

Region

A region is a service node of Alibaba Cloud. By deploying services in different Alibaba Cloud regions, you can get your services closer to your customers, achieving lower access latency and a better user experience. Currently, Alibaba Cloud provides multiple regions throughout China.

Project

A project is a basic management unit in Log Service and is used to isolate and control resources. You can use a project to manage all the logs and related log sources of an application.

Logstore

A Logstore is a unit in Log Service to collect, store, and consume logs. Each Logstore belongs to one project, and each project can have multiple Logstores. You can create multiple Logstores for a project as needed. Typically, an independent Logstore is created for each type of logs of an application. For example, you have a game application big-game, and three types of logs exist on the server: operation_log, application_log, and access_log. In this case, you can create a project named big-game and then create three Logstores in this project to collect, store, and consume the three types of logs respectively.

Log

A log is the minimum data unit processed in Log Service. Log Service uses a semi-structured data mode to define a log. The specific data model is as follows:
  • Topic: a user-defined field used to mark multiple logs. For example, access logs can be marked based on the sites. By default, this field is an empty string, which is also a valid topic.
  • Time: a reserved field in the log, which is used to indicate the log generation time (the number of seconds since 1970-1-1 00:00:00 UTC). Generally, this field is generated directly based on the time in the log.
  • Content: a field used to record the specific log content. The content consists of one or more content items. Each content item is a key-value pair.
  • Source: a field used to indicate the source of a log. For example, this field indicates the IP address of the host where the log is generated. This field is empty by default.

Log Service has different requirements on the values of different fields, as described in the following table.

Table 1. Log fields
Field Requirement
time An integer in the standard UNIX timestamp format, in seconds.
topic A UTF-8 encoded string up to 128 bytes in length.
source A UTF-8 encoded string up to 128 bytes in length.
content One or more key-value pairs. A key is a UTF-8 encoded string up to 128 bytes in length. It can contain only letters, underscores (_), and digits and cannot start with a digit. A value is a UTF-8 encoded string up to 1024 × 1024 bytes in length.
tags Tags appear in alphabetical order. The key and value of each tag are strings.
Note The key in the content field cannot use any of the following keywords: __time__, __source__, __topic__, __partition_time__, _extract_others_, and __extract_others__.

Topic

Logs in one Logstore can be grouped by log topics. You can specify the topic when writing a log. For example, you can use your ID as the log topic when writing logs. If you do not need to group logs in a Logstore, specify the same log topic for all of the logs.

Note By default, the log topic is an empty string, which is also a valid topic.

Various log formats are used in actual scenarios. The following example shows how to map a raw NGINX access log to the log data model of Log Service. Assume that the IP address of your NGINX server is 10.10.10.1. A raw log on this server is as follows:

10.1.1.1 - - [01/Mar/2012:16:12:07 +0800] "GET /Send? AccessKeyId=82251054** HTTP/1.1" 200 5 "-" "Mozilla/5.0 (X11; Linux i686 on x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"

Map the raw log to the log data model of Log Service, as described in the following table.

Table 2. Log data model of Log Service
Field Content Description
topic "" Use the default value (empty string).
time 1330589527 The precise time when the log is generated, in seconds. The time is converted from the timestamp of the raw log.
source "10.10.10.1" Use the IP address of the server as the log source.
content Key-value pair The specific content of the log.

You can decide how to extract the raw log content and combine the extracted content into key-value pairs, as described in the following table.

Table 3. Custom key-value pairs
Key Value
ip "10.1.1.1"
method "GET"
status "200"
length "5"
ref_url "-"
browser "Mozilla/5.0 (X11; Linux i686 on x86_64; rv:10.0.2) Gecko/20100101 Firef"

Logs

A collection of multiple logs.

Log group

A group of logs.

LogGroupList

A collection of log groups used to return results.

Encoding method

The following table lists the content encoding method that the system currently supports. More encoding methods will be available in the future. The supported encoding methods are specified in the Content-Type field in the RESTful API.

Table 4. Supported encoding method
Definition Content-Type
Protobuf The data model is encoded by Protobuf. application/x-protobuf

For more information about the Protobuf format, see Data encoding method.

Note

Protobuf does not require the key-value pair to be unique. However, you cannot use the same key. Otherwise, the behavior is undefined.

Protobuf must follow the order of field number to encode fields in one message. Otherwise, data may fail to be parsed.