Logs are records of changes that occur in a system when the system is running. The records contain information about the operations that are performed on specific objects and the results of the operations. The records are ordered by time.
Formats
Log data is stored in different formats such as log files, events, binary logs, and metrics. Log Service uses a semi-structured data model to define logs. A log consists of the following data domains: log topic, log time, log content, log source, and log tag. Log Service has different format requirements for each data domain. The following table describes the data domains and format requirements.
Data domain | Description | Format |
---|---|---|
Log topic | The reserved field __topic__ in Log Service specifies the topic of a log. You can specify different log topics. For example, you can specify access_log and operation_log for website logs based on the types of the logs. For more information, see Topic. | The field value is a string of up to 128 bytes. The field value can be an empty string. If the field value is an empty string, no log topic is specified. |
Log time | The reserved field __time__ in Log Service specifies the time of a log. For more information, see Reserved fields. | The field value is a UNIX timestamp. |
Log content | The content of a log consists of one or more items. Each item is a key-value pair. If you use Logtail to collect single-line or multi-line logs in simple mode, Logtail does not parse the content of the logs. Each raw log is uploaded to the content field. | A key-value pair must comply with the following requirements:
|
Log source | The reserved field __source__ in Log Service specifies the source of a log. For example, the field value can be the IP address of the server for which a log is generated. | The field value is a string of up to 128 bytes. |
Log tag | The tags of a log. Valid values:
| A tag field value is in the dictionary format. The keys and the values in a dictionary are strings. A tag field is prefixed with __tag__: in logs. |
Examples
In the following example, a website access log is used as the raw log. The following example shows the mapping between the raw log and the new log that is collected to Log Service based on a specified data model.
- Raw log
127.0.0.1 - - [01/Mar/2021:12:36:49 0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
- New log obtained after the raw log is collected to Log Service in simple mode
The raw log is uploaded to the content field.
- New log obtained after the raw log is collected to Log Service in full regex mode
Logtail splits the raw log into multiple key-value pairs based on the regular expression that you specify.