Logs are records of changes that occur in a system when the system is running. The records contain information about the operations that are performed on specific objects and the results of the operations. The records are ordered by time.
Log data is stored in different formats such as log files, events, binary logs, and metrics. Log Service uses a semi-structured data model to define logs. A log consists of the following data domains: log topic, log time, log content, log source, and log tag. Log Service has different format requirements for each data domain. The following table describes the data domains and format requirements.
|Log topic||The reserved field ||The field value is a string of up to 128 bytes. The field value can be an empty string. |
If the field value is an empty string, no log topic is specified.
|Log time||The reserved field ||The field value is a UNIX timestamp.|
|Log content||The content of a log consists of one or more items. Each item is a |
If you use Logtail to collect single-line or multi-line logs in simple mode, Logtail does not parse the content of the logs. Each raw log is uploaded to the content field.
|Log source||The reserved field ||The field value is a string of up to 128 bytes.|
|Log tag||The tags of a log. Valid values: ||A tag field value is in the dictionary format. The keys and the values in a dictionary are strings. A tag field is prefixed with __tag__: in logs.|
In the following example, a website access log is used as the raw log. The following example shows the mapping between the raw log and the new log that is collected to Log Service based on a specified data model.
- Raw log
127.0.0.1 - - [01/Mar/2021:12:36:49 0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
- New log obtained after the raw log is collected to Log Service in simple mode
The raw log is uploaded to the content field.
- New log obtained after the raw log is collected to Log Service in full regex mode
Logtail splits the raw log into multiple key-value pairs based on the regular expression that you specify.