Logs are records of changes that occur in a system when the system is running. The records contain information about the operations that are performed on specific objects and the results of the operations. The records are ordered by time.

Formats

Log data is stored in different formats such as log files, events, binary logs, and metrics. Log Service uses a semi-structured data model to define logs. A log consists of the following data domains: log topic, log time, log content, log source, and log tag. Log Service has different format requirements for each data domain. The following table describes the data domains and format requirements.

Data domainDescriptionFormat
Log topicThe reserved field __topic__ in Log Service specifies the topic of a log. You can specify different log topics. For example, you can specify access_log and operation_log for website logs based on the types of the logs. For more information, see Topic. The field value is a string of up to 128 bytes. The field value can be an empty string.

If the field value is an empty string, no log topic is specified.

Log timeThe reserved field __time__ in Log Service specifies the time of a log. For more information, see Reserved fields. The field value is a UNIX timestamp.
Log contentThe content of a log consists of one or more items. Each item is a key-value pair.

If you use Logtail to collect single-line or multi-line logs in simple mode, Logtail does not parse the content of the logs. Each raw log is uploaded to the content field.

A key-value pair must comply with the following requirements:
  • The key specifies the name of the field. The key must be a UTF-8 encoded string. The key can contain letters, digits, and underscores (_). The key cannot start with a digit. The string is 1 to 128 bytes in length. You cannot use the following fields:
    • __time__
    • __source__
    • __topic__
    • __partition_time__
    • _extract_others_
    • __extract_others__
  • The value specifies the value of the field. The value can be a string of up to 1 MB.
Log sourceThe reserved field __source__ in Log Service specifies the source of a log. For example, the field value can be the IP address of the server for which a log is generated. The field value is a string of up to 128 bytes.
Log tagThe tags of a log. Valid values:
  • Custom tags: the tags that you add when you call the PutLogs operation to write logs to a specified Logstore.
  • System tags: the tags that are added by Log Service. The tags include __client_ip__ and __receive_time__.
A tag field value is in the dictionary format. The keys and the values in a dictionary are strings. A tag field is prefixed with __tag__: in logs.

Examples

In the following example, a website access log is used as the raw log. The following example shows the mapping between the raw log and the new log that is collected to Log Service based on a specified data model.

  • Raw log
    127.0.0.1 - - [01/Mar/2021:12:36:49  0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
  • New log obtained after the raw log is collected to Log Service in simple mode

    The raw log is uploaded to the content field.

    Sample log
  • New log obtained after the raw log is collected to Log Service in full regex mode

    Logtail splits the raw log into multiple key-value pairs based on the regular expression that you specify.

    Sample log