This topic describes the parameters that you can set when you ship logs from Log Service to OSS and store the log data in the CSV format.

CSV keys

On the consumption preview page or index query page of Log Service, you can view multiple key-value pairs of a log. You can also enter the field names (keys) that you want to ship to OSS.
Note The keys that you add in the CSV Keys section must be unique.
CSV keys
The following table describes the parameters for the CSV storage format.
Parameter Type Description
Delimiter String A single-character string that is used to separate keys.
Escape character String A single-character string. If a key contains a delimiter or a line break, use double quotation marks ("") to enclose the key. This ensures key separation and prevents data reading errors.
Invalid Fields String If the specified key value does not exist, this string is entered in the key to indicate that the key is null.
Display Key Boolean Indicates whether to add the key name to the first line of the CSV file.
For more information, visit CSV standard and PostgreSQL documentation.
During log shipping from Log Service to OSS, you can use the reserved keys that are listed in the following table.
Reserved key Description
__time__ The Unix timestamp of a log. The value represents the number of seconds that have elapsed since the epoch time January 1, 1970, 00:00:00 UTC. This value is calculated based on the time key of the log.
__topic__ The topic of a log.
__source__ The client IP address of the log source.
Note You can select these reserved keys as required when you ship data to OSS and store the data in the CSV format. For example, if you require the log topic, you can set the key name to __topic__.

OSS storage addresses

Compression type File extension Example
Uncompressed .csv oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.csv
SNAPPY .snappy.csv oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.snappy.csv
  • CSV is a readable format. A CSV file can be downloaded from OSS and viewed as a TXT file.
  • For more information about SNAPPY compression, see Snappy compression.

Configuration example

If you use HybridDB for MySQL to consume CSV file data shipped to OSS, we recommend that you use the following configurations:

  • Delimiter: comma (,).
  • Escape Character: double quotation marks (").
  • Invalid Fields: null.
  • Display Key: turned off. By default, no key name is added in the first line of the CSV file in HybridDB for MySQL.