This topic describes how to ship data from Log Service to OSS and store the data in the comma-separated values (CSV) format.

Configuration parameters

You can set the storage format of data that is shipped to OSS. The following table shows how to set the storage format to CSV. For more information, see Configure a data shipping rule.

CSV fields
The following table describes the configuration parameters in the preceding figure. For more information, visit Common Format and MIME Type for CSV Files or CSV Format in PostgreSQL.
Parameter Description
CSV Fields The names of the log fields that you want to ship to OSS. You can view log fields on the Raw Logs tab of a Logstore and enter the names of the fields that you want to ship to OSS in the Key Name column. The log fields that you can ship to OSS include the fields in the log content and the reserved fields such as __time__, __topic__, and __source__. For more information about reserved fields, see Reserved fields.
Note The keys that you enter in the CSV Fields field must be unique.
Delimiter You can use commas (,), vertical bars (|), spaces, or tabs to delimit fields.
Escape Character If a field contains a delimiter, you must use an escape character to enclose the field. This ensures that the field is not delimited.
Invalid Fields If a key that you specify in the CSV Fields field does not exist, enter the value of the key in the Invalid Fields field.
Shipped Fields If you turn on the Shipped Fields switch, field names are written in the CSV file.

Directories in OSS buckets

The following table lists the directories in OSS buckets that store data shipped from Log Service.

Compressed File extension Example Description
No .csv oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.csv You can download the raw JSON object to the local host and open the object as a text file.
snappy .snappy.csv oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.snappy.csv For more information about the Snappy utility, see Snappy compression.

Examples

To use HybridDB for MySQL or HybridDB for PostgreSQL to consume CSV files in OSS, set the fields as described in the following list:
  • Delimiter: Select comma (,).
  • Escape Character: Select double quotation marks (").
  • Invalid Fields: Null.
  • Shipped Fields: Turn off the switch. By default, no key name is added in the first line of the CSV file in HybridDB for MySQL or HybridDB for PostgreSQL.