This topic describes how to ship data from Log Service to OSS and store the data in the comma-separated values (CSV) format.
Configuration parameters
You can set the storage format of data that is shipped to OSS. The following table shows how to set the storage format to CSV. For more information, see Configure a data shipping rule.
The following table describes the configuration parameters in the preceding figure. For more information, visit
Common Format and MIME Type for CSV Files or
CSV Format in PostgreSQL.
| Parameter | Description |
|---|---|
| CSV Fields | The names of the log fields that you want to ship to OSS. You can view log fields on the Raw Logs tab of a Logstore and enter the names of the fields that you want to ship to OSS in the Key Name column. The log fields that you can ship to OSS include the fields in the log content and the reserved fields such as __time__, __topic__, and __source__. For more information about reserved fields, see Reserved fields.
Note The keys that you enter in the
CSV Fields field must be unique.
|
| Delimiter | You can use commas (,), vertical bars (|), spaces, or tabs to delimit fields. |
| Escape Character | If a field contains a delimiter, you must use an escape character to enclose the field. This ensures that the field is not delimited. |
| Invalid Fields | If a key that you specify in the CSV Fields field does not exist, enter the value of the key in the Invalid Fields field. |
| Shipped Fields | If you turn on the Shipped Fields switch, field names are written in the CSV file. |
Directories in OSS buckets
The following table lists the directories in OSS buckets that store data shipped from Log Service.
| Compressed | File extension | Example | Description |
|---|---|---|---|
| No | .csv | oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.csv | You can download the raw JSON object to the local host and open the object as a text file. |
| snappy | .snappy.csv | oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.snappy.csv | For more information about the Snappy utility, see Snappy compression. |
Examples
To use HybridDB for MySQL or HybridDB for PostgreSQL to consume CSV files in OSS, set the fields as described in the following list:
- Delimiter: Select comma (,).
- Escape Character: Select double quotation marks (").
- Invalid Fields: Null.
- Shipped Fields: Turn off the switch. By default, no key name is added in the first line of the CSV file in HybridDB for MySQL or HybridDB for PostgreSQL.