After log data is shipped from Log Service to Object Storage Service (OSS), the log data can be stored in different formats. This topic describes the CSV format.
Parameters
When you configure a shipping rule, you can select csv for Storage Format. The following table describes the parameters that you must configure if you select csv. For more information about how to specify the storage format, see Configure a data shipping rule.
For more information, see Common Format and MIME Type for Comma-Separated Values (CSV) Files and PostgreSQL 9.4.26 Documentation.
| Parameter | Description |
|---|---|
| CSV Fields | The names of the log fields that you want to ship to OSS. You can view log fields
on the Raw Logs tab of a Logstore. You must add the log fields in the Key Name column one by one.
The log fields that you can ship to OSS include the fields in the log content and
the reserved fields such as __time__, __topic__, and __source__. For more information about reserved fields, see Reserved fields.
Note The log fields that you add in the CSV Fields field must be unique.
|
| Delimiter | The delimiter used to delimit log fields. You can use commas (,), vertical bars (|), spaces, or tabs to delimit log fields. |
| Escape Character | If a log field contains a delimiter, use an escape character to enclose the field. This way, the field is not delimited. |
| Invalid Fields | If a log field that you specify in the CSV Fields field does not exist in logs, the value that you specify in the Invalid Fields field is shipped. |
| Shipped Fields | If you turn on Shipped Fields, the names of log fields are written to CSV files. |
Directories of OSS buckets
The following table describes the directories of OSS buckets that store data shipped from Log Service.
| Compression type | File extension | Example | Description |
|---|---|---|---|
| Not compressed | .csv | oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.csv | You can download the CSV file to your computer and open the files as text files. |
| Snappy | .snappy.csv | oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.snappy.csv | For more information about how to open Snappy compressed files, see Decompression tools for Snappy compressed files. |
Examples
If you want to use HybridDB for PostgreSQL to consume data in CSV files in OSS, you
must configure the parameters based on the following list:
- Delimiter: Select comma (,).
- Escape Character: Select double quotation marks (").
- Invalid Fields: Leave it empty.
- Shipped Fields: Turn off the switch. By default, no field names are added to the first line of the CSV file in OSS.