Store logs in CSV format when shipping logs to OSS - Simple Log Service

When shipping logs to OSS, you can store them in CSV format with configurable delimiters, escape characters, and other CSV parameters.

Parameter configuration

When you create an OSS shipping job, set Storage Format to csv and configure the following parameters.

The following table describes the parameters. For more information, see the CSV standard and the PostgreSQL CSV documentation.

Parameter	Description
CSV Keys	On the Raw Logs tab, find the log fields and enter the field names to ship to OSS in the desired order. Simple Log Service also provides reserved fields such as __time__, __topic__, and __source__. For more information, see Reserved fields. Note Specify each log field only once in the CSV Keys setting. If your logs contain duplicate field names (for example, two fields named request_time), Log Service may display one as request_time_0 in the console. The underlying stored name remains request_time. Use the original field name request_time when configuring the shipping rule. With duplicate field names, the system randomly ships only one value. Avoid duplicate field names for predictable results.
Delimiter	The character used to separate fields in a log entry. Supported characters: comma (,), vertical bar (\|), space, and tab.
Quote	If a field contains the delimiter, enclose it with the escape character to prevent incorrect parsing.
Invalid Key Value	The placeholder for a field specified in CSV Keys that does not exist in the raw logs.
Shipping Field Name	When enabled, the field names are written as the first line (header) of the CSV file.

OSS object URL

Shipped log objects in OSS use the following URL structure.

Note

If you specify a custom object suffix when you create the OSS shipping job, that suffix is used.
If you do not specify a custom object suffix, the compression type determines the suffix.

Compression type	Object suffix	URL example	Description
Not compressed	If you specify a custom object suffix, such as .suffix, the object name ends with that suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix	Uncompressed CSV files can be downloaded and opened with a text editor.
Not compressed	If you do not specify a custom object suffix, the object name ends with .csv.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.csv
snappy	If you specify a custom object suffix, such as .suffix, the object name ends with that suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix	For instructions on opening Snappy-compressed files, see Open Snappy-compressed files.
snappy	If you do not specify a custom object suffix, the object name ends with .snappy.csv.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.snappy.csv
gzip	If you specify a custom object suffix, such as .suffix, the object name ends with that suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix	Gzip-compressed files can be decompressed by using the gzip command.
gzip	If you do not specify a custom object suffix, the object name ends with .gz.csv.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.gz.csv
zstd	If you specify a custom object suffix, such as .suffix, the object name ends with that suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix	Zstandard-compressed files can be decompressed by using the zstd command.
zstd	If you do not specify a custom object suffix, the object name ends with .zst.csv.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.zst.csv