Store logs in CSV format when shipping logs to OSS - Simple Log Service

Simple Log Service supports storing shipped logs in Object Storage Service (OSS) in CSV format. You can configure delimiters, escape characters, and other CSV parameters when creating an OSS shipping job.

Parameter configuration

When you create an OSS shipping job, set Storage Format to csv and configure the following parameters.

The following table lists the parameters. For more information, see the CSV standard and the PostgreSQL CSV documentation.

Parameter	Description
CSV Keys	View the log fields on the Raw Logs tab and enter the names of the fields to ship to OSS in the desired order. Simple Log Service also provides reserved fields such as __time__, __topic__, and __source__. For more information about reserved fields, see Reserved fields. Note Specify each log field only once in the CSV Keys setting. If your logs contain duplicate field names (for example, two fields named request_time), Log Service may display one as request_time_0 in the console. The underlying stored name remains request_time. Use the original field name request_time when configuring the shipping rule. With duplicate field names, the system randomly ships only one value. Avoid duplicate field names for predictable results.
Delimiter	The character that separates fields in a log entry. Supported characters include comma (,), vertical bar (\|), space, and tab.
Quote	If a field contains the delimiter, enclose the field with the escape character to prevent incorrect parsing.
Invalid Key Value	The placeholder value for a field specified in CSV Keys that does not exist in the raw logs.
Shipping Field Name	When enabled, Simple Log Service writes the field names as the first line (header) of the CSV file.

OSS object URL

OSS objects that contain shipped logs use the following URL structure.

Note

If you specify a custom object suffix when you create the OSS shipping job, that suffix is used.
If you do not specify a custom object suffix, the compression type determines the suffix.

Compression type	Object suffix	URL example	Description
Not compressed	If you specify a custom object suffix, such as .suffix, the object name ends with that suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix	You can download uncompressed CSV files and open them with a text editor.
Not compressed	If you do not specify a custom object suffix, the object name ends with .csv.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.csv
snappy	If you specify a custom object suffix, such as .suffix, the object name ends with that suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix	To learn how to open Snappy-compressed files, see Open Snappy-compressed files.
snappy	If you do not specify a custom object suffix, the object name ends with .snappy.csv.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.snappy.csv
gzip	If you specify a custom object suffix, such as .suffix, the object name ends with that suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix	You can decompress Gzip-compressed files by using the gzip command.
gzip	If you do not specify a custom object suffix, the object name ends with .gz.csv.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.gz.csv
zstd	If you specify a custom object suffix, such as .suffix, the object name ends with that suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix	You can decompress Zstandard-compressed files by using the zstd command.
zstd	If you do not specify a custom object suffix, the object name ends with .zst.csv.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.zst.csv