CSV format: parameters and file paths - Simple Log Service

Log Service can ship logs to OSS in CSV format with configurable delimiters, escape characters, and field mappings.

Important

The old version of shipping logs to OSS is discontinued. Refer to the new version.

Parameters

When you configure a shipping rule, set Storage Format to csv and configure the following parameters.

Parameter	Description
CSV Fields	The log field keys to ship to OSS. Key order determines CSV column order. Find field keys on the Raw Logs tab. You can also include reserved fields such as __time__, __topic__, and __source__. Note Each key can be specified only once in CSV Fields. If your logs contain duplicate field names (for example, two fields named request_time), Log Service may display one as request_time_0 in the console. The underlying stored name remains request_time. Use the original field name request_time when configuring the shipping rule. With duplicate field names, the system randomly ships only one value. Avoid duplicate field names for predictable results.
Delimiter	The character that separates fields. Supported: comma (,), vertical bar (\|), space, and tab.
Escape Character	Encloses field values that contain the delimiter, preventing parsing errors.
Invalid Fields	The placeholder value when a key from CSV Fields is not found in a log entry.
Shipped Fields	When the shipped fields switch is enabled, field names are included in the CSV file.

Shipped log files in OSS use the following path structure.

Compression type	File extension	Example OSS file path	Description
Uncompressed	.csv	oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.csv	Download and open directly with any text editor.
Snappy	.snappy.csv	oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.snappy.csv	Decompress before viewing. Open Snappy-compressed files.