Parquet format - Ship logs to OSS | Simple Log Service - Simple Log Service

Important

The old version of shipping logs to OSS is discontinued. Refer to the new version.

Parameters

When you configure a shipping rule and set Storage Format to Parquet, configure the following parameters. In the Parquet Fields section, add two fields: set Key Name to __topic__ with Type string, and Key Name to content with Type string. Click + to add more fields or Delete to remove fields.

The following table describes the parameters.

Parameter

Description

Key Name

Enter field names (keys) from the Raw Logs tab in the order you want them shipped. The keys become Parquet column names. In addition to log content keys, you can use reserved fields __time__, __topic__, and __source__ (Reserved fields). A column value is null in the following cases.

The specified key does not exist in the log entry.
The field has a non-string type (such as double or int64) and the value cannot be converted.

Note

Each key can appear only once in the Parquet Fields section.
If your logs contain duplicate field names (for example, two fields named request_time), Log Service may display one as request_time_0 in the console. The underlying stored name remains request_time. Use the original field name request_time when configuring the shipping rule.

With duplicate field names, the system randomly ships only one value. Avoid duplicate field names for predictable results.

Type

Supported types: string, boolean, int32, int64, float, and double. During shipping, string data is stored as the byte_array physical type. Simple Log Service does not set the logical_type field in Parquet metadata.

OSS object paths

Log objects shipped to OSS use the following path formats.

Compression type	Object suffix	Example object path	Description
Not compressed	.parquet	oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.parquet	Download the object to your local machine. Consume data.
Snappy	.snappy.parquet	oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.snappy.parquet	Download the object to your local machine. Consume data.

Consume data

Use E-MapReduce, Spark, or Hive. Refer to the Hive DDL documentation.

Use local inspection tools.

Install parquet-tools to validate, inspect, and read Parquet files.

pip3 install parquet-tools

View specific columns in a Parquet file.

Command

Example: view the remote_addr and body_bytes_sent columns.

parquet-tools show -n 2 -c remote_addr,body_bytes_sent 44_1693464263000000000_2288ff590970d092.parquet

Output

+----------------+-------------------+
| remote_addr    |   body_bytes_sent |
|----------------+-------------------|
| 61.243.1.63    |           b'1904' |
| 112.235.74.182 |           b'4996' |
+----------------+-------------------+

Convert a Parquet file to CSV:

Command

parquet-tools csv -n 2 44_1693464263000000000_2288ff590970d092.parquet

Output

remote_addr,body_bytes_sent,time_local,request_method,request_uri,http_user_agent,remote_user,request_time,request_length,http_referer,host,http_x_forwarded_for,upstream_response_time,status
b'61.**.**.63',b'1904',b'31/Aug/2023:06:44:01',b'GET',b'/request/path-0/file-7',"b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.801.0 Safari/535.1'",b'uh2z',b'49',b'4082',b'www.kwm.mock.com',b'www.ap.mock.com',b'222.**.**.161',b'2.63',b'200'
b'112.**.**.182',b'4996',b'31/Aug/2023:06:44:01',b'GET',b'/request/path-1/file-5',b'Mozilla/5.0 (Windows NT 6.1; de;rv:12.0) Gecko/20120403211507 Firefox/12.0',b'tix',b'71',b'1862',b'www.gx.mock.com',b'www.da.mock.com',b'36.**.**.237',b'2.43',b'200'

View file metadata.

Command

parquet-tools inspect 44_1693464263000000000_2288ff590970d092.parquet

Output

############ file meta data ############
created_by: SLS version 1
num_columns: 14
num_rows: 4661
num_row_groups: 1
format_version: 1.0
serialized_size: 2345
############ Columns ############
remote_addr
body_bytes_sent
time_local
request_method
request_uri
http_user_agent
remote_user
request_time
request_length
http_referer
host
http_x_forwarded_for
upstream_response_time
status
############ Column(remote_addr) ############
name: remote_addr
path: remote_addr
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: None
converted_type (legacy): NONE
compression: UNCOMPRESSED (space_saved: 0%)
......