Ship logs to OSS in ORC format - Simple Log Service - Alibaba Cloud - Simple Log Service

Parameters

When you create an OSS data shipping job (new version), set the Storage Format to orc. Then, set Compression to snappy. Supported compression methods include none, snappy, gzip, and zstd. In the ORC Fields section, add a field with the Key Name __topic__ and Type string, and another field with the Key Name concent and Type string. You can click the blue + button to add a new field, or click Delete to remove an existing field.

The following table describes the ORC-specific parameters.

Parameter

Description

Key Name

The log fields to ship to OSS. You can view available fields on the Raw Logs tab of a Logstore. Add field names one by one. The shipping job preserves the field order and uses the field names as ORC column names.

The log fields that you can ship to OSS include the reserved fields such as __time__, __topic__, and __source__. For more information, see Reserved fields.

In the following cases, a column value in the ORC file is null:

The specified field name does not exist in the Logstore.
The specified log field fails to be converted from the STRING type to a non-STRING type such as DOUBLE or INT64.

Note

Each log field can be configured as an ORC field only once.
If your logs contain duplicate field names (for example, two fields named request_time), Log Service may display one as request_time_0 in the console. The underlying stored name remains request_time. Use the original field name request_time when configuring the shipping rule.

With duplicate field names, the system randomly ships only one value. Avoid duplicate field names for predictable results.

Type

The data type of the log field. Supported ORC types: STRING, BOOLEAN, INT32, INT64, FLOAT, and DOUBLE.

During shipping, log fields are converted from STRING to the specified ORC type. If conversion fails, the column value is null.

Sample URLs of OSS objects

Shipped logs are stored as objects in OSS buckets. The following table shows sample object URLs.

Note

If you specify an object suffix when you create a data shipping job, the OSS objects use the suffix.
If you do not specify an object suffix when you create a data shipping job, the OSS objects use the suffix that is generated based on the compression type.

Compression type	Object suffix	Sample URL	Example
Not compressed	If you specify an object suffix, the specified suffix takes effect. Example: .suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix	You can download the OSS object to your computer and use ORC tools to open the object.
Not compressed	If you do not specify an object suffix, .orc is used as the object suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.orc
Snappy	If you specify an object suffix, the specified suffix takes effect. Example: .suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix
Snappy	If you do not specify an object suffix, .snappy.orc is used as the object suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.snappy.orc
Zstandard	If you specify an object suffix, the specified suffix takes effect. Example: .suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix
Zstandard	If you do not specify an object suffix, .zst.orc is used as the object suffix.	oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.zst.orc

Data consumption

You can consume ORC data shipped to OSS by using E-MapReduce, Spark, or Hive. For more information, see LanguageManual DDL.

You can also inspect ORC files directly with command-line tools.

Use ORC tools to view ORC file metadata and read data. Download orc-tools-1.7.2-uber.jar from the Maven repository to verify shipped data.

View metadata

Run the following command:

java -jar ~/Downloads/orc-tools-1.7.2-uber.jar meta -p file.orc

Output:

Processing data file /Users/xx/file.orc [length: 200779]
Structure for /Users/xx/file.orc
File Version: 0.12 with ORC_CPP_ORIGINAL by ORC C++ 1.7.2
Rows: 124022
Compression: ZSTD
Compression size: 65536
Calendar: Julian/Gregorian
Type: struct<bucket:string,bucket_region:string>
Stripe Statistics:
  Stripe 1:
    Column 0: count: 124022 hasNull: false
    Column 1: count: 124022 hasNull: false min: bucket0 max: sls-training-data sum: 1468133
    Column 2: count: 0 hasNull: true
File Statistics:
  Column 0: count: 124022 hasNull: false
  Column 1: count: 124022 hasNull: false min: bucket0 max: sls-training-data sum: 1468133
  Column 2: count: 0 hasNull: true
Stripes:
  Stripe: offset: 3 data: 199856 rows: 124022 tail: 97 index: 578
    Stream: column 0 section ROW_INDEX start: 3 length 102
    Stream: column 1 section ROW_INDEX start: 105 length 367
    Stream: column 2 section ROW_INDEX start: 472 length 109
    Stream: column 0 section PRESENT start: 581 length 25
    Stream: column 1 section PRESENT start: 606 length 25
    Stream: column 1 section LENGTH start: 631 length 38989
    Stream: column 1 section DATA start: 39620 length 160794
    Stream: column 2 section PRESENT start: 200414 length 23
    Stream: column 2 section LENGTH start: 200437 length 0
    Stream: column 2 section DATA start: 200437 length 0
    Encoding column 0: DIRECT
    Encoding column 1: DIRECT_V2
    Encoding column 2: DIRECT_V2
File length: 200779 bytes
Padding length: 0 bytes
Padding ratio: 0%

Read data

Run the following command:

java -jar ~/Downloads/orc-tools-1.7.2-uber.jar data -n 5 file.orc

Output:

Processing data file /Users/xx/file.orc [length: 200779]
{"bucket":"bucket3","bucket_region":"cn-hangzhou"}
{"bucket":"bucket3","bucket_region":"cn-hangzhou"}
{"bucket":"bucket4","bucket_region":"cn-hangzhou"}
{"bucket":"dashboard-bucket","bucket_region":"cn-hangzhou"}
{"bucket":"bucket2","bucket_region":null}

For more information, run the java -jar orc-tools-1.7.2-uber.jar command or see ORC tools documentation.