Create OSS data shipping job - Simple Log Service - Simple Log Service

Simple Log Service (SLS) can ship log data from a LogStore to an Object Storage Service (OSS) bucket for long-term storage and offline analysis. This topic describes how to create an OSS data shipping job (new version).

Prerequisites

A project and a LogStore are created.
Log data is collected.
An OSS bucket is created in the region where the SLS project resides. For more information, see []]().
The new version of OSS data shipping is available only in specific regions. Make sure your project resides in a supported region.

Supported regions

SLS ships data to OSS within the same region. The OSS bucket must reside in the region where the SLS project resides.

Important

The new version of data shipping to OSS is supported only in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hohhot), China (Ulanqab), China (Chengdu), China (Shenzhen), China (Heyuan), China (Guangzhou), China (Hong Kong), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), Philippines (Manila), Thailand (Bangkok), Japan (Tokyo), US (Silicon Valley), US (Virginia), Saudi Arabia (Riyadh), and Saudi virtual network operator.

Create a data shipping job

Log on to the Simple Log Service console.

1. On the Log Storage > Logstores tab, click > to the left of the target LogStore, and then choose Data Processing > Export > Object Storage Service.

Move the pointer over Object Storage Service and click +.

In the Data Shipping to OSS panel, configure the following parameters and click OK.

Set Shipping Version to New Version. The following table describes the key parameters.

After you create an OSS data shipping job, each shard determines its shipping frequency based on Batch Size and Batch Interval. Shipping is triggered when either condition is met.

Parameter	Description
Job Name	The unique name of the shipping job.
Display Name	The display name of the shipping job.
Job Description	The description of the OSS shipping job.
OSS Bucket	The name of the OSS bucket. > Important: - The bucket must already exist, must have Write Once Read Many (WORM) disabled, and must reside in the same region as the SLS project.
OSS Write RAM Role	Grants the OSS data shipping job the permissions to write data to the OSS bucket. If the LogStore and the OSS bucket belong to the same Alibaba Cloud account, select Default Role. If they belong to different accounts, select Custom Role. - Default Role: Authorizes the OSS data shipping job to write data to the OSS bucket as the `AliyunLogDefaultRole` system role of Alibaba Cloud. Enter the Alibaba Cloud Resource Name (ARN) of `AliyunLogDefaultRole`. For information about how to obtain the ARN, see []](). - Custom Role: Authorizes the OSS data shipping job to write data to the OSS bucket as a custom role. Grant the custom role the permissions to write data to the OSS bucket, and then enter the ARN of the custom role in OSS Write RAM Role. Obtain the ARN based on the following scenarios: - If the LogStore and the OSS bucket belong to the same Alibaba Cloud account, see []](). - If the LogStore and the OSS bucket belong to different Alibaba Cloud accounts, see []]().
LogStore Read RAM Role	Grants the OSS data shipping job the permissions to read data from the LogStore. If the LogStore and the OSS bucket belong to the same Alibaba Cloud account, select Default Role. If they belong to different accounts, select Custom Role. - Default Role: Authorizes the OSS data shipping job to read data from the LogStore as the `AliyunLogDefaultRole` system role of Alibaba Cloud. Enter the ARN of `AliyunLogDefaultRole`. For information about how to obtain the ARN, see []](). - Custom Role: Authorizes the OSS data shipping job to read data from the LogStore as a custom role. Grant the custom role the permissions to read data from the LogStore, and then enter the ARN of the custom role in LogStore Read RAM Role.
Storage Format	After data is shipped to OSS, the data can be stored in different file formats.
Compress	The compression method used to store data in OSS. - No Compress (none): does not compress data. - Compress (snappy): compresses data by using the snappy algorithm to reduce the storage space of the OSS bucket. - Compress (zstd): compresses data by using the zstd algorithm to reduce the storage space of the OSS bucket. - Compress (gzip): compresses data by using the gzip algorithm to reduce the storage space of the OSS bucket.
Ship Tags	The tag field is a reserved field of SLS.
Batch Size	The amount of uncompressed log data that must accumulate in a shard before shipping starts. Valid values: 5 to 256. Unit: MB. This value controls the approximate size of each OSS object. Batch Size measures the data accumulated after SLS starts reading, not the data already written to SLS. Data is read and shipped only after the Batch Interval condition is also met.
Batch Interval	The maximum time a shard waits before triggering a shipping operation. Shipping starts when the specified time has elapsed since the first log entry arrived in the current batch. Default value: 300. Valid values: 300 to 900. Unit: seconds.
Shipping Latency	The delay before data is shipped. For example, if you set this parameter to 3600, data is shipped one hour later. Data generated at 10:00:00 on June 5, 2023 is not written to the specified OSS bucket earlier than 11:00:00 on June 5, 2023.
Start Time Range	The time range of the OSS data shipping job. The time range is based on the time when logs are received. Valid values: - All: ships data from the time when the LogStore received the first log until the job is manually stopped. - From Specific Time: ships data from the specified start time until the job is manually stopped. - Specific Time Range: ships data from the specified start time to the specified end time. The job automatically stops at the specified end time. The time range is based on the `__tag__:__receive_time__` field.
Time Zone	The time zone used to format time values. If you configure both Time Zone and Partition Format, subdirectories in the OSS bucket are generated based on your settings.

After the job is created, check the job status and verify that data appears in the destination OSS bucket to confirm that the shipping job works as expected.

View OSS data

After data is shipped to OSS, you can access the data in the OSS console or by using the OSS API, an SDK, or other methods.

Partition formats

Each shipping operation maps to an OSS object URL in the oss://OSS-BUCKET/OSS-PREFIX/PARTITION-FORMAT_RANDOM-ID format. The following table describes the partition formats, based on a shipping job created at 19:50:43 on January 20, 2022.

OSS-BUCKET is the name of the OSS bucket. OSS-PREFIX is the directory prefix. PARTITION-FORMAT is the partition format, which is calculated from the shipping time by using the strptime API. RANDOM-ID is the unique identifier of a shipping operation.

OSS Bucket	OSS Prefix	Partition format	Object suffix	URL of the OSS object
test-bucket	test-table	%Y/%m/%d/%H/%M	.suffix	oss://test-bucket/test-table/2022/01/20/19/50_1484913043351525351_2850008.suffix
test-bucket	`log_ship_oss_example`	year=%Y/mon=%m/day=%d/log_%H%M	.suffix	oss://test-bucket/log_ship_oss_example/year=2022/mon=01/day=20/log_1950_1484913043351525351_2850008.suffix
test-bucket	`log_ship_oss_example`	ds=%Y%m%d/%H	.suffix	oss://test-bucket/log_ship_oss_example/ds=20220120/19_1484913043351525351_2850008.suffix
test-bucket	`log_ship_oss_example`	%Y%m%d/	.suffix	oss://test-bucket/log_ship_oss_example/20220120/_1484913043351525351_2850008.suffix > Note: This format may cause parsing failures on platforms such as Hive. We recommend that you do not use this format.

Note

Data is shipped to OSS in batches. Each shipping operation writes one object that contains a batch of data. The object path is determined by the earliest receive_time (the time when the data arrives in SLS) in the batch. Take note of the following scenarios:

When you ship real-time data, for example, once every five minutes, a shipping operation performed at 00:00:00 on January 22, 2022 ships the data that was written to a shard of SLS after 23:55 on January 21, 2022. To analyze all data of January 22, 2022, check all objects in the 2022/01/22 subdirectory of the OSS bucket, and check whether the last objects in the 2022/01/21 subdirectory contain data of January 22, 2022.
When you ship historical data, if the amount of data in the LogStore is small, a single pull may return data of multiple days. As a result, objects in the 2022/01/22 subdirectory may contain all data of January 23, 2022, and no object exists in the 2022/01/23 subdirectory.
When you use a big data platform such as Hive or MaxCompute, or Alibaba Cloud Data Lake Analytics (DLA), to analyze OSS data, you can set PARTITION-FORMAT in the object name to the key=value format if you want to use partition information. Example: oss://test-bucket/log_ship_oss_example/year=2022/mon=01/day=20/log_195043_1484913043351525351_2850008.parquet. In this example, three partition columns are configured: year, mon, and day.