Simple Log Service:Create an OSS data shipping job (new version)

Supported regions

Create a data shipping job

1. Log on to the Simple Log Service console.

2. In the Projects section, click the one you want.

   

3. On the Log Storage > Logstores tab, find the logstore, click >, and then choose Data Processing > Export > Object Storage Service (OSS).

4. Move the pointer over Object Storage Service (OSS) and click the + icon.

5. In the Create Data Shipping Job panel, select OSS Export and click OK.

6. In the Data Shipping to OSS panel, configure the parameters and click OK.

   Important

   • Data in a shard is shipped to OSS when either the Shipping Size or Shipping Time condition is met.

   • After you create the job, verify the job status and the shipped data in OSS.

   Parameter	Description
   Job Name	The name of the shipping job.
   Display Name	The display name of the shipping job.
   Job Description	The description of the shipping job.
   OSS Bucket	The OSS bucket to ship data to. Important The bucket must have Write Once Read Many (WORM) disabled and reside in the same region as your SLS project. Bucket-level retention policy (BucketWorm) Supported storage classes: Standard, Infrequent Access (IA), Archive, Cold Archive, and Deep Cold Archive. Shipped objects inherit the bucket's storage class. Overview of storage classes Non-Standard storage classes enforce minimum storage periods and minimum billable sizes. Choose a storage class based on your requirements. Comparison of storage classes
   File Delivery Directory	The directory in the OSS bucket to ship data to. The name cannot start with a forward slash (/) or backslash (\). Data from the Logstore is shipped to this directory.
   Object Suffix	The suffix appended to OSS object names. If not specified, SLS generates a suffix based on the storage format and compression type. Example: .suffix.
   Partition Format	The format used to generate subdirectories in the OSS bucket based on shipping time. Default: %Y/%m/%d/%H/%M. Cannot start with a forward slash (/). Partition formats. For partition format parameters, see strptime API.
   OSS Write RAM Role	The authorization method for writing data to the OSS bucket. Valid values: Default Role: Uses the AliyunLogDefaultRole system role. The Alibaba Cloud Resource Name (ARN) is auto-populated. Access data by using a default role Custom Role: Uses a custom RAM role with write permissions to the OSS bucket. Grant the custom role write permissions to the OSS bucket, then enter the ARN in the OSS Write RAM Role field. Obtain the ARN based on your scenario: Same account: Step 2: Grant write permissions for an OSS bucket Cross-account: Step 2: Grant write permissions in Account B
   Logstore Read RAM Role	The authorization method for reading data from the Logstore. Valid values: Default Role: Uses the AliyunLogDefaultRole system role. The ARN is auto-populated. Access data by using a default role Custom Role: Uses a custom RAM role with read permissions to the Logstore. Grant the custom role read permissions to the Logstore, then enter the ARN in the Logstore Read RAM Role field. Obtain the ARN based on your scenario: Same account: Step 1: Grant read permissions for a Logstore Cross-account: Step 1: Grant the role-a RAM role the permissions to read data from a logstore
   Storage Format	The format for storing shipped data in OSS. CSV format, JSON format, Parquet format, and ORC format.
   Compress	Whether to compress shipped data. Valid values: No Compress(none): No compression. Compress(snappy): Compresses data using the snappy algorithm to reduce storage usage. Compress(zstd): Compresses data using the zstd algorithm to reduce storage usage. Compress(gzip): Compresses data using the gzip algorithm to reduce storage usage.
   Ship Tag	Reserved field. Reserved fields
   Batch Size	Shipping starts when the raw data in a shard reaches this threshold. Also determines the size of raw data in each OSS object. Valid values: 5 to 256. Unit: MB. Note This parameter specifies the raw data amount read from a shard, not the stored data amount. Shipping starts only after the Batch Interval condition is also met.
   Batch Interval	Shipping starts when the time span between the first and nth log received in a shard reaches this value. Valid values: 300 to 900. Unit: seconds. Default value: 300.
   Shipping Time	The interval between shipping operations for a shard. Valid values: 300 to 900. Default value: 300. Unit: seconds.
   Shipping Latency	The delay before shipping data. For example, a value of 3600 delays shipping by 1 hour: data generated at 10:00:00 on June 5, 2023 is written to OSS at 11:00:00. Configuration items
   Start Time Range	The time range of data to ship, based on log receive time. Valid values: All: Ships all data in the Logstore from the first log until the job is manually stopped. From Specific Time: Ships data from the specified start time until the job is manually stopped. Specific Time Range: Ships data between the specified start time and end time. Note The time range is based on the __tag__:__receive_time__ field. Reserved fields
   Time Zone	The time zone for formatting time values. When both Time Zone and Shard Format are configured, subdirectories in the OSS bucket are generated accordingly.

View OSS data

After data is shipped, view it in the OSS console, or access it through the OSS API or SDK. Manage objects

oss://OSS-BUCKET/OSS-PREFIX/PARTITION-FORMAT_RANDOM-ID

Partition formats

To analyze OSS data on big data platforms such as Hive, MaxCompute, or Data Lake Analytics (DLA), set PARTITION-FORMAT in key=value format. Example: oss://test-bucket/log_ship_oss_example/year=2022/mon=01/day=20/log_195043_1484913043351525351_2850008.parquet. In this example, year, month, and day are partition key columns.