OSS data shipping (new version) provides high availability, auto-scaling, and data consistency. Review these limits before configuring a shipping job.
Stability
Read from SLS
|
Item |
Description |
|
Availability |
High availability with built-in retry logic. If a job fails to read data from SLS, it retries at least 10 times. If all retries fail, the job reports an error and restarts. |
Write to OSS
|
Item |
Description |
|
Concurrency |
Shipping instances are created per shard and scale out automatically. When source Logstore shards split, new instances start within seconds. |
|
Data consistency |
Shipping scales out using consumer groups to ensure data consistency. Offsets commit only after data reaches OSS, preventing data loss. |
Monitoring and alerting
|
Item |
Description |
|
Monitoring and alerting |
Track metrics such as latency and traffic in real time. Set up alerts for issues such as insufficient export instances or exceeded network quotas. Configure alert monitoring rules to monitor data shipping jobs of the new version for OSS. |
Limits
Network
|
Item |
Description |
|
Network type |
Uses the Alibaba Cloud internal network for fast, stable transfers. |
Permission management
|
Item |
Description |
|
Authorization |
Grant the required permissions to ship and access data in OSS. Authorization overview. |
|
Server-side encryption |
If server-side encryption is enabled, grant additional permissions to the RAM role. OSS configuration documentation. |
Read traffic
|
Item |
Description |
|
Read traffic |
SLS enforces upper limits on read traffic per project and per shard. For more information, see Data read and write. If a limit is exceeded, split shards or request a quota increase. A job that fails due to an exceeded limit retries at least 10 times. If all retries fail, it reports an error and restarts. |
Write to OSS
|
Item |
Description |
|
Concurrent instances |
Equals the total shard count, including read/write and read-only shards. |
|
Data shipping |
|
|
Time partition |
Each batch writes one OSS object. The object path uses the earliest receive_time (when SLS receives the data) in the batch. |
|
File format |
Supports CSV, JSON, Parquet, and ORC formats. For more information, seeJSON format, CSV format, Parquet format, and ORC format. |
|
Compression method |
Supported: Snappy, gzip, zstd, or none. |
|
OSS bucket |
|
Configuration items
|
Item |
Description |
|
Shipping latency |
The shipping latency must not exceed the Logstore data retention period. Reserve a buffer. For example, if data retention is 30 days, set shipping latency to 25 days or less. |
Data shipping management
|
Item |
Description |
|
Job pause and resume |
A paused job records the cursor of the last shipped entry and resumes from that point:
|