This topic describes the stability and limits of the new version of Object Storage Service (OSS) data shipping.
Stability
Data read from Simple Log Service
Item | Description |
Availability | High availability is provided. If an OSS data shipping job fails to read data from Simple Log Service due to an error in Simple Log Service, the job is retried at least 10 times. If the job still fails, an error is reported, and the job is restarted. |
Data write to OSS
Item | Description |
Concurrency | Data shipping instances can be created based on shards, and the resources that are used for data shipping can be scaled out. If shards in the source Logstore of a data shipping instance are split, the required resources can be scaled out within a few seconds to accelerate the data export process. |
Data consistency | The required resources are scaled out based on specified consumer groups to ensure data consistency. An offset is submitted only after data is shipped to OSS. This helps ensure that all data is shipped to OSS. |
Monitoring and alerting
Item | Description |
Monitoring and alerting | You can monitor data shipping jobs in real time based on metrics such as the latency and traffic of data shipping jobs. You can configure custom alerts based on your business requirements to report exceptions that occur during data shipping at the earliest opportunity. For example, if the data shipping instances that are used to export data are insufficient or the network quota is exceeded, alerts are triggered. For more information, see Configure alert monitoring rules to monitor data shipping jobs of the new version for OSS. |
Limits
Network
Item | Description |
Network type | Data transfer over an internal network of Alibaba Cloud is fast and stable. |
Permission management
Item | Description |
Authorization | The permissions to ship data to OSS and access data must be granted. For more information, see Authorization overview. |
Server-side encryption | If server-side encryption is enabled, you must grant additional permissions to the Resource Access Management (RAM) role that is involved. For more information, see OSS configuration documentation. |
Read traffic
Item | Description |
Read traffic | Simple Log Service specifies upper limits on read traffic in a single project and a single shard. For more information, see Data read and write. If a limit is exceeded, you must split shards or apply for a limit increase in your project. If an OSS data shipping job fails to read data because a limit is exceeded, the job is retried at least 10 times. If the job still fails, an error is reported, and the job is restarted. |
Data write to OSS
Item | Description |
Concurrent instances | The number of concurrent instances must be the same as the number of shards. The shards include readwrite shards and readonly shards. |
Data shipping |
|
Time partition | In a data shipping job, data is shipped to OSS by performing multiple shipping operations. Each data shipping operation ships data to OSS and stores the data to a different OSS object. The path to an OSS object is determined by the earliest point in time at which Simple Log Service receives the data shipped to the OSS object. This point in time is specified by receive_time. |
File format | After data is shipped to OSS, the data can be stored in one of the following formats: CSV, JSON, Parquet, and ORC. For more information, see JSON format, CSV format, Parquet format, and ORC format. |
Compression method | The following compression methods are supported: snappy, gzip, and zstd. Non-compression is also supported. |
OSS bucket |
|
Configuration items
Item | Description |
Shipping latency | The shipping latency cannot exceed the data retention period of the Logstore from which you want to ship data. We recommend that you reserve a buffer period to prevent data loss. For example, if the data retention period of a Logstore is 30 days, we recommend that you set the shipping latency to a value that is less than or equal to 25 days. |
Data shipping management
Item | Description |
Pause of a data shipping job | If you pause a data shipping job, the job records the cursor of the last log that is shipped. After you resume the job, the job continues to ship logs from the recorded cursor. Simple Log Service implements the following mechanism when you pause a data shipping job:
|