Simple Log Service:Stability and limitations

Item

Description

Availability

High availability.

If an Simple Log Service read error occurs, the MaxCompute shipping task automatically retries at least 10 times. If the issue persists, the task reports an error and then restarts.

Item

Description

Concurrency

The service creates a shipping instance for each Simple Log Service shard, which enables rapid scaling.

If a shard in the source Simple Log Service is split, new shipping instances are created within seconds to accelerate data export.

No data loss

Data shipping tasks use a consumer group to provide consistency guarantees. The task commits the offset only after successfully shipping the data. This at-least-once delivery model ensures that no data is lost.

Schema changes

If you add a new column to the MaxCompute table during the shipping process, the task writes the new column only to new partitions, not to existing or current ones.

Error type

Count as failure

Description

Partition error

Yes

Common causes include an invalid partition or a non-existent partition key column. The task does not write this data row to MaxCompute.

Invalid data column

No

Common causes include a data type mismatch or a type conversion failure. The task does not write data in this column to MaxCompute, but it writes data in other columns of the same row normally.

Data column too long

No

A common cause is that the data exceeds the length limit for the string or varchar data type. The task truncates the data in this column before writing it to MaxCompute. Data in other columns of the same row is written as normal.

Item

Description

Monitoring and alerts

Data shipping includes comprehensive monitoring features to track metrics such as task latency and traffic in real time. You can configure custom alerts based on your business requirements to promptly detect shipping issues, such as an insufficient number of shipping instances or network quota limits. For more information, see Configure alerts for a MaxCompute shipping task (New Version).

Item

Description

Too many partitions

If a task restarts while writing to a large number of partitions, write operations that take longer than 5 minutes to complete may cause data duplication.

Data write failure

If a task restarts after a write failure to MaxCompute due to authorization or network errors, partial data duplication may occur.

Limitation

Description

Network for intra-region shipping

Intra-region shipping uses the Alibaba Cloud internal network for data transmission. This provides higher network stability and speed.

Limitation

Description

Read traffic

Each project and shard has a maximum read traffic limit. For more information, see Data reads and writes.

If this limit is exceeded, you must split the shard or submit a request to increase the read traffic limit for the project. If the limit is exceeded, the shipping task fails and then retries at least 10 times. If the issue persists, the task reports an error and then restarts.

Limitation

Description

Concurrent instances

A maximum of 64 concurrent shipping instances are supported.

If the number of Simple Log Service shards exceeds 64, the service consolidates multiple shards into a single shipping instance. The service attempts to distribute the shards evenly across instances.

Write threshold

• The maximum write throughput for a single MaxCompute partition is 10 MB/s.

• The maximum write throughput for a single MaxCompute project varies by region. For example, the limit in the China (Hangzhou) region is 300 MB/s (1 MB/s × 300). The formula is described as follows:

  • 1 MB/s: The write throughput of a single slot to MaxCompute. For more information, see Limits of MaxCompute Tunnel Service.

  • 300: The maximum number of available slots for a single project in the China (Hangzhou) region. For more information, see Shared resources.

Table schema modification

MaxCompute shipping (New Version) uses the MaxCompute Tunnel Service for stream writing. During this process, the MaxCompute Tunnel Service prohibits schema modifications on the target table, such as inserting, updating, or deleting columns, or changing the column order. For more information, see Overview of MaxCompute Tunnel Service.

This restriction prevents you from using both the new and legacy versions of data shipping to write to the same MaxCompute table concurrently.

Unsupported table types

Data shipping to MaxCompute external tables, transactional tables, or clustered tables is not supported.

Table schema changes

To apply schema changes to the target MaxCompute table, pause the shipping task for 20 minutes and then restart it.

Start time

• Start at a specific time: Data shipping is based on the __receive_time__. The start time cannot be more than 24 hours earlier than the current time.

• Specific time range: Data shipping is based on the __receive_time__. The start time cannot be more than 24 hours earlier than the current time and must be before the end time.

Limitation

Description

Write authorization

You can grant write permissions using a RAM user or a RAM role, which requires a separate configuration in MaxCompute.

Limitation

Description

Pausing a shipping task

A shipping task records the log cursor of the last shipped item. When the task resumes, it continues from the recorded cursor. Therefore, the following behavior applies when you pause a shipping task:

• If you pause a task for a duration that is within the data retention period, the task resumes from the last saved cursor when restarted. No data is lost.

• If you pause a task for a duration that exceeds the data retention period, the task resumes from the earliest available data point after the last saved cursor.

Limitation

Description

Enabling an IP whitelist in your MaxCompute project, such as a classic network IP whitelist, may cause shipping tasks to fail.

Run the following commands in MaxCompute to resolve shipping failures caused by the IP whitelist.

1. Run setproject; select 1; to query the current whitelist configuration.

2. Append AliyunLogSLRService,AliyunLogDefaultService to the existing service whitelist. The complete command is as follows:

   setproject odps.security.ip.whitelist.services=AliyunLogSLRService,AliyunLogDefaultService;

3. After the configuration, run setproject; select 1; again to verify that the whitelist is updated.

For more information, see Resolve shipping failures caused by an IP whitelist.