Simple Log Service uses shards to control the read and write capacity of Logstores, EventStores, and MetricStores. All data is stored in shards. You can split or merge shards to adjust the number of active shards and control the maximum read and write capacity. Shards can be split automatically but must be merged manually.
Terms
Shard range
Each shard has a range, which is a left-closed, right-open MD5 interval of [BeginKey, EndKey). The ranges of shards do not overlap and are part of the entire MD5 range of [00000000000000000000000000000000, ffffffffffffffffffffffffffffffff). When you create a Logstore or MetricStore and specify the number of shards, Simple Log Service automatically divides the entire MD5 range equally among the shards.
BeginKey: The start of the shard range. This value is included in the range.
EndKey: The end of the shard range. This value is not included in the range.
For example, Logstore A contains four shards. The ranges of the shards are listed in the following table.
Table 1. Shard ranges
Shard ID | Range |
Shard0 | [00000000000000000000000000000000,40000000000000000000000000000000) |
Shard1 | [40000000000000000000000000000000,80000000000000000000000000000000) |
Shard2 | [80000000000000000000000000000000,c0000000000000000000000000000000) |
Shard3 | [c0000000000000000000000000000000,ffffffffffffffffffffffffffffffff) |
When you read data from a shard, you must specify the shard ID. When you write data, you can use either the load balancing mode or the specified hash key mode.
Load balancing mode: Each data packet is randomly written to an available shard.
You can use the load balancing mode if the write traffic is greater than the service capacity of a single shard.
Specified hash key mode: You can specify an MD5 key. The data is written to the shard whose range contains the key.
For example, as shown in Shard ranges, if you write data with an MD5 key value of 5F, the data is written to Shard1 because its range contains 5F. If you write data with an MD5 key value of 8C, the data is written to Shard2 because its range contains 8C.
Shard read and write capabilities
Each shard provides a certain service capacity. For more information, see Data reads and writes.
You should plan the number of shards based on your actual data traffic. If the data traffic exceeds the read and write capacity, you should split shards to increase their number and achieve higher read and write capacity. If the data traffic is far below the maximum read and write capacity of the shards, you should merge shards to reduce their number and lower the billing costs for active shards.
If the API for writing data continuously returns 403 or 500 errors, check the traffic and status codes in Cloud Monitor for the Logstore to determine whether to increase the number of shards.
If read or write operations exceed the service capacity of a shard, Simple Log Service provides the service on a best-effort basis, but service quality is not guaranteed.
Shard status
Shard statuses include readwrite and readonly.
When shards are created, they are all in the readwrite state. After a split or merge operation, the status of the original shard or shards changes to readonly, and a new shard in the readwrite state is created. The shard status does not affect data read performance. A readwrite shard guarantees data write performance. A readonly shard does not provide data write services.
Split and merge
Simple Log Service supports splitting and merging shards.
A split operation divides one shard into two new shards. The two new shards are in the readwrite state. They are arranged after the original shard, and their combined MD5 range covers the MD5 range of the original shard.
When you split a shard, you must specify a shard that is in the readwrite state. After the split, the status of the original shard changes from readwrite to readonly. The data in this shard can still be read, but you cannot write new data to it.
A merge operation combines two contiguous shards into one new shard. The new shard is in the readwrite state. It is arranged after the original shards, and its MD5 range covers the MD5 ranges of the two original shards.
When you merge shards, you must specify a readwrite shard that is not the last one in the arrangement. Simple Log Service automatically finds the shard that is contiguous to the right of the specified shard and merges them. After the merge, the status of the two original shards changes from readwrite to readonly. The data in these shards can still be read, but you cannot write new data to them.
Split a shard
You should plan the number of shards based on your service traffic. Each shard supports a write speed of 5 MB/s or 500 writes per second, and a read speed of 10 MB/s or 100 reads per second. These are not hard limits. If a limit is exceeded, the system provides the service on a best-effort basis, but service quality is not guaranteed. If the data read and write traffic exceeds a shard's capacity, you should split the shard to increase the number of shards and achieve higher read and write capacity.
If the API for writing data continuously returns 403 or 500 errors, check the traffic and status codes in Cloud Monitor for the Logstore to determine whether to increase the number of shards.
Console operations
Log on to the Simple Log Service console.
In the Projects section, click the one you want.
Click the
Log Storage icon. In the Logstores list, hover over the destination Logstore and choose 
> Modify.On the Logstore Properties page, click Modify.
Select the shard to split and click Split.
ImportantYou can split only a shard that is in the readwrite state.
Select the number of new shards and click OK.
Click Save.
View shards
Console operations
Log on to the Simple Log Service console.
In the Projects section, click the one you want.
Click the
Log Storage icon. In the Logstores list, hover over the destination Logstore and choose > Modify.On the Logstore Properties page, you can view the shard list of the current Logstore.
Automatic shard splitting
Simple Log Service supports automatic shard splitting to help you handle unexpected increases in service traffic. Shards are automatically split only if all of the following conditions are met:
The Auto Split Shard switch is turned on.
The write throughput exceeds the write service capacity of the current shard for more than 5 minutes.
The number of readwrite shards in the Logstore does not exceed the specified maximum total number of shards.
New shards that were created by a split within the last 15 minutes are not automatically split.
You can enable automatic shard splitting and set the maximum number of shards when you create or modify a Logstore.
Auto Split Shard
For example, if a Logstore has four shards, Simple Log Service independently determines whether each shard meets the splitting conditions. Shards that meet the conditions are split. The total number of shards after splitting does not exceed the maximum number that you set.
Maximum Shards
The maximum total number of shards after automatic splitting. After you enable automatic shard splitting, a Logstore can be split into a maximum of 256 readwrite shards.
Merge shards
If the read and write traffic is far below the maximum read and write capacity of a shard, you can merge shards to reduce their number and lower the billing costs for active shards. Simple Log Service finds the shard that is contiguous to the right of the specified shard and merges them. Shards can only be merged manually and cannot be merged automatically.
When you merge shards, you must specify a readwrite shard. This shard cannot be the last readwrite shard.
Click the
Log Storage icon. In the Logstores list, hover over the destination Logstore and choose > Modify.On the Logstore Properties page, click Modify.
Select the shard to merge and click Merge.
Click Save.
After the merge is complete, the status of the specified shard and its contiguous shard on the right changes to readonly. A new readwrite shard is created. The MD5 range of the new shard covers the ranges of the two original shards.
Delete a shard
After a shard is deleted, it cannot be restored. Proceed with caution.
Automatic deletion
If you set a data retention period when you create a Logstore, the shards and the data in them are automatically deleted after the retention period ends.
Manual deletion
If you turn on Permanent Storage when you create a Logstore, we recommend that you delete the shards and data in the Logstore by deleting the Logstore itself. For more information, see Stop billing or delete a Logstore.
Shard API operations
Operation | Interface |
Split a shard | |
Merge shards | |
Query shards |