Logstore read/write logs must be saved in a certain shard. Each Logstore is divided into several shards and each shard is composed of MD5 left-closed, right-open intervals. These intervals do not overlap and the interval ranges add up to the entire MD5 value range.
When creating a Logstore, the entire MD5 range is automatically divided evenly based on the specified number of shards. Each shard has a certain range within the following value range: [00000000000000000000000000000000,ffffffffffffffffffffffffffffffff).
Each shard is composed of the following keys.
- BeginKey: indicates the start of the shard. This key is included in the shard range.
- EndKey: indicates the end of the shard. This key is excluded from the shard range.
With the shard range, you can write logs by specifying Hash Key, as well as split or merge shards. The corresponding shard must be specified when reading data from it. You can use load balancing mode or specified hash key mode when writing data. In load balancing mode, a data packet is written to an available shard at random. In specified Hash Key mode, data is written to the shard whose range includes the specified key.
In the following example, assume that the MD5 value range of the Logstore is [00,ff), and the Logstore contains 4 shards with the following ranges.
If you write a log and specify the MD5 key as 5F, the log data will be written into shard1 that contains the MD5 key 5F. If you specify the MD5 Key as 8C, the log data will be written into shard2 that contains the MD5 key 8C.
Each shard has certain service capacities.
- Write: 5 MB/s, 2000 times/s
- Read: 10 MB/s, 100 times/s
You can calculate the number of shards needed based on the traffic. If the data traffic exceeds the read and write capabilities, split the Shard in time to increase the number of Shards, so as to achieve greater read and write capabilities. If data traffic is far less than the maximum read and write capabilities of the partition, we recommend that you merge the partitions to reduce the number of partitions, and save the zoning costs.
For example, if you have two Shards in readwrite status, and provide 10MB/s of data writting in maximum. But data-writing traffic has reached to 14MB, we recommended to split one of the Shards so that the number of readwrite Shards reaches three. If data-writing traffic is only 3MB/s, then one Shard meet the needs, we recommend that you merge two Shards.
- When writing logs, if the API consistently reports 403 or 500 error, refer to Logstore CloudMonitor metrics to view the traffic and status code and determine whether you need to increase the number of shards.
- For read/write operations that exceed a shard’s service capacities, the system will attempt to provide the needed services, but the service quality cannot be ensured.
- readwrite: capable of reading and writing
- readonly: read-only data
When you create a partition, all partition states are in readwrite status. Split or merge operations change the partition state to readonly and generate a new readwrite partition. The Shard state does not affect the performance of its data read, while the readwrite Shard maintains normal data write performance, and the readonly state partition does not provide data-writing services.
When splitting a Shard, you need to specify a ShardId in a readwrite status and an MD5. The MD5 must be greater than the Shard’s BeginKey and less than EndKey. Split operations can split two other Shards from one, that is, the number of Shards increase 2 after division. After the split is complete, the original Shardstate specified by split is changed from readwrite to readonly, and the data can still be consumed, but no new data can be written. The two newly generated Shard status are readwrite, behind the original Shard, and the MD5 range of the two Shards covers the range of the original Shard.
In the merge operation, you must specify a Shard in readwrite status, and the specified Shard cannot be the last readwrite Shard. Log Service will automatically find the specified Shard on the right side of the Shard, and combine the two Shard. After the merge is complete, the specified Shard and its right side adjacent Shard become readable, the data can still be consumed, but new data cannot be write in. At the same time a new readwrite Shard is generated, and MD5 scope of the new Shard covers the original two Shards.
On the Log Service console, you can perform the following shard operations.