Lindorm:Erasure coding

Background information

Erasure coding (EC) is a redundancy mechanism for data storage. Erasure coding splits original data into k blocks and generates m redundant blocks by using specific algorithms. The original data can be fully recovered based on any k blocks in the k+m blocks. If the original data is split into four blocks and two redundant blocks are generated in erasure coding, the original data can be fully recovered as long as no more than two blocks are corrupted. This way, data recovery can be implemented by storing only 1.5 replicas of the original data.

Compared with multi-replica data recovery, erasure coding can achieve data recovery with the same availability level by using less storage capacity. This way, your storage costs can be reduced.

Prerequisites

• The LindormTable version of the Lindorm instance is 2.5.4 or later. For more information about how to view or upgrade the LindormTable version, see Release notes of LindormTable and Upgrade the minor engine version of a Lindorm instance.

• The version of LindormDFS is 4.3.4 or later.

Limits

• Limits on storage types: Erasure coding is supported only on Lindorm instances that use local HDDs. Lindorm instances that use cloud disks do not support this feature.

  Important

  If you want to enable erasure coding for a single-zone or multi-zone Lindorm instance that uses local SSDs, contact the technical support (DingTalk ID: s0s3eg3) to check whether erasure coding can be enabled for your instance.

• Limits on the number of nodes: The Lindorm instance for which you want to enable erasure coding must contain at least seven nodes.

  Note

  For example, if the RS-4-2 algorithm is used in erasure coding, data in Lindorm is distributed to six nodes after erasure coding is enabled for the instance. Furthermore, an additional redundant node is required to ensure that data can be written to the instance when one of the six nodes fails. Therefore, the instance must contain at least seven nodes.

Enable erasure coding

You can configure the EC_POLICY parameter when you create a table to enable erasure coding in LindormTable. After erasure coding is enabled, data is automatically converted to the EC format when the compaction operation is triggered.

CREATE TABLE dt (p1 integer, p2 integer,primary KEY(p1)) WITH (EC_POLICY = 'RS-4-2');

Manage erasure coding

• Change the algorithm used in erasure coding

  ALTER TABLE dt SET 'EC_POLICY' = 'RS-4-2';

• Delete the algorithm used in erasure coding

  ALTER TABLE dt SET 'EC_POLICY' = '';

View the EC data written to the instance

1. Go to the Instance Monitoring page of the Lindorm console. For more information, see View monitoring information in the Lindorm console.

2. Click Cluster Storage Details. In the EC file logical data chart, view the EC data written to the instance.

FAQ

• Q: What do I do if EC data cannot be written to the instance after erasure coding is enabled?

  A: Perform the following checks:

  • Check whether the LindormTable version is 2.5.4 or later.

  • Check whether the instance contains sufficient nodes. At least seven nodes are required.

  • Check whether the major compaction operation is triggered.

  If no problem is found in the preceding checks, contact the technical support (DingTalk ID: s0s3eg3).

• Q: Does Lindorm support other algorithms rather than RS-4-2 in erasure coding, such as 8+2 or 8+3?

  A: Yes. You can contact the technical support (DingTalk ID: s0s3eg3) to configure other algorithms in erasure coding.

• Q: Why cannot I remove a node?

  A: The storage capacity of Lindorm instances that use local disks cannot be linearly scaled down. To reduce the storage capacity of these instances, you can only remove storage nodes from them. However, an instance for which erasure coding is enabled has a minimum requirement on the number of nodes based on the algorithm used in erasure coding. For example, if the RS-4-2 algorithm is used in erasure coding, at least seven (4+2+1) nodes are required for the instance. Therefore, before you remove a node from an instance, you must delete the algorithm used in erasure coding, trigger the major compaction operation, and then wait until the number of data blocks becomes 0.