Migration Hub:Migration plan from Huawei Cloud GeminiDB Redis to Alibaba Cloud Tair (Redis-compatible)

Validation principle:

compatibility-test-suite-for-redis, or ctsr, works as follows:

• Client behavior simulation: It simulates a standard Redis client and sends various Redis commands to the system under test (SUT).

• Protocol and command validation: It validates the ability of the SUT to parse the Redis communication protocol (RESP). It also checks whether the response to each command is as expected.

• Behavioral consistency comparison: It compares the response of the SUT, including return values, data types, and error messages, with the response from a reference Redis instance that has known correct behavior.

• Data structures and semantics: It thoroughly validates the correctness of the operations of the SUT on various Redis data structures, such as strings, lists, and hashes. It also checks whether the behavior of advanced features, such as command atomicity, time-to-live (TTL), transactions, publish/subscribe, and Lua scripts, is consistent with the official Redis.

• Error handling: It tests whether the SUT returns the correct error messages when it encounters invalid commands, parameter errors, or type errors.

Prerequisites:

ctsr is written in Python. The test machine must have Python 3.7 or a later version installed.

The machine that runs ctsr must have network connectivity to both GeminiDB Redis and Tair (Redis-compatible).

Risks and notes:

ctsr performs many write operations, such as SET, LPUSH, and HSET. It also typically uses the FLUSHDB or FLUSHALL command to clear data. Running ctsr in a production environment or a staging environment with important data can cause production data to be deleted or overwritten. Do not run ctsr directly in a production environment or a staging environment that contains production data. Always run tests in an isolated, non-production, and disposable staging environment. Examples include a developer machine, a temporary container in a CI/CD pipeline, or a dedicated test virtual machine.

Migration principle:

You can use the real-time migration feature of the Huawei Cloud DRS tool. This feature supports migrating data from GeminiDB Redis to other Redis services. This allows data migration from Huawei Cloud GeminiDB Redis to Alibaba Cloud Tair (Redis-compatible).

Prerequisites:

The GeminiDB Redis core version must be 4.2.0 or a later version.

To use the migration feature, you must disable the keyspace notifications for expired events.

Risks and notes:

DRS inserts a key with the prefix DRS_REDIS_TIMESTAMP_HEARTBEAT into GeminiDB Redis to record the update timestamp. If GeminiDB Redis has a cluster architecture, DRS inserts this key on each shard of the source database. This key is filtered out during synchronization and expires after the sync task ends. During incremental synchronization, this key exists in the source database but not in the destination database. You can ignore this key during comparison. If your service also uses keys with the prefix DRS_REDIS_TIMESTAMP_HEARTBEAT, data inconsistency may occur.

Using Data Transmission Service (DTS)

Using RedisShake

Deploy the Redis-full-check tool

wget https://github.com/tair-opensource/RedisFullCheck/releases/download/release-v1.4.8-20200212/redis-full-check-1.4.8.tar.gz
tar -zxcf redis-full-check-1.4.8.tar.gz 
cd redis-full-check-1.4.8/

Run the Redis-full-check tool

Parameter descriptions

  -s, --source=SOURCE               Source Redis address (ip:port). For a cluster, separate different DBs with semicolons (;). You only need to configure one of the primary or replica nodes. For example: 10.1.1.1:1000;10.2.2.2:2000;10.3.3.3:3000.
  -p, --sourcepassword=Password     Source Redis password.
      --sourceauthtype=AUTH-TYPE    Source database management permissions. This parameter is not used for open source Redis.
      --sourcedbtype=               Source database type. 0: DB (standalone or primary/replica), 1: cluster, 2: Alibaba Cloud.
      --sourcedbfilterlist=         Whitelist of logical DBs to fetch from the source. Separate with semicolons (;). For example, 0;5;15 means DB0, DB5, and DB15 will be fetched.
  -t, --target=TARGET               Destination Redis address (ip:port).
  -a, --targetpassword=Password     Destination Redis password.
      --targetauthtype=AUTH-TYPE    Destination database management permissions. This parameter is not used for open source Redis.
      --targetdbtype=               See sourcedbtype.
      --targetdbfilterlist=         See sourcedbfilterlist.
  -d, --db=Sqlite3-DB-FILE          Location of the SQLite3 DB file for storing different keys. Default is result.db.
      --comparetimes=COUNT          Number of comparison rounds.
  -m, --comparemode=                Comparison mode. 1: full comparison. 2: compare only the value length. 3: compare only if the key exists. 4: full comparison but ignore large keys.
      --id=                         Used for metrics.
      --jobid=                      Used for metrics.
      --taskid=                     Used for metrics.
  -q, --qps=                        QPS rate limit threshold.
      --interval=Second             Interval in seconds between rounds.
      --batchcount=COUNT            Batch aggregation count.
      --parallel=COUNT              Number of concurrent coroutines for comparison. Default is 5.
      --log=FILE                    Log file.
      --result=FILE                 Record inconsistent results to the result file. Format: 'db    diff-type    key    field'.
      --metric=FILE                 Metric file.
      --bigkeythreshold=COUNT       Threshold for splitting large keys. Used for comparemode=4.
  -f, --filterlist=FILTER           List of keys to compare, separated by semicolons (;). For example, "abc*|efg|m*" compares 'abc', 'abc1', 'efg', 'm', and 'mxyz', but not 'efgh' or 'p'.
  -v, --version

Example

For example, the source Redis is 113.47.XX.XX:6379 and the destination is r-bp1dql854dakxxxxxx.redis.rds.aliyuncs.com:6379.

./redis-full-check -s 113.47.XX.XX:6379 -p mock_source_password -t r-bp1dql854dakxxxxxx.redis.rds.aliyuncs.com:6379 -a mock_target_password --comparemode=1 --comparetimes=1 --log log --result result

The execution results are saved to a SQLite3 database file. If you do not specify a path, the system creates a result.db.<N> file in the current working directory, where N is the comparison round number.
The database design includes the following:

• Primary table (key): Summarizes inconsistent key names.

• Details table (field): Records detailed inconsistency information for members of Hash, Set, and ZSet types, or indexes of List types. It is linked to the primary table using the key_id foreign key.

• Process tables (key_<N>, field_<N>): Store the intermediate comparison status for each iteration round for traceability.

Example result

[ecs-assist-user@iZbp1daxq5tl4bibxxxxxx redis-full-check-1.4.8]$ sqlite3 result.db.1
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .headers on
sqlite> .mode column
sqlite> select * from key;
id          key         type        conflict_type  db          source_len  target_len
----------  ----------  ----------  -------------  ----------  ----------  ----------
1           user:1006   string      value          0           11          9         
2           user:1001   string      value          0           11          9         
3           user:1010   string      value          0           11          9         
4           user:1008   string      value          0           11          9         
5           user:1009   string      value          0           11          9         
6           user:1004   string      value          0           11          9         
7           user:1005   string      value          0           11          9         
8           user:1007   string      value          0           11          9         
9           user:1002   string      value          0           11          9         
10          user:1011   string      lack_target    0           11          0         
11          user:1003   string      value          0           11          9         
12          user:1002   string      lack_target    1           11          0         
13          user:1008   string      lack_target    1           11          0         
14          user:1003   string      lack_target    1           11          0         
15          user:1006   string      lack_target    1           11          0         
16          user:1007   string      lack_target    1           11          0         
17          user:1004   string      lack_target    1           11          0         
18          user:1005   string      lack_target    1           11          0         
19          user:1009   string      lack_target    1           11          0         
20          user:1011   string      lack_target    1           11          0         
21          user:1010   string      lack_target    1           11          0         
22          user:1001   string      lack_target    1           11          0         
23          user:1006   string      lack_target    2           11          0         
24          user:1003   string      lack_target    2           11          0         
25          user:1010   string      lack_target    2           11          0         
26          user:1011   string      lack_target    2           11          0         
27          user:1007   string      lack_target    2           11          0         
28          user:1002   string      lack_target    2           11          0         
29          user:1001   string      lack_target    2           11          0         
30          user:1004   string      lack_target    2           11          0         
31          user:1005   string      lack_target    2           11          0         
32          user:1009   string      lack_target    2           11          0         
33          user:1008   string      lack_target    2           11          0         
sqlite> .exit