Back Up and Restore PolarSearch Data to OSS for Disaster Recovery - PolarDB - Alibaba Cloud - PolarDB

PolarSearch enables you to use the snapshot feature to back up index data from a cluster to your own Object Storage Service (OSS) bucket or restore data from an OSS bucket. This feature can be used for cross-cluster data migration and custom backup and recovery for PolarSearch clusters. This provides a flexible and low-cost solution for data protection and transfer.

Note

This feature is currently in preview. To use this feature, submit a ticket to enable it.

Prerequisites

Your PolarSearch version must be 3.0.0.0.0 or later.

Note
You can access PolarSearch through the search endpoint to check your PolarSearch version in the version.number field.
You have enabled Alibaba Cloud Object Storage Service (OSS) and created a bucket to store snapshots.
A RAM user with the AliyunOSSFullAccess policy attached is required. For more information, see Add permissions to a RAM user.

Billing

The snapshot feature is free. Storing snapshot files in your OSS bucket incurs storage and request fees. For more information, see OSS billing overview.

Register a snapshot repository

Before you use the snapshot feature, you must register a snapshot repository and associate it with your OSS bucket. Use the following API to create the repository:

PUT /_snapshot/{repo-name}
{
    "type": "oss",
    "settings": {
      "endpoint": "{endpoint}",
      "bucket": "{bucket-name}",
      "base_path": "{path-name}",
      "region": "{region}",
      "access_key": "{your-AccessKey-ID}", 
      "secret_key": "{your-AccessKey-Secret}",
      "session_token": "{your-STS-Token}",
      "compress": true,
      "chunk_size": "512mb"
    }
}

Parameter description

Parameter	Description
`{repo-name}`	A custom name for the repository.
`type`	The type of the repository. This must be set to `oss`.
`endpoint`	The endpoint for your OSS bucket. For more information, see Regions and Endpoints.
`bucket`	The name of your OSS bucket.
`base_path`	(Optional) The root directory path in the OSS bucket where snapshot files are stored.
`region`	The region where the bucket is located.
`access_key`	Your AccessKey ID.
`secret_key`	Your AccessKey Secret.
`session_token`	(Optional) The STS Token for your RAM role. Important If you use an STS Token, you must also set the `access_key` and `secret_key` parameters to the values from the STS Token. If you do not use an STS Token, use your long-term AccessKey ID and AccessKey Secret.
`compress`	(Optional) Specifies whether to compress snapshot metadata files, such as index mappings and settings. This parameter does not affect data files. The default is false.
`chunk_size`	(Optional) The size limit for chunked uploads during the snapshot process. Data larger than this size is uploaded to OSS in chunks. The default is 1 GB.

Example

Replace the parameters in the following command with your own information.

curl -X PUT "https://{pc-endpoint}:3001/_snapshot/{repo-name}" \
  -u "{username}:{passwd}" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "oss",
    "settings": {
      "endpoint": "{endpoint}",
      "bucket": "{bucket-name}",
      "base_path": "{path-name}",
      "region": "{region}",
      "access_key": "{your-AccessKey-ID}", 
      "secret_key": "{your-AccessKey-Secret}"
    }
  }'

Create a snapshot: back up data to OSS

After you register a snapshot repository, create a snapshot for a specific index using the following API:

PUT /_snapshot/{repo-name}/{snapshot-name}?wait_for_completion=true
{
    "indices": "{index-name}",
    "ignore_unavailable":false
}

Parameter description

Parameter category	Parameter name	Description
Request parameter	`wait_for_completion`	Specifies whether to wait for the snapshot operation to complete. The default value is `false`. Synchronous execution: If you set `wait_for_completion=true`, the command waits for the snapshot to be created before it returns a result. Asynchronous execution: If you set `wait_for_completion=false`, the command returns immediately and the snapshot is created in the background. You can check the snapshot status by running the following command. The operation is complete when the `state` field in the response is `SUCCESS`. `GET /_snapshot/{repo-name}/{snapshot-name}/_status`
Request body parameters	`indices`	The indexes to back up. You can use the wildcard character (``) and separate multiple index names with commas (`,`). By default, all indexes are backed up. Note* Using the wildcard character (`*`) backs up system tables. To avoid backing up or restoring system tables, specify only the indexes you need, or use a hyphen (`-`) to exclude the system tables.
	`ignore_unavailable`	Specifies whether to ignore a nonexistent index and proceed with the snapshot. The default value is `false`, which causes the operation to fail.
	`partial`	Specifies whether to allow partial snapshots. If set to `true`, data from successful shards is saved even if some shards fail. The default value is `false`.

Example

Replace the parameters in the following command with your own information.

curl -X PUT "https://{pc-endpoint}:3001/_snapshot/{repo-name}/{snapshot-name}?wait_for_completion=true" \
  -u "{username}:{passwd}" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": "{index-name}",
    "ignore_unavailable": false
  }'

View snapshots

Use the following API to view information about all snapshots in your OSS snapshot repository:

GET /_snapshot/{repo-name}/_all?pretty

Example

Replace the parameters in the following command with your own information.

curl -X GET "https://{pc-endpoint}:3001/_snapshot/{repo-name}/_all?pretty" -u "{username}:{passwd}"

Restore data

Run the following command to restore index data from a specific snapshot:

Note

To restore a snapshot from a PolarSearch cluster to another PolarSearch cluster, the target PolarSearch cluster must register the same snapshot repository as the source PolarSearch cluster. If the restoration occurs within the same PolarSearch cluster, you do not need to register again and can execute data restoration directly.

POST /_snapshot/{repo-name}/{snapshot-name}/_restore?wait_for_completion=true
{
  "indices": "{index-name}",
  "ignore_unavailable": true
}

Parameters

Parameter category	Parameter name	Description
Request parameter	`wait_for_completion`	Specifies whether to wait for the snapshot restoration to complete. The default value is `false`. Synchronous execution: If you set `wait_for_completion=true`, the command waits for the restoration to complete before returning. Asynchronous execution: If you set `wait_for_completion=false`, the command returns immediately, and the restore job runs in the background. You can check the index restoration progress by running the following command. The restoration is complete when the `stage` field is `DONE`. `GET /{index-name}/_recovery`
Request body parameters	`indices`	Specifies the indices to restore. The wildcard character `*` is supported. Multiple indices are separated by a comma `,`. The default is all indices.
	`ignore_unavailable`	Specifies whether to ignore an index and continue creating the snapshot if the index does not exist. The default is `false`, which causes the operation to fail.
	`partial`	Specifies whether to create a partial snapshot. If set to `true`, data from successful shards is saved even if some shards fail. The default is `false`.
	`index_settings`	Overrides the index settings from the snapshot during restoration. For example, you can change the number of replicas to match the configuration of the destination cluster.
	`ignore_index_settings`	A list of index settings to ignore during restoration. This is typically used to ignore settings that are specific to the source cluster.

Example

Replace the parameters in the following command with your own information.

curl -X POST "https://{pc-endpoint}:3001/_snapshot/{repo-name}/{snapshot-name}/_restore?wait_for_completion=true" \
  -u "{username}:{passwd}" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": "{index-name}",
    "ignore_unavailable": true
  }'