This topic describes how to use CloudOps Orchestration Service (OOS) to periodically back up the data of ApsaraDB for Redis instances to an Object Storage Service (OSS) bucket. This method helps you resolve issues, such as ensuring data security, simplifying O&M, and reducing costs.
Prerequisites
Function Compute is activated. For more information, see Quickly create a function.
Background information
ApsaraDB for Redis provides high-performance memory data storage solutions to meet your requirements for the storage of growing business data. Scheduled backup of data is important for data security. The scheduled backup feature of ApsaraDB for Redis is an important part of database management. The feature can be used in scenarios such as disaster recovery, historical data query, and environment clone. You can specify a backup cycle to automatically back up the data of an ApsaraDB for Redis instance based on your business requirements. For more information, see Automatic or manual backup. However, this backup method has the following disadvantages:
Data security risks: After the instance is deleted, the backup data cannot be permanently stored and may be lost.
Complex O&M operations: The backup operation, including the download and upload of data, must be manually performed. This is time-consuming and prone to errors.
Increased O&M costs: The O&M costs are increased because the backup process must be manually performed. This consumes more time and resources, and is prone to errors.
Low efficiency: You cannot use a backup plan to back up multiple instances at a time or back up instances across regions in a centralized manner. This is inefficient, and the management is complex.
Insufficient monitoring and alerting capabilities: The status of a backup task cannot be controlled due to the lack of an efficient monitoring and alerting mechanism. Errors or failures cannot be dealt with at the earliest opportunity.
To handle these disadvantages, Alibaba Cloud provides an OOS-based backup solution that allows you to back up the data of ApsaraDB for Redis instances to OSS in an efficient and secure manner. The solution has the following advantages:
Improved data security: Backup files are automatically uploaded to OSS. The redundant storage mechanism of OSS ensures data durability and availability during hardware failures. In addition, the backup files can be stored permanently even if the ApsaraDB for Redis instances that you back up are deleted.
Fully automated O&M: After the configuration is complete, backup files are automatically uploaded to OSS based on the specified backup cycle without the need of manual operations. This greatly reduces the O&M workload and operation errors.
Centralized management of multiple instances across regions: You can specify multiple ApsaraDB for Redis instances at a time for a scheduled backup task. In addition, OOS allows you to store backup files to OSS across regions. You can back up the data of ApsaraDB for Redis instances that reside in different regions to the same OSS bucket.
Flexible and cost-effective backup policies: You can specify the backup frequency based on your business requirements and use the lifecycle policy of OSS to automatically clear expired backup files. This helps you control costs in a more efficient manner.
Monitoring and alerting: You can use OSS and CloudMonitor to monitor the backup status in real time. If a backup task fails or other issues occur, you can receive alert notifications in a timely manner to ensure data security.
Procedure
You may be charged during data backup. For more information, see Automatic or manual backup and Billing overview.
Log on to the OOS console.
In the left-side navigation pane, choose
. On the Scheduled O&M page, click Create.In the Set Scheduled Task section, configure the Scheduled Task Type parameter.
Valid values: Execute Now, Executed Once at the Specified Time, and Executed Periodically. If you select Execute Now, the scheduled O&M task is executed immediately after it is created. If you select Executed Once at the Specified Time, the scheduled O&M task is executed only once at the specified point in time. If you select Executed Periodically, the scheduled O&M task is periodically executed based on the scheduled time. For example, you can specify that the scheduled O&M task is executed on an hourly basis.
In this example, Executed Periodically is selected. You can click Select or use a CRON expression to specify the execution frequency. Click Select. In the Select Execution Frequency dialog box, click the Hour tab, select Hourly, and then click OK.
In the Select Template section, search for and select the ACS-Redis-BulkyCreateBackupAndUploadToOSS template.
In the Configure Template Parameters section, select the region in which the ApsaraDB for Redis instances that you want to back up reside from the RegionId drop-down list. Click Select Instances. In the Select Instances dialog box, select the instances that you want to back up. Then, click OK.
Configure the OSSRegionId and OSSBucketName parameters.
Configure the Resource Access Management (RAM) role that is required to execute the O&M task.
Create a RAM role for OOS. For more information, see Use RAM to grant permissions to OOS. Attach the following policy to the RAM role:
{ "Version": "1", "Statement": [ { "Action": [ "kvstore:CreateBackup", "kvstore:DescribeBackupTasks", "kvstore:DescribeBackups", "kvstore:DescribeInstances" ], "Resource": "*", "Effect": "Allow" }, { "Action": [ "ros:CreateStack", "ros:DeleteStack", "ros:GetStack" ], "Resource": "*", "Effect": "Allow" }, { "Action": "oos:StartExecution", "Resource": "*", "Effect": "Allow" }, { "Action": [ "fc:CreateFunction", "fc:CreateService", "fc:DeleteFunction", "fc:DeleteService", "fc:GetFunction", "fc:GetService", "fc:InvokeFunction" ], "Resource": "*", "Effect": "Allow" }, { "Condition": { "StringEquals": { "acs:Service": "fc.aliyuncs.com" } }, "Action": "ram:PassRole", "Resource": "*", "Effect": "Allow" } ] }
Create the AliyunFCDefaultRole RAM role that is assumed by Function Compute to download backup files. You can click Create Role to quickly create the RAM role that is required by Function Compute.
Click Create. In the Parameter Confirmation message, click OK. The O&M task is created.
View the task execution results.
a. You can wait until the O&M task is triggered at the specified time and view the results.
b. You can also go to the details page of the O&M task, click Trigger in the upper-right corner, and then view the results.
After the task is triggered, you can view the run logs to check the progress of the task.
View the backup files that are stored in OSS.
Wait until the execution enters the Ended state. Then, click the execution ID.
In the Execution Steps and Results section, click the Child Execution tab. Find the child execution that you want to view and click the child execution ID.
Copy the URL that is displayed in the Value column of the Output tab.
Log on to the OSS console by using the copied URL and view the stored backup files.
The names of the backup files are in the following format:
Backup_[Instance ID]/[Date in the YYYY-MM-DD format]/[OOS backup task ID]/[Backup file name].rdb
If the ApsaraDB for Redis instance that you back up does not use a cluster architecture, one RDB file is generated for each backup.
If the ApsaraDB for Redis instance that you back up uses a cluster architecture, the number of RDB files that are generated for each backup is the same as the number of shards. For example, eight RDB files are generated for a cluster instance that has eight shards.
The backup files are not automatically deleted. You can retain or delete the backup files based on your business requirements.
Appendix
Execution flowchart
The following figure shows the entire execution process.
The following figure shows the process of creating backup files for ApsaraDB for Redis instances and uploading the backup files to OSS.
Template
The following sample code shows the content of the ACS-Redis-BulkyCreateBackupAndUploadToOSS template that is used in this example:
FormatVersion: OOS-2019-06-01
Description:
en: Create Redis backups in batches and upload them to OSS. Backing up redis and using fc to upload the backup to OSS may incur charges. For billing details, please refer to <a href='https://www.alibabacloud.com/help/en/redis/user-guide/automatic-or-manual-backup'>Redis Backup Fee Instructions</a> and <a href='https://www.alibabacloud.com/help/en/functioncompute/fc-2-0/product-overview/billing-overview'>Function calculation billing instructions</a>
name-en: ACS-Redis-BulkyCreateBackupAndUploadToOSS
Parameters:
regionId:
Label:
en: RegionId
Type: String
AssociationProperty: RegionId
Default: '{{ACS::RegionId}}'
targets:
Type: Json
Label:
en: TargetInstance
AssociationProperty: Targets
AssociationPropertyMetadata:
ResourceType: ALIYUN::Redis::Instance
RegionId: regionId
OSSRegionId:
Label:
en: OSSRegionId
Type: String
AssociationProperty: RegionId
OSSBucketName:
Label:
en: OSSBucketName
Type: String
AssociationProperty: ALIYUN::OSS::Bucket::BucketName
AssociationPropertyMetadata:
RegionId: ${OSSRegionId}
rateControl:
Label:
en: RateControl
Type: Json
AssociationProperty: RateControl
Default:
Mode: Concurrency
MaxErrors: 0
Concurrency: 10
OOSAssumeRole:
Label:
en: OOSAssumeRole
Type: String
Default: ''
RamRole: '{{ OOSAssumeRole }}'
Tasks:
- Name: GetInstance
Description:
en: Get the redis instances
Action: ACS::SelectTargets
Properties:
ResourceType: ALIYUN::Redis::Instance
RegionId: '{{ regionId }}'
Filters:
- '{{ targets }}'
Outputs:
InstanceIds:
Type: List
ValueSelector: Instances.Instance[].InstanceId
- Name: CreateBackupAndUploadToOSS
Action: ACS::Redis::CreateBackupAndUploadToOSS
Description:
en: Create backup and upload to OSS
Properties:
regionId: '{{ regionId }}'
instanceId: '{{ ACS::TaskLoopItem }}'
OSSRegionId: '{{ OSSRegionId }}'
OSSBucketName: '{{ OSSBucketName }}'
Loop:
RateControl: '{{ rateControl }}'
Items: '{{ GetInstance.InstanceIds }}'
Outputs:
OSSObjectURLs:
AggregateType: Fn::ListJoin
AggregateField: OSSObjectURL
Outputs:
OSSObjectURL:
Type: String
ValueSelector: .OSSObjectURL + "{{ACS::ExecutionId}}/" | split(".t0") | .[0]
Outputs:
OSSObjectURLs:
Type: List
Value: '{{ CreateBackupAndUploadToOSS.OSSObjectURLs }}'
The following sample code shows the content of Action: ACS::Redis::CreateBackupAndUploadToOSS:
FormatVersion: OOS-2019-06-01
Description:
en: Create redis backup and upload to OSS
name-en: ACS::Redis::CreateBackupAndUploadToOSS
Parameters:
regionId:
Label:
en: RegionId
Type: String
AssociationProperty: RegionId
instanceId:
Label:
en: InstanceId
Type: String
AssociationProperty: ALIYUN::Redis::Instance::InstanceId
AssociationPropertyMetadata:
RegionId: ${regionId}
OSSRegionId:
Label:
en: OSSRegionId
Type: String
AssociationProperty: RegionId
OSSBucketName:
Label:
en: OSSBucketName
Type: String
AssociationProperty: ALIYUN::OSS::Bucket::BucketName
AssociationPropertyMetadata:
RegionId: ${OSSRegionId}
Default: ''
Tasks:
- Name: CreateBackup
Action: ACS::ExecuteApi
Description:
en: Create backup
Properties:
Service: r-kvstore
API: CreateBackup
Parameters:
RegionId: '{{ regionId }}'
InstanceId: '{{ instanceId }}'
Outputs:
BackupJobID:
Type: String
ValueSelector: .BackupJobID
- Name: WaitForBackupCreated
Action: ACS::WaitFor
Description:
en: Wait for backup created
Retries: 30
DelayType: Exponential
Delay: 2
BackOff: 2
Properties:
Service: r-kvstore
API: DescribeBackupTasks
Parameters:
RegionId: '{{ regionId }}'
InstanceId: '{{ instanceId }}'
BackupJobId: '{{ CreateBackup.BackupJobID }}'
DesiredValues:
- Finished
NotDesiredValues: []
StopRetryValues: []
PropertySelector: .BackupJobs[].BackupProgressStatus
- Name: GetBackupDownloadUrl
Action: ACS::ExecuteApi
Description:
en: Get backup download url
Properties:
Service: r-kvstore
API: DescribeBackups
Parameters:
RegionId: '{{ regionId }}'
InstanceId: '{{ instanceId }}'
BackupJobId: '{{ CreateBackup.BackupJobID }}'
StartTime:
Fn::FormatUTCTime:
- Fn::AddHour:
- '{{ ACS::CurrentUTCTime }}'
- -1
- '%Y-%m-%dT%H:%MZ'
EndTime:
Fn::FormatUTCTime:
- '{{ ACS::CurrentUTCTime }}'
- '%Y-%m-%dT%H:%MZ'
Outputs:
BackupDownloadURL:
Type: List
ValueSelector: .Backups.Backup[].BackupDownloadURL
- Name: UploadBackupToOSS
Action: ACS::FC::ExecuteScript
Description:
en: Upload backup to OSS
Properties:
runtime: python3.10
handler: index.handler
role: acs:ram::{{ACS::AccountId}}:role/AliyunFcDefaultRole
script: |-
import oss2
import requests
def handler(event, context):
auth = oss2.StsAuth(context.credentials.access_key_id, context.credentials.access_key_secret, context.credentials.security_token)
endpoint = 'https://oss-{{OSSRegionId}}.aliyuncs.com'
bucket = oss2.Bucket(auth, endpoint, '{{OSSBucketName}}')
unique_identifier = '{{ACS::TaskLoopItem}}'.split('?')[0].split('/')[-1]
execution_id = '{{ACS::ExecutionId}}'.split('.')[0]
input = requests.get('{{ACS::TaskLoopItem}}')
bucket.put_object(f'Backup_{{InstanceId}}/{{ACS::CurrentDate}}/{execution_id}/{unique_identifier}', input)
Loop:
Items: '{{ GetBackupDownloadUrl.BackupDownloadURL }}'
RateControl:
Mode: Concurrency
MaxErrors: 0
Concurrency: 20
Outputs:
OSSObjectURL:
Type: String
Value: https://oss.console.aliyun.com/bucket/oss-{{OSSRegionId}}/{{OSSBucketName}}/object?path=Backup_{{InstanceId}}/{{ACS::CurrentDate}}/