All Products
Search
Document Center

CloudOps Orchestration Service:Periodically back up the data of ApsaraDB for Redis instances to an OSS bucket

Last Updated:Sep 11, 2024

This topic describes how to use CloudOps Orchestration Service (OOS) to periodically back up the data of ApsaraDB for Redis instances to an Object Storage Service (OSS) bucket. This method helps you resolve issues, such as ensuring data security, simplifying O&M, and reducing costs.

Prerequisites

Function Compute is activated. For more information, see Quickly create a function.

Background information

ApsaraDB for Redis provides high-performance memory data storage solutions to meet your requirements for the storage of growing business data. Scheduled backup of data is important for data security. The scheduled backup feature of ApsaraDB for Redis is an important part of database management. The feature can be used in scenarios such as disaster recovery, historical data query, and environment clone. You can specify a backup cycle to automatically back up the data of an ApsaraDB for Redis instance based on your business requirements. For more information, see Automatic or manual backup. However, this backup method has the following disadvantages:

  1. Data security risks: After the instance is deleted, the backup data cannot be permanently stored and may be lost.

  2. Complex O&M operations: The backup operation, including the download and upload of data, must be manually performed. This is time-consuming and prone to errors.

  3. Increased O&M costs: The O&M costs are increased because the backup process must be manually performed. This consumes more time and resources, and is prone to errors.

  4. Low efficiency: You cannot use a backup plan to back up multiple instances at a time or back up instances across regions in a centralized manner. This is inefficient, and the management is complex.

  5. Insufficient monitoring and alerting capabilities: The status of a backup task cannot be controlled due to the lack of an efficient monitoring and alerting mechanism. Errors or failures cannot be dealt with at the earliest opportunity.

To handle these disadvantages, Alibaba Cloud provides an OOS-based backup solution that allows you to back up the data of ApsaraDB for Redis instances to OSS in an efficient and secure manner. The solution has the following advantages:

  1. Improved data security: Backup files are automatically uploaded to OSS. The redundant storage mechanism of OSS ensures data durability and availability during hardware failures. In addition, the backup files can be stored permanently even if the ApsaraDB for Redis instances that you back up are deleted.

  2. Fully automated O&M: After the configuration is complete, backup files are automatically uploaded to OSS based on the specified backup cycle without the need of manual operations. This greatly reduces the O&M workload and operation errors.

  3. Centralized management of multiple instances across regions: You can specify multiple ApsaraDB for Redis instances at a time for a scheduled backup task. In addition, OOS allows you to store backup files to OSS across regions. You can back up the data of ApsaraDB for Redis instances that reside in different regions to the same OSS bucket.

  4. Flexible and cost-effective backup policies: You can specify the backup frequency based on your business requirements and use the lifecycle policy of OSS to automatically clear expired backup files. This helps you control costs in a more efficient manner.

  5. Monitoring and alerting: You can use OSS and CloudMonitor to monitor the backup status in real time. If a backup task fails or other issues occur, you can receive alert notifications in a timely manner to ensure data security.

Procedure

Important

You may be charged during data backup. For more information, see Automatic or manual backup and Billing overview.

  1. Log on to the OOS console.

  2. In the left-side navigation pane, choose Automated Task > Scheduled O&M. On the Scheduled O&M page, click Create.

  3. In the Set Scheduled Task section, configure the Scheduled Task Type parameter.

    Valid values: Execute Now, Executed Once at the Specified Time, and Executed Periodically. If you select Execute Now, the scheduled O&M task is executed immediately after it is created. If you select Executed Once at the Specified Time, the scheduled O&M task is executed only once at the specified point in time. If you select Executed Periodically, the scheduled O&M task is periodically executed based on the scheduled time. For example, you can specify that the scheduled O&M task is executed on an hourly basis.

    image.png

    In this example, Executed Periodically is selected. You can click Select or use a CRON expression to specify the execution frequency. Click Select. In the Select Execution Frequency dialog box, click the Hour tab, select Hourly, and then click OK.

  4. In the Select Template section, search for and select the ACS-Redis-BulkyCreateBackupAndUploadToOSS template.

    image.png

  5. In the Configure Template Parameters section, select the region in which the ApsaraDB for Redis instances that you want to back up reside from the RegionId drop-down list. Click Select Instances. In the Select Instances dialog box, select the instances that you want to back up. Then, click OK.

    image.png

    image.png

  6. Configure the OSSRegionId and OSSBucketName parameters.

    image.png

  7. Configure the Resource Access Management (RAM) role that is required to execute the O&M task.

    image.png

    Create a RAM role for OOS. For more information, see Use RAM to grant permissions to OOS. Attach the following policy to the RAM role:

    {
        "Version": "1",
        "Statement": [
            {
                "Action": [
                    "kvstore:CreateBackup",
                    "kvstore:DescribeBackupTasks",
                    "kvstore:DescribeBackups",
                    "kvstore:DescribeInstances"
                ],
                "Resource": "*",
                "Effect": "Allow"
            },
            {
                "Action": [
                    "ros:CreateStack",
                    "ros:DeleteStack",
                    "ros:GetStack"
                ],
                "Resource": "*",
                "Effect": "Allow"
            },
            {
                "Action": "oos:StartExecution",
                "Resource": "*",
                "Effect": "Allow"
            },
            {
                "Action": [
                    "fc:CreateFunction",
                    "fc:CreateService",
                    "fc:DeleteFunction",
                    "fc:DeleteService",
                    "fc:GetFunction",
                    "fc:GetService",
                    "fc:InvokeFunction"
                ],
                "Resource": "*",
                "Effect": "Allow"
            },
            {
                "Condition": {
                    "StringEquals": {
                        "acs:Service": "fc.aliyuncs.com"
                    }
                },
                "Action": "ram:PassRole",
                "Resource": "*",
                "Effect": "Allow"
            }
        ]
    }

    Create the AliyunFCDefaultRole RAM role that is assumed by Function Compute to download backup files. You can click Create Role to quickly create the RAM role that is required by Function Compute.image

  8. Click Create. In the Parameter Confirmation message, click OK. The O&M task is created.

    image.png

  1. View the task execution results.

    a. You can wait until the O&M task is triggered at the specified time and view the results.

    b. You can also go to the details page of the O&M task, click Trigger in the upper-right corner, and then view the results.

    image.png

    After the task is triggered, you can view the run logs to check the progress of the task.image.png

  1. View the backup files that are stored in OSS.

    Wait until the execution enters the Ended state. Then, click the execution ID.

    image.png

    In the Execution Steps and Results section, click the Child Execution tab. Find the child execution that you want to view and click the child execution ID.

    image.png

    Copy the URL that is displayed in the Value column of the Output tab.

    image.png

    Log on to the OSS console by using the copied URL and view the stored backup files.

    The names of the backup files are in the following format:

    Backup_[Instance ID]/[Date in the YYYY-MM-DD format]/[OOS backup task ID]/[Backup file name].rdb
    • If the ApsaraDB for Redis instance that you back up does not use a cluster architecture, one RDB file is generated for each backup.

    • If the ApsaraDB for Redis instance that you back up uses a cluster architecture, the number of RDB files that are generated for each backup is the same as the number of shards. For example, eight RDB files are generated for a cluster instance that has eight shards.

    • The backup files are not automatically deleted. You can retain or delete the backup files based on your business requirements.

      image.png

Appendix

Execution flowchart

The following figure shows the entire execution process.

image.png

The following figure shows the process of creating backup files for ApsaraDB for Redis instances and uploading the backup files to OSS.image.png

Template

The following sample code shows the content of the ACS-Redis-BulkyCreateBackupAndUploadToOSS template that is used in this example:

FormatVersion: OOS-2019-06-01
Description:
  en: Create Redis backups in batches and upload them to OSS. Backing up redis and using fc to upload the backup to OSS may incur charges. For billing details, please refer to <a href='https://www.alibabacloud.com/help/en/redis/user-guide/automatic-or-manual-backup'>Redis Backup Fee Instructions</a> and <a href='https://www.alibabacloud.com/help/en/functioncompute/fc-2-0/product-overview/billing-overview'>Function calculation billing instructions</a>
    
  name-en: ACS-Redis-BulkyCreateBackupAndUploadToOSS
   
Parameters:
  regionId:
    Label:
      en: RegionId
       
    Type: String
    AssociationProperty: RegionId
    Default: '{{ACS::RegionId}}'
  targets:
    Type: Json
    Label:
      en: TargetInstance
       
    AssociationProperty: Targets
    AssociationPropertyMetadata:
      ResourceType: ALIYUN::Redis::Instance
      RegionId: regionId
  OSSRegionId:
    Label:
      en: OSSRegionId
       
    Type: String
    AssociationProperty: RegionId
  OSSBucketName:
    Label:
      en: OSSBucketName
       
    Type: String
    AssociationProperty: ALIYUN::OSS::Bucket::BucketName
    AssociationPropertyMetadata:
      RegionId: ${OSSRegionId}
  rateControl:
    Label:
      en: RateControl
       
    Type: Json
    AssociationProperty: RateControl
    Default:
      Mode: Concurrency
      MaxErrors: 0
      Concurrency: 10
  OOSAssumeRole:
    Label:
      en: OOSAssumeRole
       
    Type: String
    Default: ''
RamRole: '{{ OOSAssumeRole }}'
Tasks:
  - Name: GetInstance
    Description:
      en: Get the redis instances
       
    Action: ACS::SelectTargets
    Properties:
      ResourceType: ALIYUN::Redis::Instance
      RegionId: '{{ regionId }}'
      Filters:
        - '{{ targets }}'
    Outputs:
      InstanceIds:
        Type: List
        ValueSelector: Instances.Instance[].InstanceId
  - Name: CreateBackupAndUploadToOSS
    Action: ACS::Redis::CreateBackupAndUploadToOSS
    Description:
      en: Create backup and upload to OSS
       
    Properties:
      regionId: '{{ regionId }}'
      instanceId: '{{ ACS::TaskLoopItem }}'
      OSSRegionId: '{{ OSSRegionId }}'
      OSSBucketName: '{{ OSSBucketName }}'
    Loop:
      RateControl: '{{ rateControl }}'
      Items: '{{ GetInstance.InstanceIds }}'
      Outputs:
        OSSObjectURLs:
          AggregateType: Fn::ListJoin
          AggregateField: OSSObjectURL
    Outputs:
      OSSObjectURL:
        Type: String
        ValueSelector: .OSSObjectURL + "{{ACS::ExecutionId}}/"  | split(".t0") | .[0]
Outputs:
  OSSObjectURLs:
    Type: List
    Value: '{{ CreateBackupAndUploadToOSS.OSSObjectURLs }}'

The following sample code shows the content of Action: ACS::Redis::CreateBackupAndUploadToOSS:

FormatVersion: OOS-2019-06-01
Description:
  en: Create redis backup and upload to OSS
   
  name-en: ACS::Redis::CreateBackupAndUploadToOSS
   
Parameters:
  regionId:
    Label:
      en: RegionId
       
    Type: String
    AssociationProperty: RegionId
  instanceId:
    Label:
      en: InstanceId
       
    Type: String
    AssociationProperty: ALIYUN::Redis::Instance::InstanceId
    AssociationPropertyMetadata:
      RegionId: ${regionId}
  OSSRegionId:
    Label:
      en: OSSRegionId
       
    Type: String
    AssociationProperty: RegionId
  OSSBucketName:
    Label:
      en: OSSBucketName
       
    Type: String
    AssociationProperty: ALIYUN::OSS::Bucket::BucketName
    AssociationPropertyMetadata:
      RegionId: ${OSSRegionId}
    Default: ''
Tasks:
  - Name: CreateBackup
    Action: ACS::ExecuteApi
    Description:
      en: Create backup
       
    Properties:
      Service: r-kvstore
      API: CreateBackup
      Parameters:
        RegionId: '{{ regionId }}'
        InstanceId: '{{ instanceId }}'
    Outputs:
      BackupJobID:
        Type: String
        ValueSelector: .BackupJobID
  - Name: WaitForBackupCreated
    Action: ACS::WaitFor
    Description:
      en: Wait for backup created
       
    Retries: 30
    DelayType: Exponential
    Delay: 2
    BackOff: 2
    Properties:
      Service: r-kvstore
      API: DescribeBackupTasks
      Parameters:
        RegionId: '{{ regionId }}'
        InstanceId: '{{ instanceId }}'
        BackupJobId: '{{ CreateBackup.BackupJobID }}'
      DesiredValues:
        - Finished
      NotDesiredValues: []
      StopRetryValues: []
      PropertySelector: .BackupJobs[].BackupProgressStatus
  - Name: GetBackupDownloadUrl
    Action: ACS::ExecuteApi
    Description:
      en: Get backup download url
       
    Properties:
      Service: r-kvstore
      API: DescribeBackups
      Parameters:
        RegionId: '{{ regionId }}'
        InstanceId: '{{ instanceId }}'
        BackupJobId: '{{ CreateBackup.BackupJobID }}'
        StartTime:
          Fn::FormatUTCTime:
            - Fn::AddHour:
                - '{{ ACS::CurrentUTCTime }}'
                - -1
            - '%Y-%m-%dT%H:%MZ'
        EndTime:
          Fn::FormatUTCTime:
            - '{{ ACS::CurrentUTCTime }}'
            - '%Y-%m-%dT%H:%MZ'
    Outputs:
      BackupDownloadURL:
        Type: List
        ValueSelector: .Backups.Backup[].BackupDownloadURL
  - Name: UploadBackupToOSS
    Action: ACS::FC::ExecuteScript
    Description:
      en: Upload backup to OSS
       
    Properties:
      runtime: python3.10
      handler: index.handler
      role: acs:ram::{{ACS::AccountId}}:role/AliyunFcDefaultRole
      script: |-
        import oss2
        import requests

        def handler(event, context):
          auth = oss2.StsAuth(context.credentials.access_key_id, context.credentials.access_key_secret, context.credentials.security_token)
          endpoint = 'https://oss-{{OSSRegionId}}.aliyuncs.com'
          bucket = oss2.Bucket(auth, endpoint, '{{OSSBucketName}}')
          unique_identifier = '{{ACS::TaskLoopItem}}'.split('?')[0].split('/')[-1]
          execution_id = '{{ACS::ExecutionId}}'.split('.')[0]
          input = requests.get('{{ACS::TaskLoopItem}}')
          bucket.put_object(f'Backup_{{InstanceId}}/{{ACS::CurrentDate}}/{execution_id}/{unique_identifier}', input)
    Loop:
      Items: '{{ GetBackupDownloadUrl.BackupDownloadURL }}'
      RateControl:
        Mode: Concurrency
        MaxErrors: 0
        Concurrency: 20
Outputs:
  OSSObjectURL:
    Type: String
    Value: https://oss.console.aliyun.com/bucket/oss-{{OSSRegionId}}/{{OSSBucketName}}/object?path=Backup_{{InstanceId}}/{{ACS::CurrentDate}}/