Use a DMS data integration task to export an Oracle database table to an Object Storage Service (OSS) bucket. The task runs a full-load transfer and writes data to the bucket in Delta or Parquet format, with optional KMS encryption.
Limitations
Confirm this feature fits your use case before you start.
| Constraint | Detail |
|---|---|
| Availability | Canary release — available to specific customers only |
| Region | Singapore only |
| Table limit | One table per integration task |
| Oracle type | Pluggable Oracle databases are not supported |
| Network | Source database must be accessible over the Internet |
| Account | OSS bucket, KMS instance, RAM user, and DMS tenant must belong to the same Alibaba Cloud account |
Supported capabilities
| Capability | Supported | Notes |
|---|---|---|
| Output formats | Yes | Delta, Parquet |
| KMS encryption | Yes | Optional; requires a KMS instance |
| Multiple tables per task | No | One table per task |
| Pluggable Oracle (PDB) | No | |
| RAC architecture | Yes | Enter /<Service Name> in the SID field |
Prerequisites
Before you begin, ensure that you have:
-
An Oracle database accessible over the Internet
-
The CIDR blocks of Data Transmission Service (DTS) and Data Management (DMS) servers added to the Oracle database security settings (security group rules, firewall policies, and IP address whitelists)
For the CIDR block lists, see Add the CIDR blocks of DTS servers and Add DMS IP addresses and CIDR blocks to security settings.
Overview
The setup involves the following steps:
-
Create an OSS bucket and a directory in the bucket
-
Create a RAM user and grant the required permissions
-
(Optional) Purchase and configure a KMS instance for data encryption
-
Register the Oracle database in DMS
-
Register the OSS instance in DMS
-
Configure and run the data integration task
-
Verify the result in OSS
Step 1: Prepare resources
Create an OSS bucket
-
Log on to the OSS console.
-
In the left-side navigation pane, click Buckets. On the Buckets page, click Create Bucket.
-
Configure the bucket:
-
Bucket Name: Enter a name.
-
Region: Select the region where the data integration task will run.
-
ACL: Select Private.
-
Configure other parameters as needed. For details, see Create a bucket.
-
-
Click OK.
-
Create a directory in the bucket: For naming requirements, see Manage directories.
-
In the left-side navigation pane, click Buckets, then click the bucket you created.
-
Choose Object Management > Objects.
-
Click Create Directory.
-
Enter a Directory Name, then click OK.
-
Create a RAM user and grant permissions
-
Go to the Users page and click Create User.
-
Configure the user:
-
Logon Name: Enter a name (for example,
dts_testuser). -
Access Mode: Select Using permanent AccessKey to access.
-
-
Click OK. Record the AccessKey ID and AccessKey Secret — they are displayed only once.
-
Find the RAM user you created and click Add Permissions in the Actions column.
-
Add the following policies:
Policy Description AliyunDTSFullAccess Manage DTS resources AliyunOSSFullAccess Manage OSS resources. Alternatively, grant AliyunOSSReadOnlyAccess and configure a bucket policy in the OSS console with Full Access or Read/Write permissions on the bucket. For details, see Method 1: Configure bucket policies by using the GUI. AliyunKMSReadOnlyAccess (Optional) Read-only access to KMS AliyunKMSSecretUserAccess (Optional) Retrieve secrets from KMS AliyunKMSCryptoUserAccess (Optional) Use KMS keys for encryption and decryption -
Click Grant permissions, then click Close.
(Optional) Purchase and configure a KMS instance
Skip this section if you do not need to encrypt the integrated data.
Step 1: Purchase a KMS instance
-
Log on to the KMS console and select your target region.
-
In the left-side navigation pane, choose Resource > Instances.
-
On the Instances page, click Create Instance.
-
Set Instance Type to Software Key Management and Region to the region of your data integration task. Configure other parameters as needed, then click Buy Now and follow the prompts.
For parameter details, see Purchase and enable a KMS instance.
Step 2: Enable the KMS instance
-
Return to the Instances page and click the Software Key Management tab.
-
Find the KMS instance and click Enable in the Actions column.
-
Complete the configuration and follow the prompts.
For details, see Enable the KMS instance.
Step 3: Create a customer master key (CMK)
-
In the left-side navigation pane, choose Resource > Keys.
-
On the Keys tab, set the Instance ID to the KMS instance from step 1, then click Create Key.
-
Configure the key:
-
KMS Instance: Select the KMS instance from step 1.
-
Key Type: Select Symmetric Key.
-
Key Specifications: Select Aliyun_AES_256.
-
Configure other parameters as needed.
-
-
Click OK and follow the prompts.
For parameter details, see Create a CMK.
Step 4: Create a generic secret
-
In the left-side navigation pane, choose Resource > Secrets.
-
On the Generic Secrets tab, set the Instance ID to the KMS instance from step 1, then click Create Secret.
-
Configure the secret:
-
Secret Value: Select Plain Text and enter a value in the Plain Text field. > Note: Run
openssl rand 32 | base64to generate a random secret value. -
CMK: Select the key created in step 3.
-
Configure other parameters as needed.
-
-
Click OK.
For details, see Manage and use generic secrets.
Step 5: Create an application access policy
-
In the left-side navigation pane, choose Application Access > AAPs.
-
Click the Policy tab, then click Create Policy.
-
Configure the policy:
-
Scope: Select the KMS instance from step 1.
-
RBAC Permissions: Select all available options.
-
Accessible Resources: Select the CMK from step 3 and the secret from step 4.
-
Configure other parameters as needed.
-
-
Click OK.
For details, see Create an AAP.
Step 2: Register the Oracle database
-
Log on to the DMS console V5.0.
-
Click the
icon next to Database Instances in the left-side navigation pane.Alternatively, choose Data Assets > Instances in the top navigation bar, then click New on the Instance List tab.
-
In the Add Instance dialog box, click the Third-party Cloud/Self-managed tab.
-
In the Relational Database section, click Oracle and configure the parameters: For other parameters, see Register a database hosted on a third-party cloud service or a self-managed database.
Parameter Description Database Type Select Oracle Instance Source Select Internet Instance Region The region where the Oracle database resides Login Address The public endpoint of the Oracle database Port Number The service port of the Oracle database SID The system ID (SID) of the Oracle database. For a Real Application Clusters (RAC) deployment, enter /<Service Name>Database Account The account that owns the schema to integrate Database Password The password for the database account -
Click Submit.
Step 3: Register the OSS instance
-
Log on to the DMS console V5.0.
-
Click the
icon next to Database Instances in the left-side navigation pane.Alternatively, choose Data Assets > Instances in the top navigation bar, then click New on the Instance List tab.
-
In the Add Instance dialog box, click the Alibaba Cloud tab.
-
In the File and Log section, click OSS and configure the parameters: For other parameters, see Register an Alibaba Cloud database instance.
Parameter Description File and Log Storage Select OSS Instance Region Select the region of the bucket created in Step 1: Prepare resources Connection Method Select Connection String Address Connection String Address Auto-populated by DMS based on the selected region. For example, if you set the Instance Region parameter to China (Hangzhou), the Connection String Address parameter is automatically set to oss-cn-hangzhou.aliyuncs.com Bucket (Optional) Select the bucket created in Step 1: Prepare resources AccessKey ID The AccessKey ID of the RAM user created in Step 1: Prepare resources AccessKey Secret The AccessKey Secret of the RAM user created in Step 1: Prepare resources -
Click Submit.
Step 4: Configure and run the data integration task
Create a task flow
-
Log on to the DMS console V5.0.
-
In the top navigation bar, choose Data Development > Data Development > Task Orchestration.
In simple mode, move the pointer over the
icon in the upper-left corner and choose All Features > Data Development > Data Development > Task Orchestration. -
Click Create Task Flow. In the dialog box, set the Business Scenario and Task Flow Name, then click OK.
Configure the integration task
-
In the Task Type list on the left, drag Data Lake Data Integration from the Data Integration Nodes section to the canvas.
-
Double-click the Data Lake Data Integration node.
-
Configure the source and destination:
Section Parameter Description Source Database Database Type Select Oracle Access Method Set to Public IP Address Instance Region The region of the Oracle database registered with DMS DMS Database Instance The Oracle database registered with DMS Destination Database Database Type Set to OSS Access Method Set to Alibaba Cloud Instance Instance Region The region of the OSS bucket registered with DMS DMS Database Instance The OSS instance registered with DMS OSS Bucket The bucket created in Step 1: Prepare resources -
Click Next Step: Select Objects.
-
Configure the objects to migrate:
Parameter Description Destination OSS Database Integration Format Select Delta or Parquet Destination OSS Database Path The directory path in the bucket (created in Step 1: Prepare resources) Source Objects Select a table, then click
to add it to Selected Objects. Only one table is supported.Selected Objects Data is stored at /<Destination OSS database path>/<Table name>by default. To change the subdirectory or configure KMS encryption, right-click the table and use the Edit Table dialog box. For parameter details, see KMS encryption parameters. -
Click OK.
Run the task
In the upper-left corner of the canvas, click Try Run.
On the Execution Logs tab at the bottom of the canvas:
-
status SUCCEEDED — the task completed successfully.
-
status FAILED — the task failed. Review the error details, correct the configuration, and run the task again.
Step 5: Verify the result
-
Log on to the OSS console.
-
In the left-side navigation pane, click Buckets.
-
Search for the bucket by name and click it.
-
Navigate to the
/<Destination OSS database path>/<Table name>directory to confirm the data files are present.
The <Destination OSS database path> is the Destination OSS Database Path you set in Configure the integration task. The <Table name> is the Table Name from the Edit Table dialog box (see KMS encryption parameters).
KMS encryption parameters
When you right-click a table in Selected Objects and open the Edit Table dialog box, you can configure the following parameters:
| Parameter | Description | Source |
|---|---|---|
| Schema Name | Leave blank in this example | — |
| Table Name | The subdirectory name mapped to the data stored in the OSS bucket | — |
| Enable KMS-based Encryption | Whether to encrypt the integrated data using KMS | — |
| Enable Footer Encryption | Whether to encrypt the footer (column name metadata) in the OSS bucket | — |
| KMS Instance ID | The KMS instance created in Step 1, (Optional) Purchase and configure a KMS instance | Step 1 (KMS step 1) |
| KMS-encrypted Secret ID | The generic secret created in Step 1, (Optional) Purchase and configure a KMS instance | Step 1 (KMS step 4) |
| KMS-based Encryption Algorithms | The encryption algorithm to use | — |
| Encrypt Column | Whether to encrypt individual column data | — |