Cross-cloud deployment

Use cases

This feature is for workspaces using legacy Data Studio to migrate objects like nodes, functions, resources, and components from a source workspace to a destination workspace across different accounts, regions, or cloud platforms (for example, Alibaba Finance Cloud or Alibaba Gov Cloud).

Note

To ensure tasks run correctly in the destination workspace, read Deployment change logic to understand the mapping logic between the source and destination workspaces before you proceed.

Limitations

Permission requirements

You need permissions for both the source and destination workspaces to perform a cross-cloud deployment. For more information about how to add members to a workspace and grant permissions, see Manage permissions for workspace-level modules.

Environment requirements

The following table describes the environment requirements for a cross-cloud deployment from a source workspace to a destination workspace.

Source workspace		Destination workspace
Workspace mode	Data Studio version	Basic mode New Data Studio	Basic mode Legacy Data Studio	Standard mode New Data Studio	Standard mode Legacy Data Studio
basic mode	New Data Studio	Not supported	Not supported	Not supported	Not supported
basic mode	legacy Data Studio	Not supported	Supported	Not supported	Supported
standard mode	New Data Studio	Not supported	Not supported	Not supported	Not supported
standard mode	legacy Data Studio	Not supported	Supported	Not supported	Supported

Deployment process

Note

For a cross-cloud deployment, after you create a deployment package, you must manually download it to your local machine and then import it into the destination workspace for deployment.

Procedure in the source workspace

Step 1: Create a deployment environment

When you deploy nodes from the source workspace to the destination workspace, the destination is a completely new deployment environment. Therefore, you must map information such as engine instances, resource groups, and dependencies from the source to the destination. After the mapping is configured, the system automatically replaces the environment used by the deployed nodes with the mapped environment information when you deploy a deployment package, without requiring additional configuration in the destination workspace.

Note

Only members with the Workspace Administrator role can create and configure a deployment environment.

1. Go to Deployment Center

Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.
In the upper-left corner of the page, click the icon and choose All Products > More > Deployment Center to go to Deployment Center.

2. Configure the deployment environment

On the Deployment Center page, click Deployment Environment Management in the left-side navigation pane.

Click Create Deployment Environment and configure the environment information.

Configure the basic environment information.

Parameter	Description
Environment Name	The name of the new deployment environment.
Deployment Type	For this scenario, select Cross-cloud Deployment.
Destination workspace	Enter the name of the destination workspace.

Configure Computing Resource Mappings.

Configure the mapping between compute resources in the source workspace and compute resources in the destination workspace. After the mapping is configured, the system deploys the nodes, resources, functions, components, and other related content from the source compute resource to the mapped destination compute resource when you deploy tasks.

Note

At least one Computing Resource in Source Workspace must be retained in the Computing Resource Mappings section, and a mapping to the destination must be configured for it.
Determine whether you need to map source compute resources in the destination workspace based on your actual business requirements.

Parameter	Description
Computing Resource in Source Workspace	Displays all compute resources associated with the source workspace.
Computing Resource Type	The type of compute resource associated with the source workspace.
Computing Resource in Destination Workspace	Select the destination compute resource to which the source workspace compute resource is mapped. You can go to the Deployment Center > Deployment Environment Management page of the destination workspace and click View Workspace Environment Information in the upper-right corner to view compute resource information.
Ignore Mapping	After this feature is enabled, the destination workspace does not map the corresponding source workspace compute resource, and the nodes, resources, functions, components, and other content under the source compute resource will not be packaged for deployment. When the nodes, resources, functions, components, and other content under an engine instance in the source workspace do not need to be deployed, you can enable Ignore Mapping for the engine instance.

Configure Resource Group Mappings.

Map resource groups from the source workspace to the destination workspace. When you deploy a task, the system uses the mapped destination resource group instead of the source resource group to run the task.

Note

DataWorks only supports mapping between resource groups of the same type.
When mapping a Data Integration resource group, ensure its configurations, such as allowlists and network connectivity, match those of the source resource group. Otherwise, Data Integration tasks may fail to run as expected. For more information about network connectivity configurations, see Overview of network connection solutions.

Parameter	Description
Resource Group in Source Workspace	Displays all resource groups bound to the source workspace.
Resource Group Type	The type of the resource group bound to the source workspace.
Unique Resource Group Identifier in Destination Workspace	Select the destination resource group to map the source resource group to. You must enter the resource group ID. You can navigate to the Deployment Center > Deployment Environment Management page of the destination workspace and click View Workspace Environment Information in the upper-right corner to view resource group information.
Ignore Mapping	If you enable this option, the mapping for this resource group is ignored, and deployed tasks will continue to use the resource group from the source workspace. Note Default resource groups and the shared resource group do not support the Ignore Mapping feature. If the source workspace has more resource groups than the destination workspace, you can enable Ignore Mapping for the extra resource groups. When a task is deployed, it will continue to use the source resource group.

Configure Dependency Mappings.

This section displays the names of all upstream projects that have cross-project dependencies in the source workspace. You can configure the target project to which each project is mapped. After the mapping is configured, the system sets the target project as the upstream dependency of the deployed nodes in the destination workspace based on the mapping, generating new upstream and downstream dependency relationships.

Note

The Dependency Mappings section displays dependencies that are automatically updated on a T+1 basis by default. To view real-time dependencies, click Refresh Dependencies..
If you configure a cross-project dependency mapping to a new project, DataWorks modifies the node dependency based on the mapping. Make sure that the mapped workspace supports adding dependencies properly. For common scenarios where dependencies cannot be added, see Node commit error: Dependent ancestor output name not found.
After dependency mappings are configured for MaxCompute engine tasks (such as ODPS SQL and ODPS Spark), the system also updates the task code accordingly.

Parameter	Description
Source Project Name	Displays the names of all projects with cross-project dependencies in the source workspace.
Property	The project category.
Destination Project Name	Select the target project to which the source project is mapped.
Ignore Mapping	After this feature is enabled, the system ignores the dependency mapping for this project during deployment and continues to use the source project as the cross-project dependency project. The upstream and downstream dependency relationships of the deployed nodes remain unchanged.

Configure Parameter Mappings.
Synchronize the parameter configurations from the source workspace to the destination workspace. During deployment, the system references the parameters configured here (including workflow parameters and scheduling parameters) to the destination workspace. You can define parameter names and values by using the visual interface or expressions. For more information about parameter configuration, see Scheduling parameter formats.
Note
Make sure that the parameter names are unique in each deployment. Avoid assigning the same parameter to different tasks.
Click Create to create the deployment environment.

Step 2: Commit a node

On the node editing page in Data Studio, click the Commit () button in the top toolbar.

Step 3: Create a deployment package

After you enter any DataWorks module, click the icon in the upper-left corner of the page and select All Products > More > Deployment Center to open the Deployment Center.
In the left-side navigation pane, click Nodes to Deploy to view the objects that are ready for deployment. In the Actions column for an object, click Add to Deployment Package. After you add an object, the number of objects in the package is displayed on the View Deployment Package button.

Note
You can also select multiple objects and click Batch Add to Deployment Package at the bottom of the page.
After you add objects to the deployment package, click View Deployment Package to confirm that the objects in the package are correct. You can click Remove to remove any unwanted objects from the package.

Note
Currently, you can view only objects from legacy Data Studio. Objects from New Data Studio cannot be viewed.
On the Pending Deployment page, click Deployment Package Creation and follow the on-screen instructions to create the package.

Step 4: Download the deployment package

Using an account with the Operator or Workspace Administrator role, navigate to the Deployment Center of the source workspace.

You can enter any DataWorks module, click the icon in the upper-left corner of the page, and select All Products > More > Deployment Center to open the Deployment Center.
In the left-side navigation pane, click Deployment Packages to view the generated deployment packages.
In the Actions column of the target deployment package, click Download to download the package to your local machine.

Note
A deployment package is retained for only 30 days after creation. Download it promptly, as expired packages cannot be downloaded and must be recreated.

After you download the deployment package, you need to import it into the destination workspace for deployment.

Procedure in the destination workspace

Step 1: Import the deployment package

Go to the Deployment Center page of the destination workspace. In the left-side navigation pane, click Nodes to Deploy to navigate to the Nodes to Deploy page.
In the upper-right corner of the page, click Import Deployment Package to open the import panel.

Note
If you cannot find this button on the Nodes to Deploy page, check whether your workspace has New Data Studio enabled. Workspaces with New Data Studio enabled do not support cross-cloud deployment.

You can upload a package from a local file or from Object Storage Service (OSS).

A warning message at the top of the dialog box states that a first-time deployment overwrites any task with the same name in the same directory. You must confirm this to proceed. The form requires a Deployment Package Name and an optional description.
- Local file: Directly upload the downloaded deployment package. You can use this method if the package is 30 MB or smaller.
  
  Note
  The maximum file size for a local file upload is 30 MB.
- OSS upload: You must first upload the downloaded deployment package to OSS and then obtain its storage address. You must use this method if the package size exceeds 30 MB.
  
  To upload the deployment package to OSS, see Upload objects. To obtain the OSS storage address, see Obtain the URL of an object.

Step 2: Deploy the package

Once the package is uploaded, click Preview Deployment Package to review the environment and object list. If the information is correct, click Deploy.

Important

The first deployment of a task overwrites any existing task in the destination workspace that has the same name. Confirm that this is your intended action before proceeding.

Step 3: View the deployment results

After the cross-cloud deployment package is deployed, you can navigate to the Deployment Packages page of the destination workspace to view the deployment results.
If the deployment is successful, you can navigate to the Operation Center of the destination workspace to view the deployed tasks.