Prerequisites for MMS - MaxCompute - Alibaba Cloud Documentation Center

Step 1: Create a MaxCompute project

A project is the basic organizational unit in MaxCompute.

Before you migrate data, you must select a target MaxCompute project to which the data source will be mapped.
During the migration, MMS runs various migration jobs in the MaxCompute project.

Therefore, you must create a MaxCompute project before you migrate data. You can skip this step if a project is already available. To create a project in the MaxCompute console, follow these steps:

Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose Manage Configurations > Projects.
On the Internal Project tab, click Create Project.
In the Create Project dialog box, configure the settings and click OK.

For more information about the parameters, see Create a MaxCompute project.

Step 2: Create a VPC network connection

You can skip this step if a VPC network connection already exists in the current region.

Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose Manage Configurations > Network Connection .
On the Network Connection page, click Add Network Connection.

In the Add Network Connection dialog box, configure the parameters as prompted and click OK. When you add a network connection for the first time, you must first grant authorization to allow the MaxCompute platform proxy to request network interface cards. Otherwise, the connection cannot be created.

The following table describes the parameters.

Parameter	Required	Description
Connection Name:	Required	A custom name for the connection. The name must meet the following requirements: Start with a letter. Contain only letters, underscores (_), and digits. Be 1 to 63 characters in length.
Type:	Required	The default value is Passthrough. A Passthrough connection is a VPC connection.
Region:	Required	The system automatically populates this parameter based on the region that you select in the upper-left corner. For more information, see Supported regions.
VPC Selected:	Required	A virtual private cloud (VPC) is an isolated virtual network. It provides a secure and configurable private network space, similar to a traditional data center. The ID of the VPC. To create a new VPC, see Create or delete a VPC. To obtain the ID: Log in to the Virtual Private Cloud console. In the navigation pane on the left, choose VPC. In the top-left corner, select a region. On the VPC page, find the Instance ID/Name of the VPC. If you connect to an HBase or Hadoop cluster, you can find this information in the network connection details on the corresponding console.
Switch:	Required	A vSwitch is used to create a subnet. Different vSwitches in the same VPC can communicate with each other over the internal network. You can deploy cloud resources across vSwitches in different zones to protect your application from failures in a single zone. The ID of the vSwitch that is attached to the VPC. If no vSwitches are available, see Create or delete a vSwitch. To obtain the ID: Log in to the Virtual Private Cloud console. In the navigation pane on the left, choose vSwitch. In the top-left corner, select a region. On the vSwitch page, find the Instance ID/Name of the vSwitch. If you connect to an HBase or Hadoop cluster, you can find this information in the network connection details on the corresponding console.
Security group:	Required	A security group acts as a virtual firewall for your cloud resources. By managing security groups and their rules, you can implement fine-grained network isolation and access control. The ID of the security group. To create a security group, see Create a security group.

Data migration jobs need to access the data source. Therefore, you must ensure that the VPC can access the data source either over the internet or through a dedicated line.
- If your data source is on the public internet, use a NAT gateway to enable public network access. For more information, see Use a NAT gateway as a unified internet egress.
- If your data source is in an on-premises data center, connect it to Alibaba Cloud by using an Express Connect circuit. For more information, see Use Express Connect.

Step 3: Create a service-linked role

Before you use MMS for the first time, you must create the service-linked role AliyunServiceRolePolicyForMaxComputeMMS. MMS uses this role to access resources in MaxCompute and other cloud services.

If you are using an Alibaba Cloud account, you can create the service-linked role directly.

If you are a RAM user, you must first have the AliyunRAMFullAccess permission. For more information, see Manage permissions for a RAM role.
You can create the service-linked role in the MaxCompute console or the RAM console.
MaxCompute console
1. Log in to the MaxCompute console and select a region in the upper-left corner.
2. In the left-side navigation pane, choose Data Transfer > Migration Service.
3. On the Data Source tab, click Add Data Source.
4. To create the service-linked role, click OK in the MaxCompute Service-linked Role dialog box. If this dialog box does not appear, the role already exists.
RAM console
1. Log in to the RAM console.
2. In the left navigation bar, select Identities > Roles.
3. On the Roles page, click Create Role.
4. In the upper-right corner of the Create Role page, click Create Service Linked Role.
5. On the Create Service Linked Role page, under Select Service, select AliyunServiceRoleForMaxComputeMMS, and click Create Service Linked Role.

Step 4: Configure permissions

Grant the MMS service-linked role permissions to operate on MaxCompute data

MaxCompute console

Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose Manage Configurations > Projects.
On the Projects page, click Manage in the Actions column for the target project.

Configure data operation permissions

Admin role

On the Project Settings page, click the Role Permissions tab.
Find the admin role and click Manage Members in the Actions column.
In the Manage Members dialog box, enter RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS in the Add Member Manually text box. Replace <account_id> with your Alibaba Cloud account ID.
Click OK.

Create a custom role

Create a project role in MaxCompute and grant it only the data operation permissions required by MMS.

On the Project Settings page, click the Role Permissions tab.

Click Create Project-level Role, configure the role as prompted, and click Confirm.

The following table describes the parameters.

Parameter	Required	Description
Role Name	Required	The name must be 1 to 64 characters in length, and can contain letters and digits. The name must start with a letter.
Role Type	Required	Admin: A management role. You can grant management permissions by using a policy, but you cannot grant resource permissions. You cannot grant management permissions by using an ACL. Resource: A resource-type role. You can grant resource permissions by using a policy or an ACL, but you cannot grant management permissions.
Authorization Method	Required	Access Control List (ACL): A whitelist-based authorization mechanism that allows users or roles to perform specific actions on specific objects. Policy: A mechanism that supports both allow and deny rules, suitable for large enterprises and complex scenarios that require fine-grained control.

Grants a user permissions to perform operations on projects, tables, models, resources, functions, or instances. The following table lists the permissions that you can grant.

Object	Permissions
Table	Describe, Select, Update, Alter, Drop, ShowHistory, Download
Resource	Read, Write, Download, Delete
Function	Read, Write, Download, Execute, Delete
Package	Read
Project	Read, Write, List, CreateTable, CreateInstance

Return to the Role Permissions tab, select the new project-level role, and click Manage Members in the Actions column. Select the Alibaba Cloud account or RAM user to authorize and click OK. If you cannot find the account when you authorize it for the first time, you can add it in the Add Member Manually section.

Via SQL

Connect to MaxCompute by using the odpscmd client and add the service-linked role that you created.

USE <target_projectname>;

-- Add the service-linked role to the user.
Add USER  `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

Grant permissions to the user.

MaxCompute supports permission control at various levels. During project operations, you can grant permissions based on the scope of operations.

Coarse-grained authorization: Directly grant the admin role to the user. This way, you avoid having to grant permissions for each table individually.
```
GRANT admin TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
```

Fine-grained authorization: Grant permissions at the project, table, or instance level.

Project-level permissions

-- Grant project-level data operation permissions.
GRANT <actions> 
  ON project <project_name> TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
-- Supported actions
Read,Write,List,CreateTable,CreateInstance,CreateFunction,CreateResource,All
-- Example 1
GRANT Read,Write,List,CreateTable,CreateInstance,CreateFunction,CreateResource
  ON project <project_name> TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
-- Example 2
GRANT ALL 
  ON project <project_name> TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

Table-level permissions

-- Grant table-level data operation permissions.
GRANT <actions> 
  ON table <table_name> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
-- Supported actions
Describe,Select,Alter,Update,Drop,ShowHistory,All
-- Example 1
GRANT Describe,Select,Alter,Update,Drop,ShowHistory
  ON table <table_name> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
-- Example 2
GRANT All
  ON table <table_name> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

Instance-level permissions

-- Grant instance-level data operation permissions. 
GRANT <actions> 
  ON instance <instance_id> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
-- Supported actions
Read,Write,All 

-- Example 1
GRANT Read,Write 
  ON instance <instance_id> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

-- Example 2
GRANT All 
  ON instance <instance_id> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

Parameter description:

target_projectname: Required. The name of the target project.
account_id: Required. The ID of your Alibaba Cloud account.
privileges: Required. The name of the operation permission to be granted. You can specify multiple operations at a time. Separate them with commas (,).

Configure RAM permissions

This step grants the RAM user performing the data migration the permissions to operate MMS, such as managing MMS data, metadata, and migration jobs. If you use an Alibaba Cloud account to perform the migration, you can skip this step.

If the current RAM user has the AliyunMaxComputeFullAccess permissions to manage MaxCompute, you can skip this step.

If you want to grant the RAM user only the permissions to operate MMS, follow these steps:

Log in to the RAM console.
In the left navigation bar, select Manage Permissions > Policies.
On the Policies page, click Create Policy.

On the Create Policy page, select JSON Editor. Enter the following policy:

All MMS operations

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "odps:ListMmsDataSources",
        "odps:CreateMmsDataSource",
        "ram:GetRole",
        "odps:GetMmsDataSource",
        "odps:UpdateMmsDataSource",
        "odps:DeleteMmsDataSource",
        "odps:CreateMmsFetchMetadataJob",
        "odps:GetMmsFetchMetadataJob",
        "odps:ListMmsFetchMetadataJobLogs",
        "odps:ListMmsDbs",
        "odps:GetMmsDb",
        "odps:ListMmsTables",
        "odps:GetMmsTable",
        "odps:ListMmsPartitions",
        "odps:GetMmsPartition",
        "odps:ListMmsJobs",
        "odps:GetMmsJob",
        "odps:CreateMmsJob",
        "odps:DeleteMmsJob",
        "odps:StartMmsJob",
        "odps:StopMmsJob",
        "odps:RetryMmsJob",
        "odps:ListMmsTasks",
        "odps:GetMmsTask",
        "odps:ListMmsTaskLogs",
        "odps:StopMmsTask",
        "odps:StartMmsTask",
        "odps:RetryMmsTask",
        "odps:GetMmsAsyncTask",
        "odps:GetMmsProgress",
        "odps:GetMmsSpeed",
        "odps:CreateMmsAuthFile",
        "odps:ListMmsAgents",
        "odps:ListMmsTimers",
        "odps:GetMmsTimer",
        "odps:UpdateMmsTimer",
        "odps:ListMmsTimerLogs",
        "odps:CreateMmsTimer",
        "odps:UpdateMmsTables",
        "odps:UpdateMmsTable",
        "odps:UpdateMmsDb",
        "odps:ListNetworkLinks"
      ],
      "Resource": "*"
    }
  ]
}

Source data management

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "odps:ListMmsDataSources",
        "odps:CreateMmsDataSource",
        "ram:GetRole",
        "odps:GetMmsDataSource",
        "odps:UpdateMmsDataSource",
        "odps:DeleteMmsDataSource",
        "odps:CreateMmsFetchMetadataJob",
        "odps:GetMmsFetchMetadataJob",
        "odps:ListMmsFetchMetadataJobLogs",
        "odps:ListMmsDbs",
        "odps:GetMmsDb",
        "odps:ListMmsTables",
        "odps:GetMmsTable",
        "odps:ListMmsPartitions",
        "odps:GetMmsPartition",
        "odps:GetMmsAsyncTask",
        "odps:GetMmsProgress",
        "odps:GetMmsSpeed",
        "odps:CreateMmsAuthFile",
        "odps:ListMmsAgents",
        "odps:UpdateMmsTables",
        "odps:UpdateMmsTable",
        "odps:UpdateMmsDb"
      ],
      "Resource": "*"
    }
  ]
}

Migration job management

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "odps:ListMmsDataSources",
        "odps:GetMmsDataSource",
        "odps:CreateMmsFetchMetadataJob",
        "odps:GetMmsFetchMetadataJob",
        "odps:ListMmsFetchMetadataJobLogs",
        "odps:ListMmsDbs",
        "odps:GetMmsDb",
        "odps:ListMmsTables",
        "odps:GetMmsTable",
        "odps:ListMmsPartitions",
        "odps:GetMmsPartition",
        "odps:ListMmsJobs",
        "odps:GetMmsJob",
        "odps:CreateMmsJob",
        "odps:DeleteMmsJob",
        "odps:StartMmsJob",
        "odps:StopMmsJob",
        "odps:RetryMmsJob",
        "odps:ListMmsTasks",
        "odps:GetMmsTask",
        "odps:ListMmsTaskLogs",
        "odps:StopMmsTask",
        "odps:StartMmsTask",
        "odps:RetryMmsTask",
        "odps:GetMmsAsyncTask",
        "odps:GetMmsProgress",
        "odps:GetMmsSpeed",        
        "odps:ListMmsTimers",
        "odps:GetMmsTimer",
        "odps:UpdateMmsTimer",
        "odps:ListMmsTimerLogs",
        "odps:CreateMmsTimer"
      ],
      "Resource": "*"
    }
  ]
}

Click OK. In the Create Policy dialog box, enter a custom Policy Name. You can also add a description and tags.
Attach the custom MMS policy to the RAM user. For more information, see Manage permissions for a RAM user.

Step 5: Prepare quota resources

Data migration jobs consume computing resources. Therefore, you must associate a computing quota with the MaxCompute project that runs the migration jobs.

Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose Manage Configurations > Quotas .
Associate a computing quota with the MaxCompute project for migration tasks.
- Use the default computing quota:
  
  A project is assigned a default computing quota. Jobs initiated in the project use this default quota without additional configuration. When you create a new project, you can directly associate it with the default quota.
- Switch the computing quota:
  1. In the left-side navigation pane, choose Manage Configurations > Projects.
  2. On the Projects page, click Manage in the Actions column for the target project.
  3. On the Project Settings page, click the Parameter Configuration tab.
  4. In the Basic Properties section, click Edit to modify the configuration as needed.
  5. In the Authorized Tunnel Quota drop-down list, select the computing quota for the project and click Submit.
When you add a data source, for the project to run migration tasks setting, select the project to which you associated the computing quota in the previous step.
1. In the left-side navigation pane, choose Data Transfer > Migration Service.
2. On the Migration Service page, click the Data Source tab.
3. On the Data Source tab, click Add Data Source.
4. From the MaxCompute Project for Migration Jobs drop-down list, select the project for which you associated the computing quota in the previous step.

MaxCompute:Prerequisites

Supported regions

Procedure

Step 1: Create a MaxCompute project

Step 2: Create a VPC network connection

Step 3: Create a service-linked role

MaxCompute console

RAM console

Step 4: Configure permissions

MaxCompute console

Admin role

Create a custom role

Via SQL

Project-level permissions

Table-level permissions

Instance-level permissions

All MMS operations

Source data management

Migration job management

Step 5: Prepare quota resources