Preparations before using the MMS service - MaxCompute - Alibaba Cloud Documentation Center

This topic describes the preparations you must complete before using MaxCompute Migration Service (MMS).

Usage notes

The MMS service is available only in the China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Ulanqab), China (Shenzhen), China (Chengdu), and Singapore regions.

Important

Before you start a data migration, stop writing data to the source tables and partitions. This prevents data validation failures after the migration is complete.

Procedure

Step 1: Create a target MaxCompute project

A project is the basic organizational unit in MaxCompute.

Before you migrate data, you must specify a target MaxCompute project for the data source.
During the migration, the MaxCompute project is used to execute migration jobs.

Therefore, you must create a MaxCompute project before you migrate data. You can skip this step if you have an existing project. To create a project in the MaxCompute console, perform the following steps:

Log on to the MaxCompute console and select a region in the top-left corner.
In the navigation pane on the left, choose Manage Configurations > Projects.
On the Internal Project tab, click Create Project.
In the Create Project dialog box, configure the parameters and click Confirm.
For more information about the parameters, see Create a MaxCompute project.

Step 2: Enable VPC network connectivity

You can skip this step if you have an existing VPC connection in the current region.

Log on to the MaxCompute console and select a region in the top-left corner.
In the navigation pane on the left, choose Manage Configurations > Network Connection.
On the Network Connection page, click Add Network Connection.

In the Add Network Connection dialog box, configure the parameters as prompted and click OK. When you add a network connection for the first time, you must first grant authorization to allow the MaxCompute platform proxy to request network interface cards. Otherwise, the connection cannot be created.

The following table describes the parameters.

Parameter	Required	Description
Connection Name:	Required	Enter a custom name for the connection.
Type:	Required	The default value is Passthrough.
Region:	Required	The system automatically populates this parameter based on the region you selected in the upper-left corner.
VPC Selected:	Required	A virtual private cloud (VPC) is an isolated virtual network. It provides a secure and configurable private network space similar to a traditional data center. To create a new VPC, see Create or delete a VPC.
Switch:	Required	A vSwitch defines a subnet. Service interconnection is enabled between different vSwitches in the same VPC. Deploy resources across vSwitches in different zones to protect your application from failures in a single zone. If no vSwitch is available, see Create or delete a vSwitch.
Security group:	Required	A security group acts as a virtual firewall for your resources. Manage security groups and their rules to implement fine-grained network isolation and access control. To create a security group, see Create a security group.

Subsequent data migration jobs need to access the data source. Ensure that the VPC can access the Internet or the leased line to the source data.
- If the data source is on the public Internet, use a NAT Gateway to enable public network access. For more information, see Use a NAT Gateway for a unified egress for Internet traffic.
- If the data source is in a local data center, connect to Alibaba Cloud through a leased line. For more information, see Use a leased line connection.

Step 3: Create a service-linked role

Before you use Migration Service (MMS) for the first time, you must create the AliyunServiceRoleForMaxComputeMMS service-linked role. MMS uses this role to access resources in MaxCompute and other cloud products.

If you are using an Alibaba Cloud account, you can directly create the service-linked role.
If the current user is a RAM user, you must first grant this user the <a href="https://ram.console.alibabacloud.com/policies/detail?policyType=System&policyName=AliyunRAMFullAccess" id="72a764ec52c0q" target="_blank">AliyunRAMFullAccess</a> permission. For more information, see Granting permissions to a RAM role.
You can create the service-linked role in two ways: through the MaxCompute console or through the RAM console. The steps are as follows:
Use the MaxCompute console
1. Log on to the MaxCompute console and select a region in the top-left corner.
2. In the navigation pane on the left, choose Data Transfer > Migration Service.
3. On the Data Source tab, click Add Data Source.
4. In the MaxCompute Service-linked Role dialog box, click OK to create the role. If this dialog box does not appear, this means the role has already been created.
Use the RAM console
1. Log on to the Resource Access Management (RAM) console.
2. In the navigation pane on the left, choose Identities > Roles.
3. On the Roles page, click Create Role.
4. In the upper-right corner of the Create Role page, click Create Service Linked Role.
5. On the Create Service Linked Role page, set Select Service to AliyunServiceRoleForMaxComputeMMS and click Create Service Linked Role.

Step 4: Configure permissions

Configure MaxCompute data operation permissions for the user

Use SQL

Connect to MaxCompute using a local client (odpscmd) and add the created service-linked role.

USE <target_projectname>;

-- Add the service-linked role to the user.
Add USER  `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

You can grant permissions to the user.

MaxCompute supports fine-grained access control. This lets you grant fine-grained permissions based on a user's scope of operations within a project.

-- Grant project-level data operation permissions.
GRANT Read|Write|List|CreateTable|CreateInstance|CreateFunction|CreateResource|All 
  ON project <project_name> TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
  
-- Grant table-level data operation permissions.
GRANT Describe|Select|Alter|Update|Drop|ShowHistory|All 
  ON table <table_name> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

-- Grant instance-level data operation permissions. 
GRANT Read|Write|All  
  ON instance <instance_id> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

Metric description:

target_projectname: Required. The name of the target project.
account_id: Required. The UID.
privileges: Required. The names of the permissions to be granted. You can specify multiple permissions. Separate them with commas (,).

Use the MaxCompute console

Log on to the MaxCompute console and select a region in the top-left corner.
In the navigation pane on the left, choose Manage Configurations > Projects.
On the Projects page, click Manage in the Actions column of the target project.
On the Project Settings page, click the Role Permissions tab.

In the Create Role dialog box, configure the parameters and click OK.

This step creates a new project role in MaxCompute. You can then grant data operation permissions specific to MMS to this role. You can also use the built-in admin role. If you use the admin role, you do not need to create a new role and can proceed to the next step.

The following are the configuration parameters:

Parameter	Required	Description
Role Name	Required	The name must be 1 to 64 characters in length, and can contain letters and digits. The name must start with a letter.
Role Type	This parameter is required.	Admin: An administrative role. You can grant administrative permissions using a Policy. You cannot grant resource permissions. You cannot grant administrative permissions using an ACL. Resource: A resource-level role. You can grant resource permissions using a Policy or an ACL. You cannot grant administrative permissions.
Authorization Method	Required	ACL (Access Control Lists): A whitelist-based authorization mechanism that allows a user or role to perform specified operations on a specified object. Policy: A mechanism that supports whitelists and blacklists. It is suitable for large enterprises and complex scenarios that require fine-grained control.

You can grant permissions to perform operations on projects, tables, models, resources, functions, or instances. The following table lists the available permissions.

Object	Permission
Table	Describe, Select, Update, Alter, Drop, ShowHistory, Download
Resource	Read, Write, Download, Delete
Function	Read, Write, Download, Execute, Delete
Package	Read
Project	Read, Write, List, CreateTable, CreateInstance

Select the target project-level role and click Manage Members in the Actions column. In the dialog box that appears, select the Alibaba Cloud account or RAM user to which you want to grant the role, and then click OK. If you cannot find the account by search, you can add it in the Add Account section.

Configure RAM permissions

In this step, you configure MMS operation permissions for the RAM user who performs the data migration. These permissions include MMS data and metadata management, and migration job management. If you use an Alibaba Cloud account to perform data migration, you can skip this step.

If the current RAM user has the permission to manage MaxCompute, that is, <a href="https://ram.console.alibabacloud.com/policies/detail?policyType=System&policyName=AliyunMaxComputeFullAccess" id="58faac6875ati" target="_blank">AliyunMaxComputeFullAccess</a>, you can skip this step.

If you want to grant the RAM user only MMS operation permissions, perform the following steps:

Log on to the Resource Access Management (RAM) console.
In the navigation pane on the left, choose Permissions > Policies.
On the Policies page, click Create Policy.

On the Create Policy page, select JSON. Enter the following access policy:

Access policy for all MMS operations

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "odps:ListMmsDataSources",
        "odps:CreateMmsDataSource",
        "ram:GetRole",
        "odps:GetMmsDataSource",
        "odps:UpdateMmsDataSource",
        "odps:DeleteMmsDataSource",
        "odps:CreateMmsFetchMetadataJob",
        "odps:GetMmsFetchMetadataJob",
        "odps:ListMmsFetchMetadataJobLogs",
        "odps:ListMmsDbs",
        "odps:GetMmsDb",
        "odps:ListMmsTables",
        "odps:GetMmsTable",
        "odps:ListMmsPartitions",
        "odps:GetMmsPartition",
        "odps:ListMmsJobs",
        "odps:GetMmsJob",
        "odps:CreateMmsJob",
        "odps:DeleteMmsJob",
        "odps:StartMmsJob",
        "odps:StopMmsJob",
        "odps:RetryMmsJob",
        "odps:ListMmsTasks",
        "odps:GetMmsTask",
        "odps:ListMmsTaskLogs",
        "odps:StopMmsTask",
        "odps:StartMmsTask",
        "odps:RetryMmsTask",
        "odps:GetMmsAsyncTask",
        "odps:GetMmsProgress",
        "odps:GetMmsSpeed",
        "odps:CreateMmsAuthFile",
        "odps:ListMmsAgents"
      ],
      "Resource": "*"
    }
  ]
}

Access policy for MMS source data management

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "odps:ListMmsDataSources",
        "odps:CreateMmsDataSource",
        "ram:GetRole",
        "odps:GetMmsDataSource",
        "odps:UpdateMmsDataSource",
        "odps:DeleteMmsDataSource",
        "odps:CreateMmsFetchMetadataJob",
        "odps:GetMmsFetchMetadataJob",
        "odps:ListMmsFetchMetadataJobLogs",
        "odps:ListMmsDbs",
        "odps:GetMmsDb",
        "odps:ListMmsTables",
        "odps:GetMmsTable",
        "odps:ListMmsPartitions",
        "odps:GetMmsPartition",
        "odps:GetMmsAsyncTask",
        "odps:GetMmsProgress",
        "odps:GetMmsSpeed",
        "odps:CreateMmsAuthFile",
        "odps:ListMmsAgents"
      ],
      "Resource": "*"
    }
  ]
}

Access policy for MMS migration job management

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "odps:ListMmsDataSources",
        "odps:GetMmsDataSource",
        "odps:CreateMmsFetchMetadataJob",
        "odps:GetMmsFetchMetadataJob",
        "odps:ListMmsFetchMetadataJobLogs",
        "odps:ListMmsDbs",
        "odps:GetMmsDb",
        "odps:ListMmsTables",
        "odps:GetMmsTable",
        "odps:ListMmsPartitions",
        "odps:GetMmsPartition",
        "odps:ListMmsJobs",
        "odps:GetMmsJob",
        "odps:CreateMmsJob",
        "odps:DeleteMmsJob",
        "odps:StartMmsJob",
        "odps:StopMmsJob",
        "odps:RetryMmsJob",
        "odps:ListMmsTasks",
        "odps:GetMmsTask",
        "odps:ListMmsTaskLogs",
        "odps:StopMmsTask",
        "odps:StartMmsTask",
        "odps:RetryMmsTask",
        "odps:GetMmsAsyncTask",
        "odps:GetMmsProgress",
        "odps:GetMmsSpeed"
      ],
      "Resource": "*"
    }
  ]
}

Click OK. In the Create Policy dialog box, enter a custom Policy Name. Optionally, add a description and attach tags.
Attach the custom MMS access policy to the RAM user. For more information, see Grant permissions to a RAM user.

Step 5: Prepare quota resources

Data migration jobs consume computing resources. Therefore, you must assign computing quota resources to the MaxCompute project that runs the migration jobs.

Log on to the MaxCompute console and select a region in the top-left corner.
In the navigation pane on the left, choose Manage Configurations > Quotas.
On the Quotas page, click New Quota.
On the Purchase page, for Specification Type, select Data Transmission Service.
We recommend that you purchase subscription computing resources to reduce migration costs.
You can skip this step if you have existing Data Transmission Service quota resources.
Assign computing quota resources to the MaxCompute project that runs the migration jobs.
- Use the default computing quota:
  A project is assigned a default computing quota. Jobs initiated by this project use the default quota resources by default. When you create a new project, you can directly associate it with the default quota.
- Switch the computing quota:
  1. In the navigation pane on the left, choose Manage Configurations > Projects.
  2. On the Projects page, click Manage in the Actions column of the target project.
  3. On the Project Settings page, click the Parameter Configuration tab.
  4. In the Basic Properties section, click Edit.
  5. From the Authorized Tunnel Quota drop-down list, select a computing quota for the project and click Submit.
When you add a data source, set the project to which you assigned computing quota resources as the project to run migration jobs.
1. In the navigation pane on the left, choose Data Transfer > Migration Service.
2. On the Migration Service page, click the Data Source tab.
3. On the Data Source tab, click Add Data Source.
4. From the MaxCompute Project To Run Migration Jobs drop-down list, select the project to which you assigned computing quota resources.

Usage notes

Procedure

Step 1: Create a target MaxCompute project

Step 2: Enable VPC network connectivity

Step 3: Create a service-linked role

Use the MaxCompute console

Use the RAM console

Step 4: Configure permissions

Use SQL

Use the MaxCompute console

Access policy for all MMS operations

Access policy for MMS source data management

Access policy for MMS migration job management

Step 5: Prepare quota resources