In EMR Data Platform, a project organizes your data development work. Each project holds jobs and workflows, and connects to the clusters, RAM users, and variables it needs to run.
This page covers: creating a project, viewing its details, enabling security mode, managing RAM user access, associating clusters, and defining custom variables.
Prerequisites
Before you begin, ensure that you have:
An EMR cluster. See Create a cluster
An Alibaba Cloud account (required for most administrative actions — see Limits below)
Limits
Only an Alibaba Cloud account can perform the following actions. The corresponding UI controls are hidden from RAM users.
| Action | Alibaba Cloud account | RAM user |
|---|---|---|
| Create a project | Yes | No |
| Add or remove project members | Yes | No |
| Associate or disassociate clusters | Yes | No |
| View projects | All projects in the account | Only projects with development permissions |
To grant a RAM user development permissions on a project, see Manage RAM users.
Create a project
Log on to the EMR console with your Alibaba Cloud account.
In the top navigation bar, select the region where your cluster resides and select a resource group.
Click the Data Platform tab.
In the upper-right corner of the Projects section, click Create Project.
In the Create Project dialog box, enter a Project Name and Project Description and select an existing resource group from the Select Resource Group drop-down list, and select a resource group from the Select Resource Group drop-down list.
If you leave Select Resource Group blank, the project is added to the default resource group. For details on resource groups, see Use resource groups.
Click Create.
The project appears in the Projects section.
View project details
Log on to the EMR console.
In the top navigation bar, select the region and resource group.
Click the Data Platform tab.
In the Projects section, click the project ID.
Click the Projects tab.
The Basic Information page shows the project name, creation time, description, and the user who created the project.
Enable security mode
Security mode lets you control which EMR user accounts can submit jobs within a project.
If you submit a job with your Alibaba Cloud account, the job runs as the
hadoopuser.If you submit a job as a RAM user, the job runs as the EMR user account with the same name as the RAM user.
After enabling security mode, add the relevant EMR user accounts on the Users page. See Manage user accounts.
Notice: After you enable security mode, Shell and Hive jobs cannot run in the project.
To enable or disable security mode:
Log on to the EMR console with your Alibaba Cloud account.
In the top navigation bar, select the region and resource group.
Click the Data Platform tab.
In the Projects section, click the project ID, then click the Projects tab.
In the left navigation pane, click General Configuration.
Turn Security Mode on or off.
Manage RAM users
Add RAM users to a project to grant them development permissions (view and develop jobs and workflows). Only an Alibaba Cloud account can manage project members.
Log on to the EMR console with your Alibaba Cloud account.
In the top navigation bar, select the region and resource group.
Click the Data Platform tab.
In the Projects section, click the project ID, then click the Projects tab.
In the left navigation pane, click Users.
Add or remove users:
Add a user: Click Add User in the upper-right corner. In the Add User dialog box, select the RAM user and click Add.
Remove a user: Find the RAM user in the list and click Delete in the Actions column.
Configure cluster resources
Associate one or more clusters with a project so that jobs in the project can run on those clusters. Only subscription and pay-as-you-go clusters are supported. Clusters created from a cluster template are not supported.
Associate a cluster
Log on to the EMR console with your Alibaba Cloud account.
In the top navigation bar, select the region and resource group.
Click the Data Platform tab.
In the Projects section, click the project ID, then click the Projects tab.
In the left navigation pane, click Cluster Settings.
Click Add Cluster in the upper-right corner.
In the Add Cluster dialog box, select a resource group and a cluster from the Select Cluster drop-down list, then click OK.
The cluster appears on the Cluster Settings page.
Change cluster configuration
After associating a cluster, you can configure how jobs are submitted to it.
On the Cluster Settings page, find the cluster and click Change Configuration in the Actions column.
In the Change Configuration dialog box, set the following parameters:
Parameter Description Default Default job submission user The user that submits jobs to this cluster. Unique per project. hadoopDefault job submission queue The queue that jobs are submitted to. defaultJob submission user whitelist Users allowed to submit jobs. Separate multiple entries with commas. — Job submission queue whitelist Queues that jobs can be submitted to. Separate multiple entries with commas. — Client whitelist Nodes from which jobs can be submitted. Select the master node of an existing EMR cluster, or a node from an associated gateway cluster. Self-managed gateway clusters on ECS instances are not supported. — Click OK.
Disassociate a cluster
On the Cluster Settings page, find the cluster and click Delete in the Actions column.
Define custom variables
Custom variables are project-level global variables that jobs in the project can reference using the ${VariableName} syntax.
For example, if you define a variable named ENV_ABC with the value 12345, running echo ${ENV_ABC} in a job returns 12345. This is equivalent to running export ENV_ABC=12345 in the job environment.
Add a variable
Log on to the EMR console.
In the top navigation bar, select the region and resource group.
Click the Data Platform tab.
In the Projects section, click the project ID, then click the Projects tab.
In the left navigation pane, click Custom Variable.
Click Add in the upper-right corner.
In the Add Custom Variable dialog box, enter a Variable Name and Value.
To encrypt the variable value, turn on Set as Password.
Click OK.
The variable appears on the Custom Variable page.
Remove a variable
On the Custom Variable page, find the variable and click Delete in the Action column.
What's next
Manage user accounts — add EMR user accounts to use with security mode
Use resource groups — organize your EMR resources by group