E-MapReduce (EMR) automatically binds an ECS application role called MetaService to every cluster you create. Jobs and applications running on the cluster use this role to access Alibaba Cloud resources — no AccessKey pair required in configuration files or code, which eliminates the risk of credential exposure.
MetaService supports access to the following services without an AccessKey pair:
| Service | Default enabled |
|---|---|
| Object Storage Service (OSS) | Yes |
| Log Service | No — requires additional RAM permissions |
| Message Service (MNS) | No — requires additional RAM permissions |
To enable Log Service or MNS access, grant the required permissions to the AliyunEmrEcsDefaultRole role in the RAM console. For more information, see Grant permissions to a RAM role.
Prerequisites
Before you begin, ensure that you have:
-
Authorized the role. For more information, see Assign roles to an Alibaba Cloud account
How MetaService works
When a job or application calls a supported Alibaba Cloud service, MetaService provides a Security Token Service (STS) temporary credential in the background. No AccessKey pair is needed in your code or configuration.
STS credential rotation: A new STS temporary credential is generated 30 minutes before the current one expires. Both credentials remain valid during the 30-minute overlap window.
Default permissions
The AliyunEmrEcsDefaultRole role is configured with the policy AliyunEmrECSRolePolicy, which grants the following permissions.
OSS permissions
| Permission (Action) | Description |
|---|---|
| oss:PutObject | Uploads a file or folder |
| oss:GetObject | Queries a file or folder |
| oss:ListObjects | Queries files |
| oss:DeleteObject | Deletes a file |
| oss:ListBuckets | Queries buckets |
| oss:AbortMultipartUpload | Terminates a multipart upload event |
| oss:ListMultipartUploads | Queries all ongoing multipart upload events |
| oss:RestoreObject | Restores an Archive or Cold Archive object |
| oss:GetBucketInfo | Queries the information about a bucket |
| oss:ListObjectVersions | Queries the versions of all objects in a bucket, including delete markers |
| oss:DeleteObjectVersion | Deletes a specific version of an object |
| oss:PostDataLakeStorageFileOperation | Accesses OSS-HDFS |
Tablestore permissions
| Permission (Action) | Description |
|---|---|
| ots:CreateTable | Creates a table based on the specified table schema |
| ots:DeleteTable | Deletes a specific table from the current instance |
| ots:GetRow | Reads data in a single row based on a specific primary key |
| ots:PutRow | Inserts data into a specific row |
| ots:UpdateRow | Updates data in a specific row |
| ots:DeleteRow | Deletes a row of data |
| ots:GetRange | Reads data within a specific value range of the primary key |
| ots:BatchWriteRow | Inserts, modifies, or deletes multiple rows of data from one or more tables at a time |
| ots:BatchGetRow | Reads multiple rows of data from one or more tables at a time |
| ots:ComputeSplitPointsBySize | Logically splits data in a table into several shards whose sizes are close to the specified size, and returns the split points between the shards and the prompt about hosts where the partitions reside |
| ots:StartLocalTransaction | Creates a local transaction based on a specified partition key value and queries the ID of the local transaction |
| ots:CommitTransaction | Commits a local transaction |
| ots:AbortTransaction | Aborts a local transaction |
Data Lake Formation (DLF) permissions
| Permission (Action) | Description |
|---|---|
| dlf:BatchCreatePartitions | Creates multiple partitions at a time |
| dlf:BatchCreateTables | Creates multiple tables at a time |
| dlf:BatchDeletePartitions | Deletes multiple partitions at a time |
| dlf:BatchDeleteTables | Deletes multiple tables at a time |
| dlf:BatchGetPartitions | Queries information about multiple partitions at a time |
| dlf:BatchGetTables | Queries information about multiple tables at a time |
| dlf:BatchUpdatePartitions | Updates multiple partitions at a time |
| dlf:BatchUpdateTables | Updates multiple tables at a time |
| dlf:CreateDatabase | Creates a database |
| dlf:CreateFunction | Creates a function |
| dlf:CreatePartition | Creates a partition |
| dlf:CreateTable | Creates a table |
| dlf:DeleteDatabase | Deletes a database |
| dlf:DeleteFunction | Deletes a function |
| dlf:DeletePartition | Deletes a partition |
| dlf:DeleteTable | Deletes a table |
| dlf:GetDatabase | Queries information about a database |
| dlf:GetFunction | Queries information about a function |
| dlf:GetPartition | Queries information about a partition |
| dlf:GetTable | Queries information about a table |
| dlf:ListCatalogs | Queries catalogs |
| dlf:ListDatabases | Queries databases |
| dlf:ListFunctionNames | Queries the names of the functions |
| dlf:ListFunctions | Queries functions |
| dlf:ListPartitionNames | Queries the names of the partitions |
| dlf:ListPartitions | Queries partitions |
| dlf:ListPartitionsByExpr | Queries metadata table partitions by conditions |
| dlf:ListPartitionsByFilter | Queries metadata table partitions by conditions |
| dlf:ListTableNames | Queries the names of tables |
| dlf:ListTables | Queries tables |
| dlf:RenamePartition | Renames a partition |
| dlf:RenameTable | Renames a table |
| dlf:UpdateDatabase | Updates a database |
| dlf:UpdateFunction | Updates a function |
| dlf:UpdateTable | Updates a table |
| dlf:UpdateTableColumnStatistics | Updates the statistics of a metadata table |
| dlf:GetTableColumnStatistics | Queries the statistics of a metadata table |
| dlf:DeleteTableColumnStatistics | Deletes the statistics of a metadata table |
| dlf:UpdatePartitionColumnStatistics | Updates the statistics of a partition |
| dlf:GetPartitionColumnStatistics | Queries the statistics of a partition |
| dlf:DeletePartitionColumnStatistics | Deletes the statistics of a partition |
| dlf:BatchGetPartitionColumnStatistics | Queries the statistics of multiple partitions at a time |
| dlf:CreateLock | Creates a metadata lock |
| dlf:UnLock | Unlocks a specific metadata lock |
| dlf:AbortLock | Aborts a metadata lock |
| dlf:RefreshLock | Refreshes a metadata lock |
| dlf:GetLock | Queries information about a metadata lock |
| dlf:GetAsyncTaskStatus | Queries the status of an asynchronous task |
| dlf:DeltaGetPermissions | Queries permissions |
| dlf:GetPermissions | Queries information about data permissions |
| dlf:GetServiceInfo | Queries information about a service |
| dlf:GetRoles | Queries information about roles in data permissions |
| dlf:CheckPermissions | Verifies data permissions |
Use MetaService
Choose the approach that matches your workload:
| Scenario | Approach | What you get |
|---|---|---|
| Jobs running on the cluster (Hadoop, Hive, Spark) | Use simplified OSS paths | Access OSS without credentials in the path |
| Self-managed services or custom applications | Call the MetaService HTTP API | STS temporary credentials for OSS, Log Service, and MNS |
Access OSS from cluster jobs
When MetaService is active, use the simplified path format in any cluster job:
oss://<bucket-name>/<object-path>MetaService handles authentication automatically. You do not need to embed credentials in the path. This also improves your experience when interacting with OSS resources, because the OSS path you need to enter is significantly shorter.
Hadoop — list objects:
# Without MetaService (credentials embedded in path)
hadoop fs -ls oss://ZaH******As1s:Ba23N**************sdaBj2@bucket.oss-cn-hangzhou-internal.aliyuncs.com/a/b/c
# With MetaService (simplified path)
hadoop fs -ls oss://bucket/a/b/cHive — create an external table backed by OSS:
-- Without MetaService (credentials embedded in path)
CREATE EXTERNAL TABLE test_table(id INT, name string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '/t'
LOCATION 'oss://ZaH******As1s:Ba23N**************sdaBj2@bucket.oss-cn-hangzhou-internal.aliyuncs.com/a/b/c';
-- With MetaService (simplified path)
CREATE EXTERNAL TABLE test_table(id INT, name string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '/t'
LOCATION 'oss://bucket/a/b/c';Spark — read OSS data:
// Without MetaService (credentials embedded in path)
val data = sc.textFile("oss://ZaH******As1s:Ba23N**************sdaBj2@bucket.oss-cn-hangzhou-internal.aliyuncs.com/a/b/c")
// With MetaService (simplified path)
val data = sc.textFile("oss://bucket/a/b/c")Get STS credentials from self-managed services
MetaService exposes an HTTP API at http://localhost:10011 on each cluster node. Call these endpoints to retrieve the STS temporary credential your application needs to access Alibaba Cloud resources without an AccessKey pair.
Example — get the cluster region:
curl http://localhost:10011/cluster-regionAvailable endpoints:
| Endpoint | Returns |
|---|---|
/cluster-region |
Region where the cluster resides |
/cluster-role-name |
Role name |
/role-access-key-id |
AccessKey ID of the STS credential |
/role-access-key-secret |
AccessKey secret of the STS credential |
/role-security-token |
Security token of the STS credential |
/cluster-network-type |
Network type |
Usage notes
Modify or delete the AliyunEmrEcsDefaultRole role with caution. Deleting or misconfiguring this role causes cluster creation failures and job failures. To minimize security risk, follow the principle of least privilege when configuring permissions in the RAM console.