Document Center
E-MapReduce
E-MapReduce
Announcements & Updates
Document updates
Product Introduction
What is E-MapReduce?
Benefits
Architecture
Use scenarios
Limits
Release version
Overview
Release notes (EMR V5.X series)
Release notes for EMR V5.3.X
Release notes for EMR V5.2.X
Release notes (EMR V4.X series)
Release notes for EMR V4.9.X
Release notes for EMR V4.8.X
Release notes for EMR V4.6.X
Release notes of EMR V4.5.X
Release notes of EMR V4.4.1
Release notes of EMR V4.4.0
Release notes of EMR V4.3.x
Release notes (EMR V3.X series)
Release notes for EMR V3.37.X
Release notes for EMR V3.36.X
Release notes for EMR V3.35.X
Release notes for EMR V3.34.X
Release notes for EMR V3.33.X
Release notes of EMR V3.32.X
Release notes of EMR V3.30.X
Release notes of EMR V3.29.X
Release notes of EMR V3.28.X
Release notes of EMR V3.27.X
Release notes of EMR V3.26.X
Release notes of EMR V3.25.X
Release notes of EMR V3.24.X
Release notes of EMR V3.23.X
Release notes of EMR V3.22.X
Release notes of versions earlier than E-MapReduce V3.22.X
Pricing
Billing overview
Billing rules
Subscription
Pay-as-you-go
Switch from pay-as-you-go to subscription
Renewal
Expiration and overdue payments
View billing information
Refunds
Billing FAQ
Quick start
O&M and Management Guide
Cluster Management
Cluster planning
Select configurations
Node categories
Gateway clusters
ECS instances
Storage
Supported instances with local disks
Disaster recovery in EMR clusters
Configure clusters
Create a cluster
Create a gateway cluster
Manage permissions
Assign roles
EMR service role
ECS application role (used in EMR V3.32.0 and earlier V3.X.X versions as well as in EMR V4.5.0 and earlier V4.X.X versions)
ECS application role (used in V3.X.X versions later than EMR V3.32.0 as well as in V4.X.X versions later than EMR V4.5.0)
Use a custom ECS application role to access cloud resources of the same account
Grant permissions to RAM users
Manage user accounts
Manage and use tags
Use resource groups
Manage security groups
Manage Metadata
Configure an independent ApsaraDB RDS for MySQL database
Manage bootstrap actions
Cluster scripts
Manage cluster resources
Cluster resource overview
Use Fair Scheduler
Use Capacity Scheduler
Cluster management page
View the cluster list and cluster details
Manage services
View the running status of services
Add a service
Restart a service
Manage parameters for services
Customize software configurations
View the deployment information of service components
Access the Web UI
Create an SSH tunnel to access web UIs of open source components
Access the web UIs of open source components
Cluster Operations
Common file paths
Log on to a cluster
Scale out a cluster
Scale in a cluster
Auto Scaling
Overview
Create an auto scaling machine group
Manage auto scaling
Configure auto scaling rules by time
View auto scaling history
Configure auto scaling monitoring alerts
Expand disks
Add node groups
Remove abnormal ECS instances
Upgrade node configurations
Status list
Cluster O&M guide
Release a cluster
FAQ about cluster management
Metadata Management
Manage Hive metadata
Manage Hive metadata in a centralized manner
Basic operations on Hive metadata
Migrate Hive metadata
Migrate data from a unified metadatabase to an ApsaraDB RDS instance
Manage Kafka metadata
FAQ about metadata management
Monitoring Dashboard (new)
Create a threshold-triggered alert rule
Create an event-triggered alert rule
Monitoring and Alarms
Monitoring dashboard
Overview
Dashboard
Events
Cluster monitoring
Service monitoring
Overview
HDFS monitoring
YARN monitoring
Hive monitoring
ZooKeeper monitoring
Spark monitoring
Druid monitoring
Kafka service
Job monitoring
Configure event subscription
Subscription event encoding table
Log search
Query custom metrics
Alarm management
Cluster alert management
Use CloudMonitor to monitor service status
EMR Development Guide
Data Development
Overview
Manage projects
Edit jobs
Edit a workflow
Perform ad hoc queries
Scheduling center
Create a cluster template
Event codes for Cloud Monitor
Job configuration
Configure job time and date
Configure a Shell job
Configure a Hive job
Configure a Hive SQL job
Configure a Spark job
Configure a Spark SQL job
Configure a Spark Shell job
Configure a Spark Streaming job
Configure a Hadoop MapReduce job
Configure a Sqoop job
Configure a Pig job
Configure a VVR-based Flink job
Configure a Streaming SQL job
Configure a Presto SQL job
Configure an Impala SQL job
FAQ about data development
Data Developer Guide
Preparations
Development preparations
Configure the OSS URI to use E-MapReduce
Use the sample project
Install Python
Spark
Preparations
Parameters
Use Spark to access OSS
Use Spark to access MaxCompute
Use Spark to access Message Queue for Apache RocketMQ
Use Spark to access Tablestore
Use Spark to consume Log Service data
Consume data in real time
Consume data offline
Consume data in Log Service
Use Spark to access MNS
Use Spark to write data to HBase
Use Spark to access Kafka
Use Spark to access MySQL
Configure spark-submit parameters
Spark Streaming SQL
Common keywords
Streaming query
Job template (EMR V3.23.0 and later)
Job template
Configurations
DDL overview
CREATE TABLE
CREATE SCAN
CREATE STREAM
Query overview
SELECT
WHERE
GROUP BY
JOIN
WATERMARK
UNION ALL
DML overview
MERGE INTO
INSERT INTO
Window function
Overview
Tumbling window
Sliding window
Data source
Overview
HBase data source
JDBC data source
Kafka data source
LogHub data source
Tablestore data source
Redis data source
Druid data source
DataHub data source
Hadoop
Parameters
Develop a MapReduce job
Create and run a Hive job
Create and run a Pig job
Hadoop Streaming
Process Table Store data in Hive
HBase
Access HBase
Back up an HBase cluster
Component Operation Guide
Zeppelin
Zepplin overview
SmartData
Oozie
OpenLDAP
Sqoop
Knox
Superset
Tez
Livy
Phoenix
TensorFlow
HDFS
Overview
Basic usage
Common commands
Advanced use
Enable HDFS authorization
Configure quotas
Snapshots
HDFS balancer
HaAdmin tool
HDFS
Developer guide
Best practices
JVM memory tuning
Optimization in real-time computing scenarios
HDFS usage optimization
FAQ
HBase
HBase overview
HBase
Hudi
Overview
Basic operations
Advanced usage
Integrate Hudi with Spark SQL
Developer guide
DDL statements
DML statements
Configure Hudi-related parameters
FAQ
Alluxio
Overview
Basic usage
Common commands
Advanced usage
Common administrative commands
Cache policies and lifecycle management commands
Configure permissions
FAQ
Kudu
Overview
Basic usage
Common commands
Integrate Impala with Kudu
Developer guide
Manage tables
FAQ
YARN
Overview
YARN
Best Practices
Use cgroups with YARN to control the CPU usage
Spark
Overview
Advanced use
Manage LDAP authentication
FAQ
Hive
Overview
Enhanced features of Hive in EMR
Basic usage
Use Hive to perform basic operations
Connect to Hive
Advanced use
Use Kerberos authentication to access Hive
Hive
Manage LDAP authentication
Optimize Hive jobs
Developer guide
UDFs
Best practices
Use Hive to access HBase data in EMR
Use Hive to access Phoenix data in EMR
Use Hive to access data of Delta Lake and Hudi
Use Hive jobs to process Tablestore data
Run the TPC-DS benchmark in an EMR cluster
Use JDBC to connect to HiveServer2
FAQ
Impala
Overview
Developer guide
Use JDBC to connect to Impala
Use the Impala shell tool
Advanced usage
Manage LDAP authentication
FAQ
Zookeeper
Overview
Use ZooKeeper
FAQ
ESS
Overview
Flink
Overview
Use Flink (VVR) on YARN
Access the web UI of Flink-VVP
Flink SQL Reference
Kafka
Overview
Access Kafka across clusters
Integrate Kafka with Ranger
Access Kafka over SSL
FAQ about Kafka
Use Kafka Manager
DeltaLake
Overview
Use Delta Lake
Developer guide
Delta Lake parameter
Batch reads and writes
Streaming reads and writes
Manage data
Scenarios
Scenario 1: Stream ingestion
Scenario 2: Data synchronization
Scenario 3: Hot and cold data separation
Scenario 4: Slowly changing dimensions
FAQ
Presto
Overview
Architecture
Terms
Use a CLI to access the Presto console
Use JDBC
Manage LDAP authentication
Flume
Overview
Advanced usage
Developer guide
Install a third-party plug-in
Configure a custom source
Configure a custom sink
Best practices
Synchronize audit logs to HDFS
Synchronize data from an EMR Kafka cluster to HDFS
Synchronize data from an EMR Kafka cluster to Hive
Synchronize data from an EMR Kafka cluster to HBase
Synchronize data from an EMR Kafka cluster to OSS
Synchronize data from LogHub to HDFS
FAQ
Hue
Use Hue
Basic usage
Manage accounts
Add configuration items
Configure YARN queues
Advanced usage
Connect Hue to an LDAP server
Configure Hue to connect to the execution engines for which LDAP authentication is enabled
Implement multi-instance load balancing for Hue
Manage LDAP authentication
Best practices
Configure Hue to support access to Presto
Use the HBase service on the Hue web UI
Use File Browser on the Hue web UI
Use editors on the Hue web UI
Ranger
Overview
Integrate components with Ranger
HDFS
HBase
Hive
Spark
Presto
Impala
YARN
JindoFS configuration
Advanced features
Integrate Ranger UserSync with an LDAP server
Integrate Ranger Admin with an LDAP server
Manage LDAP authentication
Mask Hive data
Filter Hive data by row
View audit logs on the Ranger web UI
Configure security zones
Kerberos
Introduction to Kerberos
Configure MIT Kerberos authentication
Configure cross-realm trust
RAM authentication
Execution plan authentication
Druid
Introduction to Druid
Quick start
Ingestion Spec
Kafka Indexing Service
LOG Indexing Service
Common Druid problems
SmartData
Overview
Overview and usage of JindoFS
SmartData 3.6.x
SmartData 3.6.X overview
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Use Jindo AuditLog
Access the web UI of JindoFS
Manage JindoFS permissions
Data management policies
Analyze metadata offline
Use a JindoFS credential provider
Use JindoFS in block storage mode to store encrypted files
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo AuditLog
Use Jindo Job Committer
Use JindoFS OSS credential providers
Access the web UI of JindoFS
Manage JindoFS permissions
JindoTable
Enable query acceleration based on a native engine
Use JindoTable
Archive and unarchive data in SDK mode
Use the MoveTo command of JindoTable
Use JindoTable to collect access frequency statistics on tables and partitions
Use JindoTable to collect infrequent-access statistics of tables and partitions
Toolset
Use Jindo SQL
Use JindoFS FUSE
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
Use tiered storage commands of JindoFS
SmartData 3.5.x
SmartData 3.5.X
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Use Jindo AuditLog
Access the web UI of JindoFS
Manage JindoFS permissions
Data management policies
Analyze metadata offline
Use a JindoFS credential provider
Use JindoFS in block storage mode to store encrypted files
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo AuditLog
Use Jindo Job Committer
Use JindoFS OSS credential providers
Access the web UI of JindoFS
Manage JindoFS permissions
JindoTable
Enable query acceleration based on a native engine
Use JindoCube
Use JindoTable
Use JindoTable to collect access frequency statistics on tables and partitions
Use JindoTable to collect infrequent-access statistics of tables and partitions
Toolset
Use JindoFS FUSE
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
Use tiered storage commands of JindoFS
SmartData 3.4.x
SmartData 3.4.X
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Use Jindo AuditLog
Access the web UI of JindoFS
Manage JindoFS permissions
Data management policies
Analyze metadata offline
Use a JindoFS credential provider
Use JindoFS in block storage mode to store encrypted files
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo AuditLog
Use Jindo Job Committer
Use JindoFS OSS credential providers
Access the web UI of JindoFS
Manage JindoFS permissions
JindoTable
Enable query acceleration for ORC files
Use JindoTable
Use JindoCube
Use JindoTable to collect access frequency statistics on tables and partitions
Toolset
Use JindoFS FUSE
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
Use tiered storage commands of JindoFS
SmartData 3.2.X
SmartData 3.2.X
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Use Jindo AuditLog
Access the web UI of JindoFS
Manage JindoFS permissions
Data management policies
Analyze metadata offline
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo AuditLog
Use Jindo Job Committer
Use a credential provider
Access the web UI of JindoFS
Manage JindoFS permissions
JindoTable
Enable query acceleration for ORC files
Use JindoTable
Use JindoCube
Use JindoTable to collect access frequency statistics on tables and partitions
Toolset
Use JindoFS FUSE
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
Use tiered storage commands of JindoFS
SmartData 3.1.x
SmartData 3.1.x
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Use Jindo AuditLog
Access the web UI of JindoFS
Manage JindoFS permissions
Data management policies
Analyze metadata offline
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo AuditLog
Use Jindo Job Committer
Use a credential provider
Access the web UI of JindoFS
Manage JindoFS permissions
JindoTable
Use JindoTable
Use JindoTable to collect access frequency statistics of tables and partitions
Use JindoCube
Toolset
Use JindoFS FUSE
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
Use tiered storage commands of JindoFS
SmartData 3.0.x
SmartData 3.0.0
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Access the web UI of JindoFS
Manage JindoFS permissions
Use Jindo AuditLog
Analyze metadata offline
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Access the web UI of JindoFS
Manage JindoFS permissions
Use Jindo Job Committer
Use Jindo AuditLog
Use a credential provider
JindoTable
Use JindoTable
Use JindoCube
Toolset
Use JindoFS FUSE
Use tiered storage commands of JindoFS
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
SmartData 2.7.301
JindoFS in block storage mode
Use JindoFS in block storage mode
Use Tablestore instances to store metadata
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Manage JindoFS permissions
Use Jindo AuditLog
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo Job Committer
Manage JindoFS permissions
Use Jindo AuditLog
JindoTable
Use JindoCube
Toolset
Use Jindo DistCp
Use JindoFS FUSE
SmartData 2.6.0-2.7.2
SmartData 2.6.X
JindoFS in block storage mode
Use JindoFS in block storage mode
Use Tablestore instances to store metadata
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Manage JindoFS permissions
JindoFS in cache mode
Use JindoFS in cache mode
Manage JindoFS permissions
Use Jindo Job Committer
JindoTable
Use JindoCube
Toolset
Use Jindo DistCp
SmartData 2.2.x and earlier
Use JindoFS in EMR V3.20.0 to V3.22.0 (V3.22.0 excluded)
Use JindoFS in EMR V3.22.0 to V3.26.3
Use the block storage mode
Use the cache mode
Use the password-free feature of JindoFS SDK
Use the external client
Best Practices
Migrate data from HDFS to JindoFS
Use MapReduce to process data in JindoFS
Use Hive to query data in JindoFS
Use Spark to process data in JindoFS
Use Flink to process data in JindoFS
Use Impala or Presto to query data in JindoFS
Use JindoFS as the storage backend of HBase
Store the logs of YARN MapReduce and Spark jobs
Import data from Kafka to JindoFS
Access JindoFS across clusters
Change the HDFS client path for the Jindo system
SmartData FAQ
Best Practices
Data analysis
Adaptive execution framework of Spark SQL
Data migration and synchronization
Use Presto to query table data in an ApsaraDB RDS for MySQL database
E-MapReduce data migration solution
Use Flink jobs to process OSS data
Use EMR to transmit MySQL binary logs in near real time
Use Spark Streaming jobs to process Kafka data
Use Kafka Connect to migrate data
Use PyFlink jobs to process Kafka data
API Reference
List of operations by function
Description of calls
Request structure
Common parameters
Sign signatures
Responses
Cluster
Create a cluster
Create a cluster
Create a cluster by using templates
Delete a cluster Template
Queries the details of a cluster template
Queries the basic information of a cluster
Query cluster information
Queries the host list of a cluster
Query the list of clusters
Queries the list of cluster templates
Query quick links of a service
Queries host groups in a cluster
Queries the information list of available clusters
Query available resources
View EMR Version information
Modify a cluster Template
ModifyClusterName
Release a cluster
Scale out or scale up an instance
Add to a resource group
Cluster scale-in
Cluster service
Add a service
View cluster operation logs
Query Service information
Query Service configuration details
Queries the service configuration tags of a cluster
Query the list of components
Query the operation history of a cluster
View the task list for a specified host
A list of hosts whose operation history is queried
Query the list of services installed on a cluster
Queries the task list for a specified host
View the list of services supported by a cluster
Queries the Service list of a cluster
View a list of health information
Query the modification history of a service configuration
Modifies the configuration of a specified service of a cluster
Run specified actions
Auto Scaling
Create a scaling group
Create a Auto Scaling configuration items
Modify a scaling group
Modify a scaling configuration items
View scaling groups
View scaling configuration items
View scaling activities
Querying Scaling configuration items Details
View scaling group instance details
Query details about a scaling Group
Perform operations on scaling Group instances
Delete Auto Scaling configuration items
Manage resources
Create Resource Pool
Create a resource queue
Deletes a specified resource pool
Delete Resource Queue
Queries the list of resource pools
Update resource pools
Modifies the scheduling type of a resource pool
Modify resource queue
Synchronize resource pools and configure to clusters
View resource pool policy types
Tags
Query tag list
Create and bind a tag
Unbind a tag
Data development
Create a job
Clone a job
Create a workflow
Clone workflow
Create a data development project
Create project cluster settings
Create a workflow directory folder
Add a project user
Delete a workflow
Delete workflow Directory
Delete a job
Delete a data development project
Delete project cluster settings
Delete a project user
Queries the information of a workflow
Query directory information
Obtain directory tree
Obtains the information about a workflow instance
Query job information
Queries the details of a node instance
Queries the container logs of a node instance
Query the initiator logs of a node instance
Query project details
Query the details of Project settings
Terminate job instance
Queries the workflow list
Query the list of clusters available in a project
Query the list of clusters available for data development
Obtain the commit agent node list
Queries the list of clients that can submit jobs
Query the list of workflow instances
ListJobs
Query the list of running job instances
Queries the node instance list of a workflow
Queries the container status details of a node instance
Queries the SQL results of a node instance
Queries the list of the specified projects
Queries the cluster settings list of a project
Queries the user information of a project
Modify workflows
Query the directory of a workflow
Modify project cluster settings
Rename a directory
Workflow for modifying graph information
Modify a data development project
ModifyJob
Retry workflow
Resume workflow
Submit a running workflow
Submit and run a job
Suspend a workflow
Common APIs
Create a cluster
Modify the configurations of a cluster
Scale out a cluster
Create and manage a project
Run a job
Run a workflow
SDK Reference
E-MapReduce SDK release notes
Java SDK
Download SDKs
Quick start
Reference project
Python SDK
Install SDK
Sample code
FAQ
all-products-head
This Product
This Product
All Products
All Products
Show more
Show less