Document Center
E-MapReduce
E-MapReduce
Announcements & Updates
Document updates
Product Introduction
What is E-MapReduce?
Benefits
Architecture
Use scenarios
Limits
Release version
Overview
Release notes (EMR V4.X series)
Release notes for EMR V4.9.X
Release notes for EMR V4.8.X
Release notes for EMR V4.6.X
Release notes of EMR V4.5.X
Release notes of EMR V4.4.1
Release notes of EMR V4.4.0
Release notes of EMR V4.3.x
Release notes (EMR V3.X series)
Release notes for EMR V3.35.X
Release notes for EMR V3.34.X
Release notes for EMR V3.33.X
Release notes of EMR V3.32.X
Release notes of EMR V3.30.X
Release notes of EMR V3.29.X
Release notes of EMR V3.28.X
Release notes of EMR V3.27.X
Release notes of EMR V3.26.X
Release notes of EMR V3.25.X
Release notes of EMR V3.24.X
Release notes of EMR V3.23.X
Release notes of EMR V3.22.X
Release notes of versions earlier than E-MapReduce V3.22.X
Pricing
Billable items
Subscription
Pay-as-you-go
Expiration and overdue payments
Renewal
Switch from pay-as-you-go to subscription
on ECS
Quick Start
Overview
Make preparations
Create a cluster
Create and run a job
Cluster Management
Cluster planning
Select configurations
Node categories
Gateway clusters
ECS instances
Storage
Supported instances with local disks
Disaster recovery in EMR clusters
Configure clusters
Create a cluster
Create a gateway cluster
Manage permissions
Authorize roles
EMR service role
ECS application role (used in EMR V3.32.0 and earlier V3.X.X versions as well as in EMR V4.5.0 and earlier V4.X.X versions)
ECS application role (used in V3.X.X versions later than EMR V3.32.0 as well as in V4.X.X versions later than EMR V4.5.0)
Use a custom ECS application role to access cloud resources of the same account
Authorize RAM users
Manage user accounts
Manage and use tags
Use resource groups
Security groups
Manage Metadata
Configure an independent ApsaraDB RDS instance
Manage bootstrap actions
Cluster scripts
Manage cluster resources
Cluster resource overview
Use Fair Scheduler
Use Capacity Scheduler
Cluster management page
View the cluster list and cluster details
Manage services
View the running status of services
Add a service
Restart a service
Manage parameters for services
Customize software configurations
View the deployment information of service components
Access the Web UI
Create an SSH tunnel to access web UIs of open source components
Access the web UIs of open source components
Cluster Operations
Common file paths
Log on to a cluster
Scale out a cluster
Scale in a cluster
Auto Scaling
Overview
Create an auto scaling machine group
Manage auto scaling
Configure auto scaling rules by time
View auto scaling history
Configure auto scaling monitoring alerts
Expand disks
Add node groups
Remove abnormal nodes
Cluster renewal
Upgrade node configurations
Status list
Guide for E-MapReduce cluster operations
Release a cluster
FAQ about cluster planning and configuration
Metadata Management
Manage Hive metadata
Manage Hive metadata in a centralized manner
Basic operations on Hive metadata
Migrate Hive metadata
Migrate data from a unified metadatabase to an ApsaraDB RDS instance
Manage Kafka metadata
FAQ
Data Development
Overview
Manage projects
Edit jobs
Edit a workflow
Implement ad hoc queries
Scheduling center
Manage cluster templates
Event codes for Cloud Monitor
Jobs
Configure job time and date
Configure a Shell job
Configure a Hive job
Configure a Hive SQL job
Configure a Spark job
Configure a Spark SQL job
Configure a Spark Shell job
Configure a Spark Streaming job
Configure a Hadoop MapReduce job
Configure a Sqoop job
Configure a Pig job
Configure a VVR-based Flink job
Configure a Streaming SQL job
Configure a Presto SQL job
Configure an Impala SQL job
FAQ about data development
Monitoring Dashboard (new)
Create a threshold-triggered alert rule
Create an event-triggered alert rule
Monitoring and Alarms
Monitoring dashboard
Overview
Dashboard
Events
Cluster monitoring
Service monitoring
Overview
HDFS monitoring
YARN monitoring
Hive monitoring
ZooKeeper monitoring
Spark monitoring
Druid monitoring
Kafka service
Job monitoring
Configure event subscription
Subscription event encoding table
Log search
Query custom metrics
Alarm management
Cluster alert management
Use CloudMonitor to monitor service status
Component Operation Guide
SmartData
HBase
Zeppelin
Oozie
OpenLDAP
Sqoop
Knox
Superset
Tez
Livy
Phoenix
Use Kafka Manager
TensorFlow
HDFS
Introduction
Hudi
Overview
Alluxio
Overview
Kudu
Introduction
YARN
Overview
Best Practices
Use cgroups with YARN to control the CPU usage
Spark
Overview
Manage LDAP authentication
Hive
Overview
Use Hive to perform basic operations
Connect to Hive
Manage LDAP authentication
Impala
Overview
Developer guide
Connect to Impala
Advanced usage
Manage LDAP authentication
Zookeeper
Basic operation
ESS
Overview
Flink
Overview
Kafka
Overview
Access Kafka across clusters
Integrate Kafka with Ranger
Access Kafka over SSL
FAQ
Delta
Overview
Quick start
Scenarios
Scenario 1: Stream ingestion
Scenario 2: Data synchronization
Scenario 3: Hot and cold data separation
Basic operations
Batch reads and writes
Streaming reads and writes
Data deletes, updates, and merges
Optimize tables
Convert tables
Modify tables
Data quality and schema evolution
Table history access
Use Hive to read data from a Delta table
Use Presto to read Delta tables
Appendixes
Appendix 1: Important parameters
Appendix 2: Optimization tips
Appendix 3: FAQ
Presto
Overview
Architecture
Terms
Use a CLI to access the Presto console
Use JDBC
Manage LDAP authentication
Flume
Overview
Best practices
Synchronize audit logs to HDFS
Synchronize data from an EMR Kafka cluster to HDFS
Synchronize data from an EMR Kafka cluster to Hive
Synchronize data from an EMR Kafka cluster to HBase
Synchronize data from an EMR Kafka cluster to OSS
Synchronize data from LogHub to HDFS
Configure Flume
Hue
Use Hue
Advanced usage
Connect Hue to an LDAP server
Configure Hue to connect to the execution engines for which LDAP authentication is enabled
Implement multi-instance load balancing for Hue
Manage LDAP authentication
Best practices
Configure Hue to support access to Presto
Ranger
Overview
Integrate components with Ranger
HDFS
HBase
Hive
Spark
Presto
JindoFS configuration
Connect to LDAP for Ranger
Integrate Ranger UserSync with an LDAP server
Integrate Ranger Admin with an LDAP server
Manage LDAP authentication
Mask Hive data
Kerberos
Introduction to Kerberos
Configure MIT Kerberos authentication
RAM authentication
Execution plan authentication
Cross-region access
Druid
Introduction to Druid
Quick start
Ingestion Spec
Kafka Indexing Service
LOG Indexing Service
Common Druid problems
Component authorization
HDFS
YARN
Hive
HBase
Developer Guide
Preparations
Development preparations
Configure the OSS URI to use E-MapReduce
Use the sample project
Install Python
Spark
Preparations
Parameters
Use Spark to access OSS
Use Spark to access MaxCompute
Use Spark to access Message Queue for Apache RocketMQ
Use Spark to access Tablestore
Use Spark to consume Log Service data
Consume data in real time
Consume data offline
Consume data in Log Service
Use Spark to access MNS
Use Spark to write data to HBase
Use Spark to access Kafka
Use Spark to access MySQL
Configure spark-submit parameters
Spark Streaming SQL
Common keywords
Streaming query
Job template (EMR V3.23.0 and later)
Job template
Configurations
DDL overview
CREATE TABLE
CREATE SCAN
CREATE STREAM
Query overview
SELECT
WHERE
GROUP BY
JOIN
WATERMARK
UNION ALL
DML overview
MERGE INTO
INSERT INTO
Window function
Overview
Tumbling window
Sliding window
Data source
Overview
HBase data source
JDBC data source
Kafka data source
LogHub data source
Tablestore data source
Redis data source
Druid data source
DataHub data source
Hadoop
Parameters
Develop a MapReduce job
Create and run a Hive job
Create and run a Pig job
Hadoop Streaming
Process Table Store data in Hive
HBase
Access HBase
Back up an HBase cluster
SmartData
Overview
Overview and usage of JindoFS
SmartData 3.5.x
SmartData 3.5.X
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Use Jindo AuditLog
Access the web UI of JindoFS
Manage JindoFS permissions
Data management policies
Analyze metadata offline
Use a JindoFS credential provider
Use JindoFS in block storage mode to store encrypted files
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo AuditLog
Use Jindo Job Committer
Use JindoFS OSS credential providers
Access the web UI of JindoFS
Manage JindoFS permissions
JindoTable
Enable query acceleration based on a native engine
Use JindoCube
Use JindoTable
Use JindoTable to collect access frequency statistics on tables and partitions
Use JindoTable to collect infrequent-access statistics of tables and partitions
Toolset
Use JindoFS FUSE
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
Use tiered storage commands of JindoFS
SmartData 3.4.x
SmartData 3.4.X
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Use Jindo AuditLog
Access the web UI of JindoFS
Manage JindoFS permissions
Data management policies
Analyze metadata offline
Use a JindoFS credential provider
Use JindoFS in block storage mode to store encrypted files
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo AuditLog
Use Jindo Job Committer
Use JindoFS OSS credential providers
Access the web UI of JindoFS
Manage JindoFS permissions
JindoTable
Enable query acceleration for ORC files
Use JindoTable
Use JindoCube
Use JindoTable to collect access frequency statistics on tables and partitions
Toolset
Use JindoFS FUSE
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
Use tiered storage commands of JindoFS
SmartData 3.2.X
SmartData 3.2.X
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Use Jindo AuditLog
Access the web UI of JindoFS
Manage JindoFS permissions
Data management policies
Analyze metadata offline
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo AuditLog
Use Jindo Job Committer
Use a credential provider
Access the web UI of JindoFS
Manage JindoFS permissions
JindoTable
Enable query acceleration for ORC files
Use JindoTable
Use JindoCube
Use JindoTable to collect access frequency statistics on tables and partitions
Toolset
Use JindoFS FUSE
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
Use tiered storage commands of JindoFS
SmartData 3.1.x
SmartData 3.1.x
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Use Jindo AuditLog
Access the web UI of JindoFS
Manage JindoFS permissions
Data management policies
Analyze metadata offline
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo AuditLog
Use Jindo Job Committer
Use a credential provider
Access the web UI of JindoFS
Manage JindoFS permissions
JindoTable
Use JindoTable
Use JindoTable to collect access frequency statistics of tables and partitions
Use JindoCube
Toolset
Use JindoFS FUSE
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
Use tiered storage commands of JindoFS
SmartData 3.0.x
SmartData 3.0.0
JindoFS in block storage mode
Use JindoFS in block storage mode
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Access the web UI of JindoFS
Manage JindoFS permissions
Use Jindo AuditLog
Analyze metadata offline
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Access the web UI of JindoFS
Manage JindoFS permissions
Use Jindo Job Committer
Use Jindo AuditLog
Use a credential provider
JindoTable
Use JindoTable
Use JindoCube
Toolset
Use JindoFS FUSE
Use tiered storage commands of JindoFS
Use Jindo DistCp
Use Jindo DistCp in specific scenarios
SmartData 2.7.301
JindoFS in block storage mode
Use JindoFS in block storage mode
Use Tablestore instances to store metadata
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Manage JindoFS permissions
Use Jindo AuditLog
JindoFS in cache mode
Use JindoFS in cache mode
Use the password-free feature of JindoFS SDK
Use Jindo Job Committer
Manage JindoFS permissions
Use Jindo AuditLog
JindoTable
Use JindoCube
Toolset
Use Jindo DistCp
Use JindoFS FUSE
SmartData 2.6.0-2.7.2
SmartData 2.6.X
JindoFS in block storage mode
Use JindoFS in block storage mode
Use Tablestore instances to store metadata
Use RocksDB to store metadata
Use Raft-RocksDB-Tablestore to store metadata
Manage JindoFS permissions
JindoFS in cache mode
Use JindoFS in cache mode
Manage JindoFS permissions
Use Jindo Job Committer
JindoTable
Use JindoCube
Toolset
Use Jindo DistCp
SmartData 2.2.x and earlier
Use JindoFS in EMR V3.20.0 to V3.22.0 (V3.22.0 excluded)
Use JindoFS in EMR V3.22.0 to V3.26.3
Use the block storage mode
Use the cache mode
Use the password-free feature of JindoFS SDK
Use the external client
Best Practices
Migrate data from HDFS to JindoFS
Use MapReduce to process data in JindoFS
Use Hive to query data in JindoFS
Use Spark to process data in JindoFS
Use Flink to process data in JindoFS
Use Impala or Presto to query data in JindoFS
Use JindoFS as the storage backend of HBase
Store the logs of YARN MapReduce and Spark jobs
Import data from Kafka to JindoFS
Access JindoFS across clusters
API Reference
List of operations by function
Description of calls
Request structure
Common parameters
Sign signatures
Responses
Cluster
Create a cluster
Create a cluster
Create a cluster by using templates
Delete a cluster Template
Queries the details of a cluster template
Queries the basic information of a cluster
Query cluster information
Queries the host list of a cluster
Query the list of clusters
Queries the list of cluster templates
Query quick links of a service
Queries host groups in a cluster
Queries the information list of available clusters
Query available resources
View EMR Version information
Modify a cluster Template
ModifyClusterName
Release a cluster
Scale out or scale up an instance
Join resource groups
Cluster scale-in
Cluster service
Add a service
View cluster operation logs
Query Service information
Query Service configuration details
Queries the service configuration tags of a cluster
Query the list of components
Query the operation history of a cluster
View the task list for a specified host
A list of hosts whose operation history is queried
Query the list of services installed on a cluster
Queries the task list for a specified host
View the list of services supported by a cluster
Queries the Service list of a cluster
View a list of health information
Query the modification history of a service configuration
Modifies the configuration of a specified service of a cluster
Run specified actions
Auto Scaling
Create a scaling group
New auto scaling configuration items
Modify a scaling group
Modify a scaling configuration items
View scaling groups
View scaling configuration items
View scaling activities
Query details about a scaling configuration items
View scaling group instance details
Query details about a scaling Group
Perform operations on scaling Group instances
Delete Auto Scaling configuration items
Manage resources
Create Resource Pool
Create a resource queue
Deletes a specified resource pool
Delete Resource Queue
Queries the list of resource pools
Update resource pools
Modifies the scheduling type of a resource pool
Modify resource queue
Synchronize resource pools and configure to clusters
View resource pool policy types
Tags
Query tag list
Create and bind a tag
Unbind a tag
Data development
Clone workflow
Clone a job
Create a workflow directory folder
Create a workflow
Create a job
Create a data development project
Create project cluster settings
Add a project user
Delete a workflow
Delete workflow Directory
Delete a job
Delete a data development project
Delete project cluster settings
Delete a project user
Queries the information of a workflow
Query directory information
Obtain directory tree
Obtains the information about a workflow instance
Query job information
Queries the details of a node instance
Queries the container logs of a node instance
Query the initiator logs of a node instance
Query project details
Query the details of Project settings
Terminate job instance
Queries the workflow list
Query the list of clusters available in a project
Query the list of clusters available for data development
Obtain the commit agent node list
Queries the list of clients that can submit jobs
Queries the workflow instance list
ListJobs
Query the list of job running instances
Queries the node instance list of a workflow
Queries the container status details of a node instance
Queries the SQL results of a node instance
Queries the list of the specified projects
Queries the cluster settings list of a project
Queries the user information of a project
Modify workflows
Query the directory of a workflow
Modify project cluster settings
Rename a directory
Workflow for modifying graph information
Modify a data development project
ModifyJob
Retry workflow
Resume workflow
Submit a running workflow
Submit a running job
Suspend a workflow
Common APIs
Create a cluster
Modify the configurations of a cluster
Scale out a cluster
Create and manage a project
Run a job
Run a workflow
SDK Reference
E-MapReduce SDK release notes
Java SDK
Download SDKs
Quick start
Reference project
Python SDK
Install SDK
Sample code
Best Practices
Data analysis
Adaptive execution of Spark SQL
Data migration and synchronization
Use Presto to query table data in an ApsaraDB RDS for MySQL database
E-MapReduce data migration solution
Use Flink jobs to process OSS data
Use EMR to transmit MySQL binary logs in near real time
Run Flume on a Gateway node to synchronize data
Use Spark Streaming jobs to process Kafka data
Use Kafka Connect to migrate data
Use Hive jobs to process Tablestore data
Use JDBC to connect to HiveServer2
Use PyFlink jobs to process Kafka data
Run the TPC-DS benchmark in an EMR cluster
FAQ
FAQ
all-products-head
This Product
This Product
All Products
E-MapReduce
E-MapReduce
All Products
E-MapReduce
E-MapReduce
Show more
Show less
Announcements & Updates
Document updates
Show more
Show less
Product Introduction
What is E-MapReduce?
Benefits
Architecture
Use scenarios
Limits
Release version
Show more
Show less
Pricing
Billable items
Subscription
Pay-as-you-go
Expiration and overdue payments
Renewal
Switch from pay-as-you-go to subscription
Show more
Show less
on ECS
Quick Start
Cluster Management
Metadata Management
Data Development
Monitoring Dashboard (new)
Monitoring and Alarms
Component Operation Guide
Developer Guide
SmartData
API Reference
SDK Reference
Best Practices
FAQ
Show more
Show less
Free Trial
Free Trial