Document Center
E-MapReduce
Release notes
Version overview
Release notes
Release notes of E-MapReduce 3.23.0
Release notes of EMR 3.22.0
Release notes of EMR 3.x
Product Introduction
What is E-MapReduce?
Benefits
Architecture
Scenarios
Pricing
Overview
Price of Subscription E-MapReduce, not including ECS (dollar/month per core)
Price of Pay-As-You-Go E-MapReduce, not including ECS (dollar/month per core)
Expiration and overdue payments
Renewal
Quick Start
Step 1: Select configuration
Step 2: Prerequisites
Step 3: Create a cluster
Step 4: Create and run a job
Cluster Planning and Configurations
Cluster planning
Role authorization
User management
Instance types
Components deployed on different instance nodes
Gateway clusters
ECS instances
Storage guide
D1 series
Enable access between classic networks and VPCs
Disaster recovery in E-MapReduce clusters
Configure clusters
Create a cluster
Cluster list and details
Expand a cluster
Release a cluster
Cluster renewal
Connect to a cluster using SSH
Service list
Cluster scripts
Security groups
Access links and ports
Create a gateway cluster
Manage cluster resources
Auto Scaling
Overview
Configure Auto Scaling by time
Preemptible instances in AutoScaling
Auto Scaling records
VPC
MetaService
Modify configurations
Upgrade node configuration
Expand disks
Convert your billing method
Remove abnormal nodes
Multiple host groups
Third-party software
Bootstrap actions
Software configuration
Configure parameters
Data Development
Manage a workflow project
Edit a job
Edit a workflow
Ad hoc queries
Manage cluster templates
Event codes for CloudMonitor
Jobs
Job operations
Time and date variables
Configure a Shell job
Configure a Hive job
Configure a Hive SQL job
Configure a Spark job
Configure a Spark SQL job
Configure a Spark Shell job
Configure a Spark Streaming job
Configure a Hadoop MapReduce job
Configure a Sqoop job
Configure a Pig job
Configure a Flink job
Configure a Streaming SQL job
Metadata management
Manage Hive metadata
Manage Kafka metadata
Monitoring and Alarms
Alarm management
Cluster alarm management
Best Practices
Cluster management
Isolate OSS data of different users
Use cgroups with YARN to control the CPU usage
Use EMR Data Science clusters for deep learning
Data Development
Use E-MapReduce to collect metrics from a Kafka client
E-MapReduce data migration solution
Run Flume on a Gateway node to synchronize data
Use EMR for real-time MySQL binlog transmission
Adaptive execution of Spark SQL
Configure a network connection for using Sqoop to transfer data from a database to an EMR cluster
Connect to ApsaraDB for HBase using E-MapReduce Hive
Use E-MapReduce to submit a Spark Streaming job for consuming Kafka data
Use E-MapReduce to process offline jobs
Use Flink jobs to process OSS data
Submit Storm topologies to process data in Kafka on E-MapReduce
Use ES-Hadoop on E-MapReduce
Use Mongo-Hadoop on E-MapReduce
Deep learning with Analytics Zoo on E-MapReduce
Use Kafka Connect to migrate data
Developer Guide
Use Python on E-MapReduce
Preparations
Development preparations
Configure the OSS URI to use E-MapReduce
Use sample projects
Spark
Preparations
Parameter description
Operate OSS files
Use Spark to write data to OSS
Use MaxCompute in Spark
Use Spark Streaming to consume MQ data
Consume Table Store data in Spark
Use Spark to consume Log Service data
Consume Log Service data
Consume data in real time
Consume data offline
Use Spark Streaming to consume MNS data
Use Spark to write data to HBase
Use Spark Streaming to process Kafka data
Use Spark and write data to MySQL
Configure spark-submit parameters
Spark Streaming SQL
Preface
Keyword
Streaming query
Streaming query configuration
Job template
Job template (new)
DDL overview
SCAN statement
STREAM statement
DDL overview
DML overview
DML overview
Query overview
SELECT statement
WHERE statement
GROUP BY statement
JOIN statement
UNION ALL statement
Window function
Overview
Tumbling window
Sliding window
Data source
Overview
HBase data table instructions
JDBC data table instructions
Kafka data table instructions
LogHub data table instructions
Table Store data table instructions
Druid data table instructions
Redis data table instructions
Hadoop
Parameter description
Create and run MapReduce jobs
Create and run a Hive job
Create and run a Pig job
Create a Hadoop streaming job
Process Table Store data in Hive
Process Table Store data in MR
Create an HBase cluster and use the HBase storage service
Back up HBase
Guide for E-MapReduce cluster operations
FAQ
Q&A about E-MapReduce
Error messages
FAQ about using E-MapReduce
Job exception
Use execution plans
How do I set the authentication mode of HiveServer2 to LDAP?
FAQ about the HDFS balancer in E-MapReduce
O&M FAQ
Errors and Troubleshooting
Appendix
Cluster management page
Notes on E-MapReduce versions
Error code list
Status list
Configure cluster ports
Common E-MapReduce file paths
Open Source Components
Hue
Oozie
Zeppelin
ZooKeeper
Kafka
Quick start
Access Kafka across clusters
Kafka Ranger
Kafka SSL
Common Kafka problems
Kafka Manager
Druid
Introduction to Druid
Quick start
Ingestion Spec
Kafka Indexing Service
LOG Indexing Service
Superset
Common Druid problems
Presto
What is Presto?
Overview
Quick start
System structure
Basic concepts
Command line tool
Uses JDBC
Implement authentication with ApacheDS
Data types
Common connectors
JMX connector
Kafka connector
System connector
SQL statements
SQL statement overview
ALTER SCHEMA
ALTER TABLE
CALL
COMMIT
CREATE SCHEMA
CREATE TABLE
CREATE TABLE AS
CREATE VIEW
DEALLOCATE PREPARE
DELETE
DESCRIBE
DESCRIBE INPUT
DESCRIBE OUTPUT
DROP SCHEMA
DROP TABLE
DROP VIEW
EXECUTE
EXPLAIN
EXPLAIN ANALYZE
GRANT
INSERT
PREPARE
RESET SESSION
REVOKE
ROLLBACK
SET SESSION
SHOW CATALOGS
SHOW COLUMNS
SHOW CREATE TABLE
SHOW CREATE VIEW
SHOW FUNCTIONS
SHOW GRANTS
SHOW PARTITIONS
SHOW SCHEMAS
SHOW SESSION
SHOW TABLES
START TRANSACTION
USE
VALUES
SELECT
WITH clause
GROUP BY clause
HAVING clause
Set operations
ORDER BY clause
LIMIT clause
TABLESAMPLE
UNNEST
Joins
Subquery
Common functions and operators
Decimal function
Binary functions
Bitwise functions
String functions
Mathematical functions and operators
Date and time processing functions
Conditional expressions
Regular expression
Comparison functions and operators
Aggregate functions
Conversion functions
Logical operators
Flume
Use Flume
Configure Flume
Use LogHub Source to move data from non-EMR clusters to HDFS of EMR clusters
Kudu
Use Kudu
Sqoop
Knox
TensorFlow
Component authorization
HDFS authorization
YARN authorization
Hive authorization
HBase authorization
Kafka authorization
Ranger
Introduction to Ranger
Integrate Ranger into HDFS
Integrate Ranger into Hive
Integrate Spark with Ranger
HBase configurations
Integrate Ranger into Kafka
Data masking in Hive
Kerberos authentication
Introduction to Kerberos
Compatible with the MIT Kerberos authentication protocol
RAM authentication
LDAP authentication
Execution plan authentication
Cross-region access
API Reference
Description of calls
Request structure
Sign signatures
Common parameters
Responses
Cluster
CreateClusterV2
CreateClusterTemplate
CreateClusterWithTemplate
DeleteClusterTemplate
DescribeClusterTemplate
DescribeClusterV2
DescribeEmrMainVersion
DescribeClusterBasicInfo
ListClusterHost
ListClusters
ListClusterTemplates
ListClusterServiceQuickLink
ListClusterHostGroup
ListEmrAvailableConfig
ListEmrAvailableResource
ListEmrMainVersion
ModifyClusterTemplate
ModifyClusterName
ReleaseCluster
ResizeClusterV2
Cluster service
ListClusterInstalledService
ListClusterSupportService
ModifyResourcePoolSchedulerType
ModifyResourceQueue
ModifyClusterServiceConfig
CreateResourcePool
CreateResourceQueue
DeleteResourcePool
DeleteResourceQueue
RefreshClusterResourcePool
RunClusterServiceAction
DescribeClusterOperationHostTaskLog
ListClusterOperationHostTask
ListClusterOperationHost
ModifyResourcePool
ListClusterServiceConfigHistory
DescribeClusterServiceConfig
Describeclusterclusternetworkoolschedulertype
ListResourcePool
ListClusterServiceComponentHealthInfo
ListClusterHostComponent
ListClusterOperation
ListClusterService
DescribeClusterService
AddClusterService
DescribeClusterServiceConfigTag
Data development
CloneFlow
CloneFlowJob
CreateFlowCategory
CreateFlowForWeb
CreateFlowJob
CreateFlowProject
CreateFlowProjectClusterSetting
CreateFlowProjectUser
DeleteFlow
DeleteFlowCategory
DeleteFlowJob
DeleteFlowProject
DeleteFlowProjectClusterSetting
DeleteFlowProjectUser
DescribeFlow
DescribeFlowCategory
DescribeFlowCategoryTree
DescribeFlowInstance
DescribeFlowJob
DescribeFlowNodeInstance
DescribeFlowNodeInstanceContainerLog
DescribeFlowNodeInstanceLauncherLog
DescribeFlowProject
DescribeFlowProjectClusterSetting
KillFlowJob
ListFlow
ListFlowCluster
ListFlowClusterAll
ListFlowClusterAllHosts
ListFlowClusterHost
ListFlowInstance
ListFlowJob
ListFlowJobHistory
ListFlowNodeInstance
ListFlowNodeInstanceContainerStatus
ListFlowNodeSqlResult
ListFlowProject
ListFlowProjectClusterSetting
ListFlowProjectUser
ModifyFlow
ListFlowCategory
ModifyFlowProjectClusterSetting
ModifyFlowCategory
ModifyFlowForWeb
ModifyFlowProject
ModifyFlowJob
RerunFlow
ResumeFlow
SubmitFlow
SubmitFlowJob
SuspendFlow
Data type
ClusterInfo
EcsInstance
EcsOrder
BootstrapAction
ExecutionPlanInfo
ExecutionPlanInstance
JobInfo
JobInstance
JobInstanceWorker
SoftwareInfo
FailReason
Enumerations
SDK Reference
E-MapReduce SDK release notes
Create an AccessKey
Java SDK
Download SDKs
Get started
Reference project
Python SDK
Install SDK
Sample code
Kernel Enhancement
Get started with JindoFS
Use JindoFS in E-MapReduce V3.20.0 to V3.22.0 (V3.22.0 excluded)
Use JindoFS in E-MapReduce V3.22.0 or later
Use the block storage mode
Use the cache mode
Use the external client
JindoFS ecosystem
Migrate data from HDFS to JindoFS
Use MapReduce to process data in JindoFS
Use Hive to query data in JindoFS
Use Spark to process data in JindoFS
Use Flink to process data in JindoFS
Use Impala or Presto to query data in JindoFS
Use JindoFS as the storage back end of HBase
Store the logs of YARN MapReduce and Spark jobs
Import data from Kafka to JindoFS
General Reference
Guidelines for the use of documentation
Alibaba Cloud documentation
Instructions for open source documentation
Open-Source Documentation Code of Conduct
Editing Open-Source Documentation
Alibaba Cloud console
Regions and zones
Service limits
Create an AccessKey
ICP Filing
Glossary
all-products-head
This Product
This Product
All Products
E-MapReduce
E-MapReduce
All Products
Release notes
Version overview
Release notes
Show more
Show less
Product Introduction
Download PDF
What is E-MapReduce?
Benefits
Architecture
Scenarios
Show more
Show less
Pricing
Download PDF
Overview
Price of Subscription E-MapReduce, not including ECS (dollar/month per core)
Price of Pay-As-You-Go E-MapReduce, not including ECS (dollar/month per core)
Expiration and overdue payments
Renewal
Show more
Show less
Quick Start
Download PDF
Step 1: Select configuration
Step 2: Prerequisites
Step 3: Create a cluster
Step 4: Create and run a job
Show more
Show less
Cluster Planning and Configurations
Download PDF
Cluster planning
Configure clusters
Modify configurations
Third-party software
Show more
Show less
Data Development
Download PDF
Manage a workflow project
Edit a job
Edit a workflow
Ad hoc queries
Manage cluster templates
Event codes for CloudMonitor
Jobs
Show more
Show less
Metadata management
Manage Hive metadata
Manage Kafka metadata
Show more
Show less
Monitoring and Alarms
Download PDF
Alarm management
Show more
Show less
Best Practices
Download PDF
Cluster management
Data Development
Show more
Show less
Developer Guide
Download PDF
Use Python on E-MapReduce
Preparations
Spark
Spark Streaming SQL
Hadoop
Create an HBase cluster and use the HBase storage service
Back up HBase
Guide for E-MapReduce cluster operations
Show more
Show less
FAQ
Download PDF
Q&A about E-MapReduce
Error messages
FAQ about using E-MapReduce
Job exception
Use execution plans
How do I set the authentication mode of HiveServer2 to LDAP?
FAQ about the HDFS balancer in E-MapReduce
O&M FAQ
Show more
Show less
Errors and Troubleshooting
Download PDF
Appendix
Show more
Show less
Open Source Components
Download PDF
Hue
Oozie
Zeppelin
ZooKeeper
Kafka
Druid
Presto
Flume
Kudu
Sqoop
Knox
TensorFlow
Component authorization
Kerberos authentication
Show more
Show less
API Reference
Download PDF
Description of calls
Cluster
Cluster service
Data development
Data type
Enumerations
Show more
Show less
SDK Reference
Download PDF
E-MapReduce SDK release notes
Create an AccessKey
Java SDK
Python SDK
Show more
Show less
Kernel Enhancement
Download PDF
Get started with JindoFS
JindoFS ecosystem
Show more
Show less
Free Trial
Free Trial