EMR on ECS runs Apache Hadoop, Spark, Flink, Hive, and other open-source big data components on Alibaba Cloud ECS instances, with self-developed components such as OSS-HDFS and JindoCache and integrations with DataWorks and DLF. Both subscription and pay-as-you-go billing are supported, letting you spin up elastic big data clusters in minutes.
EMR is built on community open-source services such as Apache Hadoop, Spark, Flink, and StarRocks, and provides the cluster management platform around them. EMR on ECS is semi-managed: cluster resources belong to your Alibaba Cloud account, and you are responsible for the routine O&M of the open-source services, including capacity planning, parameter tuning, and troubleshooting. Plan for in-house big data O&M expertise to keep your workloads running. For the full support boundary, see Scope and methods of technical support.
Architecture
EMR integrates Alibaba Cloud services and open-source components alongside self-developed components, and provides cluster management capabilities. For details on component types and supported use cases, see Components and Use scenarios.
Alibaba Cloud services
-
Clusters run on ECS instances.
-
Data can be stored in Object Storage Service (OSS).
-
EMR integrates with DataWorks, which uses EMR as its job computing and data storage engine.
-
EMR Workflow is a fully managed service for scheduling workflows and tasks.
Open-source components
EMR bundles open-source big data components across six categories: data integration, data storage, resource management, compute engines, data development, and data service.
Self-developed components
EMR provides three self-developed components that extend open-source capabilities on Alibaba Cloud infrastructure:
-
OSS-HDFS: An object storage solution compatible with Hadoop Distributed File System (HDFS) APIs. Big data computing tasks access OSS data directly over the standard HDFS protocol.
-
JindoCache: A distributed cache solution that caches data blocks in memory to improve read performance and reduce pressure on the underlying storage system.
-
DLF-Auth: A component from Data Lake Formation (DLF) that enables DLF's data permission management.
Cluster management
-
Create clusters, scale them out, and configure auto scaling rules.
-
Manage cluster and service configurations; perform operations and maintenance (O&M) on nodes and services.
-
Monitor cluster health with multi-dimensional metrics, cluster report analysis, and alerting.
Benefits
|
Capability |
Description |
|
Stable open-source components |
Each EMR release ships with the latest open-source component versions and eliminates cross-component version compatibility issues. The Alibaba Cloud deployment environment delivers much higher performance than standard community builds. For more information about the services that are supported by EMR clusters of different versions, see Release version. |
|
Ease of use |
Spin up a big data computing environment in minutes without manually deploying or starting services. Adjust cluster size with a few clicks. Built-in monitoring and alerting with intelligent diagnosis reduces troubleshooting time. |
|
Cost-effectiveness |
Pay only for the compute resources you use. Hot and cold data is tiered across storage layers to lower unit storage costs. Various O&M tools, intelligent diagnosis, and big data platforms further reduce operational overhead. |
|
Scalability |
Scale cluster resources up or down based on load or on a schedule. Auto scaling completes within minutes and supports multiple elastic resource types. |
|
Deep integration |
Deploy clusters on ECS or Container Service for Kubernetes (ACK). Various ECS instance types are supported. For more information, see ECS instances. Integrate with DataWorks for job scheduling. Use DLF for centralized metadata management across multiple engines in data lake scenarios. |
|
AI-assisted O&M |
The built-in EMR AI Assistant reads your cluster's runtime state and monitoring data to answer questions, diagnose performance, run health checks, and push alerts. Talk to it in the console, or add it to a DingTalk or Lark group. For more information, see Introduction to the EMR AI Assistant Feature. |
For a detailed comparison with self-managed Hadoop clusters, see Comparison between EMR clusters and self-managed Hadoop clusters.
EMR AI Assistant
EMR AI Assistant is the intelligent O&M service built into EMR. Backed by a large language model, it connects to your cluster and reads runtime state and monitoring data before drawing a conclusion, so its diagnoses and recommendations reflect your actual cluster rather than generic advice.
Capabilities
-
Performance diagnosis: When queries time out or latency spikes, the assistant traces the cause and recommends a fix. Ask which queries were slow recently, and it returns the top SQL statements with a bottleneck analysis for each.
-
Health checks: Inspects cluster configuration and runtime state on the schedule you set, then reports what it finds.
-
Proactive alerts: After you turn on daily reports and alerting, the assistant monitors the cluster continuously and pushes issues to DingTalk or Lark, with no one on watch.
-
Q&A: Ask in natural language. Multi-turn conversations let you drill into a problem within a single session.
How to use it
The assistant is available three ways: the chat window in the console, a DingTalk or Lark group, and the API and SDK.
-
To get started, see Get started with EMR AI Assistant.
-
For the full feature set and supported scenarios, see Introduction to the EMR AI Assistant Feature.
-
For editions and billing, see EMR AI Assistant Billing Details.
Billing
EMR on ECS cluster fees consist of EMR service fees and ECS instance fees. Fees for other Alibaba Cloud services used by the cluster (such as OSS, DLF, and EMR Workflow) follow each service's own billing rules: Billing overview, Billing.
EMR on ECS supports two billing methods:
-
Subscription: Pay upfront based on a fixed duration. Reserve capacity in advance at discounted rates. See Subscription.
-
Pay-as-you-go: Pay after use. Purchase and release resources as your workload demands, with no advance commitment. See Pay-as-you-go.
Internet traffic generated by cluster nodes is always billed at pay-as-you-go rates, regardless of the cluster billing method.
For full pricing details, see Billing.
What's next
-
Use EMR with DataWorks: Develop and govern data lakes from a single interface. See Getting started with DataWorks on EMR.
-
Explore use cases: See Use scenarios.
-
Check component versions: See Services supported by EMR clusters of different versions.
-
Monitor and operate clusters: See Cluster O&M and Cluster monitoring.