Migrate Hadoop clusters to DataLake clusters

Background information

E-MapReduce (EMR) Next-Gen Console is EMR’s next-generation cloud-native open-source big data platform. It delivers a new user experience, development platform, resource model, and analytics scenarios. For details about its features, see Announcement: EMR Next-Gen Console is now available.

EMR on ECS, one of EMR’s primary resource models, includes multiple functional updates. In particular, the EMR Next-Gen Console introduces new cluster scenarios—DataLake, Dataflow, OLAP, and Custom—which significantly improve cluster management efficiency and engine performance compared to legacy cluster scenarios (such as Hadoop and Data Science). DataLake clusters are an upgraded version of legacy Hadoop-based data lake clusters. After upgrading to DataLake clusters, you gain multiple benefits. For a detailed feature comparison, see DataLake clusters.

Preparations

Review the legacy cluster architecture

Review your current big data architecture, clarify the legacy cluster’s application scenario, and note the following:

• Service scope and versions: Record the services and their versions running on each legacy cluster to evaluate upgrade compatibility and feature update requirements.

• Metadata type: Confirm the metadata type used by the legacy cluster (DLF or self-managed RDS) to plan integration and migration strategies for the metadata management system in the new architecture.

• Data storage architecture: Analyze the legacy cluster’s data storage architecture (local HDFS, OSS, or JindoFS block mode) to inform the data migration path design.

• User authentication & authorization architecture: Confirm whether services such as OpenLDAP, Ranger, or Kerberos are used so the new architecture can seamlessly inherit existing security mechanisms.

• Scheduling system: Confirm your current development and scheduling system to maintain consistent and smooth task scheduling during migration.

If you have multiple legacy clusters to upgrade, migrate them one at a time to ensure business continuity and stability. Based on actual business needs and priorities, create a practical migration sequence and plan to smoothly transition legacy clusters to new clusters.

Review legacy cluster details

• View cluster instance configuration information

  When creating a new cluster on the new platform, you can reuse basic instance information from the legacy cluster. For details, see Create a new cluster. Apart from software and hardware configurations—which require special attention—you can use identical settings for other parameters in both legacy and new clusters.

  On the legacy EMR on ECS cluster’s Basic Information and Nodes pages, review cluster and node group configurations, focusing on the following:

  Configuration type	Configuration name	Details to review
  Software configuration	Cluster version Service version List of components actually used Hive metadata type	List of services currently used in the cluster and their corresponding versions.
  Hardware configuration	Zone where the cluster resides Instance type and billing method for each node group	The cluster’s zone and hardware specifications for each node group—for example, CPU, memory, system disk, and data disk.

• (Optional) Export cluster service component configurations

  During routine O&M, users often customize default service component configurations or add custom settings based on specific needs. To efficiently migrate these configurations across clusters, use EMR’s configuration export feature to batch-export all configuration settings from the legacy cluster. Then, during new cluster initialization, batch-import these settings to quickly replicate service component configurations. Alternatively, after successfully launching the new cluster, you can manually adjust each service configuration in the service management interface.

  1. Export service configurations.

     See Export and import service configurations to export configuration files for target service components with one click.

     Note

     • Select Configuration Files: Select only edited configuration files. Multiple selections are allowed.

     • Export Mode: The current Hadoop cluster does not support exporting only custom or modified configurations.

     • Export Format: Select JSON format for easy import into the new cluster.

     The exported configuration file parameters are described in the following table.

     Parameter	Description
     ApplicationName	Service name.
     ConfigFileName	Configuration file name.
     ConfigItemKey	Configuration item name.
     ConfigItemValue	Specific value set for the configuration item.

  2. Edit the configuration file.

     Carefully review the exported configuration information, retain only the necessary items applicable to the new environment, and delete unnecessary configurations.

     • When adjusting YARN-related resource parameters, align closely with the cluster’s actual hardware specifications. Ensure the imported parameter values are reasonable for the new cluster.

     • If JindoFS-related configurations (for example, Credential Provider) exist, adjust them according to OSS/OSS-HDFS configurations. For details, see Configure OSS/OSS-HDFS Credential Provider.

  3. Apply the edited configuration file to (Optional) Custom software configuration as preset configurations for the new cluster.

• (Optional) Review bootstrap actions

  During the View cluster instance configuration information phase, check whether the legacy cluster has bootstrap scripts configured. If bootstrap actions exist, carefully evaluate each script’s function to determine if it should be reused in the new cluster.

  For scripts needed in the new cluster, adjust them as follows to ensure proper execution:

  • Update JAR package names and paths related to open-source component versions. For file paths in legacy and new platforms, see Common file paths.

  • If the script downloads files from OSS, update the corresponding OSS commands. For details, see Bootstrap action execution scripts.

  After modifying the script, upload it to OSS and enter the updated OSS address during new cluster creation.

  Important

  Before deploying the bootstrap script to a production cluster, verify it in a staging environment.

• (Optional) Review legacy cluster Auto Scaling rules

  If Auto Scaling rules are configured for the legacy cluster, review them—focusing on maximum instance count, minimum instance count, graceful shutdown, trigger method, and trigger rules—and reconfigure Auto Scaling after creating the new cluster. For details, see Create a custom Auto Scaling policy.

  1. Go to the Auto Scaling page.

     1. Log on to the E-MapReduce console.

     2. Click the target cluster name.

     3. Click the Auto Scaling tab.

  2. Review Auto Scaling rules.

     On the Auto Scaling tab, click Configure Rule in the Actions column for the node group with configured Auto Scaling rules. Focus on the following:

     • Maximum instance count

     • Minimum instance count

     • Graceful shutdown

     • Trigger method

     • Trigger rules (scale-out, scale-in)

  Note

  Parameters such as instance selection method, billing type, instance type, and graceful shutdown are configured in the node group properties panel in the new cluster. For details, see Manage node groups.

• (Optional) Review legacy cluster load metrics

  Review cluster resource load metrics to observe daily resource usage in the legacy cluster and assess hardware requirements for the new cluster. Alternatively, perform a smooth migration by initially matching the legacy cluster’s hardware specifications exactly, then adjust the new cluster’s hardware configuration later based on actual resource utilization.

  • Method 1: View cluster monitoring

    Check cluster load metrics, focusing on YARN and HDFS usage. For details, see View service monitoring metrics.

  • Method 2: View EMR Doctor daily reports

    EMR Doctor daily reports provide global analysis of compute resources, YARN scheduling resources, and HDFS storage resources. You can review total data volume, hot/cold data distribution, and compute task distribution. For details, see View cluster daily reports and analysis.

    Note

    EMR Doctor must be installed on legacy clusters. For details, see Enable EMR Doctor (Hadoop cluster type).

Define the migration plan and timeline

Based on your current big data workload and each legacy cluster’s configuration, define the final migration target and clarify the following key points. Use this information to plan human resources and timeline accordingly.

• New cluster product version and service scope. For service compatibility, see Product versions and optional services.

• New cluster metadata option (DLF or self-managed RDS)

• New cluster storage solution (OSS-HDFS or OSS)

Step 1: Build the new environment cluster

Create a new cluster

For detailed steps and parameter descriptions, see Create a cluster. Fill in parameters based on the cluster configuration details gathered during View cluster instance configuration information. Pay special attention to the following parameters.

• Product versions and optional services

  Based on the service list gathered during View cluster instance configuration information, consult compatibility information to select the appropriate service scope and versions for the new cluster.

  • Component compatibility notes

    As open-source community services evolve, some services in DataLake scenarios use higher versions than those in Hadoop. The following table shows backward compatibility ranges. Use your legacy cluster’s software versions along with this table to determine new cluster service versions.

    Legacy cluster service	Backward compatibility range 1	Backward compatibility range 2	Backward compatibility range 3	Backward compatibility range 4
    Spark	2.x	3.x	-	-
    Hive	2.x	3.x	-	-
    Tez	Fully compatible across all versions	-	-	-
    Delta Lake	0.6.x	0.8.0–1.1.0	-	-
    Iceberg	0.12.x	0.13.x	-	-
    Hudi	0.6.x	0.8.x	0.9.x	0.10.x
    Sqoop	Fully compatible across all versions	-	-	-
    Ranger	1.x	2.x	-	-
    OpenLDAP	Fully compatible across all versions	-	-	-

    Note

    • Backward compatibility range indicates that within this range, higher versions are compatible with lower versions.

    • The above compatibility information is for reference only. Refer to official open-source community documentation for authoritative details.

    • Due to changes in open-source community activity and technology evolution, some open-source services are no longer supported on the new EMR platform.

      For example, Hue, Zeppelin, and Oozie. Migrate to EMR Notebook or EMR Workflow, or deploy the corresponding engines manually on the cluster.

  • Select an appropriate product version

    Important

    If software version requirements are met, choose the latest EMR product version to access richer features.

    In DataLake scenarios, Alibaba Cloud EMR offers two series: EMR-3.x and EMR-5.x. Each series includes multiple product versions with varying integrated services and versions. When building a new cluster, select a product version based on your data lake application scenario and target service compatibility requirements. The following table shows the mapping between legacy and new platform versions.

    Legacy cluster version	Corresponding new cluster version
    EMR-3.35.0: YARN 2.8.5, HDFS 2.8.5, Hive 2.3.7, Spark 2.4.7	EMR-3.x series
    EMR-5.6.0: YARN 3.2.1, HDFS 3.2.1, Hive 3.1.2, Spark 3.2.1	EMR-5.x series

  • HDFS & OSS-HDFS selection notes

    In EMR versions 5.12.1 and later, and 3.46.1 and later, you can choose HDFS or OSS-HDFS as the cluster’s storage method in optional services.

    Based on the storage solution defined in Define the migration plan and timeline, select the corresponding service when configuring optional services during new cluster creation.

    New platform cluster storage method	Service selection
    OSS	OSS-HDFS
    OSS-HDFS	OSS-HDFS

    Note

    When you select OSS-HDFS in the Optional Services parameter, configure the Root Storage Directory of Cluster by selecting a Bucket with OSS-HDFS enabled as the cluster’s root storage path.

• Select metadata

  The new EMR platform supports the following metadata storage options. Choose based on the metadata storage solution defined in Define the migration plan and timeline. If metadata migration is required, see Metadata migration after creating the new cluster.

  Metadata storage method	Description
  Unified DLF metadata (recommended)	Metadata is stored in Data Lake Formation (DLF). If the legacy platform already uses DLF, configure the same DLF data catalog. The new cluster automatically connects to the same metadata after creation, eliminating the need for migration.
  Self-managed RDS	Use a self-managed Alibaba Cloud RDS instance as the metadatabase. When selecting this option, configure existing RDS parameters. For details, see Configure self-managed RDS.
  Built-in MySQL	Metadata is stored in a local MySQL database on the cluster. Important This option is for testing only. Do not use it in production environments.

• (Optional) Custom software configuration

  If you exported service configurations from the legacy cluster or plan to preconfigure settings during cluster creation, enable custom software configuration in the new cluster creation workflow and paste the edited configurations into the input box. For details, see Configure custom software.

• Hardware configuration

  During the View cluster instance configuration information phase, you can fully understand hardware configurations for each node. For different node types—Master, Core, and Task—select appropriate hardware resources based on business needs.

  • When creating a new cluster, use the latest ECS instance families and cloud disk types to leverage newer hardware features.

  • To add more node groups with the same role, use the add node group feature after cluster creation.

  • Attach public network: Enable this per node group. Once enabled, all nodes in the group receive public IP addresses.

    Enable public network attachment for the Master node group to log on to the master node over the public network or use EMR’s access links and port mapping features.

(Optional) Create a new Gateway

Gateways are primarily used to submit jobs to compute clusters and provide isolation. If you used a Gateway cluster on the legacy platform, create a Gateway on the new platform as follows.

The new platform offers more flexible Gateway deployment options, allowing you to deploy Gateways on existing ECS instances with automatic synchronization of compute cluster configurations. To simplify Gateway deployment, EMR provides a tool called EMR-CLI, which helps you easily set up a Gateway on existing Alibaba Cloud ECS instances. For details, see Use EMR-CLI to customize Gateway deployment.

Step 2: Migration and validation

After building the new cluster environment, migrate metadata, data, and jobs from the legacy cluster.

Metadata migration

Both legacy and new EMR platforms support three metadata management methods: self-managed RDS, DLF, and built-in MySQL. For the new platform, we strongly recommend using DLF metadata service. Based on the metadata management methods used in legacy and new clusters, use the following migration approaches.

Data migration

After creating the new cluster, use the following migration methods based on storage differences between legacy and new clusters to ensure accurate and successful data migration.

Data correctness validation

After completing data migration, validate the correctness of HDFS data and Hive databases/tables. If data inconsistency is detected, take immediate action—such as rerunning affected jobs or restoring missing data.

Choose a validation method based on your specific requirements.

Job migration

To ensure legacy cluster jobs run correctly on the new cluster, apply the following migration strategies based on your scheduling system and environment:

• If you use EMR’s legacy Data Studio, migrate to EMR Workflow. For details, see Announcement: Migrate from EMR legacy Data Studio.

• If you use another development environment (for example, Alibaba Cloud DataWorks or a self-built platform), follow the migration guide provided by that environment.

  Refer carefully to the specific migration documentation for your platform and adjust configurations—such as switching compute cluster information—to ensure jobs schedule and execute correctly on the new cluster.

Step 3: Parallel dual-run validation

To minimize potential impact on live business during cluster migration, perform dual-run validation between legacy and new clusters. This involves replicating live traffic to the new cluster and running jobs simultaneously to comprehensively validate data consistency and business correctness.

Since dual-run validation methods vary based on your development environment, business characteristics, and data processing needs, we strongly recommend designing a flexible dual-run validation plan tailored to your specific scenario and requirements.

Step 4: Decommission the legacy cluster

After successfully validating data and business operations, proceed with cutover. Gradually shift job workloads from the legacy cluster to the new cluster, incrementally increasing processing volume on the new cluster until all business runs stably there.

Once all business is fully migrated and the legacy cluster is idle, safely decommission it by following the Release cluster procedure.