All Products
Search

This Product

    E-MapReduce:Switch the metadata storage type

    Document Center

    E-MapReduce:Switch the metadata storage type

    Last Updated:Sep 21, 2026

    In Alibaba Cloud E-MapReduce (EMR), you can select a metadata storage type, such as built-in MySQL, Data Lake Formation (DLF), or self-managed ApsaraDB RDS, when you create a cluster. You can also change the metadata storage type after a cluster is created. This topic introduces Data Lake Formation and describes how to switch the metadata storage type for an EMR cluster.

    Background information

    DLF is a fully managed service that is developed by Alibaba Cloud to help you easily build and manage a cloud-native data lake. DLF allows you to manage metadata and user permissions in a centralized manner, ingest data into data lakes with ease, and explore data in data lakes with a few clicks. For more information, see Overview.

    DLF also allows you to connect the cloud-native data lake to various compute engines. This way, you can break down data silos and gain insight into the business value of data.

    Prerequisites

    The DLF service is activated in the DLF console.

    Limitations

    • DLF unified metadata storage is compatible with Hive 2.x, Hive 3.x, Presto, and Spark SQL in EMR.

    • Only EMR 3.30.0 or later and EMR 4.5.0 or later support DLF unified metadata storage as a Hive database.

    • Switching is supported in both directions. You can switch from a MySQL metastore (built-in MySQL, unified metadatabase, or self-managed RDS) to DLF unified metadata storage, and from DLF unified metadata storage back to a MySQL metastore, which is also referred to as the Hive metastore (HMS).

    • Switching the metadata storage type changes only the way in which Hive accesses metadata. Existing metadata is not migrated automatically. After the switch, Hive can access only the databases and tables that already exist in the target metastore. To retain your existing metadata, migrate the metadata before you switch.

    Procedure

    1. Go to the Hive service page.

      1. Log on to the E-MapReduce console.

      2. In the top navigation bar, select a region and a resource group based on your business requirements.

      3. On the EMR on ECS page, find the target cluster and click Services in the Actions column.

      4. In the Hive service section, click Configure.

    2. In the configuration search box, enter the parameter name hive.imetastoreclient.factory.class and click the search icon.

      • To switch to a MySQL metastore (including built-in MySQL, unified metadatabase, and self-managed RDS):

        Set hive.imetastoreclient.factory.class to org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientFactory.

        If the cluster currently uses DLF unified metadata storage, Hive can access only the databases and tables that already exist in the HMS backend database after the switch. To prevent missing databases and tables or job failures, complete the following preparations before you switch:

        1. Suspend the jobs that are running on the cluster.

        2. Compare the metadata between DLF and HMS, and write the databases and tables that were created or modified after the switch to DLF back to the HMS backend database.

        3. Make sure that the HMS backend database is accessible and that the Hive metadata schema is initialized.

        4. Make sure that the HiveMetaStore component of the Hive service is running. When DLF unified metadata storage is used, Hive accesses metadata directly by using the DLF client, and the HiveMetaStore component may not be running.

        Note

        Comparing and writing back metadata affects data consistency. We recommend that you contact Alibaba Cloud technical support for assistance. For more information, see EMR metadata migration notice.

      • To switch to DLF unified metadata storage:

        Set hive.imetastoreclient.factory.class to com.aliyun.datalake.metastore.hive2.DlfMetaStoreClientFactory.

    3. Save the configuration.

      1. In the Service Configuration section of the Hive service page, click Save.

      2. In the dialog box that appears, enter an execution reason and click Save.

    4. Restart the Hive service.

      1. On the Hive service page, choose More > restart in the upper-right corner.

      2. In the dialog box that appears, enter an Execution Reason and click OK.

      3. In the Confirm dialog box, click OK.

        You can click Operation History at the top of the page to view the task progress and wait for the task to complete.

    After the Hive service is restarted, run SHOW DATABASES and SHOW TABLES to verify that all databases and tables are available before you resume your jobs.

    If other engines in the cluster, such as Spark and Presto, are also configured to use DLF unified metadata storage, update the metadata configurations of those engines accordingly. Otherwise, different engines connect to different metastores and return inconsistent databases and tables.