All Products
Search

This Product

    E-MapReduce:Manage logs

    Document Center

    E-MapReduce:Manage logs

    Last Updated:Jun 20, 2026

    Open-source components in E-MapReduce (EMR) generate numerous logs during operation. The Log Management feature integrates EMR with Simple Log Service (SLS), allowing you to query these component logs directly in the EMR console.

    Prerequisites

    Limitations

    • This topic applies only to clusters for the following scenarios: DataLake, DataFlow, Online Analytical Processing (OLAP), DataServing, and DataLake (Hadoop).

    • Log shipping is supported for the following services: hdfs, yarn, yarn_application, hive, spark, jindodata, tez, flink, hbase, zookeeper, kafka, presto, kudu, impala, flume, starrocks, clickhouse, kyuubi, rss, and host.

    Usage notes

    • Simple Log Service (SLS) separately charges for items such as log storage and indexed traffic at standard rates. For more information, see Billable items.

    • If you are using a RAM user, log on to the RAM console with your Alibaba Cloud account and grant the AliyunLogFullAccess permission to the RAM user. For more information, see Manage permissions for a RAM user.

    Enable log shipping

    1. Go to the Log Management page.

      1. Log on to the E-MapReduce console.

      2. In the top navigation bar, select a region and a resource group as needed.

      3. On the EMR on ECS page, click the Cluster ID of the target cluster.

      4. Click the Logs tab.

    2. Configure the log shipping scope.

      1. On the Logs page, click Set Log Shipping Scope in the upper-right corner.

      2. In the dialog box that appears, configure the Project for log storage.

        You can select an existing Project or create a new one.

        Important

        You cannot change the Project once it is selected.

      3. Select the services whose logs you want to ship, and then click OK.

    3. View service logs.

      1. From the Please select EMR service drop-down list, select the service whose logs you want to view.

      2. Analyze logs on the log page.

        You can specify a time range and a query statement to perform real-time queries. For example, you can quickly analyze the distribution of a specific field over a period, or filter and aggregate access records based on specified conditions.

    Disable log shipping

    Important

    Disabling real-time log shipping does not automatically delete the Project. To avoid unnecessary charges after you disable log shipping, go to the Simple Log Service console and delete the configured Project. For more information, see Manage a Project.

    If you no longer need to retain log data, follow these steps to disable log shipping.

    1. On the Logs page, click Disable Log Shipping in the upper-right corner.

    2. In the dialog box that appears, click OK.

    Manage an SLS Project

    On the Logs page, you can click Go to Log Service Console in the upper-right corner to open the Simple Log Service (SLS) console. In the SLS console, you can set log retention periods and modify Logstore configurations.

    Log fields

    Parameter

    Description

    __source__

    The private IP address of the node.

    __tag__:__hostname__

    The hostname of the node.

    __tag__:__path__

    The log storage path.

    __tag__:__user_defined_id__

    The custom identifier that SLS defines for the cluster.

    __tag__:_aliyun_user_id_

    The UID of the Alibaba Cloud account.

    __tag__:_ecs_id_

    The ID of the ECS instance.

    __tag__:_main_cluster_id_

    The EMR cluster ID.

    __tag__:_role_

    The name of the node. For example, core-1-1.

    __topic__

    The log topic, used to categorize log entries.

    content

    The raw content of a log entry.

    level

    The log level, which indicates the severity of the log entry, such as INFO, WARN, ERROR, or DEBUG.

    message

    The main log message, typically extracted after parsing.

    thread

    The thread that generated the log entry.

    time

    The timestamp when the log entry was generated.

    Assets

    • Custom Projects and dedicated Logstores

      You can specify a custom Project. The system then creates a dedicated Logstore named emr_<service_name>_log in that Project. For example, if you ship host service logs, the corresponding Logstore is emr_host_log.

      Important

      • Before disabling the Log Management feature, do not delete the SLS Projects and Logstores used for EMR logs. Otherwise, existing logs will be deleted and new logs cannot be shipped to SLS.

      • If you previously used the pay-by-ingested-data billing method, the system creates a dedicated Logstore that uses this method by default. To switch to the pay-by-feature billing method, you must modify the Logstore configuration. For more information, see Modify Logstore configurations.

    • Dedicated dashboards

      No dedicated dashboards are provided. You can create custom dashboards to display query results. For more information, see Create a dashboard.

    Billing

    • The Log Management feature is currently free of charge in EMR.

    • If a Logstore uses the pay-by-feature billing method, SLS charges for shipped EMR logs based on items such as storage space, read traffic, requests, data processing, and data shipping. For more information, see Billable items of the pay-by-feature billing method.

    • If a Logstore uses the pay-by-ingested-data billing method, SLS charges for shipped EMR logs based on the volume of raw ingested data. For more information, see Billable items of the pay-by-ingested-data billing method.

    FAQ

    Q: I have activated SLS and granted the required permissions. Why does the console still display the message "Service log collection is not enabled for this resource. Go to the Log Management page of the cluster in the EMR console to enable it."?

    A: No services are configured for log shipping in the current cluster. Click Set Log Shipping Scope in the upper-right corner to complete the configuration.