E-MapReduce (EMR) clusters generate logs from open source components like HDFS, Spark, and Hive. Log Management integrates EMR with Simple Log Service (SLS) so you can query these logs directly in the EMR console—without switching tools or manually configuring log collection.
Prerequisites
Before you begin, ensure that you have:
A cluster created in the EMR console. For more information, see Create a cluster.
Simple Log Service activated.
Limitations
Log Management is available only for DataLake, DataFlow, Online Analytical Processing (OLAP), DataServing, and Hadoop clusters.
Log shipping is supported for the following services: Hadoop Distributed File System (HDFS), YARN, YARN application, Hive, Spark, JindoData, Tez, Flink, HBase, ZooKeeper, Kafka, Presto, Kudu, Impala, Flume, StarRocks, ClickHouse, Kyuubi, RSS, and host.
Usage notes
SLS charges for log storage and indexed traffic at standard rates. For more information, see Billable items.
If you use a Resource Access Management (RAM) user, grant the AliyunLogFullAccess permission to the RAM user from the RAM console using your Alibaba Cloud account. For more information, see Grant permissions to a RAM user.
Enable log shipping
Go to the Logs page.
Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
In the top menu bar, select a region and resource group.
On the EMR on ECS page, click the Cluster ID of the target cluster.
In the left navigation pane, click the Logs tab.
Configure the log shipping scope.
On the Logs page, click Set Log Shipping Scope in the upper-right corner.
In the dialog box, configure the project for log storage.
Select an existing project or create a new one.
ImportantA project cannot be changed after it is selected.
Select the services for which you want to ship logs, then click OK.
View service logs.
From the Please select EMR service drop-down list, select the service whose logs you want to view.
On the log analysis page, analyze the logs. Specify a time range and a search statement to run real-time queries. For example, analyze the distribution of a field over a period of time, or filter and count access records based on specific conditions.
Disable log shipping
Disabling real-time log shipping does not automatically delete the project. To avoid ongoing SLS charges, go to the Simple Log Service console and delete the project you configured for log shipping. For more information, see Manage a project.
On the Logs page, click Disable Log Shipping in the upper-right corner.
In the dialog box, click OK.
Manage SLS projects
On the Logs page, click Go To Log Service Console to open the SLS console. In the SLS console, set the log retention period and modify Logstore configurations.
Log field details
| Field name | Description |
|---|---|
__source__ | The internal IP address of the node. |
__tag__:__hostname__ | The hostname of the node. |
__tag__:__path__ | The log storage path. |
__tag__:__user_defined_id__ | The custom identifier that SLS defines for the cluster. |
__tag__:_aliyun_user_id_ | The UID of the Alibaba Cloud account. |
__tag__:_ecs_id_ | The ID of the ECS instance. |
__tag__:_main_cluster_id_ | The ID of the EMR cluster. |
__tag__:_role_ | The node name. Example: core-1-1. |
__topic__ | The log topic. Used to classify log content. |
content | The raw log content, which is usually a complete log record. |
level | The log level: INFO, WARN, ERROR, or DEBUG. |
message | The log message, which is the main information part after parsing. |
thread | The thread that generated the log. |
time | The time when the log was generated. |
Asset details
Custom projects and dedicated Logstores
Specify a custom project. The system then creates a dedicated Logstore named emr_<EMR_service_name>_log in that project. For example, the Logstore for the host service is emr_host_log.
Do not delete the SLS projects and Logstores associated with EMR logs before disabling Log Management. Doing so purges stored logs and prevents new logs from being delivered to SLS.
If you previously enabled the pay-by-ingested-data billing method, the system creates a dedicated Logstore that uses this billing method by default. To switch to the pay-by-feature billing method, modify the Logstore configuration. For more information, see Manage Logstore configurations.
Dedicated dashboards
No dedicated dashboards are provided. Create custom dashboards to display query results. For more information, see Create a dashboard.
Billing
EMR does not charge for Log Management.
When a Logstore uses the pay-by-feature billing method, SLS charges for storage space, read traffic, requests, data transformation, and data shipping after logs are delivered. For more information, see Billable items of the pay-by-feature billing method.
When a Logstore uses the pay-by-ingested-data billing method, SLS charges for the raw ingested data volume and other items after logs are delivered. For more information, see Billable items of the pay-by-ingested-data billing method.
FAQ
Q: SLS is activated and the required permissions are granted. Why do I still see "Service log collection is not enabled for this resource. Go to the Log Management page of the cluster in the EMR console to enable it."?
The services for log shipping haven't been configured for this cluster yet. Click Set Log Shipping Scope to complete the configuration.