Open source components that run in an E-MapReduce (EMR) cluster generate a large volume of logs. The log management feature integrates EMR with Simple Log Service (SLS), which lets you query the logs of open source components directly in the EMR console.
Prerequisites
A cluster has been created in the EMR console. For more information, see Create a cluster.
Limitations
The operations described in this topic apply only to DataLake, DataFlow, Online Analytical Processing (OLAP), DataServing, and Hadoop clusters.
Log shipping is supported for the following services: HDFS, YARN, YARN application, Hive, Spark, JindoData, Tez, Flink, HBase, ZooKeeper, Kafka, Presto, Kudu, Impala, Flume, StarRocks, ClickHouse, Kyuubi, RSS, and host.
Precautions
Simple Log Service (SLS) charges you for log storage and indexed traffic at standard rates. For more information, see Billable items.
If you use a Resource Access Management (RAM) user, log on to the RAM console with your Alibaba Cloud account and grant the AliyunLogFullAccess permission to the RAM user. For more information, see Grant permissions to a RAM user.
Enable log shipping
Go to the Log Management page.
Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
In the top menu bar, select a region and resource group as needed.
On the EMR on ECS page, click the Cluster ID of the target cluster.
In the navigation pane on the left, click the Logs tab.
Configure the log shipping scope.
On the Logs page, click Set Log Shipping Scope in the upper-right corner.
In the dialog box that appears, configure the project for log storage.
You can select an existing project or create a new one.
ImportantA project cannot be changed after it is selected.
Select the services for which you want to ship logs, and then click OK.
View service logs.
From the Please select EMR service drop-down list, select the service whose logs you want to view.
On the log analysis page, analyze the logs.
You can specify a time range and a search statement to run real-time queries. For example, you can quickly analyze the distribution of a field over a period of time. You can also filter or count access records based on specific conditions.
Disable log shipping
Disabling real-time log shipping does not automatically delete the project. To prevent future charges, go to the Simple Log Service console and delete the project that you configured for log shipping. For more information, see Manage a project.
If you no longer need the log data, follow these steps to disable log shipping.
On the Logs page, click Disable Log Shipping in the upper-right corner.
In the dialog box that appears, click OK.
Manage SLS projects
On the Logs page, click Go To Log Service Console to open the Simple Log Service console. In the SLS console, you can set the log retention period and modify Logstore configurations.
Log field details
Field name | Description |
__source__ | The internal IP address of the node. |
__tag__:__hostname__ | The hostname of the node. |
__tag__:__path__ | The log storage path. |
__tag__:__user_defined_id__ | The custom identifier that SLS defines for the cluster. |
__tag__:_aliyun_user_id_ | The UID of the Alibaba Cloud account. |
__tag__:_ecs_id_ | The ID of the ECS instance. |
__tag__:_main_cluster_id_ | The ID of the EMR cluster. |
__tag__:_role_ | The node name. Example: core-1-1. |
__topic__ | The log topic. Used to classify log content. |
content | The raw log content, which is usually a complete log record. |
level | The log level. Indicates the severity of the log, such as |
message | The log message, which is usually the main information part after parsing. |
thread | The thread that generated the log. |
time | The time when the log was generated. |
Asset details
Custom projects and dedicated Logstores
You can specify a custom project. The system then creates a dedicated Logstore named
emr_<EMR_service_name>_login the project. For example, if you want to ship logs for the host service, the corresponding Logstore isemr_host_log.ImportantDo not delete the Simple Log Service projects and Logstores that are related to EMR logs before you disable the log management feature. Otherwise, stored logs are purged and new logs cannot be delivered to Simple Log Service.
If you previously enabled the pay-by-ingested-data billing method, the system creates a dedicated Logstore that uses this billing method by default. To switch to the pay-by-feature billing method, modify the Logstore configuration. For more information, see Manage Logstore configurations.
Dedicated dashboards
No dedicated dashboards are provided. You can create custom dashboards to display query results. For more information, see Create a dashboard.
Billing details
Currently, EMR does not charge for the log management feature.
When a Logstore uses the pay-by-feature billing method, Simple Log Service charges you for storage space, read traffic, requests, data transformation, and data shipping after EMR logs are delivered to Simple Log Service. For more information, see Billable items of the pay-by-feature billing method.
When a Logstore uses the pay-by-ingested-data billing method, Simple Log Service charges you for the raw ingested data volume and other items after EMR logs are delivered to Simple Log Service. For more information, see Billable items of the pay-by-ingested-data billing method.
FAQ
Q: Simple Log Service is activated and the required permissions are granted. Why do I still see the message "Service log collection is not enabled for this resource. Go to the Log Management page of the cluster in the EMR console to enable it."?
A: This message appears because the services for which to ship logs are not configured for the current cluster. Click Set Log Shipping Scope to complete the configuration.