This topic summarizes common issues with Hive.
- How to resolve a job that is stuck in a waiting state
- How to check if the map stage is reading small files
- How to determine if a long-running reduce task is caused by data skew
Job stuck in a waiting state
Follow these steps to identify the problem:
- In the E-MapReduce console, go to the Access Links and Ports page and click the YARN UI link.
- Click the application ID.
- Click the Tracking URL.This page displays multiple jobs in a waiting state.
- In the left navigation pane, click Scheduler.In the Application Queues section, check the Capacity and Used percentages for each queue to identify resource constraints. If a queue is constrained, consider moving the job to an idle queue. Otherwise, you may need to optimize your code.
Small files in the map stage
Follow these steps to identify the problem:
- In the E-MapReduce console, go to the Access Links and Ports page and click the YARN UI link.
- Click the application ID.This opens the details page for the map tasks, which shows the amount of data read by each map task. For example, a task might read a record of only 2 bytes. If most map tasks are processing very small amounts of data, you should consider small file merging.
You can also check the logs for each map task to get more information.
Data skew in reduce tasks
Follow these steps to identify the problem:
- In the E-MapReduce console, go to the Access Links and Ports page and click the YARN UI link.
- Click the application ID.
- On the Reduce tasks page, sort the tasks by completion time in descending order to find the longest-running ones. Check the State column to confirm the task status. Then, check the Successful Attempt, Shuffle Finish Time, Merge Finish Time, and Finish Time columns to analyze the time spent in each phase and determine if data skew is occurring.
- Click the name of a long-running task.
- On the task details page, click Counters in the left navigation pane.
Find the Map-Reduce Framework section. Check the values of Reduce input records and Reduce shuffle bytes. If these values are significantly larger for one task compared to others, it indicates data skew.