Use the Jindo Job Committer (SmartData 2.6.0–2.7.2) - E-MapReduce

Enable the Jindo Job Committer in MapReduce and Spark jobs to write data to OSS with higher performance by eliminating rename operations.

Background information

A job committer is a fundamental component of distributed computing frameworks such as MapReduce and Spark. It ensures data consistency when a job writes data.

Jindo Job Committer is a high-performance job committer developed by Alibaba Cloud E-MapReduce to write data to Object Storage Service (OSS). It uses the OSS multipart upload API and customized file system features. Tasks write data directly to the final destination directory, which eliminates rename operations and keeps intermediate data hidden until the job is committed to ensure data consistency.

Important

OSS copy performance varies depending on factors such as bandwidth and feature configuration. For specific issues, contact OSS technical support.
After all tasks are complete, the MapReduce Application Master or Spark Driver performs the final job commit. This phase includes a brief time window whose duration is proportional to the number of files. You can increase the value of fs.oss.committer.threads to improve concurrent processing speed.
Frameworks such as Hive and Presto do not use the Hadoop job committer.
JindoOssCommitter is enabled by default in E-MapReduce clusters.

Use Jindo Job Committer in MapReduce

Navigate to the mapred-site tab in the YARN service.
1. Log on to the EMR console.
2. In the top navigation bar, select a region and a resource group based on your requirements.
3. Click the Cluster Management tab.
4. On the Cluster Management page, find your cluster and click Details in the Actions column.
5. In the left-side navigation pane, choose Services > YARN.
6. Click the Configure tab.
7. In the Service Configuration section, click the mapred-site tab.
Configure the following parameters in the YARN service based on your Hadoop version.
- For Hadoop 2.x versions
  
  On the mapred-site tab of the YARN service, set mapreduce.outputcommitter.class to com.aliyun.emr.fs.oss.commit.JindoOssCommitter.
- For Hadoop 3.x versions
  
  On the mapred-site tab of the YARN service, set mapreduce.outputcommitter.factory.scheme.oss to com.aliyun.emr.fs.oss.commit.JindoOssCommitterFactory.
Save the configuration.
1. In the upper-right corner, click Save.
2. In the Confirm dialog box, enter a reason for the modification and enable Auto-update Configuration.
3. Click OK.
Navigate to the smartdata-site tab of the SmartData service.
1. In the left-side navigation pane, choose Services > SmartData.
2. Click the Configure tab.
3. In the Service Configuration section, click the smartdata-site tab.
On the smartdata-site tab of the SmartData service, set fs.oss.committer.magic.enabled to true.
Save the configuration.
1. In the upper-right corner, click Save.
2. In the Confirm dialog box, enter a reason for the modification and enable Auto-update Configuration.
3. Click OK.

Note

After you set mapreduce.outputcommitter.class to com.aliyun.emr.fs.oss.commit.JindoOssCommitter, you can use the fs.oss.committer.magic.enabled switch to control which job committer is used. If this switch is enabled, MapReduce jobs use the Jindo Oss Magic Committer, which eliminates the need for rename operations. If this switch is disabled, JindoOssCommitter behaves like FileOutputCommitter.

Use Jindo Job Committer in Spark

Navigate to the spark-defaults tab of the Spark service.
1. In the left-side navigation pane, choose Services > Spark.
2. Click the Configure tab.
3. In the Service Configuration section, click the spark-defaults tab.

On the spark-defaults tab of the Spark service, set the following parameters.

Parameter	Value
spark.sql.sources.outputCommitterClass	com.aliyun.emr.fs.oss.commit.JindoOssCommitter
spark.sql.parquet.output.committer.class	com.aliyun.emr.fs.oss.commit.JindoOssCommitter
spark.sql.hive.outputCommitterClass	com.aliyun.emr.fs.oss.commit.JindoOssCommitter

These parameters set the job committer for Spark DataSource tables, Parquet-formatted DataSource tables, and Hive tables.

Save the configuration.
1. In the upper-right corner, click Save.
2. In the Confirm dialog box, enter a reason for the modification and enable Auto-update Configuration.
3. Click OK.
Navigate to the smartdata-site tab of the SmartData service.
1. In the left-side navigation pane, choose Services > SmartData.
2. Click the Configure tab.
3. In the Service Configuration section, click the smartdata-site tab.
On the smartdata-site tab of the SmartData service, set fs.oss.committer.magic.enabled to true.

Note
You can use the fs.oss.committer.magic.enabled switch to control which job committer is used. If this switch is enabled, Spark jobs use the Jindo Oss Magic Committer, which eliminates the need for rename operations. If this switch is disabled, JindoOssCommitter behaves like FileOutputCommitter.
Save the configuration.
1. In the upper-right corner, click Save.
2. In the Confirm dialog box, enter a reason for the modification and enable Auto-update Configuration.
3. Click OK.

Optimize Jindo Job Committer performance

To improve commit performance for MapReduce or Spark jobs that write many files, adjust the number of concurrent commit threads in the MapReduce Application Master or Spark Driver.

Navigate to the smartdata-site tab of the SmartData service.
1. In the left-side navigation pane, choose Services > SmartData.
2. Click the Configure tab.
3. In the Service Configuration section, click the smartdata-site tab.
On the smartdata-site tab of the SmartData service, set fs.oss.committer.threads to 8.

The default value is 8.
Save the configuration.
1. In the upper-right corner, click Save.
2. In the Confirm dialog box, enter a reason for the modification and enable Auto-update Configuration.
3. Click OK.