All Products
Search

This Product

    E-MapReduce:Jindo Job Committer user guide

    Document Center

    E-MapReduce:Jindo Job Committer user guide

    Last Updated:Mar 26, 2026

    Jindo Job Committer is a high-performance job committer developed by the Alibaba Cloud E-MapReduce (EMR) team for writing job output to Object Storage Service (OSS). It uses the OSS multipart upload feature and the OSS file system customization feature to write task output directly to the destination directory — no rename operations required. Intermediate data stays invisible to other readers until the job commits, ensuring data consistency.

    EMR clusters enable Jindo OSS Committer by default.

    Hive and Presto jobs do not use the Hadoop job committer and are unaffected by these settings.

    Which committer to use

    EMR provides three committers. Choose based on whether bucket versioning is enabled on your OSS bucket.

    Committer When to use Bucket versioning required
    Jindo OSS Committer (default) General MapReduce and Spark jobs writing to OSS No
    Jindo OSS Magic Committer High-throughput jobs writing many files; eliminates rename operations entirely No
    Jindo OSS Direct Committer Jobs writing to a versioning-enabled bucket; writes output directly with no temporary files No (recommended when bucket versioning is enabled)

    The fs.oss.committer.magic.enabled parameter on the SmartData service controls the switch between Jindo OSS Committer and Jindo OSS Magic Committer:

    • true — jobs use Jindo OSS Magic Committer (no rename operations)

    • false — jobs use Jindo OSS Committer (same behavior as File Output Committer)

    Usage notes

    After all tasks complete, MapReduce Application Master or Spark Driver commits the job. During this commit, there is a brief window where only some result files appear in the destination directory. The length of this window scales with the number of output files.

    To reduce the commit window, increase fs.oss.committer.threads (see Optimize committer performance).

    OSS bandwidth and certain advanced OSS features may affect data copy performance. Performance varies per user and per bucket. Contact OSS technical support if you encounter unexpected performance.

    Configure Jindo OSS Committer for MapReduce jobs

    Before you begin, ensure that you have:

    Configure the committer

    1. In the EMR console, click the Cluster Management tab.

    2. Find your cluster and click Details in the Actions column.

    3. In the left-side navigation pane, choose Cluster Service > YARN.

    4. Click the Configure tab. In the Service Configuration section, click the mapred-site tab.

    5. Set the job committer parameter for your Hadoop version:

      Hadoop version Parameter Value
      Hadoop 2.X mapreduce.outputcommitter.class com.aliyun.emr.fs.oss.commit.JindoOssCommitter
      Hadoop 3.X mapreduce.outputcommitter.factory.scheme.oss com.aliyun.emr.fs.oss.commit.JindoOssCommitterFactory

    6. Save the configuration: click Save in the upper-right corner of the Service Configuration section. In the Confirm Changes dialog box, enter a description, turn on Auto-update Configuration, and click OK.

    7. In the left-side navigation pane, choose Cluster Service > SmartData.

    8. Click the Configure tab. In the Service Configuration section, click the smartdata-site tab.

    9. Set fs.oss.committer.magic.enabled to true to use Jindo OSS Magic Committer (no rename operations). Set it to false to use the standard Jindo OSS Committer.

    10. Save the configuration: click Save, complete the Confirm Changes dialog box, and click OK.

    Configure Jindo OSS Committer for Spark jobs

    Before you begin, ensure that you have:

    Configure the committer

    1. In the EMR console, click the Cluster Management tab.

    2. Find your cluster and click Details in the Actions column.

    3. In the left-side navigation pane, choose Cluster Service > Spark.

    4. Click the Configure tab. In the Service Configuration section, click the spark-defaults tab.

    5. Set the following parameters to com.aliyun.emr.fs.oss.commit.JindoOssCommitter:

      Parameter Description
      spark.sql.sources.outputCommitterClass Committer for writing to data source tables
      spark.sql.parquet.output.committer.class Committer for writing Parquet-format data source tables
      spark.sql.hive.outputCommitterClass Committer for writing to Hive tables

    6. Save the configuration: click Save in the upper-right corner of the Service Configuration section. In the Confirm Changes dialog box, enter a description, turn on Auto-update Configuration, and click OK.

    7. In the left-side navigation pane, choose Cluster Service > SmartData.

    8. Click the Configure tab. In the Service Configuration section, click the smartdata-site tab.

    9. Set fs.oss.committer.magic.enabled to true to use Jindo OSS Magic Committer (no rename operations). Set it to false to use the standard Jindo OSS Committer.

    10. Save the configuration: click Save, complete the Confirm Changes dialog box, and click OK.

    Optimize committer performance

    If your MapReduce or Spark jobs write a large number of files, increase the number of concurrent commit threads to speed up the commit phase.

    1. In the left-side navigation pane, choose Cluster Service > SmartData.

    2. Click the Configure tab. In the Service Configuration section, click the smartdata-site tab.

    3. Set fs.oss.committer.threads to a value higher than the default.

      Parameter Description Default
      fs.oss.committer.threads Number of threads for concurrent commit operations 8

    4. Save the configuration: click Save, complete the Confirm Changes dialog box, and click OK.

    Manage temporary directories when bucket versioning is enabled

    When bucket versioning is enabled on OSS, Magic Committer creates and deletes temporary directories during the commit. In OSS, deleting a versioned object creates a delete marker rather than removing the object. Over time, accumulated delete markers degrade the performance of list operations on the directory.

    Starting with SmartData 3.4.0, Jindo OSS Magic Committer automatically cleans up historical versions of temporary directories and files after a commit, preventing delete markers from accumulating and keeping list operation performance stable. This cleanup is enabled by default.

    To disable automatic cleanup, set fs.jfs.cache.oss.delete-marker.dirs in the smartdata-site configuration or in your job code:

    Value Effect
    (blank) Disables cleanup entirely
    temporary,.staging,.hive-staging,__magic Disables auto-clearing by specifying directories explicitly (equivalent to the default active state)

    Use Jindo OSS Direct Committer

    Jindo OSS Direct Committer (available in SmartData 3.4.X and later) writes output directly to the destination directory without generating any temporary files. Use it when bucket versioning is enabled and you want to avoid the overhead of temporary file management.

    To switch to Jindo OSS Direct Committer, replace the committer class in your configuration with com.aliyun.emr.fs.oss.commit.direct.JindoOssDirectCommitter.

    For MapReduce jobs (mapred-site tab of the YARN service):

    Parameter Value
    mapreduce.outputcommitter.class com.aliyun.emr.fs.oss.commit.direct.JindoOssDirectCommitter

    For Spark jobs (spark-defaults tab of the Spark service):

    Parameter Value
    mapreduce.outputcommitter.class com.aliyun.emr.fs.oss.commit.direct.JindoOssDirectCommitter
    spark.sql.parquet.output.committer.class com.aliyun.emr.fs.oss.commit.direct.JindoOssDirectCommitter
    spark.sql.hive.outputCommitterClass com.aliyun.emr.fs.oss.commit.direct.JindoOssDirectCommitter
    spark.sql.sources.outputCommitterClass com.aliyun.emr.fs.oss.commit.direct.JindoOssDirectCommitter

    Follow the same save procedure described in the configuration sections above to apply the changes.