Hadoop-compatible MapReduce SDK
MaxCompute MapReduce implements the Hadoop MapReduce APIs but does not fully support every interface. Before you migrate existing Hadoop MapReduce jobs to MaxCompute, review the table below to understand how each interface behaves.
This table details the compatibility between MaxCompute MapReduce and Hadoop MapReduce.
Type | API | Compatible |
Mapper | void map(KEYIN key, VALUEIN value, org.apache.hadoop.mapreduce.Mapper.Context context) | Yes |
Mapper | void run(org.apache.hadoop.mapreduce.Mapper.Context context) | Yes |
Mapper | void setup(org.apache.hadoop.mapreduce.Mapper.Context context) | Yes |
Reducer | void cleanup(org.apache.hadoop.mapreduce.Reducer.Context context) | Yes |
Reducer | void reduce(KEYIN key, VALUEIN value, org.apache.hadoop.mapreduce.Reducer.Context context) | Yes |
Reducer | void run(org.apache.hadoop.mapreduce.Reducer.Context context) | Yes |
Reducer | void setup(org.apache.hadoop.mapreduce.Reducer.Context context) | Yes |
Partitioner | int getPartition(KEY key, VALUE value, int numPartitions) | Yes |
MapContext (extends TaskInputOutputContext) | InputSplit getInputSplit() | No (throws an exception) |
ReduceContext | nextKey() | Yes |
ReduceContext | getValues() | Yes |
TaskInputOutputContext | getCurrentKey() | Yes |
TaskInputOutputContext | getCurrentValue() | Yes |
TaskInputOutputContext | getOutputCommitter() | No (throws an exception) |
TaskInputOutputContext | nextKeyValue() | Yes |
TaskInputOutputContext | write(KEYOUT key, VALUEOUT value) | Yes |
TaskAttemptContext | getCounter(Enum<?> counterName) | Yes |
TaskAttemptContext | getCounter(String groupName, String counterName) | Yes |
TaskAttemptContext | setStatus(String msg) | Empty implementation |
TaskAttemptContext | getStatus() | Empty implementation |
TaskAttemptContext | getTaskAttemptID() | No (throws an exception) |
TaskAttemptContext | getProgress() | No (throws an exception) |
TaskAttemptContext | progress() | Yes |
Job | addArchiveToClassPath(Path archive) | No |
Job | addCacheArchive(URI uri) | No |
Job | addCacheFile(URI uri) | No |
Job | addFileToClassPath(Path file) | No |
Job | cleanupProgress() | No |
Job | createSymlink() | No (throws an exception) |
Job | failTask(TaskAttemptID taskId) | No |
Job | getCompletionPollInterval(Configuration conf) | Empty implementation |
Job | getCounters() | Yes |
Job | getFinishTime() | Yes |
Job | getHistoryUrl() | Yes |
Job | getInstance() | Yes |
Job | getInstance(Cluster ignored) | Yes |
Job | getInstance(Cluster ignored, Configuration conf) | Yes |
Job | getInstance(Configuration conf) | Yes |
Job | getInstance(Configuration conf, String jobName) | Empty implementation |
Job | getInstance(JobStatus status, Configuration conf) | No (throws an exception) |
Job | getJobFile() | No (throws an exception) |
Job | getJobName() | Empty implementation |
Job | getJobState() | No (throws an exception) |
Job | getPriority() | No (throws an exception) |
Job | getProgressPollInterval(Configuration conf) | Empty implementation |
Job | getReservationId() | No (throws an exception) |
Job | getSchedulingInfo() | No (throws an exception) |
Job | getStartTime() | Yes |
Job | getStatus() | No (throws an exception) |
Job | getTaskCompletionEvents(int startFrom) | No (throws an exception) |
Job | getTaskCompletionEvents(int startFrom, int numEvents) | No (throws an exception) |
Job | getTaskDiagnostics(TaskAttemptID taskid) | No (throws an exception) |
Job | getTaskOutputFilter(Configuration conf) | No (throws an exception) |
Job | getTaskReports(TaskType type) | No (throws an exception) |
Job | getTrackingURL() | Yes |
Job | isComplete() | Yes |
Job | isRetired() | No (throws an exception) |
Job | isSuccessful() | Yes |
Job | isUber() | Empty implementation |
Job | killJob() | Yes |
Job | killTask(TaskAttemptID taskId) | No |
Job | mapProgress() | Yes |
Job | monitorAndPrintJob() | Yes |
Job | reduceProgress() | Yes |
Job | setCacheArchives(URI[] archives) | No (throws an exception) |
Job | setCacheFiles(URI[] files) | No (throws an exception) |
Job | setCancelDelegationTokenUponJobCompletion(boolean value) | No (throws an exception) |
Job | setCombinerClass(Class<? extends Reducer> cls) | Yes |
Job | setCombinerKeyGroupingComparatorClass(Class<? extends RawComparator> cls) | Yes |
Job | setGroupingComparatorClass(Class<? extends RawComparator> cls) | Yes |
Job | setInputFormatClass(Class<? extends InputFormat> cls) | Empty implementation |
Job | setJar(String jar) | Yes |
Job | setJarByClass(Class<?> cls) | Yes |
Job | setJobName(String name) | Empty implementation |
Job | setJobSetupCleanupNeeded(boolean needed) | Empty implementation |
Job | setMapOutputKeyClass(Class<?> theClass) | Yes |
Job | setMapOutputValueClass(Class<?> theClass) | Yes |
Job | setMapperClass(Class<? extends Mapper> cls) | Yes |
Job | setMapSpeculativeExecution(boolean speculativeExecution) | Empty implementation |
Job | setMaxMapAttempts(int n) | Empty implementation |
Job | setMaxReduceAttempts(int n) | Empty implementation |
Job | setNumReduceTasks(int tasks) | Yes |
Job | setOutputFormatClass(Class<? extends OutputFormat> cls) | No (throws an exception) |
Job | setOutputKeyClass(Class<?> theClass) | Yes |
Job | setOutputValueClass(Class<?> theClass) | Yes |
Job | setPartitionerClass(Class<? extends Partitioner> cls) | Yes |
Job | setPriority(JobPriority priority) | No (throws an exception) |
Job | setProfileEnabled(boolean newValue) | Empty implementation |
Job | setProfileParams(String value) | Empty implementation |
Job | setProfileTaskRange(boolean isMap, String newValue) | Empty implementation |
Job | setReducerClass(Class<? extends Reducer> cls) | Yes |
Job | setReduceSpeculativeExecution(boolean speculativeExecution) | Empty implementation |
Job | setReservationId(ReservationId reservationId) | No (throws an exception) |
Job | setSortComparatorClass(Class<? extends RawComparator> cls) | No (throws an exception) |
Job | setSpeculativeExecution(boolean speculativeExecution) | Yes |
Job | setTaskOutputFilter(Configuration conf, org.apache.hadoop.mapreduce.Job.TaskStatusFilter newValue) | No (throws an exception) |
Job | setupProgress() | No (throws an exception) |
Job | setUser(String user) | Empty implementation |
Job | setWorkingDirectory(Path dir) | Empty implementation |
Job | submit() | Yes |
Job | toString() | No (throws an exception) |
Job | waitForCompletion(boolean verbose) | Yes |
task execution & environment | mapreduce.map.java.opts | Empty implementation |
task execution & environment | mapreduce.reduce.java.opts | Empty implementation |
task execution & environment | mapreduce.map.memory.mb | Empty implementation |
task execution & environment | mapreduce.reduce.memory.mb | Empty implementation |
task execution & environment | mapreduce.task.io.sort.mb | Empty implementation |
task execution & environment | mapreduce.map.sort.spill.percent | Empty implementation |
task execution & environment | mapreduce.task.io.soft.factor | Empty implementation |
task execution & environment | mapreduce.reduce.merge.inmem.thresholds | Empty implementation |
task execution & environment | mapreduce.reduce.shuffle.merge.percent | Empty implementation |
task execution & environment | mapreduce.reduce.shuffle.input.buffer.percent | Empty implementation |
task execution & environment | mapreduce.reduce.input.buffer.percent | Empty implementation |
task execution & environment | mapreduce.job.id | Empty implementation |
task execution & environment | mapreduce.job.jar | Empty implementation |
task execution & environment | mapreduce.job.local.dir | Empty implementation |
task execution & environment | mapreduce.task.id | Empty implementation |
task execution & environment | mapreduce.task.attempt.id | Empty implementation |
task execution & environment | mapreduce.task.is.map | Empty implementation |
task execution & environment | mapreduce.task.partition | Empty implementation |
task execution & environment | mapreduce.map.input.file | Empty implementation |
task execution & environment | mapreduce.map.input.start | Empty implementation |
task execution & environment | mapreduce.map.input.length | Empty implementation |
task execution & environment | mapreduce.task.output.dir | Empty implementation |
JobClient | cancelDelegationToken(Token<DelegationTokenIdentifier> token) | No (throws an exception) |
JobClient | close() | Empty implementation |
JobClient | displayTasks(JobID jobId, String type, String state) | No (throws an exception) |
JobClient | getAllJobs() | No (throws an exception) |
JobClient | getCleanupTaskReports(JobID jobId) | No (throws an exception) |
JobClient | getClusterStatus() | No (throws an exception) |
JobClient | getClusterStatus(boolean detailed) | No (throws an exception) |
JobClient | getDefaultMaps() | No (throws an exception) |
JobClient | getDefaultReduces() | No (throws an exception) |
JobClient | getDelegationToken(Text renewer) | No (throws an exception) |
JobClient | getFs() | No (throws an exception) |
JobClient | getJob(JobID jobid) | No (throws an exception) |
JobClient | getJob(String jobid) | No (throws an exception) |
JobClient | getJobsFromQueue(String queueName) | No (throws an exception) |
JobClient | getMapTaskReports(JobID jobId) | No (throws an exception) |
JobClient | getMapTaskReports(String jobId) | No (throws an exception) |
JobClient | getQueueAclsForCurrentUser() | No (throws an exception) |
JobClient | getQueueInfo(String queueName) | No (throws an exception) |
JobClient | getQueues() | No (throws an exception) |
JobClient | getReduceTaskReports(JobID jobId) | No (throws an exception) |
JobClient | getReduceTaskReports(String jobId) | No (throws an exception) |
JobClient | getSetupTaskReports(JobID jobId) | No (throws an exception) |
JobClient | getStagingAreaDir() | No (throws an exception) |
JobClient | getSystemDir() | No (throws an exception) |
JobClient | getTaskOutputFilter() | No (throws an exception) |
JobClient | getTaskOutputFilter(JobConf job) | No (throws an exception) |
JobClient | init(JobConf conf) | No (throws an exception) |
JobClient | isJobDirValid(Path jobDirPath, FileSystem fs) | No (throws an exception) |
JobClient | jobsToComplete() | No (throws an exception) |
JobClient | monitorAndPrintJob(JobConf conf, RunningJob job) | No (throws an exception) |
JobClient | renewDelegationToken(Token<DelegationTokenIdentifier> token) | No (throws an exception) |
JobClient | run(String[] argv) | No (throws an exception) |
JobClient | runJob(JobConf job) | Yes |
JobClient | setTaskOutputFilter(JobClient.TaskStatusFilter newValue) | No (throws an exception) |
JobClient | setTaskOutputFilter(JobConf job, JobClient.TaskStatusFilter newValue) | No (throws an exception) |
JobClient | submitJob(JobConf job) | Yes |
JobClient | submitJob(String jobFile) | No (throws an exception) |
JobConf | deleteLocalFiles() | No (throws an exception) |
JobConf | deleteLocalFiles(String subdir) | No (throws an exception) |
JobConf | normalizeMemoryConfigValue(long val) | Empty implementation |
JobConf | setCombinerClass(Class<? extends Reducer> theClass) | Yes |
JobConf | setCompressMapOutput(boolean compress) | Empty implementation |
JobConf | setInputFormat(Class<? extends InputFormat> theClass) | No (throws an exception) |
JobConf | setJar(String jar) | No (throws an exception) |
JobConf | setJarByClass(Class cls) | No (throws an exception) |
JobConf | setJobEndNotificationURI(String uri) | No (throws an exception) |
JobConf | setJobName(String name) | Empty implementation |
JobConf | setJobPriority(JobPriority prio) | No (throws an exception) |
JobConf | setKeepFailedTaskFiles(boolean keep) | No (throws an exception) |
JobConf | setKeepTaskFilesPattern(String pattern) | No (throws an exception) |
JobConf | setKeyFieldComparatorOptions(String keySpec) | No (throws an exception) |
JobConf | setKeyFieldPartitionerOptions(String keySpec) | No (throws an exception) |
JobConf | setMapDebugScript(String mDbgScript) | Empty implementation |
JobConf | setMapOutputCompressorClass(Class<? extends CompressionCodec> codecClass) | Empty implementation |
JobConf | setMapOutputKeyClass(Class<?> theClass) | Yes |
JobConf | setMapOutputValueClass(Class<?> theClass) | Yes |
JobConf | setMapperClass(Class<? extends Mapper> theClass) | Yes |
JobConf | setMapRunnerClass(Class<? extends MapRunnable> theClass) | No (throws an exception) |
JobConf | setMapSpeculativeExecution(boolean speculativeExecution) | Empty implementation |
JobConf | setMaxMapAttempts(int n) | Empty implementation |
JobConf | setMaxMapTaskFailuresPercent(int percent) | Empty implementation |
JobConf | setMaxPhysicalMemoryForTask(long mem) | Empty implementation |
JobConf | setMaxReduceAttempts(int n) | Empty implementation |
JobConf | setMaxReduceTaskFailuresPercent(int percent) | Empty implementation |
JobConf | setMaxTaskFailuresPerTracker(int noFailures) | Empty implementation |
JobConf | setMaxVirtualMemoryForTask(long vmem) | Empty implementation |
JobConf | setMemoryForMapTask(long mem) | Yes |
JobConf | setMemoryForReduceTask(long mem) | Yes |
JobConf | setNumMapTasks(int n) | Yes |
JobConf | setNumReduceTasks(int n) | Yes |
JobConf | setNumTasksToExecutePerJvm(int numTasks) | Empty implementation |
JobConf | setOutputCommitter(Class<? extends OutputCommitter> theClass) | No (throws an exception) |
JobConf | setOutputFormat(Class<? extends OutputFormat> theClass) | Empty implementation |
JobConf | setOutputKeyClass(Class<?> theClass) | Yes |
JobConf | setOutputKeyComparatorClass(Class<? extends RawComparator> theClass) | No (throws an exception) |
JobConf | setOutputValueClass(Class<?> theClass) | Yes |
JobConf | setOutputValueGroupingComparator(Class<? extends RawComparator> theClass) | No (throws an exception) |
JobConf | setPartitionerClass(Class<? extends Partitioner> theClass) | Yes |
JobConf | setProfileEnabled(boolean newValue) | Empty implementation |
JobConf | setProfileParams(String value) | Empty implementation |
JobConf | setProfileTaskRange(boolean isMap, String newValue) | Empty implementation |
JobConf | setQueueName(String queueName) | No (throws an exception) |
JobConf | setReduceDebugScript(String rDbgScript) | Empty implementation |
JobConf | setReducerClass(Class<? extends Reducer> theClass) | Yes |
JobConf | setReduceSpeculativeExecution(boolean speculativeExecution) | Empty implementation |
JobConf | setSessionId(String sessionId) | Empty implementation |
JobConf | setSpeculativeExecution(boolean speculativeExecution) | No (throws an exception) |
JobConf | setUseNewMapper(boolean flag) | Yes |
JobConf | setUseNewReducer(boolean flag) | Yes |
JobConf | setUser(String user) | Empty implementation |
JobConf | setWorkingDirectory(Path dir) | Empty implementation |
FileInputFormat | N/A | No (throws an exception) |
TextInputFormat | N/A | Yes |
InputSplit | mapred.min.split.size. | No (throws an exception) |
FileSplit | map.input.file | No (throws an exception) |
RecordWriter | N/A | No (throws an exception) |
RecordReader | N/A | No (throws an exception) |
OutputFormat | N/A | No (throws an exception) |
OutputCommitter | abortJob(JobContext jobContext, int status) | No (throws an exception) |
OutputCommitter | abortJob(JobContext context, JobStatus.State runState) | No (throws an exception) |
OutputCommitter | abortTask(TaskAttemptContext taskContext) | No (throws an exception) |
OutputCommitter | abortTask(TaskAttemptContext taskContext) | No (throws an exception) |
OutputCommitter | cleanupJob(JobContext jobContext) | No (throws an exception) |
OutputCommitter | cleanupJob(JobContext context) | No (throws an exception) |
OutputCommitter | commitJob(JobContext jobContext) | No (throws an exception) |
OutputCommitter | commitJob(JobContext context) | No (throws an exception) |
OutputCommitter | commitTask(TaskAttemptContext taskContext) | No (throws an exception) |
OutputCommitter | needsTaskCommit(TaskAttemptContext taskContext) | No (throws an exception) |
OutputCommitter | needsTaskCommit(TaskAttemptContext taskContext) | No (throws an exception) |
OutputCommitter | setupJob(JobContext jobContext) | No (throws an exception) |
OutputCommitter | setupJob(JobContext jobContext) | No (throws an exception) |
OutputCommitter | setupTask(TaskAttemptContext taskContext) | No (throws an exception) |
OutputCommitter | setupTask(TaskAttemptContext taskContext) | No (throws an exception) |
Counter | getDisplayName() | Yes |
Counter | getName() | Yes |
Counter | getValue() | Yes |
Counter | increment(long incr) | Yes |
Counter | setValue(long value) | Yes |
Counter | setDisplayName(String displayName) | Yes |
DistributedCache | CACHE_ARCHIVES | No (throws an exception) |
DistributedCache | CACHE_ARCHIVES_SIZES | No (throws an exception) |
DistributedCache | CACHE_ARCHIVES_TIMESTAMPS | No (throws an exception) |
DistributedCache | CACHE_FILES | No (throws an exception) |
DistributedCache | CACHE_FILES_SIZES | No (throws an exception) |
DistributedCache | CACHE_FILES_TIMESTAMPS | No (throws an exception) |
DistributedCache | CACHE_LOCALARCHIVES | No (throws an exception) |
DistributedCache | CACHE_LOCALFILES | No (throws an exception) |
DistributedCache | CACHE_SYMLINK | No (throws an exception) |
DistributedCache | addArchiveToClassPath(Path archive, Configuration conf) | No (throws an exception) |
DistributedCache | addArchiveToClassPath(Path archive, Configuration conf, FileSystem fs) | No (throws an exception) |
DistributedCache | addCacheArchive(URI uri, Configuration conf) | No (throws an exception) |
DistributedCache | addCacheFile(URI uri, Configuration conf) | No (throws an exception) |
DistributedCache | addFileToClassPath(Path file, Configuration conf) | No (throws an exception) |
DistributedCache | addFileToClassPath(Path file, Configuration conf, FileSystem fs) | No (throws an exception) |
DistributedCache | addLocalArchives(Configuration conf, String str) | No (throws an exception) |
DistributedCache | addLocalFiles(Configuration conf, String str) | No (throws an exception) |
DistributedCache | checkURIs(URI[] uriFiles, URI[] uriArchives) | No (throws an exception) |
DistributedCache | createAllSymlink(Configuration conf, File jobCacheDir, File workDir) | No (throws an exception) |
DistributedCache | createSymlink(Configuration conf) | No (throws an exception) |
DistributedCache | getArchiveClassPaths(Configuration conf) | No (throws an exception) |
DistributedCache | getArchiveTimestamps(Configuration conf) | No (throws an exception) |
DistributedCache | getCacheArchives(Configuration conf) | No (throws an exception) |
DistributedCache | getCacheFiles(Configuration conf) | No (throws an exception) |
DistributedCache | getFileClassPaths(Configuration conf) | No (throws an exception) |
DistributedCache | getFileStatus(Configuration conf, URI cache) | No (throws an exception) |
DistributedCache | getFileTimestamps(Configuration conf) | No (throws an exception) |
DistributedCache | getLocalCacheArchives(Configuration conf) | No (throws an exception) |
DistributedCache | getLocalCacheFiles(Configuration conf) | No (throws an exception) |
DistributedCache | getSymlink(Configuration conf) | No (throws an exception) |
DistributedCache | getTimestamp(Configuration conf, URI cache) | No (throws an exception) |
DistributedCache | setArchiveTimestamps(Configuration conf, String timestamps) | No (throws an exception) |
DistributedCache | setCacheArchives(URI[] archives, Configuration conf) | No (throws an exception) |
DistributedCache | setCacheFiles(URI[] files, Configuration conf) | No (throws an exception) |
DistributedCache | setFileTimestamps(Configuration conf, String timestamps) | No (throws an exception) |
DistributedCache | setLocalArchives(Configuration conf, String str) | No (throws an exception) |
DistributedCache | setLocalFiles(Configuration conf, String str) | No (throws an exception) |
IsolationRunner | N/A | No (throws an exception) |
profiling | N/A | Empty implementation |
debugging | N/A | Empty implementation |
data compression | Not applicable | Yes |
Skipping Bad Records | Not applicable | No, it throws an exception. |
Job Authorization | mapred.acls.enabled | No. It throws an exception. |
Job Authorization | mapreduce.job.acl-view-job | No, it throws an exception. |
Job Authorization | mapreduce.job.acl-modify-job | No, an exception is thrown. |
Job Authorization | mapreduce.cluster.administrators | No, an exception is thrown. |
Job Authorization | mapred.queue.queue-name.acl-administer-jobs | No, an exception is thrown. |
MultipleInputs | Not applicable | No, an exception is thrown. |
MultipleOutputs | Not applicable | Yes |
org.apache.hadoop.mapreduce.lib.db | Not applicable | No, an exception is thrown. |
org.apache.hadoop.mapreduce.security | Not applicable | No, an exception is thrown. |
org.apache.hadoop.mapreduce.lib.jobcontrol | Not applicable | No, an exception is thrown. |
org.apache.hadoop.mapreduce.lib.chain | Not applicable | No, an exception is thrown. |
org.apache.hadoop.mapreduce.lib.db | Not applicable | No, it throws an exception. |