MaxCompute:MapReduce

Scenarios

MapReduce supports the following scenarios:

• Search: Web crawling, inverted indexes, and PageRank.

• Analysis of web access logs:

  • Analyze and mine user behavior, such as browsing and shopping, to provide personalized recommendations.

  • Analyze user access patterns.

• Statistical analysis of texts:

  • Word count and term frequency-inverse document frequency (TF-IDF) analysis of popular novels.

  • Statistical analysis and citation analysis of academic papers and patent documents.

  • Wikipedia data analysis.

• Mining large amounts of data, such as unstructured data, spatio-temporal data, and image data.

• Machine learning: supervised learning, unsupervised learning, and classification algorithms, such as decision trees and support vector machines (SVMs).

• Natural Language Processing:

  • Training and prediction based on big data.

  • Building word co-occurrence matrices, mining frequent itemsets, and detecting duplicate documents based on a corpus.

• Advertisement recommendations: prediction of click-through rates (CTRs) and conversion rates (CVRs).

MapReduce process description

A MapReduce program processes data in two main stages: a map stage followed by a reduce stage. You can define the processing logic for these stages, but it must follow the conventions of the MapReduce framework. The complete data processing flow of MapReduce is as follows:

1. Input data: Before the map operation, the input data is partitioned into equally sized data blocks. Each data block is used as the input for a map worker. This allows multiple map workers to run concurrently.

2. Map stage: Each map worker reads and processes its assigned data block. It then assigns a key to each output record. This key determines which reduce worker receives the record.

   Note

   Data records that have the same key are sent to the same reduce worker. A single reduce worker can receive data records that have different keys.

3. Shuffle phase: Before the Reduce phase, the MapReduce framework sorts data by Key, which groups data with the same Key together. If you specify a Combine Operation (Combiner), the framework calls the Combiner to aggregate data with the same Key. You can customize the logic of the Combiner. In MaxCompute, unlike the classic MapReduce framework protocol, the input and output parameters of the Combiner must be consistent with those of the Reduce phase. This part of the process is also commonly called Shuffle.

4. Reduce stage: Data records with the same key are sent to the same reduce worker. A single reduce worker can receive data from multiple map workers. Each reduce worker performs a reduce operation on the data records that have the same key, converting them into a single value.

5. Output data: The results are generated.

The following section uses the WordCount example to explain the concepts of each stage in MaxCompute MapReduce.

Procedure

1. Input data: The text file is partitioned. The data in each partition is used as the input for a map worker.

2. Map stage: The map worker processes the input. For each digit it receives, it sets the count to 1 and outputs a <Word, Count> pair. The word is used as the key for the output data.

3. Shuffle > Combine and sort: In the early shuffle stage, the output from each map worker is first sorted by key (the word). After sorting, a combiner operation is performed. The counts for the same key (word) are aggregated to form a new <Word, Count> pair. This process is called combine and sort.

4. Shuffle > Assign to reducers: In the late shuffle stage, the data is sent to the reducers. After a reduce worker receives the data, it sorts the data again by key.

5. Reduce stage: Each reduce worker processes the data using the same logic as the combiner. It aggregates the counts for the same key (word) to obtain the final result.

6. Output data.

Limits

• For MapReduce limits, see Limits summary.

• For the limits on running MapReduce locally, see Differences between running locally and in a distributed environment.

Background of the MR2 model

The traditional MapReduce model requires that after each round of MapReduce operations, the resulting data must be stored in a distributed file system, such as Hadoop Distributed File System (HDFS) or a MaxCompute table. A MapReduce workflow usually consists of multiple MapReduce jobs. After each job is complete, its intermediate data must be written to a disk. However, a subsequent map task might only need to read this data once before the next shuffle stage. This process creates redundant disk I/O operations.

The computing and scheduling logic of MaxCompute supports more complex programming models. To address this situation, MaxCompute lets you run a reduce operation directly after another reduce operation, without an intermediate map operation. Therefore, MaxCompute provides an extended MapReduce model that supports chaining multiple reduce operations after a map operation, such as Map > Reduce > Reduce.

Comparison with Hadoop Chain Mapper and ChainReducer

Hadoop Chain Mapper and ChainReducer also support similar serialized map or reduce operations. However, they are fundamentally different from the extended MapReduce (MR2) model of MaxCompute.

Chain Mapper and ChainReducer are based on the traditional MapReduce model. They only allow you to add one or more map operations after the original map or reduce operation. You cannot add a reduce operation. The advantage of this approach is that you can reuse existing mapper business logic to split a map or reduce operation into multiple mapper stages. However, this does not fundamentally change the underlying scheduling and I/O models.