edit-icon download-icon

Friend suggestion by using MapReduce

Last Updated: Mar 27, 2018

Social network provides a predominant information platform, where you can add friends from People you may be interested in list provided by social networking website or app. People you may be interested in, or Friend Suggestion, suggests you to send requests to the people based on your mutual friends suggestions. Here, we use a simple example to describe how to implement the Friend Suggestion feature with MapReduce.

Case introduction

The following figure shows friend relationship between A, B, C, D, and E, among which solid lines shows relationship of mutual friends. So, how can we get the number of friends between two users who are not friends act as a reference of new friends suggestion?

Figure name

Follow these steps:

  1. Divide the relationships to two maps, and each map contains three friend relationships. Split each friend relationship. If the two users in the Key are friends, set the value as 0. If not, set the value as 1. Sort the results. (A B) and (B A) are regarded as the same Key (A B).

    Figure name

  2. Combine the records of two maps. If the Keys of two records are the same and the Value in each record is not 0, add 1 to the Values.

    Note:

    Keep the records whose Value is 0 during the Combine process. Otherwise, an error may occur in obtaining results during the Reduce process.

    Figure name

  3. Club the Combine results of two maps by using the Reduce method.

    1. If the Keys of two records are same and the Value in each record is not 0, add 1 to the Values.

    2. Delete records whose Values are 0.

    3. Acquire the number of mutual friends between two users who are not friends. The Key shows the two users who are not friends, and the Value shows the number of mutual friends between them. The social networking websites or apps can suggest friends based on the Value.

    Figure name

Procedure

Follow these steps to create a data table

  1. Log on to the DataWorks console, and click Enter Workspace in the corresponding project space.

  2. Click Data Development from the upper menu to enter the Data Development homepage. Click New > Create script or Create script.

    1

  3. Complete the configurations in the New Script File dialog box. Enter the file name, select ODPS SQL for the type, and click Submit. See the following figure.

    32

  4. The statements used for table creation are as follows:

    1. drop table if exists dual;--Create system dual
    2. create table dual(id bigint); --If the project does not have the pseudo table, create the table and initialize data.
    3. insert overwrite table dual select count(*)from dual;--Initialize data to the pseudo table
    4. --Create a data input table for Friend Suggestion MR, among which the uid shows a user and the friends shows the friends of the uid user.
    5. create table friends_in (uid string, friends string);
    6. --Create a data output table for Friend Suggestion MR, among which the userA shows a user, the userB shows another user, and the cnt shows the number of mutual friends between them.
    7. create table friends_out (userA string, userB string, cnt bigint);
  5. Click Run. When the log message returns success, it means the target table is created successfully.

    33

  6. Click Save to save the input SQL table creation statement.

Import local data

  1. Click Import > Import Local Data from the upper menu, and open the friends_in_data.csv local file (Click here to download).

  2. Set all configurations as default, and view the imported data. Click Next,

    Note:

    The data must be imported as txt or csv files in the real work environment.

    34

  3. Enter friends_in in Import to Table on the Import Local Data page to import the test data of this case to the friends_in input table of Friend Suggestion, and check that the Target Field matches with the Source Field. Click Import.

    35

    The data size is large, please wait for one to two minutes.

  4. After the data is imported, enter the following statement to check. See the following figure.

    36

Add MR resource

  1. Click Resource in the left-side navigation pane, and click the Upload in the upper-right corner of the list.

    37

  2. Configure the information in the Upload Resource window that appears, and select the Friends_MR to be uploaded. See the following figure.

    38

  3. Click Submit.

    You can see the uploaded Jar package friends_mr.jar, under Resource Management in the left-side navigation pane.

Test and verify Friend Suggestion

  1. Click New > New Task from the upper menu to create the MR task for this case.

  2. Set Task Type of the new task as Node Task in the dialog box that appears, and configure as follows:

    39

  3. Click Create.

  4. Enter all the configurations on the task page, as shown in the following figure:

    40

    Configuration item description:

    • MRJar package: Click the text box, and select friends_mr.jar.
    • Resource: friends_mr.jar by default.

    • Input table: Enter friends_in.

    • mapper: Enter friends_mr_odps.FriendsMapper, which is the full name of Mapper class in the Jar package.

    • reducer: Enter friends_mr_odps.FriendsReducer, which is the full name of Reducer class in the Jar package.

    • combiner: Enter friends_mr_odps.FriendsReducer, which is the full name of Combiner class in the Jar package.

    • Output table: Enter friends_out.

    • Output Key: Enter userA:String, userB:String.

    • Output Val: Enter cnt:Bigint.

  5. Save and Run the configured OPEN MR task, and check the status and result in Logs. See the following figure.

    41

  6. Enter the following SQL statement in the script file, and click Run to query the data with more than two mutual friends.

    1. SELECT * FROM friends_out WHERE cnt>2 order by cnt desc limit 100;

    42

Thank you! We've received your feedback.