E-MapReduce:Use third-party Python libraries in a notebook

Method 1: Install libraries using pip

1. Go to the notebook development page.

   1. Log in to the E-MapReduce console.

   2. In the left navigation pane, choose EMR Serverless > Spark.

   3. On the Spark page, click the name of your target workspace.

   4. On the EMR Serverless Spark page, click Development in the left navigation pane.

   5. Double-click the notebook that you created.

2. In a Python cell of the notebook, enter the following command to install the scikit-learn library, and then click the Run Cell icon .

   pip install scikit-learn

3. In another Python cell, enter the following code, and then click the Run Cell icon .

   # Import the library and prepare the dataset.
   from sklearn import datasets
   # Load a built-in dataset, such as the Iris dataset.
   iris = datasets.load_iris()
   X = iris.data  # Feature data
   y = iris.target  # Label
   # Split the dataset.
   from sklearn.model_selection import train_test_split
   # Split the data into training and test sets.
   X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
   # Train a support vector machine (SVM) model.
   from sklearn.svm import SVC
   # Create a classifier instance.
   clf = SVC(kernel='linear')  # Use a linear kernel.
   # Train the model.
   clf.fit(X_train, y_train)
   # Make predictions with the trained model.
   y_pred = clf.predict(X_test)
   # Evaluate model performance.
   from sklearn.metrics import classification_report, accuracy_score
   print(classification_report(y_test, y_pred))
   print("Accuracy:", accuracy_score(y_test, y_pred))
   

   The output is similar to the following:

                 precision    recall  f1-score   support
              0       1.00      1.00      1.00        19
              1       1.00      1.00      1.00        13
              2       1.00      1.00      1.00        13
       accuracy                           1.00        45
      macro avg       1.00      1.00      1.00        45
   weighted avg       1.00      1.00      1.00        45
   Accuracy: 1.0

Step 1: Create a runtime environment

1. Go to the Environments page.

   1. Log in to the E-MapReduce console.

   2. In the left navigation pane, choose EMR Serverless > Spark.

   3. On the Spark page, click the name of your target workspace.

   4. On the EMR Serverless Spark page, click Environment in the left navigation pane.

2. Click Create Environment.

3. On the Create Environment page, click Add Library.

   For parameter descriptions, see Manage runtime environments.

4. In the New Library dialog box, set Source Type to PyPI, specify the PyPI Package, and then click OK.

   In the PyPI Package field, enter the library name and version. If you do not specify a version, the latest version is installed. Example: scikit-learn.

5. Click create.

   After the environment is created, the system starts to initialize it.

Step 2: Use the runtime environment

1. Go to the Notebook Session tab.

   1. On the EMR Serverless Spark page, choose O&M Center > Sessions in the left navigation pane.

   2. Click the Notebook Session tab.

2. In the Actions column of the target notebook session, click Edit.

3. From the Environment drop-down list, select the runtime environment that you created in the preceding step and click Save Changes.

4. In the upper-right corner, click START.

Step 3: Classify data with Scikit-learn

1. Go to the notebook development page.

   1. On the EMR Serverless Spark page, click Development in the left navigation pane.

   2. Double-click the notebook that you created.

2. In a Python cell of the notebook, enter the following code, and then click the Run Cell icon .

   # Import the library and prepare the dataset.
   from sklearn import datasets
   # Load a built-in dataset, such as the Iris dataset.
   iris = datasets.load_iris()
   X = iris.data  # Feature data
   y = iris.target  # Label
   # Split the dataset.
   from sklearn.model_selection import train_test_split
   # Split the data into training and test sets.
   X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
   # Train a support vector machine (SVM) model.
   from sklearn.svm import SVC
   # Create a classifier instance.
   clf = SVC(kernel='linear')  # Use a linear kernel.
   # Train the model.
   clf.fit(X_train, y_train)
   # Make predictions with the trained model.
   y_pred = clf.predict(X_test)
   # Evaluate model performance.
   from sklearn.metrics import classification_report, accuracy_score
   print(classification_report(y_test, y_pred))
   print("Accuracy:", accuracy_score(y_test, y_pred))
                 

Step 1: Build and deploy a conda environment

1. Run the following commands to install Miniconda:

   wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
   chmod +x Miniconda3-latest-Linux-x86_64.sh
   ./Miniconda3-latest-Linux-x86_64.sh -b
   source miniconda3/bin/activate

2. Build a conda environment that uses Python 3.8 and numpy.

   # Create and activate the conda environment.
   conda create -y -n pyspark_conda_env python=3.8
   conda activate pyspark_conda_env
   # Install third-party libraries.
   pip install numpy \
   ipykernel~=6.29 \
   jupyter_client~=8.6 \
   jieba \
   conda-pack
   # Pack the environment.
   conda pack -f -o pyspark_conda_env.tar.gz

Step 2: Upload resource file to OSS

Upload the packaged pyspark_conda_env.tar.gz file to Alibaba Cloud OSS and note the complete OSS path. For more information, see Simple Upload.

Step 3: Configure and start the notebook session

1. Go to the Notebook Session tab.

   1. On the EMR Serverless Spark page, choose O&M Center > Sessions in the left navigation pane.

   2. Click the Notebook Session tab.

2. In the Actions column of the target notebook session, click Edit.

3. In the Spark Configuration section, add the following configurations and click Save Changes.

   spark.archives  oss://<yourBucket>/path/to/pyspark_conda_env.tar.gz#env
   spark.pyspark.python ./env/bin/python

   Note

   In the configuration, replace <yourBucket>/path/to with your actual OSS upload path.

4. In the upper-right corner, click START.

Step 4: Process text data with Jieba

1. Go to the notebook development page.

   1. On the EMR Serverless Spark page, click Development in the left navigation pane.

   2. Double-click the notebook that you created.

2. In a new Python cell, enter the following command to perform Chinese word segmentation using Jieba, and then click the Run Cell icon .

   import jieba
   strs = ["EMRServerlessSpark是一款专为大规模数据处理与分析而打造的全托管Serverless产品", "为您提供了高效的任务开发调试调度以及运维等一站式服务", "EMRServerlessSpark实现了资源调度和根据任务负载进行动态扩展"]
   sc.parallelize(strs).flatMap(lambda s: jieba.cut(s, use_paddle=True)).collect()

   The command returns the following result:

   import jieba
   strs = ["EMRServerlessSpark是一款专为大规模数据处理与分析而打造的全托管Serverless产品", "为您提供了高效的任务开发调试调度以及运维等一站式服务", "EMRServerlessSpark实现了资源调度和根据任务负载进行动态扩展..."]
   sc.parallelize(strs).flatMap(lambda s: jieba.cut(s, use_paddle=True)).collect()
   ['EMRServerlessSpark',
    '是',
    '一款',
    '专',
    '为',
    '大规模',
    '数据处理',
    '与',
    '分析',
    '而',
    '打造',
    '的',
    '全',
    '托管',
    'Serverless',
    '产品',
    '为',
    '您',
    '提供',
    '了',
    '高效',
    '的',
    '任务',
    '开发',
    '调试',
    '调度',
    '以及',
    '运维',
    '等',
    '一站式',
    '服务',
    'EMRServerlessSpark',
    '实现',
    '了',
    '资源',
    '调度',
    '和',
    '根据',
    '任务',
    '负载',
    '进行',
    '动态',
    '扩展']