E-MapReduce:Get started with notebook development

Step 1: Prepare a test file

To help you quickly get started with notebook tasks, this quickstart provides a downloadable test file for use in the following steps.

Click employee.csv to download the test file.

Step 2: Upload the test file

Upload the data file (employee.csv) to the Object Storage Service (OSS) console. For more information, see Upload files.

Step 3: Develop and run a notebook

1. On the EMR Serverless Spark page, click Development in the left-side navigation pane.

2. Create a notebook.

   1. On the Development tab, click the icon.

   2. In the dialog box that appears, enter a name, select interactive development > Notebook as the type, and then click OK.

3. In the upper-right corner, select a running notebook session instance.

   You can also select Create Notebook Session from the drop-down list to create a notebook session instance. For more information about notebook sessions, see Manage notebook sessions.

   Note

   Multiple notebooks can share a single session instance and its resources, eliminating the need to create a separate instance for each notebook.

4. Process and visualize data.

   PySpark

   1. Copy the following code into a Python cell in the new notebook.

      # Create a simple DataFrame. Replace the OSS path with the path of the file that you uploaded in Step 2.
      df = spark.read.option("delimiter", ",").option("header", True).csv("oss://path/to/file")
      # Display the first few rows of the DataFrame.
      df.show(5)
      # Perform a simple aggregation operation: calculate the total salary for each department.
      sum_salary_per_department = df.groupBy("department").agg({"salary": "sum"}).show()

   2. Click Execute All Cells to run the notebook.

      You can also run a single cell by clicking the icon next to the cell.

      # Create a simple DataFrame. Replace the OSS path with the path of the file that you uploaded in Step 2.
      df = spark.read.option("delimiter", ",").option("header", True).csv("oss://<yourBucketName>/<path>/employee.csv")
      # Display the first few rows of the DataFrame.
      df.show(5)
      +-------------+----------+------+
      |employee_name|department|salary|
      +-------------+----------+------+
      |        James|     Sales|  3000|
      |      Michael|     Sales|  4600|
      |       Robert| Marketing|  4100|
      |        Maria|   Finance|  3000|
      |        James|     Sales|  3000|
      +-------------+----------+------+
      only showing top 5 rows
      # Perform a simple aggregation operation: calculate the total salary for each department.
      sum_salary_per_department = df.groupBy("department").agg({"salary": "sum"}).show()
      +----------+-----------+
      |department|sum(salary)|
      +----------+-----------+
      |     Sales|    12600.0|
      |   Finance|     6900.0|
      | Marketing|    10400.0|
      +----------+-----------+

   3. (Optional) View the Spark UI.

      In the session drop-down list, hover over the icon for the current notebook session instance and click Spark UI to view information about your Spark jobs.

   Visualization

   Note

   Notebook sessions have the matplotlib, numpy, and pandas libraries pre-installed. If you need to use another third-party library, see Use third-party Python libraries in a notebook.

   1. Use the matplotlib library for data visualization.

      import matplotlib.pyplot as plt
      l = sc.parallelize(range(20)).collect()
      plt.plot(l)
      plt.ylabel('some numbers')
      plt.show()

   2. Click Execute All Cells to run the notebook.

      You can also run a single cell by clicking the icon next to the cell.

      pip install matplotlib

      Installation output:

      Looking in indexes: http://mirrors.cloud.aliyuncs.com/pypi/simple
      Collecting matplotlib

      import matplotlib.pyplot as plt
      l = sc.parallelize(range(20)).collect()
      plt.plot(l)
      plt.ylabel('some numbers')
      plt.show()

      After the code runs, it displays a line chart with the Y-axis labeled "some numbers".