DataWorks:MaxCompute Spark node

Prerequisites

• (Required if you use a RAM user to develop tasks) The desired RAM user is added to your DataWorks workspace as a member and is assigned the Develop or Workspace Administrator role. The Workspace Administrator role has extensive permissions. We recommend that you assign the Workspace Administrator role to a user only when necessary. For more information about how to add a member, see Add workspace members and assign roles to them.

  Note

  If you use an Alibaba Cloud account, ignore this prerequisite.

• A MaxCompute Spark node is created. For more information, see Create a node.

Limits

If an error is reported when you commit a MaxCompute Spark node that uses the Spark 3.X version, purchase a serverless resource group. For more information, see Create and use a serverless resource group.

Preparations

MaxCompute Spark nodes allow you to use Java, Scala, or Python to develop and run offline Spark on MaxCompute tasks. The operations and parameters that are required for developing the offline Spark on MaxCompute tasks vary based on the programming language that you use. You can select a programming language based on your business requirements.

Parameter descriptions

You can run offline Spark on MaxCompute tasks in cluster mode in DataWorks. In this mode, you must specify the Main method as the entry point of a custom application. A Spark job ends when Main succeeds or fails. You must add the configuration items in the spark-defaults.conf file to the configurations of the MaxCompute Spark node. For example, you must add the configuration items such as the number of executors, the memory size, and spark.hadoop.odps.runtime.end.point.

Procedure

1. Create a resource.

   1. In the left-side navigation pane of the Data Studio page, click Resource Management. In the RESOURCE MANAGEMENT: ALL pane, click the plus icon and choose Create Resource > MaxCompute Python. In the popover that appears, enter spark_is_number.py as the name of the Python resource. Sample code:

      # -*- coding: utf-8 -*-
      import sys
      from pyspark.sql import SparkSession
      
      try:
          # for python 2
          reload(sys)
          sys.setdefaultencoding('utf8')
      except:
          # python 3 not needed
          pass
      
      if __name__ == '__main__':
          spark = SparkSession.builder\
              .appName("spark sql")\
              .config("spark.sql.broadcastTimeout", 20 * 60)\
              .config("spark.sql.crossJoin.enabled", True)\
              .config("odps.exec.dynamic.partition.mode", "nonstrict")\
              .config("spark.sql.catalogImplementation", "odps")\
              .getOrCreate()
      
      def is_number(s):
          try:
              float(s)
              return True
          except ValueError:
              pass
      
          try:
              import unicodedata
              unicodedata.numeric(s)
              return True
          except (TypeError, ValueError):
              pass
      
          return False
      
      print(is_number('foo'))
      print(is_number('1'))
      print(is_number('1.3'))
      print(is_number('-1.37'))
      print(is_number('1e3'))

   2. Save the resource.

2. Configure parameters and scheduling properties for the created MaxCompute Spark node. For more information, see the Parameter descriptions section of this topic.

3. If you want to run the node on a regular basis, configure the scheduling information based on your business requirements.

4. After the node is configured, deploy the node. For more information, see Deploy nodes.

5. After you deploy the node, view the running status of the node in Operation Center. For more information, see Getting started with Operation Center.

   Note

   • DataWorks does not provide entry points for you to run MaxCompute Spark nodes in Data Studio. You must run MaxCompute Spark nodes in Operation Center in the development environment.

   • After the data backfill instance of the MaxCompute Spark node is successfully run, click tracking URL in the generated run logs to view the result.

References

• For more information about the development of Spark on MaxCompute tasks in other scenarios, see the following topics:

  • Spark 1.x examples for Java or Scala

  • Spark 2.x examples for Java or Scala

  • Develop a Spark on MaxCompute application by using PySpark

  • Access instances in a VPC from Spark on MaxCompute

  • Access OSS from Spark on MaxCompute

• Learn the FAQ about Spark: You can learn the FAQ about Spark. This way, you can identify and troubleshoot issues in an efficient manner when exceptions occur. For more information, see FAQ about Spark on MaxCompute.

• Enable the system to diagnose Spark tasks: MaxCompute provides the Logview tool and Spark Web UI. You can view the logs of Spark tasks to check whether the tasks are submitted and run as expected. For more information, see Perform job diagnostics.