Develop a MaxCompute Spark task

Limits

If an error is reported when you commit an ODPS Spark node that uses the Spark 3.X version, purchase a serverless resource group. For more information, see Create and use a serverless resource group.

Preparations

You can use a MaxCompute Spark node to run a Spark on MaxCompute offline task in Java/Scala or Python. The development steps and configuration process differ for each language. Choose a language based on your business requirements.

Descriptions of parameters

DataWorks runs Spark on MaxCompute offline tasks in cluster mode. In this mode, you must specify the entry point of your custom application in the main method. The Spark task ends when the main method finishes execution, with a status of either Success or Fail. Additionally, you must add the configurations from the spark-defaults.conf file individually to the configuration items of the MaxCompute Spark node. Examples include the number of executor instances, memory size, and the spark.hadoop.odps.runtime.end.point configuration.

Simple code editing example

This section provides a simple example to show how to use an ODPS Spark node to develop a Spark on MaxCompute task. In this example, a Spark on MaxCompute task is developed to determine whether a string can be converted into digits.

1. Create a resource.

   1. On the DataStudio page, create a new Python resource and name it spark_is_number.py. For more information, see Create and use MaxCompute resources. Use the following code:

      # -*- coding: utf-8 -*-
      import sys
      from pyspark.sql import SparkSession
      try:
          # for python 2
          reload(sys)
          sys.setdefaultencoding('utf8')
      except:
          # python 3 not needed
          pass
      if __name__ == '__main__':
          spark = SparkSession.builder\
              .appName("spark sql")\
              .config("spark.sql.broadcastTimeout", 20 * 60)\
              .config("spark.sql.crossJoin.enabled", True)\
              .config("odps.exec.dynamic.partition.mode", "nonstrict")\
              .config("spark.sql.catalogImplementation", "odps")\
              .getOrCreate()
      def is_number(s):
          try:
              float(s)
              return True
          except ValueError:
              pass
          try:
              import unicodedata
              unicodedata.numeric(s)
              return True
          except (TypeError, ValueError):
              pass
          return False
      print(is_number('foo'))
      print(is_number('1'))
      print(is_number('1.3'))
      print(is_number('-1.37'))
      print(is_number('1e3'))

   2. Save and commit the resource.

2. In the created ODPS Spark node, configure parameters and scheduling properties for the MaxCompute Spark task by referring to the Descriptions of parameters section in this topic, and save and commit the node.

   Parameter	Description
   Spark Version	Select Spark2.x.
   Language	Select Python.
   Main Python Resource	The Python resource spark_is_number.py that you created.

3. Go to Operation Center in the development environment to backfill data for the ODPS Spark node. For more information, see Backfill data and view data backfill instances (new version).

   Note

   DataWorks does not provide entry points for you to run ODPS Spark nodes in DataStudio. You must run ODPS Spark nodes in Operation Center in the development environment.

4. View the result.

   After the data backfill instance is successfully run, click tracking URL in the run logs that are generated to view the result. The following information is returned:

   False
   True
   True
   True
   True

Advanced code editing examples

For more information about the development of Spark on MaxCompute tasks in other scenarios, see the following topics:

• Spark 1.x examples for Java or Scala

• Spark 2.x examples for Java or Scala

• Develop a Spark on MaxCompute application by using PySpark

• Access instances in a VPC from Spark on MaxCompute

• Access OSS from Spark on MaxCompute