DataWorks:MaxCompute Spark node

Permissions

(Optional, required for RAM users) The RAM user used for task development must be added to the corresponding workspace and granted the developer or workspace administrator role (use caution when granting workspace administrator, as it has extensive permissions). For more information about adding members, see Add members to a workspace.

Limitations

If an error occurs when you submit a MaxCompute Spark node that uses Spark 3.x, purchase and use a serverless resource group. For more information, see Use serverless resource groups.

Before you begin

MaxCompute Spark nodes support running Spark on MaxCompute batch jobs in Java/Scala and Python. Because the development steps and configuration interface differ for each language, choose the one that best fits your business needs.

Configuration items

DataWorks runs Spark on MaxCompute batch jobs in cluster mode. In cluster mode, you must specify a custom program entry point main. When main finishes (that is, its status is Success or Fail), the corresponding Spark job ends. In addition, each configuration in spark-defaults.conf must be added individually to the configuration items of the MaxCompute Spark node, such as the number of executors, memory size, and spark.hadoop.odps.runtime.end.point.

Procedure

1. Create a resource.

   1. In the Resource Management section of the left-side navigation pane on the Data Studio page, click Create, select Create MaxCompute Spark Python resource, and name it spark_is_number.py. For more information, see Resource management. The code is as follows:

      # -*- coding: utf-8 -*-
      import sys
      from pyspark.sql import SparkSession
      
      try:
          # for python 2
          reload(sys)
          sys.setdefaultencoding('utf8')
      except:
          # python 3 not needed
          pass
      
      if __name__ == '__main__':
          spark = SparkSession.builder\
              .appName("spark sql")\
              .config("spark.sql.broadcastTimeout", 20 * 60)\
              .config("spark.sql.crossJoin.enabled", True)\
              .config("odps.exec.dynamic.partition.mode", "nonstrict")\
              .config("spark.sql.catalogImplementation", "odps")\
              .getOrCreate()
      
      def is_number(s):
          try:
              float(s)
              return True
          except ValueError:
              pass
      
          try:
              import unicodedata
              unicodedata.numeric(s)
              return True
          except (TypeError, ValueError):
              pass
      
          return False
      
      print(is_number('foo'))
      print(is_number('1'))
      print(is_number('1.3'))
      print(is_number('-1.37'))
      print(is_number('1e3'))

   2. Save the resource.

2. In the created MaxCompute Spark node, configure the node parameters and schedule settings. For more information, see Configuration items.

3. To run the node on a schedule, configure schedule information based on your business needs. For more information, see Configure node scheduling.

4. After configuring the node task, you must deploy the node. For more information, see Node and workflow deployment.

5. After the task is deployed, you can view the running status of scheduled tasks in Operation Center. For more information, see Get started with operation center.

   Note

   • Because the MaxCompute Spark node in Data Studio does not have a run entry, you must run Spark tasks from Operation Center in the development environment.

   • After the backfill data instance runs successfully, open the tracking URL in its runtime log to view the execution result.

References

• For Spark on MaxCompute task development in other scenarios, see:

  • Spark 1.x examples

  • Spark-2.x examples

  • Develop a Spark on MaxCompute application by using PySpark

  • Access instances in a VPC from Spark on MaxCompute

  • Access OSS from Spark on MaxCompute

• Spark FAQ: Learn about common issues during Spark execution so that you can quickly troubleshoot and resolve exceptions. For more information, see Spark FAQ.

• Spark job diagnostics: MaxCompute provides the Logview tool and Spark WebUI for Spark jobs. You can check the job logs to verify that the job has been submitted and run properly. For more information, see Perform job diagnostics.