DataWorks:MaxCompute Spark node

Permissions

(Optional, for RAM users) A RAM user developing tasks must be a member of the workspace with the Developer or Workspace Administrator role. The Workspace Administrator role has extensive permissions, so assign it with caution. For more information about how to add members, see Add members to a workspace and assign roles to them.

Limitations

If you select Spark 3.x for a MaxCompute Spark node and the job submission fails, you must purchase and use a Serverless resource group. For more information, see Purchase and use a Serverless resource group.

Prerequisites

MaxCompute Spark nodes support running Spark on MaxCompute batch jobs written in Java/Scala and Python. The development steps and configuration interface differ for each language. Choose the language that best fits your business needs.

Configuration properties

DataWorks runs Spark on MaxCompute batch jobs in cluster mode. In this mode, you must specify the program's entry point, which is the main class. The Spark job ends when the main method finishes with a status of Success or Fail. In addition, you must add each configuration from your spark-defaults.conf file to the Configuration Properties of the MaxCompute Spark node. Examples include the number of executor instances, memory size, and the spark.hadoop.odps.runtime.end.point property.

Steps

1. Create a resource.

   1. On the Data Development page, find Resource Management in the left-side navigation pane. Click Create and select MaxCompute Spark Python to create a resource. Name the resource spark_is_number.py. For more information, see Create and use a MaxCompute resource. Example:

      # -*- coding: utf-8 -*-
      import sys
      from pyspark.sql import SparkSession
      
      try:
          # For Python 2
          reload(sys)
          sys.setdefaultencoding('utf8')
      except:
          # Not needed for Python 3
          pass
      
      if __name__ == '__main__':
          spark = SparkSession.builder\
              .appName("spark sql")\
              .config("spark.sql.broadcastTimeout", 20 * 60)\
              .config("spark.sql.crossJoin.enabled", True)\
              .config("odps.exec.dynamic.partition.mode", "nonstrict")\
              .config("spark.sql.catalogImplementation", "odps")\
              .getOrCreate()
      
      def is_number(s):
          try:
              float(s)
              return True
          except ValueError:
              pass
      
          try:
              import unicodedata
              unicodedata.numeric(s)
              return True
          except (TypeError, ValueError):
              pass
      
          return False
      
      print(is_number('foo'))
      print(is_number('1'))
      print(is_number('1.3'))
      print(is_number('-1.37'))
      print(is_number('1e3'))

   2. Save the resource.

2. In the MaxCompute Spark node that you created, configure the node and scheduling parameters. For more information, see Configure a MaxCompute Spark node.

3. If you need to run the node periodically, configure its scheduling properties based on your business requirements. For more information, see Configure scheduling properties.

4. After configuring the node, deploy it. For more information, see Deploy tasks.

5. After deploying the task, you can view the run status of the periodic task in Operation and Maintenance Center. For more information, see View and manage auto-triggered tasks.

   Note

   • You cannot run MaxCompute Spark nodes directly from the node editor on the Data Development page. You must run the task from Operation and Maintenance Center.

   • After a Backfill Instance runs successfully, you can view the results by opening the tracking URL from the Run Log.

Related topics

• For more information about common development scenarios for Spark on MaxCompute tasks, see the following topics:

  • Java/Scala example: Spark 1.x

  • Java/Scala example: Spark 2.x

  • Python example: PySpark development

  • Use case: Access a VPC instance from Spark

  • Use case: Access OSS from Spark

• Spark FAQ: Learn about common issues when running Spark jobs to help you troubleshoot exceptions more quickly. For more information, see FAQ about Spark.

• Diagnose Spark jobs: MaxCompute provides the Logview tool and the Spark web UI for Spark jobs. You can check the job log to verify correct job submission and execution. For more information, see Diagnose Spark jobs.