MaxCompute:Develop ODPS Spark tasks

Java or Scala

Complete the following steps on your local machine before configuring the node:

1. Set up a development environment. Prepare a development environment based on your operating system:

   • Linux: Set up a Linux development environment

   • Windows: Set up a Windows development environment

2. Develop your code. Write your Spark on MaxCompute application code. Start from the sample project template to get a pre-configured project structure with the correct Spark and MaxCompute dependencies.

3. Package and upload the JAR. Package your code as a JAR and upload it to DataWorks as a MaxCompute resource. For details, see Create and use MaxCompute resources.

Python (default environment)

Write your PySpark code directly in DataWorks as a Python resource, then commit it. No local setup is needed. For examples and instructions, see Create and use MaxCompute resources and Develop a Spark on MaxCompute application by using PySpark.

If the default Python environment does not include the third-party packages your task needs, either prepare a custom Python environment (see below), or use PyODPS 2 nodes or PyODPS 3 nodes, which support a broader set of Python libraries.

Python (custom environment)

If the default Python environment does not meet your requirements:

1. Prepare a custom Python environment on your local machine. Refer to PySpark Python versions and supported dependencies to configure the environment based on your dependency requirements.

2. Package and upload the environment. Package the Python environment as a ZIP file and upload it to DataWorks as a MaxCompute resource. For details, see Create and use MaxCompute resources.

Step 1: Create and commit the Python resource

1. In the DataWorks console, go to DataStudio and create a Python resource named spark_is_number.py. For details on creating resources, see Create and use MaxCompute resources. Paste the following code into the resource:

   # -*- coding: utf-8 -*-
   import sys
   from pyspark.sql import SparkSession
   
   try:
       # Python 2 only
       reload(sys)
       sys.setdefaultencoding('utf8')
   except:
       # Python 3 does not need this
       pass
   
   if __name__ == '__main__':
       spark = SparkSession.builder\
           .appName("spark sql")\
           .config("spark.sql.broadcastTimeout", 20 * 60)\
           .config("spark.sql.crossJoin.enabled", True)\
           .config("odps.exec.dynamic.partition.mode", "nonstrict")\
           .config("spark.sql.catalogImplementation", "odps")\
           .getOrCreate()
   
   def is_number(s):
       try:
           float(s)
           return True
       except ValueError:
           pass
   
       try:
           import unicodedata
           unicodedata.numeric(s)
           return True
       except (TypeError, ValueError):
           pass
   
       return False
   
   print(is_number('foo'))
   print(is_number('1'))
   print(is_number('1.3'))
   print(is_number('-1.37'))
   print(is_number('1e3'))

2. Save and commit the resource.

Step 2: Configure the ODPS Spark node

In the ODPS Spark node, set the following parameters:

Save and commit the node.

Step 3: Run the node in Operation Center

ODPS Spark nodes cannot be run from DataStudio. Go to Operation Center in the development environment to run the node.

In Operation Center, trigger a backfill for the ODPS Spark node. For details, see Backfill data and view data backfill instances (new version).

Step 4: View the results

After the backfill instance completes successfully, click the tracking URL in the run logs to view the output:

False
True
True
True
True