DataWorks:Develop ODPS Spark jobs

Java/Scala

• A Linux or Windows development environment. For setup instructions, see Set up a Linux development environment or Set up a Windows development environment.

• Spark on MaxCompute code developed locally. Use the sample project template to get started.

• The packaged JAR uploaded to DataWorks as a MaxCompute resource. See Create and use MaxCompute resources.

Python (default environment)

• A Python resource containing your PySpark code, uploaded to DataWorks. See Create and use MaxCompute resources and Develop a Spark on MaxCompute application by using PySpark.

Python (custom environment)

• A local Python environment configured per the PySpark Python version and dependency support requirements.

• The environment compressed into a ZIP package and uploaded to DataWorks as a MaxCompute resource. This package provides the execution environment for the job. See Create and use MaxCompute resources.

Step 1: Create the Python resource

1. On the Data Development page, create a Python resource named spark_is_number.py. For more information, see Create and use MaxCompute resources.

   # -*- coding: utf-8 -*-
   import sys
   from pyspark.sql import SparkSession
   
   try:
       # for python 2
       reload(sys)
       sys.setdefaultencoding('utf8')
   except:
       # python 3 not needed
       pass
   
   if __name__ == '__main__':
       spark = SparkSession.builder\
           .appName("spark sql")\
           .config("spark.sql.broadcastTimeout", 20 * 60)\
           .config("spark.sql.crossJoin.enabled", True)\
           .config("odps.exec.dynamic.partition.mode", "nonstrict")\
           .config("spark.sql.catalogImplementation", "odps")\
           .getOrCreate()
   
   def is_number(s):
       try:
           float(s)
           return True
       except ValueError:
           pass
   
       try:
           import unicodedata
           unicodedata.numeric(s)
           return True
       except (TypeError, ValueError):
           pass
   
       return False
   
   print(is_number('foo'))
   print(is_number('1'))
   print(is_number('1.3'))
   print(is_number('-1.37'))
   print(is_number('1e3'))

2. Save and commit the resource.

Step 2: Configure the ODPS Spark node

In the ODPS Spark node, configure the following parameters, then save and commit the node.

Step 3: Run the job

Go to Operation Center for the development environment and run a Data Backfill job. For detailed instructions, see Data backfill instance O&M.

Step 4: View the results

After the Data Backfill instance completes successfully, go to its tracking URL in the run log to view the output:

False
True
True
True
True