DataWorks:Create an ODPS Spark node

Alternatively, you can find your desired workflow, right-click the workflow name, and then choose Create > MaxCompute > Resource > JAR.

Alternatively, you can find your desired workflow, right-click the workflow name, and then choose Create > MaxCompute > Resource > Python.

In the following example, the Python code defines the logic for checking whether parameter values are correct instead of the logic for processing data.

# -*- coding: utf-8 -*-
import sys
from pyspark.sql import SparkSession

try:
    # for python 2
    reload(sys)
    sys.setdefaultencoding('utf8')
except:
    # python 3 not needed
    pass

if __name__ == '__main__':
    spark = SparkSession.builder\
        .appName("spark sql")\
        .config("spark.sql.broadcastTimeout", 20 * 60)\
        .config("spark.sql.crossJoin.enabled", True)\
        .config("odps.exec.dynamic.partition.mode", "nonstrict")\
        .config("spark.sql.catalogImplementation", "odps")\
        .getOrCreate()

def is_number(s):
    try:
        float(s)
        return True
    except ValueError:
        pass

    try:
        import unicodedata
        unicodedata.numeric(s)
        return True
    except (TypeError, ValueError):
        pass

    return False

print(is_number('foo'))
print(is_number('1'))
print(is_number('1.3'))
print(is_number('-1.37'))
print(is_number('1e3'))

Alternatively, you can find your desired workflow, right-click the workflow name, and then choose Create > MaxCompute > ODPS Spark.

Three options are available for the Spark Version parameter, and two options are available for the Language parameter. The parameters displayed on the configuration tab vary based on the value of the Language parameter. You can configure the parameters as prompted.

• The following table describes the parameters that are displayed on the configuration tab after you set Language to Java/Scala.

  Parameter	Description
  Spark Version	The Spark version of the node. Valid values: Spark1.x, Spark2.x, and Spark3.x.
  Language	The programming language used by the node. Set this parameter to Java/Scala.
  Main JAR Resource	The main JAR resource that you want to reference for the node. Select the desired JAR resource from the drop-down list.
  Configuration Items	The additional items that you want to configure for the node. Click Add, and enter a key and a value to add a configuration item. Alternatively, you can directly select a key from the field that appears. In this case, a value is automatically entered for the key.
  Main Class	The class name of the node.
  Parameters	The additional parameters that you want to configure for the node. Separate multiple parameters with spaces. You can configure variables for the node in the ${Variable name} format in the Parameters field of the configuration tab, and assign scheduling parameters to the variables as values in the Parameters section of the Properties tab. For information about the value assignment formats of scheduling parameters, see Supported formats of scheduling parameters. Note After you configure the scheduling parameters, you must continue to configure the parameters for the node. The scheduling parameters and node parameters are run in sequence.
  JAR Resources	The JAR resource that you want to reference for the node. The system displays all the uploaded JAR resources. Select the desired JAR resource from the drop-down list.
  File Resources	The file resource that you want to reference for the node. The system displays all the uploaded file resources. Select the desired file resource from the drop-down list.
  Archive Resources	The archive resource that you want to reference for the node. The system displays all the uploaded archive resources that are compressed. Select the desired archive resource from the drop-down list.

• The following table describes the parameters that are displayed after you set Language to Python.

  Parameter	Description
  Spark Version	The Spark version of the node. Valid values: Spark1.x, Spark2.x, and Spark3.x.
  Language	The programming language used by the node. Set this parameter to Python.
  Main Python Resource	The main Python resource that you want to reference for the node. Select the desired Python resource from the drop-down list.
  Configuration Items	The additional items that you want to configure for the node. Click Add, and enter a key and a value to add a configuration item. Alternatively, you can directly select a key from the field that appears. In this case, a value is automatically entered for the key.
  Parameters	The additional parameters that you want to configure for the node. Separate multiple parameters with spaces. You can configure variables for the node in the ${Variable name} format in the Parameters field of the configuration tab, and assign scheduling parameters to the variables as values in the Parameters section of the Properties tab. For information about the value assignment formats of scheduling parameters, see Supported formats of scheduling parameters.
  Python Resources	The Python resource that you want to reference for the node. The system displays all the uploaded Python resources. Select the desired Python resource from the drop-down list.
  File Resources	The file resource that you want to reference for the node. The system displays all the uploaded file resources. Select the desired file resource from the drop-down list.
  Archive Resources	The archive resource that you want to reference for the node. The system displays all the uploaded archive resources that are compressed. Select the desired archive resource from the drop-down list.

If the workspace that you use is in standard mode, you must click Deploy in the upper-right corner to deploy the node after you commit it. For more information, see Deploy nodes.