DataWorks:Shell node

Node development

1. Log on to the DataWorks console. After you switch to the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select the target workspace from the drop-down list, and then click Go to DataStudio.

2. Hover over the icon, and click Create Node > General > Shell. In the Create Node dialog, enter the node name and path.

3. Enter standard shell code in the script editor (interactive syntax is not supported):

   echo "Hello DataWorks!"

4. After you complete the code, click , select the target resource group and image, and run the shell node task.

5. After the script passes debugging, click Scheduling Settings on the right side to configure production-level scheduling policies such as scheduling time and resource properties. This allows the node to run periodically as planned. For more information about configuring node scheduling properties, see Configure schedule settings.

Deploy and manage nodes

1. After you configure the scheduling properties, you can submit the node to the development environment, and then deploy it to the production environment for periodic scheduling.

2. After the node is deployed, the task runs periodically based on the schedule you configured. You can click the icon in the upper-left corner, and then select All Products > Data Development and O&M > Operation Center on the navigation page that appears to go to Operation Center. Then, in the left-side menu, navigate to Task O&M > Auto Triggered Task O&M > Periodic tasks to view the deployed scheduled tasks. For more information, see Manage scheduled tasks.

Reference resources

1. DataWorks allows you to upload resources to DataWorks through resource management for use in shell nodes. For more information, see Manage resources.

   Note

   A resource must be submitted before it can be referenced by a node. If the resource is required by a production task, you must also deploy the resource to the production environment. For more information, see Deploy nodes.

2. In the left-side directory tree of Data Studio, locate the uploaded resource.

3. Right-click the resource and select Insert Resource Path. The resource is then referenced in the current node. You can write code on the node editing page to run the resource.

   Note

   After a successful reference, the system automatically inserts a declaration comment in the format ##@resource_reference{resource_name} at the top of the script.

   This comment is a required identifier for DataWorks to recognize resource dependencies and automatically mount the corresponding resource to the execution environment at runtime. Do not manually modify or delete it.

Use scheduling parameters

Scheduling parameters are injected into shell nodes as positional parameters. Custom variable names are not supported. DataWorks passes the scheduling parameter values that you configure in the Scheduling Settings of a node to the shell script in order as positional parameters such as $1, $2, $3, and so on. When more than 9 parameters are used, use the brace syntax such as ${10} and ${11} to ensure correct parsing. Multiple parameter values must be separated by spaces, and the order must strictly match the positional references in the script.

In this example:

• The built-in parameter $1 is assigned the business date: $bizdate.

• The custom parameter $2 is assigned the business date: ${yyyymmdd}.

• The custom parameter $3 is assigned the business date: $[yyyymmdd].

Use ossutil to access OSS

The DataWorks shell node natively supports the Alibaba Cloud OSS command-line tool ossutil, which allows bucket management, file upload and download, batch operations, and other tasks. You can configure access credentials to use ossutil to access OSS through a configuration file or command-line parameters.

• To use ossutil to access OSS through command-line parameters, see Access OSS by using command-line parameters.

• To use ossutil to access OSS through a configuration file, see Access OSS by using a configuration file.

Appendix: Script exit codes

You can use script exit codes to further determine whether a script ran successfully.

• Exit code 0: Indicates success.

• Exit code -1: Indicates the process was terminated.

• Exit code 2: Indicates that the platform needs to automatically rerun the task once.

• Other exit codes: Indicate failure.

The following is a sample runtime log when the shell node runs successfully (exit code 0).

INFO  Exit code of the Shell command 0
INFO  --- Invocation of Shell command completed ---
INFO  Shell run successfully!