Use ossutil in Shell nodes - DataWorks - Alibaba Cloud Documentation Center

Shell nodes allow you to use the Object Storage Service (OSS) command-line tool ossutil. This topic describes how to use ossutil in common Shell nodes or E-MapReduce (EMR) Shell nodes.

Prerequisites

For information about the prerequisites of using an EMR Shell node, see Create an EMR Shell node.
For information about the prerequisites of using a common Shell node, see Create a Shell node.

Limits

For information about the limits imposed when you use an EMR Shell node, see Create an EMR Shell node.
Note
The EMR Shell image in a serverless resource group that is used to run an EMR Shell node does not support ossutil. If you want to use ossutil, you can use a common Shell node or use a custom image to install ossutil. For more information about how to use a custom image to install ossutil, see Manage images.
For information about the limits imposed when you use a common Shell node, see Create a Shell node.

Use ossutil in a Shell node to list the buckets within your account

DataWorks allows you to use ossutil to access OSS in the same way in EMR Shell nodes and common Shell nodes. In this example, a common Shell node is used to describe two methods for using ossutil to list the buckets within your account. You can also refer to the common commands of ossutil to meet your specific business requirements. For more information, see ossutil 1.0.

Use command-line options

Create a common Shell node.
1. Go to the DataStudio page.
  Log on to the DataWorks console. In the top navigation bar, select a desired region. In the left-side navigation pane, choose Data Development and Governance > Data Development. On the page that appears, select a desired workspace from the drop-down list and click Go to Data Development.
2. Create a common Shell node.
  On the DataStudio page, find the desired workflow, right-click the workflow name, and then choose Create Node > General > Shell. In the Create Node dialog box, configure the Name parameter and click Confirm.
On the configuration tab of the common Shell node, write node code.
```
/home/admin/usertools/tools/ossutil64 -e <OSSEndPoint> -i <AccessKeyID> -k <AccessKeySecret> ls -s 
if [[ $?  == 0 ]];then
    echo "access oss success"
else
    echo "failed"
    exit 1
fi
echo "finished"
```
Note
You must replace OSSEndPoint, AccessKeyID, and AccessKeySecret in the node code with actual values. For more information about the parameter settings, see Configure ossutil.
Verify the result.
In the top toolbar of the configuration tab of the common Shell node, click the icon and select a serverless resource group to run a task on the common Shell node. The following figure shows the result.

Use a command configuration file

Configure and upload an ossutil configuration file to OSS.
1. Prepare a configuration file.
  Create a file named myconfig.txt on your on-premises machine. The file contains the following sample code:
```
[Credentials]
language = CH
endpoint = https://oss-cn-hangzhou.aliyuncs.com
accessKeyID = <AccessKeyID>
accessKeySecret = <AccessKeySecret>
```
  Note
  You must replace https://oss-cn-hangzhou.aliyuncs.com, AccessKeyID, and AccessKeySecret in the code with actual values. For more information about the parameter settings, see Configure ossutil.
2. Upload the configuration file.
  Upload the myconfig.txt file to OSS. For more information, see Simple upload.
3. View the download link.
  Click View Details in the Actions column that corresponds to the uploaded configuration file. In the View Details panel, turn off Use HTTPS and then click Copy Object URL.

Download the configuration file to a serverless resource group.

On the Image Management page in the DataWorks console, create a custom image and download the configuration file to a serverless resource group.

Create an image.

Log on to the DataWorks console. In the top navigation bar, select a desired region. In the left-side navigation pane, click Image Management. On the Image Management page, click the Custom Images tab. Then, click Create Image. In the Create Image panel, configure parameters. The following table describes the parameters.

Parameter	Description
Image Name	The custom command name.
Image Description	The custom command description.
Reference Type	Select DataWorks Official Image. This is the default value.
Image Namespace	Select DataWorks Default. This is the default value.
Image Repository	Select DataWorks Default. This is the default value.
Image Name/ID	Select `dataworks_shell_task_pod:xxx`, which is a DataWorks official image available for Shell nodes.
Visible Scope	Select a scope in which the custom image is visible based on your business requirements. Valid values: Visible Only to Creator Visible to all
Module	Select DataStudio. This is the default value.
Supported Task Type	Select Shell. This is the default value.
Installation Package	Select `Script` from the Installation Package drop-down list. Then, enter the following command in the command box: `# Download wget 'http://<yourConfigURL>' -O /home/admin/usertools/tools/<configName>` Note Replace `http://<yourConfigURL>` with the object URL that you copied from the View Details panel in the OSS console. Replace `<configName>` with `myconfig.txt`.

Click OK.

Test and publish the image.
Click Publish in the Actions column that corresponds to the created image. In the Publish Image panel, configure the Test Resource Group parameter and click Test to the right of Test Result. After the test succeeds, click Publish.

Use ossutil based on the command configuration file.
1. Create a common Shell node.
  1. Go to the DataStudio page.
    Log on to the DataWorks console. In the top navigation bar, select a desired region. In the left-side navigation pane, choose Data Development and Governance > Data Development. On the page that appears, select a desired workspace from the drop-down list and click Go to Data Development.
  2. Create a common Shell node.
    On the DataStudio page, find the desired workflow, right-click the workflow name, and then choose Create Node > General > Shell. In the Create Node dialog box, configure the Name parameter and click Confirm.
2. On the configuration tab of the common Shell node, write node code.
```
/home/admin/usertools/tools/ossutil64 -c /home/admin/usertools/tools/myconfig.txt ls -s
```
3. Verify the result.
  In the top toolbar of the configuration tab of the common Shell node, click the icon and select the created custom image and the serverless resource group that passed the test in the image testing and publishing process to run a task on the common Shell node. The following figure shows the result.