Upload, Download & Sync OSS Buckets via ossutil in Shell Nodes - DataWorks

DataWorks Shell nodes—including common Shell nodes and EMR Shell nodes—have ossutil pre-installed at /home/admin/usertools/tools/ossutil64. No installation is required. Use it to upload and download objects, run batch operations, and manage buckets directly from your node scripts.

ossutil supports two ways to supply credentials: command-line parameters and a configuration file. Choose based on your security requirements:

Method	How credentials are supplied	Security	Best for
Command-line parameters	Passed inline with each command	Lower — credentials appear in plaintext in the script	Quick testing
Configuration file (recommended)	Stored in a file deployed to the resource group	Higher — credentials are not embedded in the script	Production workloads

Important

Passing an AccessKey pair as command-line parameters exposes your credentials in plaintext. For production use, use the configuration file method or attach a RAM role to the node. For RAM role–based access using Security Token Service (STS) temporary credentials, see What's next.

To verify that ossutil64 is available on your resource group, run:

ls -l /home/admin/usertools/tools/

The following figure shows a sample output.

Prerequisites

Before you begin, ensure that you have:

A DataWorks workspace with a Shell node or EMR Shell node
An Alibaba Cloud AccessKey pair (AccessKey ID and AccessKey secret) for a RAM user with OSS read permissions
An OSS bucket and its endpoint (for example, oss-cn-shanghai.aliyuncs.com)

Access OSS using command-line parameters

Pass the OSS endpoint and credentials directly with the ossutil64 command. Credentials are supplied at runtime — no pre-deployment setup needed.

In the Shell node editor, enter the following script. It connects to the specified OSS endpoint and lists all buckets the account can access.

Placeholder	Description	Example
`<OSSEndPoint>`	OSS endpoint for the target region	`oss-cn-shanghai.aliyuncs.com`
`<AccessKeyID>`	AccessKey ID of the RAM user	`LTAI5tXxx`
`<AccessKeySecret>`	AccessKey secret of the RAM user	`xXxXxXx`

/home/admin/usertools/tools/ossutil64 -e <OSSEndPoint> -i <AccessKeyID> -k <AccessKeySecret> ls -s
if [[ $? == 0 ]]; then
    echo "access oss success"
else
    echo "failed"
    exit 1
fi
echo "finished"

Replace the following placeholders with your actual values: For all supported parameters, see Configure ossutil.

Click the icon and select a serverless resource group to run the node. The following figure shows a sample result.

Access OSS using a configuration file

Store credentials in a configuration file and deploy it to the serverless resource group via a custom image. The ossutil64 command reads credentials from the file — no plaintext credentials in your script.

Step 1: Create the configuration file and upload it to OSS

On your local machine, create a file named myconfig.txt with the following content:
```
[Credentials]
language = EN
endpoint = <Endpoint>  # For example, oss-cn-shanghai.aliyuncs.com
accessKeyID = <AccessKeyID>
accessKeySecret = <AccessKeySecret>
```
Replace <Endpoint>, <AccessKeyID>, and <AccessKeySecret> with your OSS endpoint and the AccessKey pair of a RAM user with the required permissions. Do not enclose values in quotation marks. For parameter details, see Configure ossutil.
Upload myconfig.txt to OSS. For instructions, see Simple upload.
In the OSS console, find the uploaded file and click Details in the Actions column. Disable Use HTTPS and click Copy File URL. Save this URL — you will need it in the next step.

Step 2: Deploy the configuration file to the serverless resource group

Create a custom image that downloads the configuration file to the resource group at startup.

Log on to the DataWorks console and switch to the target region. In the left navigation pane, click Image Management. On the Custom Images tab, click Create Image and set the following parameters:

Parameter	Value
Image name	Any name you choose
Image description	Description of the custom image
Reference type	DataWorks Official Image (default)
Image namespace	DataWorks Default (default)
Image repository	DataWorks Default (default)
Image name/ID	`dataworks_shell_task_pod:xxx` — the official DataWorks image for Shell nodes
Visible scope	Visible Only to Creator or Visible to All
Using sub-products	Data Studio (default)
Supported task type	Shell (default)
Installation package	Select Script from the drop-down list, then enter the following command in the command box:

# Download the ossutil configuration file
wget 'http://<yourConfigURL>' -O /home/admin/usertools/tools/<configName>

Replace http://<yourConfigURL> with the file URL you copied from the OSS console. Set <configName> to myconfig.txt. Click Confirm to create the image.

In the Actions column for the image, click Publish. On the Publish Image page, select a Test Resource Group and click Test next to Test Result. After Test Succeeded appears, click Publish.
Click the icon in the Actions column and select Modify Workspace. Bind the image to the target workspace.

Step 3: Run the Shell node with the configuration file

Log on to the DataWorks console and switch to the target region. In the left navigation pane, choose Data Development and O&M > Data Studio. Select the target workspace and click Go to Data Studio.
In Data Studio, right-click the target business flow, choose Create Node, and select Shell in the General folder.
In the Shell node editor, enter the following script. It connects to the OSS endpoint in myconfig.txt and lists all buckets the account can access.
```
/home/admin/usertools/tools/ossutil64 -c /home/admin/usertools/tools/myconfig.txt ls -s
```
Click the icon. Select the serverless resource group you used during image testing and specify the custom image you created. Run the node. The following figure shows a sample result.
On the Scheduling Configuration tab, go to the Resource Property section. Select the target resource group and specify the custom image. Save the configuration, commit the node to the development environment, and publish it to the production environment.

What's next

In the latest version of Data Studio, you can attach a RAM role to a Shell node. The node then uses Security Token Service (STS) to obtain temporary security credentials automatically — no AccessKey pair required in your script or configuration file. For details, see Configure a node to use a specific RAM role.