DataWorks:Use ossutil in Shell nodes

Overview

DataWorks Shell nodes, including common Shell nodes and EMR Shell nodes, natively integrate the Alibaba Cloud Object Storage Service (OSS) command-line tool, ossutil. You can configure ossutil to access OSS by providing credentials in a configuration file or as command-line parameters. For more information about how to configure ossutil, see Configure ossutil.

Using command-line parameters

1. In the Shell node, edit the node content. The following sample code uses ossutil to connect to a specified OSS endpoint and lists all buckets the current account can access.

   /home/admin/usertools/tools/ossutil64 -e <OSSEndPoint> -i <AccessKeyID> -k <AccessKeySecret> ls -s 
   if [[ $? == 0 ]];then
       echo "access oss success"
   else
       echo "failed"
       exit 1
   fi
   echo "finished"

   Note

   You must replace the OSSEndPoint, AccessKeyID, and AccessKeySecret placeholders with your actual values. For more information about the parameters, see Configure ossutil.

   Important

   Passing an access key via command-line parameters exposes the key, creating a security risk.

2. Verify the result.

   Click the icon, select a serverless resource group, and then run the common Shell node task.

Using a configuration file

1. Create an ossutil configuration file and upload it to OSS.

   1. Create a configuration file.

      Create a file named myconfig.txt on your local machine. The following sample code shows the file content:

      [Credentials]
      language = CH
      endpoint = <Endpoint> # For example, oss-cn-shanghai.aliyuncs.com
      accessKeyID = <AccessKeyID>
      accessKeySecret = <AccessKeySecret>

      Note

      Replace the Endpoint, AccessKeyID, and AccessKeySecret placeholders with your actual Alibaba Cloud OSS endpoint and the access key of a RAM user with the necessary permissions. Note: Do not enclose the parameter values in quotation marks. For more information about the parameters, see Configure ossutil.

   2. Upload the configuration file.

      Upload the myconfig.txt file to OSS. For more information, see Simple upload.

   3. Obtain the download URL.

      In the Operation column of the uploaded object, click Details. In the panel that appears, turn off Use HTTPS and then click Copy Object URL.

2. Download the configuration file to a serverless resource group.

   On the Image Management page of the DataWorks console, create a custom image to download the configuration file to a serverless resource group.

   1. Create an image.

      Log on to the DataWorks console and switch to the target region. In the left-side navigation pane, click Image Management. On the Image Management page, click the Custom Image tab, and then click Create Image. In the panel that appears, configure the parameters as follows:

      Parameter	Example value
      Image Name	Enter a custom name for the image.
      Image description	Enter a custom description for the image.
      Reference type	DataWorks official image (default).
      Image namespace	DataWorks default (default).
      Image repository	DataWorks default (default).
      Image name/ID	Select dataworks_shell_task_pod:xxx. This is the official DataWorks image for Shell nodes.
      Visibility Scope	Select a scope based on your needs: Visible only to creator Visible to all
      Module	Data Development (default).
      Supported task type	Shell (default).
      Installation package	From the Installation package drop-down list, select Script. Then, enter the following command in the command box: # Download wget 'http://<yourConfigURL>' -O /home/admin/usertools/tools/<configName> Note Replace http://<yourConfigURL> with the object URL you copied from the OSS details panel. Set <configName> to the name of the file to download from OSS, such as myconfig.txt.

      Click OK to create the image.

   2. Test and publish the image.

      Click Deploy in the Operation column for the image you created. On the Publish Image page, select Test Resource Group, and click Test to the right of Test Result. After the Succeeded. message is displayed, click the Deploy button at the bottom.

   3. In the Actions column for the published custom image, click the icon and select Change Workspace. Bind the image to the target workspace.

3. Use the ossutil command-line tool with the configuration file.

   1. Create a common Shell node.

      1. Go to the Data Development page.

         Log on to the DataWorks console, switch to the target region, and then choose Data Development and O&M > Data Studio in the left-side navigation pane. From the drop-down list, select the target workspace and click Go to DataStudio.

      2. Create a common Shell node.

         On the Data Development page, right-click the target business process, choose Create Node, and select Shell under the General directory.

   2. In the Shell node, edit the node content. The following sample code uses ossutil to connect to the OSS endpoint specified in myconfig.txt and lists all buckets that the account specified in myconfig.txt can access.

      /home/admin/usertools/tools/ossutil64 -c /home/admin/usertools/tools/myconfig.txt ls -s

   3. Click the icon, select the serverless resource group you used for testing, specify the custom image you created and published, and then run the common Shell node task.

   4. On the Scheduling tab of the node, select the target resource group in the Resource Group section, specify the custom image that you created and published, save the scheduling configuration, commit the Shell node to the development environment, and then publish it to production.

Related documents

In the new version of Data Development, you can bind a RAM role to a Shell node and use STS to dynamically obtain temporary security credentials to securely access OSS by using ossutil. For more information, see Associate roles: Securely access other cloud resources.