how to configure an ecs remote command node - Data Management

When your DMS task flow needs to run custom scripts — such as Python data-processing jobs, Hadoop programs, or email-report generators — directly on an Elastic Compute Service (ECS) instance, add an ECS remote commands node. The node uses Cloud Assistant to execute shell, PowerShell, or batch scripts without requiring SSH access, and can pass JSON output to downstream nodes as variables.

Scenarios

Process data by using advanced tools, including the NumPy and scikit-learn libraries for Python, and the MLlib library of Apache Spark. The models generated after data processing can be applied to the fine sort and recommendation features of a search system.
Consume data. For example, you can generate an Excel script when you read data or generate a script that is used to automatically send emails that contain the data you read.
Call a self-built Hadoop MapReduce or Apache Spark program.

Prerequisites

Before you begin, ensure that you have:

An ECS instance. See Create an instance on the Custom Launch tab.
A tag on the ECS instance with key dms and value script-for-dms. See Modify the tags of an instance.
The Cloud Assistant agent installed on the ECS instance. See Install the Cloud Assistant Agent.

Note
ECS instances created from public images after December 1, 2017 have the Cloud Assistant agent pre-installed.
The ecs:InvokeCommand permission granted to your RAM user. To grant this permission:
1. Create a custom RAM policy using the JSON below. See Create a custom policy. The policy statement is as follows:
```
{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:InvokeCommand"
      ],
      "Resource": [
        "*"
      ],
      "Condition": {}
    }
  ]
}
```
2. Attach the policy to your RAM user. See Grant permissions to the RAM user.

Add and configure the node

Log in to DMS 5.0.
In the top navigation bar, choose DTS > Data Development > Task Orchestration.

Note
If the console is in simple mode, click the menu icon in the upper-left corner, choose All functions, then in the navigation pane choose DTS > Data Development > Task Orchestration.
Click the name of the task flow you want to edit.

Note
To create a task flow first, see Overview.
In the Task Type list on the left side of the canvas, drag the ECS remote commands node type to a blank area of the canvas.
Double-click the ECS remote commands node on the canvas to open its configuration page.

Set the following parameters.

Note

The ECS Instance ID dropdown only shows instances tagged with key dms and value script-for-dms. If your instance is not listed, verify that the tag key and value are set correctly on the instance.

Category	Parameter	Required	Description
Basic configuration	Region	Yes	The region where your ECS instance resides.
	ECS Instance ID	Yes	The ECS instance on which to run the commands. Select from the dropdown. You can select multiple instances, but they must all run the same operating system.
	Command Type	Yes	The script type: Shell (Linux) for Linux instances, Bat (Windows) or PowerShell (Windows) for Windows instances.
	Timeout	Yes	How long, in seconds, to wait before the command times out.
	Working Directory	Yes	The directory on the ECS instance where the commands run. If left blank, the default is `/root` for Linux and `C:\Windows\System32` for Windows.
Advanced settings	User	Yes	The OS user account under which the commands run. If you specify a non-root or non-system user, grant that user the required permissions first. See Run Cloud Assistant commands as a regular user.
Advanced settings	Password Name for Windows	Yes	The password for the user account on a Windows instance. See Run Cloud Assistant commands as a regular user.
Command settings	Command Settings	Yes	The commands to run on the ECS instance. For example: `python /home/admin/hello.py`.

Optional: Configure output variables so that downstream nodes can reference results from this node.
1. Click the Variable Setting tab in the right-side navigation pane.
2. Click the Output Variables tab.
3. Click Increase Variable.
4. In the Variable Name field, enter a name for the variable. See Variables for naming rules.
Note
For each selected ECS instance, if the last line of the command output is valid JSON, its keys become available as output variables. For example, if instance A outputs {"a":"hello"} and instance B outputs {"b":"world"}, both a and b are available as output variables in downstream nodes.
Click Try Run in the top toolbar to test the node.
- If the last line of the logs shows SUCCEEDED, the node ran successfully.
- If the last line shows FAILED, click Log in the top toolbar to view the full output. Identify the error, update the configuration, and run the node again.