ds-controller is a command tool that you can use to create, release, and scale an E-MapReduce (EMR) cluster. You can also use ds-controller to manage resources related to an EMR cluster, such as the virtual private cloud (VPC), vSwitch, and security group to which the EMR cluster belongs. ds-controller runs in command-line interface (CLI) mode or service mode.
Prerequisites
An EMR Data Science cluster is created. For more information, see Create a cluster.
Background information
- CLI mode: After you install ds-controller, you can run commands in the Data Science
cluster to manage resources.
For more information about use examples, see Run ds-controller in CLI mode.
- Service mode: After you install ds-controller, you can deploy ds-controller on an
ECS instance. Then, you can use ds-controller to manage resources by calling RESTful
APIs.
For more information about use examples, see Run ds-controller in service mode.
Install ds-controller
Run ds-controller in CLI mode
Run ds-controller in service mode
- Common mode
./ds-controller
- Docker-based mode
docker pull registry.cn-huhehaote.aliyuncs.com/pai-recommend2/ds-controller:0.0.1 # No configuration file is available. docker run -it -d -p 8080:8080 registry-vpc.cn-huhehaote.aliyuncs.com/pai-recommend2/ds-controller:0.0.1 # The local configuration file is used. docker run -it -d -p 8080:8080 -v /root/.ds/:/go/app/conf registry-vpc.cn-huhehaote.aliyuncs.com/pai-recommend2/ds-controller:0.0.1
- Manage an EMR cluster
- Create an EMR Data Science cluster.
curl -v -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/createclusterv2" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "EmrVer": "EMR-3.31.0", "ZoneId": "cn-huhehaote-a", "ClusterName": "<dataScience-test>", "ClusterType": "DATA_SCIENCE", "RegionId": "cn-huhehaote", "Master_NodeCount": "1", "Master_InstanceType": "ecs.g5.4xlarge", "Master_DiskType": "CLOUD_EFFICIENCY", "Master_DiskCapacity": "80", "Master_DiskCount": "1", "Master_SysDiskType": "CLOUD_ESSD", "Master_SysDiskCapacity": "120", "Core_NodeCount": "2", "Core_InstanceType": "ecs.g5.4xlarge", "Core_DiskType": "CLOUD_EFFICIENCY", "Core_DiskCapacity": "120", "Core_DiskCount": "4", "Core_SysDiskType": "CLOUD_ESSD", "Core_SysDiskCapacity": "120", "Task_NodeCount": "1", "Task_InstanceType": "ecs.g5.4xlarge", "Task_DiskType": "CLOUD_EFFICIENCY", "Task_DiskCapacity": "120", "Task_DiskCount": "4", "Task_SysDiskType": "CLOUD_ESSD", "Task_SysDiskCapacity": "120", "VSwitchId": "vsw-hp3antge3fe8p3e2s****", "SecurityGroupId": "sg-hp3hk9cues1zuyzn****", "SecurityGroupName": "emr-test", "ChargeType": "PostPaid", "VpcId":"vpc-hp3wsdsed3enmxdvi****", "NetType": "vpc", "MasterPwd": "EMRtest1234!", "IsOpenPublicIp": true, "SshEnable": true, "IoOptimized": true, "HighAvailabilityEnable": false }'
- Query the basic information about a cluster.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/describeclusterbasicinfo" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****" }'
- Query the information about a cluster.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/listclusterhostgroup" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****" }'
- Query the clusters that are in the IDLE state.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/listclusters" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterType": "DATA_SCIENCE", "Status": "IDLE" }'
- Query the EMR version.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/listemrmainversion" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "PageNumber": 1, "PageSize": 10 }'
- Modify the name of a cluster.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/modifyclustername" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****", "Name": "<dataScience-test-new>" }'
- Release a cluster.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/releasecluster" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****" }'
- Scale out a cluster. For example, you can add task nodes to the cluster.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/resizeclusterv2" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****", "HostGroupType": "TASK", "NodeCount": "1", "InstanceType": "ecs.g5.4xlarge", "DiskType": "CLOUD_EFFICIENCY", "DiskCapacity": "80", "DiskCount": "1", "SysDiskType": "CLOUD_ESSD", "SysDiskCapacity": "120", "VSwitchId": "vsw-hp3antge3fe8p3e2s****", "ChargeType": "PostPaid" }'
- Release nodes in a cluster by ECS instance group ID. For example, you can release
nodes in the cluster by the ID of a task group.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/releaseclusterhostgroup" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****", "HostGroupId": "G-08DBEB063DD4****" }'
- Release nodes in a cluster by ECS instance group type. For example, you can release
nodes in the cluster by the task group type.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/releaseclusterhostgroupbytype" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****", "HostGroupType": "TASK" }'
- Create an EMR Data Science cluster.
- Manage an ECS instance
- Apply for a public IP address.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/allocatepublicipaddress" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****" }'
- Add a security group rule. For example, add Port 22.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/authorizesecuritygroup" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "SecurityGroupId": "sg-hp3hk9cues1zuyzn****", "IpProtocol": "TCP", "PortRange": "22/22", "SourceCidrIp": "0.0.0.0/0" }'
- Create an ECS instance.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/createinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceType": "ecs.g6.large", "ImageId": "ubuntu_18_04_64_20G_alibase_20190624.vhd", "SecurityGroupId": "sg-hp3hk9cues1zuyzn****", "InstanceName": "<emr_ds_insname>", "InternetChargeType": "PayByTraffic", "Password": "EMRtest1234!", "ZoneId": "cn-huhehaote-a", "SystemDiskSize": 80, "SystemDiskCategory": "cloud_efficiency", "VSwitchId": "vsw-hp3antge3fe8p3e2s****", "InstanceChargeType": "PostPaid", "InternetMaxBandwidthIn": 1, "InternetMaxBandwidthOut": 1 }'
In the examples provided in this topic, <emr_ds_insname> indicates the name of an ECS instance.
- Create a security group.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/createsecuritygroup" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "SecurityGroupName": "<emr_ds_groupname>", "SecurityGroupType": "normal", "VpcId": "vpc-hp3wsdsed3enmxdvi****" }'
In the examples provided in this topic, <emr_ds_groupname> indicates the name of a security group.
- Delete an ECS instance.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/deleteinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****", "Force": true }'
- Query your resources in the current region.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeavailableresource" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "DestinationResource": "InstanceType" }'
- View image resources.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeimages" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "PageNumber": 1, "PageSize": 100 }'
- Query the information about ECS instances.
- Query the information about a single ECS instance.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeecsinstances" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "[\"i-hp3h472yhby69kwh****\"]" }'
- Query the information about multiple ECS instances. Separate IDs of multiple ECS instances
with commas (,).
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeecsinstances" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "[\"i-hp3h472yhby69kwh****\"],...,[\"i-hp2h472yhby69kwh****\"]" }'
- Query the information about a single ECS instance.
- Query available regions.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeregions" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- Query the quotas of ECS resources that you can create in the current region. The resources
that you can query include the numbers of security groups, elastic network interfaces
(ENIs), pay-as-you-go instance vCPUs, preemptible instance vCPUs, and dedicated hosts,
and the total capacity quota of pay-as-you-go disks. You can also query the information
such as the network type or whether real-name verification is passed by an account.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeaccountattributes" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- Query a list of zones that are provided by Alibaba Cloud. This command also returns
some information about the resources that are supported in each zone.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeregions" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- Modify the bandwidth configuration of an ECS instance.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/modifyinstancenetworkspec" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InternetMaxBandwidthOut": 6, "InternetMaxBandwidthIn": 6, "NetworkChargeType": "PayByTraffic", "InstanceId": "i-hp3h472yhby69kwh****" }'
- Restart an ECS instance.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/rebootinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****" }'
- Start an ECS instance.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/startinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****" }'
- Stop an ECS instance.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/stopinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****" }'
- Query the metadata information about an ECS instance, such as the hostname, network
type, MAC address, region, zone, and image ID. The metadata information is used to
identify the current environment.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/metadata" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "Path": "/mac" }'
- Apply for a public IP address.
- Manage a VPC
- Create a VPC.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/createvpc" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "CidrBlock": "192.168.0.0/16", "VpcName": "vpc-hp3wsdsed3enmxdvi****", "EnableIpv6": false }'
- Create a vSwitch.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/createvsw" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ZoneId": "cn-huhehaote-a", "CidrBlock": "192.168.0.0/18", "VSwitchName": "<emr_ds_vswname>", "VpcId": "vpc-hp3wsdsed3enmxdvi****" }'
In the example, <emr_ds_vswname> indicates the name of a vSwitch.
- Delete a VPC.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/deletevpc" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "VpcId": "vpc-hp3wsdsed3enmxdvi****" }'
- Delete a vSwitch.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/deletevsw" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "VSwitchId": "vsw-hp3antge3fe8p3e2s****" }'
- Query the information about VPCs.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/describevpcs" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- Query the information about a vSwitch.
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/describevsws" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- Create a VPC.
- Manage an SMS message
Use SendSms to send an SMS message.
curl -v -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/sms/sendsms" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "PhoneNumbers": "186212xxxxx,186213xxxxx", "SignName":"Datascience", "TemplateCode":"SMS_22062xxxx", "TemplateParam":"clusterid:C-CB14657FAEE2****,regionid:cn-huhehaote" }'