This topic describes how to use SDK for Java to create clusters and jobs.

Note Alibaba Cloud provides OpenAPI Explorer to simplify API usage. You can use OpenAPI Explorer to search for API operations, call API operations, and dynamically generate SDK sample code.

Before you begin

Use Alibaba Cloud E-MapReduce (EMR) OpenAPI SDK for Java in your Eclipse project.
  • Create a Maven project and add Maven dependencies.
     <dependency>
                <groupId>com.aliyun</groupId>
                <artifactId>aliyun-java-sdk-core</artifactId>
                <version>2.3.9</version>
           </dependency>
           <dependency>
                <groupId>com.aliyun</groupId>
                <artifactId>aliyun-java-sdk-emr</artifactId>
                <version>2.2.2</version>
      </dependency>
  • Download the required JAR files. Use Eclipse as an example.
    1. Download the JAR files.
    2. Copy the downloaded JAR files to your project folder.
    3. Right-click the name of your project in Eclipse and choose Properties > Java Build Path > Add JARs.
    4. Select the JAR files you copied in Step 2.

Initialize a client

IClientProfile profile = DefaultProfile.getProfile("cn-hangzhou", "<Your-AccessKeyId>", "<Your-AccessKeySecret>");
DefaultAcsClient client = new DefaultAcsClient(profile);

Sample code

  • Manage clusters
    • Create a cluster
      public static void main(String[] args) {
            IClientProfile profile = DefaultProfile.getProfile("cn-hangzhou", "<Your-AccessKeyId>", "<Your-AccessKeySecret>");
            DefaultAcsClient client = new DefaultAcsClient(profile);
            final CreateClusterRequest request = new CreateClusterRequest();
            request.setName("Your-Cluster-Name");
            request.setSecurityGroupId("Your-Security-Group-Id"); // If no security group ID is specified, a new security group with a given name is created.
            request.setAutoRenew(false);
            request.setChargeType("PostPaid"); // Specify the billing method. Pay-as-you-go is used in this example.
            request.setClusterType("HADOOP"); // Specify the cluster type.
            request.setEmrVer("EMR-1.3.0"); // Specify the EMR version.
            request.setIsOpenPublicIp(true);
            request.setLogEnable(true);
            request.setLogPath("oss://Your-Bucket/Your-Folder");
            request.setMasterPwdEnable(true); // Enable the use of a password for the master node.
            request.setMasterPwd("Aa123456789"); // Specify the password of the master node.
            request.setZoneId("cn-hangzhou-b"); // Specify the zone ID.
             // The available hardware configurations, such as ECS instance types and cloud disk types, are determined by the specified I/O optimization parameter, ECS instance series, and network type. For more information, visit the ECS buy page.
            request.setIoOptimized(true); // Set this parameter to true to enable I/O optimization.
            request.setInstanceGeneration("ecs-2"); // Set this parameter to ecs-2. Valid values: ecs-1 and ecs-2.
            request.setNetType("vpc"); // Specify the network type.
            request.setVpcId("your-vpcId");
            request.setVSwitchId("your-switchId");
            List<CreateClusterRequest.EcsOrder> ecsOrders = new ArrayList<CreateClusterRequest.EcsOrder>();
            CreateClusterRequest.EcsOrder masterOrder = new CreateClusterRequest.EcsOrder();
            masterOrder.setIndex(1);
            masterOrder.setDiskCapacity(50);
            masterOrder.setDiskCount(2);
            masterOrder.setDiskType("CLOUD_EFFICIENCY"); // Specify the disk type.
            masterOrder.setInstanceType("ecs.n1.large"); // Specify the ECS instance type.
            masterOrder.setNodeCount(1);
            masterOrder.setNodeType("MASTER"); // Specify the node type. The master node is used in this example.
            ecsOrders.add(masterOrder);
            CreateClusterRequest.EcsOrder coreOrder = new CreateClusterRequest.EcsOrder();
            coreOrder.setIndex(2);
            coreOrder.setDiskCapacity(50);
            coreOrder.setDiskCount(4);
            coreOrder.setDiskType("CLOUD_EFFICIENCY");
            coreOrder.setInstanceType("ecs.n1.large");
            coreOrder.setNodeCount(3);
            coreOrder.setNodeType("CORE");
            ecsOrders.add(coreOrder);
            request.setEcsOrders(ecsOrders);
            try {
                CreateClusterResponse response = client.getAcsResponse(request);
                String clusterId = response.getClusterId(); // cluster id
                // Perform operations on the cluster.
            } catch (Exception e) {
      
            }
        }
      • When you create a cluster, you must specify a security group for the cluster. If you do not specify the ID of a security group, you must specify a security group name. This way, a security group is automatically created during cluster creation.
      • Specify whether to enable high availability. For more information, see hardware configurations in Create a cluster.
        request.setHighAvailabilityEnable(true);
      • Configure optional software components. For more information, see software configurations in Create a cluster.
        List<String> soft = new ArrayList<String>();
        soft.add("presto");
        soft.add("oozie");
        request.setOptionSoftWareLists(soft);
      • Specify other configuration items. For more information, see Customize software configurations.
        request.setConfigurations("oss://your-bucket/your-conf.json");
      • Configure bootstrap actions. For more information, see Bootstrap actions.
        List<CreateClusterRequest.BootstrapAction> bootstrapActionLists = new ArrayList<CreateClusterRequest.BootstrapAction>();
        CreateClusterRequest.BootstrapAction bootstrapActionList = new CreateClusterRequest.BootstrapAction();
        bootstrapActionList.setName("bootstrapName");
        bootstrapActionList.setPath("oss://emr-agent-pack/bootstrap/run-if.py");
        bootstrapActionList.setArg("instance.isMaster=true mkdir -p /tmp/abc");
        bootstrapActionLists.add(bootstrapActionList);
        request.setBootstrapActions(bootstrapActionLists);
    • Query the details of a cluster
      public static void main(String[] args) {
            IClientProfile profile = DefaultProfile.getProfile("cn-hangzhou", "<Your-AccessKeyId>", "<Your-AccessKeySecret>");
            DefaultAcsClient client = new DefaultAcsClient(profile);
            final DescribeClusterRequest request = new DescribeClusterRequest();
            request.setId("C-XXXXXXXXXXXXXXXX"); // Specify the ID of the cluster.
            try {
                DescribeClusterResponse response = client.getAcsResponse(request);
                DescribeClusterResponse.ClusterInfo clusterInfo = response.getClusterInfo();
                // Perform operations on the cluster.
            } catch (Exception e) {
      
            }
        }
    • Query the list of clusters
      public static void main(String[] args) {
                IClientProfile profile = DefaultProfile.getProfile("cn-hangzhou", "<Your-AccessKeyId>", "<Your-AccessKeySecret>");
                DefaultAcsClient client = new DefaultAcsClient(profile);
                final ListClustersRequest request = new ListClustersRequest();
                request.setPageNumber(1);
                request.setIsDesc(true);
                request.setPageSize(20);
                try {
                    ListClustersResponse response = client.getAcsResponse(request);
                    List<ListClustersResponse.ClusterInfo> clusterInfos = response.getClusters();
                    for (ListClustersResponse.ClusterInfo clusterInfo : clusterInfos) {
                        // Perform operations on the cluster.
                    }
                } catch (Exception e) {
      
                }
            }
    • Release a cluster
      public static void main(String[] args) {
                  IClientProfile profile = DefaultProfile.getProfile("cn-hangzhou", "<Your AccessKeyId>", "<Your AccessKeySecret>");
                  DefaultAcsClient client = new DefaultAcsClient(profile);
                  ReleaseClusterRequest request = new ReleaseClusterRequest();
                  request.setId("C-XXXXXXXXXXXXXXXX"); // Specify the ID of the cluster.
                  try {
                      ReleaseClusterResponse response = client.getAcsResponse(request);
                  } catch (Exception e) {
      
                  }
              }
  • Manage jobs
    • Create a job
      public static void main(String[] args) {
            IClientProfile profile = DefaultProfile.getProfile("cn-hangzhou", "<Your-AccessKeyId>", "<Your-AccessKeySecret>");
            DefaultAcsClient client = new DefaultAcsClient(profile);
            final CreateJobRequest request = new CreateJobRequest();
            request.setName("Your-Job-Name");
            request.setRunParameter("--master yarn-client --driver-memory 4g --executor-memory 4g --executor-cores 2 --num-executors 4 --class com.test.RemoteDebug ossref://Your-Bucket/Resource.jar 1000\"");
            request.setFailAct("CONTINUE"); // Continue the job.
            request.setType("SPARK"); // Specify the job type. 
      try {
                  CreateJobResponse response = client.getAcsResponse(request);
                  String jobId = response.getId();
              } catch (Exception e) {
      
              }
          }
    • Delete a job
      Notice If a job is used by a workflow, it cannot be deleted. You must delete or modify the workflow first.
      public static void main(String[] args) {
                IClientProfile profile = DefaultProfile.getProfile("cn-hangzhou", "<Your-AccessKeyId>", "<Your-AccessKeySecret>");
                DefaultAcsClient client = new DefaultAcsClient(profile);
                final DeleteJobRequest request = new DeleteJobRequest();
                request.setId("J-XXXXXXXXXXXXXXXX"); // Specify the ID of the job.
                try {
                    DeleteJobResponse response = client.getAcsResponse(request);
                } catch (Exception e) {
      
                }
            }