Java SDK quick start

Steps

• Prepare the job

  • Upload the data file to OSS

  • Use the sample code

  • Compile and package the code

  • Upload the package to OSS

• Create and submit the job using the SDK

• View the results

1. Prepare the job

This job counts the occurrences of "INFO", "WARN", "ERROR", and "DEBUG" in a log file.

The job consists of three tasks: split, count, and merge.

• The split task divides the log file into three parts.

• The count task counts the occurrences in each part. This task requires an instance count of 3, meaning three instances run the count program concurrently.

• The merge task consolidates the results from the count tasks.

(1) Upload the data file to OSS

Download the data file for this example: log-count-data.txt

Upload log-count-data.txt to the following path: oss://your-bucket/log-count/log-count-data.txt

• Replace your-bucket with the name of your bucket. This quickstart assumes the region is China (Shenzhen).

(2) Use the sample code

This example uses Java to write the job tasks and Maven to compile the code. We recommend using the free Community edition of IntelliJ IDEA: https://www.jetbrains.com/idea/download/.

Download the sample program: java-log-count.zip

This is a Maven project.

(3) Compile and package

Run the following command to compile and package the code:

mvn package

This command generates the following three JAR files in the target directory:

batchcompute-job-log-count-1.0-SNAPSHOT-Split.jar
batchcompute-job-log-count-1.0-SNAPSHOT-Count.jar
batchcompute-job-log-count-1.0-SNAPSHOT-Merge.jar

Next, compress the three JAR files into a tar.gz archive with the following commands:

> cd target  # Enter the target directory.
> tar -czf worker.tar.gz *SNAPSHOT-*.jar # Package the files.

Run the following command to verify the contents of the package:

> tar -tvf worker.tar.gz
batchcompute-job-log-count-1.0-SNAPSHOT-Split.jar
batchcompute-job-log-count-1.0-SNAPSHOT-Count.jar
batchcompute-job-log-count-1.0-SNAPSHOT-Merge.jar

(4) Upload the package to OSS

Upload worker.tar.gz to your bucket in OSS: oss://your-bucket/log-count/worker.tar.gz

To run this example, create your own bucket and upload the worker.tar.gz file to it.

2. Submit the job with the SDK

(1) Create a Maven project

Add the following dependencies to your project's pom.xml file:

    <dependencies>
        <dependency>
            <groupId>com.aliyun</groupId>
            <artifactId>aliyun-java-sdk-batchcompute</artifactId>
            <version>5.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun</groupId>
            <artifactId>aliyun-java-sdk-core</artifactId>
            <version>3.2.3</version>
        </dependency>
    </dependencies>

Use the latest version of the Java SDK.

(2) Create Demo.java

When you submit a job, you must specify either a cluster ID or parameters for an auto-cluster. This example uses an auto-cluster, which requires the following two parameters:

• An available image ID. You can use a system-provided image or create a custom image.

• An instance type. For more information, see Currently supported types.

In OSS, create a path to store the program output (StdoutRedirectPath) and error logs (StderrRedirectPath). In this example, the path is oss://your-bucket/log-count/logs/.

To run this example, replace the placeholder variables in the code with your credentials and the OSS paths you created.

The following template shows how to submit a job with the Java SDK. For details about the parameters, see the SDK interface description.

Demo.java:

/*
* IMAGE_ID: The ECS image ID.
* INSTANCE_TYPE: The instance type.
* REGION_ID: The region where the job is submitted. This must be the same region as your worker package's bucket.
* ACCESS_KEY_ID: Your AccessKey ID.
* ACCESS_KEY_SECRET: Your AccessKey Secret.
* WORKER_PATH: The OSS path to your worker package.
* LOG_PATH: The OSS path for storing error logs and task output. You must create this directory in advance.
 */
 import com.aliyuncs.batchcompute.main.v20151111.*;
 import com.aliyuncs.batchcompute.model.v20151111.*;
 import com.aliyuncs.batchcompute.pojo.v20151111.*;
 import com.aliyuncs.exceptions.ClientException;

 import java.util.ArrayList;
 import java.util.List;

 public class Demo {

     static String IMAGE_ID = "img-ubuntu";;  // Enter your ECS image ID.
     static String INSTANCE_TYPE = "ecs.sn1.medium"; // Enter an instance type that is available in your region.

     static String REGION_ID = "cn-shenzhen";   // Enter the region ID.
     static String ACCESS_KEY_ID = "";  // Enter your AccessKey ID.
     static String ACCESS_KEY_SECRET = ""; // Enter your AccessKey Secret.
     static String WORKER_PATH = ""; // Enter the OSS path of your worker.tar.gz file, for example, "oss://your-bucket/log-count/worker.tar.gz".
     static String LOG_PATH = ""; // Enter the OSS path for logs, for example, "oss://your-bucket/log-count/logs/".
     static String MOUNT_PATH = ""; // For example, "oss://your-bucket/log-count/".

     public static void main(String[] args){

         /** Create a BatchCompute client. */
         BatchCompute client = new BatchComputeClient(REGION_ID, ACCESS_KEY_ID, ACCESS_KEY_SECRET);

         try{

             /** Construct a JobDescription object. */
             JobDescription jobDescription = genJobDescription();

             // Create the job.
             CreateJobResponse response = client.createJob(jobDescription);

             // After the job is created, the job ID is returned.
             String jobId = response.getJobId();

             System.out.println("Job created successfully. Job ID: "+jobId);


             // Query the job status.
             GetJobResponse getJobResponse = client.getJob(jobId);

             Job job = getJobResponse.getJob();

             System.out.println("Job state:"+job.getState());

         } catch (ClientException e) {
             e.printStackTrace();

             System.out.println("Job creation failed. Error code: "+ e.getErrCode()+", Error message: "+e.getErrMsg());
         }
     }

     private static JobDescription genJobDescription(){



         JobDescription jobDescription = new JobDescription();

         jobDescription.setName("java-log-count");
         jobDescription.setPriority(0);
         jobDescription.setDescription("log-count demo");
         jobDescription.setJobFailOnInstanceFail(true);
         jobDescription.setType("DAG");

         DAG taskDag = new DAG();


         /** Add the split task. */

         TaskDescription splitTask =  genTaskDescription();
         splitTask.setTaskName("split");
         splitTask.setInstanceCount(1);
         splitTask.getParameters().getCommand().setCommandLine("java -jar batchcompute-job-log-count-1.0-SNAPSHOT-Split.jar");
         taskDag.addTask(splitTask);

         /** Add the count task. */
         TaskDescription countTask =  genTaskDescription();
         countTask.setTaskName("count");
         countTask.setInstanceCount(3);
         countTask.getParameters().getCommand().setCommandLine("java -jar batchcompute-job-log-count-1.0-SNAPSHOT-Count.jar");
         taskDag.addTask(countTask);

         /** Add the merge task. */
         TaskDescription mergeTask =  genTaskDescription();
         mergeTask.setTaskName("merge");
         mergeTask.setInstanceCount(1);
         mergeTask.getParameters().getCommand().setCommandLine("java -jar batchcompute-job-log-count-1.0-SNAPSHOT-Merge.jar");
         taskDag.addTask(mergeTask);



         /** Add task dependencies:  split-->count-->merge  */

         List<String> taskNameTargets = new ArrayList();
         taskNameTargets.add("merge");
         taskDag.addDependencies("count", taskNameTargets);

         List<String> taskNameTargets2 = new ArrayList();
         taskNameTargets2.add("count");
         taskDag.addDependencies("split", taskNameTargets2);

         //dag
         jobDescription.setDag(taskDag);

         return jobDescription;
     }

     private static TaskDescription genTaskDescription(){

         AutoCluster autoCluster = new AutoCluster();
         autoCluster.setInstanceType(INSTANCE_TYPE);
         autoCluster.setImageId(IMAGE_ID);
         //autoCluster.setResourceType("OnDemand");

         TaskDescription task = new TaskDescription();
         //task.setTaskName("Find");

        // If you use a VPC, you must configure the cidrBlock. Make sure that the IP address range does not conflict with other ranges.
        Configs configs = new Configs();
        Networks networks = new Networks();
        VPC vpc = new VPC();
        vpc.setCidrBlock("192.168.0.0/16");
        networks.setVpc(vpc);
        configs.setNetworks(networks);
        autoCluster.setConfigs(configs);

         // The full OSS path of the uploaded job package.
         Parameters p = new Parameters();
         Command cmd = new Command();
         //cmd.setCommandLine("");
         // The full OSS path of the uploaded job package.
         cmd.setPackagePath(WORKER_PATH);
         p.setCommand(cmd);
         // The path for storing error logs.
         p.setStderrRedirectPath(LOG_PATH);
         // The path for storing the final output.
         p.setStdoutRedirectPath(LOG_PATH);

         task.setParameters(p);
         task.addInputMapping(MOUNT_PATH, "/home/input");
         task.addOutputMapping("/home/output",MOUNT_PATH);

         task.setAutoCluster(autoCluster);
         //task.setClusterId(clusterId);
         task.setTimeout(30000); /* 30,000 seconds */
         task.setInstanceCount(1); /** Run on one instance. */

         return task;
     }
 }

Sample output:

Job created successfully. Job ID: job-01010100010192397211
Job state:Waiting

3. Check the job status

You can check the job status by calling the Get job information method in the SDK:

//Query the job status.
GetJobResponse getJobResponse = client.getJob(jobId);
Job job = getJobResponse.getJob();
System.out.println("Job state:"+job.getState());

The job state can be Waiting, Running, Finished, Failed, or Stopped.

4. View the results

You can sign in to the BatchCompute console to view the job status.

After the job is complete, sign in to the OSS console and view the /log-count/merge_result.json file in your bucket.

The file content is as follows:

{"INFO": 2460, "WARN": 2448, "DEBUG": 2509, "ERROR": 2583}

You can also use the OSS SDK to retrieve the results.