This topic describes how to obtain an SDK and use it to submit a Spark job in Data Lake Analytics (DLA).

Obtain an SDK

You can obtain the latest DLA SDK package from the Maven repository.

Use the SDK to submit a Spark job

  1. Obtain the AccessKey pair of your Alibaba Cloud account. For more information, see Obtain an AccessKey pair.
  2. Obtain the ID of the region where DLA is deployed. For more information about region IDs, see Regions and zones.
  3. Determine the name of the virtual cluster and the job content in the JSON format. You can submit the Spark job in the DLA console and then determine the input and output content based on the information shown in the following figure. 12
    Sample code for using an SDK to submit a Spark job:
     * Submit a job to the serverless Spark engine of DLA.
     * @param regionId           The ID of the region where DLA is deployed.
     * @param accessKeyId        The AccessKey ID of your Alibaba Cloud account.
     * @param accessKeySecret    The AccessKey secret of your Alibaba Cloud account.
     * @param virtualClusterName The name of the virtual cluster in DLA.
     * @param jobConfig          The JSON file that describes the Spark job that you want to submit.
     * @return Spark JobId, The ID of the job that is returned after the job is submitted. The ID is used to monitor the status of the job.
     * @throws ClientException An exception is returned due to issues, such as network errors.
    public String submitSparkJob(String regionId,
                                 String accessKeyId, 
                                 String accessKeySecret, 
                                 String virtualClusterName, 
                                 String jobConfig) throws ClientException {
        // Initialize the ACS client.
        DefaultProfile profile = DefaultProfile.getProfile(regionId, accessKeyId, accessKeySecret);
        IAcsClient client = new DefaultAcsClient(profile);
        // Initialize the request by specifying the cluster name and job content.
        SubmitSparkJobRequest request = new SubmitSparkJobRequest();
        // Submit the job and return the ID of the Spark job.
        SubmitSparkJobResponse response = client.getAcsResponse(request);
        return response.getJobId();
    Notice JobConfig is a valid JSON string. We recommend that you run a small number of jobs in the DLA console, and then use the SDK to automate the submission of core services.