Create, monitor, and stop Spark jobs in the Lindorm console using Lindorm Distributed Processing System (LDPS).
Prerequisites
Before you begin, make sure you have:
LDPS activated for your Lindorm instance. See Activate LDPS and modify the configurations
A developed Spark job — either a Java job or a Python job
The job file uploaded to HDFS or OSS. See Upload files in the Lindorm console
Navigate to job management
All job operations start from the same location in the console. Complete these steps before creating, viewing, or stopping a job:
Log on to the Lindorm console.
In the upper-left corner, select the region where your instance is deployed.
On the Instances page, click the instance ID or click Manage in the Actions column.
In the left-side navigation pane, click Compute Engine.
Create a Spark job
Click the Job Management tab, then click Create Job.
In the Create Job dialog box, enter a job name and select the job type.
Configure the following parameters in the job template. Keep the default values for all other parameters. If the job file is stored in OSS, add the following entries to
configs:Parameter Description tokenThe authentication token for computing resources. To get it, go to Database Connections and click the Compute Engine tab. mainResourceThe OSS or HDFS path to the JAR package or Python file for the job. mainClassThe entry-point class for a JAR Spark job. argsArguments passed to mainClass.configsSystem parameters for the Spark job. If the job file is stored in OSS, you must also configure the OSS-specific parameters listed below. Parameter Description spark.hadoop.fs.oss.endpointThe path in which the Spark job is stored in OSS. spark.hadoop.fs.oss.accessKeyIdYour AccessKey ID for OSS access. See Obtain an AccessKey pair. spark.hadoop.fs.oss.accessKeySecretYour AccessKey secret for OSS access. See Obtain an AccessKey pair. spark.hadoop.fs.oss.implSet to org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.Click Save in the upper-right corner.
Click Run in the upper-right corner.
View Spark job details
Click the Jobs tab. The tab lists all jobs with the following information:
| Field | Description |
|---|---|
| JobId | The unique identifier of the Spark job. |
| AppName | The name of the Spark job. |
| WebUI Address | The URL for the Spark Web UI. Copy the URL, open it in a browser, and log in with your LindormTable username and password to inspect job stages, executor status, and logs. See View the information about Spark jobs. |
| Status | The status of the Spark job. |
| Details | Additional information about the job status. |
| Actions | Operations available for the job. |
Your LindormTable default username and password are on the Wide Table Engine tab of the Database Connections page.
Terminate a Spark job
Only jobs in the Starting or Running state can be terminated. Terminating a job does not affect the Lindorm instance.
On the Jobs tab, find the job and click Stop Job in the Actions column.