Manage jobs using DMS

Prerequisites

• You have activated Data Management (DMS).

• Lindorm Distributed Processing System (LDPS) is activated. For more information, see Activate the service.

• You have developed a job. For more information, see JAR job development practices or Python job development practices.

• You have uploaded the job file to HDFS or OSS. For information about how to upload a file to HDFS, see Upload files using the console.

Create a Lindorm Spark task flow

1. Log on to the DMS console V5.0.

2. Go to the Task Orchestration page.

   • Simple mode:

     1. In the Scene Guide section, click Data Transmission and Processing (DTS).

     2. In the Data Processing section on the right, click the Task Orchestration card.

   • Normal mode: In the top navigation bar, choose Integration and Development (DTS) > Data Development > Task Orchestration.

3. On the Task Orchestration page, click Create Task Flow.

4. In the Create Task Flow dialog box, enter a Task Flow Name and Description, and then click Confirm.

5. From the Task Type list on the left, drag a Lindorm Spark task node to the canvas. Connect the nodes to define the dependencies between them.

6. Configure the Lindorm Spark task node.

   1. Double-click the Lindorm Spark task node, or click the Lindorm Spark task node and select .

   2. On the node configuration page, set the basic parameters and custom parameters for the job.

      • In the Basic Configurations section, set the following basic parameters.

        Parameter	Description
        Region	Select the region where the target Lindorm instance is located.
        Lindorm Instance	Select the ID of the target Lindorm instance.
        Task Type	Select a Spark job type. The following types are supported: JAR Python

      • In the Job Configuration section, configure the custom parameters for running the job. The following sections provide configuration templates and parameter descriptions for different Spark job types.

        • JAR job configuration template and parameter descriptions:

          {
            "mainResource" : "oss://path/to/your/file.jar",
            "mainClass" : "path.to.main.class",
            "args" : [ "arg1", "arg2" ],
            "configs" : {
              "spark.hadoop.fs.oss.endpoint" : "",
              "spark.hadoop.fs.oss.accessKeyId" : "",
              "spark.hadoop.fs.oss.accessKeySecret" : "",
              "spark.hadoop.fs.oss.impl" : "org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem",
              "spark.sql.shuffle.partitions" : "20"
            }
          }

          Parameter	Type	Required	Description	Example
          mainResource	String	Yes	The HDFS or OSS path of the JAR package.	JAR package stored in HDFS: hdfs:///path/spark-examples_2.12-3.1.1.jar JAR package stored in OSS: oss://testBucketName/path/spark-examples_2.12-3.1.1.jar
          mainClass	String	Yes	The entry point class for the JAR job.	com.aliyun.ldspark.SparkPi
          args	Array	No	The arguments for the main class.	["arg1", "arg2"]
          configs	Json	No	Spark system parameters. If your job is stored in OSS, you must add the following configurations: spark.hadoop.fs.oss.endpoint: The OSS endpoint where the job is stored. spark.hadoop.fs.oss.accessKeyId: The AccessKey ID that you create in the Alibaba Cloud console. For more information, see Create an AccessKey. spark.hadoop.fs.oss.accessKeySecret: The AccessKey secret that you obtain from the Alibaba Cloud console. For more information, see Create an AccessKey. spark.hadoop.fs.oss.impl: The class used for OSS access. The value must be org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.	{ "spark.sql.shuffle.partitions": "200"}

        • Python job configuration template and parameter descriptions:

          {
            "mainResource" : "oss://path/to/your/file.py",
            "args" : [ "arg1", "arg2" ],
            "configs" : {
              "spark.hadoop.fs.oss.endpoint" : "",
              "spark.hadoop.fs.oss.accessKeyId" : "",
              "spark.hadoop.fs.oss.accessKeySecret" : "",
              "spark.hadoop.fs.oss.impl" : "org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem",
              "spark.submit.pyFiles" : "oss://path/to/your/project_file.py,oss://path/to/your/project_module.zip",
              "spark.archives" : "oss://path/to/your/environment.tar.gz#environment",
              "spark.sql.shuffle.partitions" : "20"
            }
          }

          Parameter	Type	Required	Description	Example
          mainResource	String	Yes	The OSS or HDFS path where the Python file is stored.	Python file stored in OSS: oss://testBucketName/path/spark-examples.py Python file stored in HDFS: hdfs:///path/spark-examples.py
          args	Array	No	The arguments passed to the main program.	["arg1", "arg2"]
          configs	Json	No	Spark system parameters. If you have uploaded the job to OSS, you must configure the following parameters: spark.hadoop.fs.oss.endpoint: The OSS endpoint where the job is stored. spark.hadoop.fs.oss.accessKeyId: The AccessKey ID that you create in the Alibaba Cloud console. For more information, see Create an AccessKey. spark.hadoop.fs.oss.accessKeySecret: The AccessKey secret that you obtain from the Alibaba Cloud console. For more information, see Create an AccessKey. spark.hadoop.fs.oss.impl: The class used for OSS access. The value must be org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.	{"spark.sql.shuffle.partitions": "200"}

   3. After you complete the configuration, click Try Run in the upper-left corner to check whether the job runs as expected.

7. After configuring all task nodes, publish the task flow by clicking Save in the upper-left corner of the task flow page.

View task flow history and logs

1. On the Task Orchestration page, click the name of the target task flow.

2. In the upper-right corner of the task flow page, click Go to O&M.

3. View the publishing history and logs of the task flow.

   • To view the publishing history, go to the Task Flow Information page and click the Published Tasks tab. The publishing list includes the Version ID, Publisher, Publish Time, Remarks, and Actions columns. In the Actions column, you can click Details or DAG to view the publishing details of a specific version.

   • View the logs of the task flow.

     1. On the Running History tab, select Triggered at Specified Time or Triggered Manually from the upper-left drop-down list to view the details of all task nodes in the task flow. Then, expand an execution record to view the running status, duration, and run logs of each subtask node, such as Lindorm Spark-1, Lindorm Spark-2, and Lindorm Spark-3.

     2. Click View next to a task node name to view the submission logs of the Lindorm Spark job and find the node's JobID and SparkUI.

        Note

        If the job submission fails, provide the JobID and SparkUI when you submit a ticket.

Advanced settings

Scheduling settings

Configure a scheduling policy to automatically run the task flow according to your business requirements. Follow these steps:

1. On the Task Orchestration page, click the name of the target task flow.

2. In the lower-left corner of the task flow page, click Task Flow Information.

3. In the Scheduling Settings section on the right, turn on the Enable Scheduling switch and configure the scheduling policy. The following table describes the parameters.

   Parameter	Description
   Scheduling Type	The type of scheduling. Valid values: Periodic scheduling: Schedules the task to run periodically, for example, once a week. Schedule once: Schedules the task to run only once at a specified time.
   Effective Period	The time range during which the scheduling cycle is active. The default is 1970-01-01~9999-01-01, which means the scheduling is always active.
   Scheduling Cycle	The period for scheduling the task. Valid values: Hour: Schedules the task based on hours. You must configure the timed scheduling settings. Day: Runs the task daily. You must configure the specific time for the daily schedule. Week: Runs the task on specified days of the week. You must configure the specified days and a specific time. Month: Runs the task on specified days of the month. You must configure the specified days and a specific time.
   Timed Scheduling	Two scheduling modes are supported: Fixed interval scheduling: Start Time: The start time for the task execution. Interval: The interval between task executions, in hours. End Time: The end time for the task execution. For example, if you set Start Time to 00:00, Interval to 6 hours, and End Time to 20:59, the system executes the task at 00:00, 06:00, 12:00, and 18:00. Specified Time scheduling: Select the specific time points to execute the task. For example, if you select 0 hour and 5 hour, the system executes the task at 00:00 and 05:00.
   Specified Time	If the scheduling cycle is Week, select the days of the week to execute the task. You can select multiple days. If the scheduling cycle is Month, select the days of the month to execute the task. You can select multiple days.
   Specific Time	Set the specific time to execute the task flow. For example, if you set this to 02:55, the system executes the task at 02:55 on the specified days.
   cron expression	The system generates this automatically based on your settings. No manual configuration is required.

   Example: If you need to schedule a task flow to run at 00:00 and 12:00 every day, configure the scheduling policy as follows.

   • Set Scheduling Type to Periodic scheduling.

   • Set Scheduling Cycle to Hour.

   • Set Timed Scheduling to Specified Time, and then select 0 hour and 12 hour from the Specified Time list.

Variable settings

For a periodically scheduled task flow, you can pass the scheduling time as a parameter to the task. For example, to pass the business date bizdate (the day before the run time) to a task node, follow these steps to configure a time variable.

1. On the task flow page, double-click the Lindorm Spark task node, or click the Lindorm Spark task node and select .

2. In the right-side panel, click Variable Setting.

3. On the Node Variable or Task Flow Variable tab, add a variable name.

4. Use the variable in the Job Configuration section. For more information about variables, see Variables.

Notification settings

Enable notifications to receive messages about task flow execution results. Follow these steps:

1. In the lower-left corner of the task flow page, click Notification Configuration.

2. Turn on the notification switches based on your business requirements.

   • Success Notification: Sends a notification when the task flow runs successfully.

   • Failure Notification: Sends a notification when the task flow fails.

   • Timeout Notification: Sends a notification when the task flow execution times out.

3. Optional: Configure message recipients. For more information, see Manage notifications.

Related topics

For more information about the task orchestration feature of Data Management (DMS), see Overview.