Schedule regular DLC tasks - Platform For AI - Alibaba Cloud Documentation Center

To perform continuous incremental training and model tuning with updated data or hyperparameters, use the periodic scheduling feature to submit distributed training (DLC) jobs at regular intervals. DLC integrates with DataWorks, letting you use its scheduling configuration to submit DLC jobs. This topic explains how to configure and periodically submit DLC jobs.

Background information

You can configure periodic scheduling for jobs using one of the following methods:

Method 1: Use a PAI DLC node in DataWorks to load a DLC job and configure scheduling dependencies to schedule the job periodically. For more information, see Method 1: Use a PAI DLC node.
Method 2: Create a script task and configure scheduling dependencies to schedule the task periodically. For more information, see Method 2: Create a script task.

Prerequisites

Ensure that the required permissions for DLC are granted. For more information, see Cloud product dependencies and authorization: DLC.
Grant DataWorks access to PAI.

You can go to the Authorization Page to grant the permissions with a single click. For more information about the permission policy, see AliyunServiceRoleForDataWorksEngine. Only an Alibaba Cloud account or a RAM user with the AliyunDataWorksFullAccess policy can perform this one-click authorization.
Create a workflow.

You must create a workflow before creating a node. For more information, see Create a workflow.

Considerations

Each completed PAI DLC node run generates a new DLC task on the PAI Distributed Training (DLC) platform. To avoid accumulating many identically named tasks, set a reasonable scheduling frequency and add date and time variables to the task name. Assign time-based scheduling parameters to these variables so that each task name includes a unique timestamp. For more information, see Step 2: Develop a PAI DLC task.
DataWorks does not support running PAI DLC tasks on the public scheduling resource group.

Note

The examples in this topic use the China (Shanghai) region. The user interface may vary in other regions.

Method 1: Use a PAI DLC node

Step 1: Create a DLC job

Log on to the PAI console, go to the distributed training jobs page, and create a DLC job. The following example shows how to submit a DLC job of the PyTorch framework type. For more information, see Quickly submit a single-node PyTorch transfer learning job.

Step 2: Create a PAI DLC node

Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
Right-click the target workflow and choose Create Node > Machine Learning > PAI DLC.
In the Create Node dialog box, enter a Name for the node and click OK. You can then develop and configure the task within the node.
On the node's edit page, search for and load the created DLC job by its name.
After the job is loaded, the DLC node editor generates the corresponding code based on the job's configuration in PAI. You can then edit the job using this code. For more information, see Step 2: Create a PAI DLC node.

Step 3: Configure task scheduling

Click Scheduling Settings on the right side of the node editing area. The Scheduling Settings panel contains configuration items such as Basic Properties, Scheduling Parameters, Time Properties, Resource Properties, and Scheduling Dependencies. You can configure the scheduling cycle in the Time Properties section. DataWorks will then automatically run the node task based on the configured cycle. For more information about the configuration, see Overview of task scheduling properties.

Note

You must set the Rerun and Parent Nodes properties for the node before you can commit it.
To avoid creating numerous hard-to-distinguish jobs with the same name in PAI when using periodic scheduling in DataWorks, set a reasonable scheduling cycle based on your needs when developing the DLC task. For more information, see Step 2: Develop a PAI DLC task.

Step 4: Debug the task code

Perform the following debugging operations to ensure the task runs correctly.

(Optional) Select a resource group and assign values to custom parameters.
- Click the icon in the toolbar. In the Parameter dialog box, select the scheduling resource group for debugging.
- If your task code uses scheduling parameters, you can assign values to them here for debugging. For more information about the parameter assignment logic, see What is the difference in assignment logic among Run, Advanced Run, and smoke test in the development environment?.
Save and run the code.

Click the icon in the toolbar to save the code, and then click the icon to run the task.
(Optional) Perform a smoke test.

To check whether the scheduled node task runs as expected in the development environment, perform a smoke test when you submit the node or after submission. For more information, see Perform a smoke test.

Step 5: Commit and publish the task

After the node task is configured, submit and deploy it. After deployment, the node runs periodically based on its scheduling configuration.

Click the icon in the toolbar to save the node.
Click the icon in the toolbar to submit the node task.

In the Commit Node dialog box, enter a Change Description. You can also choose whether to perform a code review after submission.
Note
- You must set the Rerun attribute and Parent Nodes for the node before you can submit the node.
- Code review helps ensure code quality and prevents erroneous tasks from being deployed to production. If you enable code review, the submitted code must be approved before deployment. For more information, see Code review.

If you use a workspace in standard mode, click Deploy in the upper-right corner of the node editing page after submission to deploy the task to production. For more information, see Deploy a task.

Step 6: View operation logs

After the task is submitted and deployed, it runs periodically based on the node's configuration. Click O&M Personnel in the upper-right corner of the editor page to view the scheduling and execution status. For more information, see Manage scheduled tasks.

Method 2: Create a script task

Step 1: Create an exclusive scheduling resource group

In the DataWorks console, create an exclusive resource group for scheduling. For more information, see Use an exclusive resource group for scheduling.

Step 2: Bind the workspace

Bind the exclusive resource group for scheduling to a workspace before you can select it. For more information, see Step 2: Bind a workspace.

Step 3: Install the DLC toolkit

To install the DLC toolkit, administrator permission is required.

Create a command.

Log on to the DataWorks console. In the left-side navigation pane, click Resource Group. The Exclusive Resource Group page appears by default.
Find the exclusive resource group whose Purpose is Data Scheduling. In the Actions column, click the icon and select O&M Assistant.

On the O&M Assistant page, click Create Command, configure the key parameters as described in the following table, and then click OK.

Parameter	Description
Command Type	Select Manually Enter.
Command Content	Enter the following commands. `wget -P /home/admin/usertools/tools/ https://dlc-release.oss-cn-zhangjiakou.aliyuncs.com/console/public/latest/dlc --no-check-certificate chmod +x /home/admin/usertools/tools/dlc`
Installation Directory	Install the toolkit to the /home/admin/usertools/tools/ directory.
Timeout	The timeout period for the command, in seconds. If the command times out, the system terminates it. A value of 60 seconds is recommended.

Run the command.
1. On the O&M Assistant page, find the command that you created and click Actions in the Run Command column.
2. In the Run Command panel, click Run.
View the command execution status.
1. In the list at the bottom of the O&M Assistant page, click View result for the corresponding command. If the execution status is Succeeded, the command to install the DLC toolkit has been successfully executed.
2. In the Command Execution Result dialog box, view the execution details. If the progress is 100%, the DLC toolkit is installed successfully.

Step 4: Create a workflow

Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and Governance > Data Development. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.

Step 5: Submit a test task

Periodic task submission requires an existing task node. Before setting up the schedule, run a smoke test to create this initial node. If a node already exists, skip to Step 6.

Edit the deployment script.

On the workflow page, double-click the created Shell node (deployment node).

On the Shell node page, enter the following commands.

# Generate a job description file.
cat << EOF > jobfile
name=dataworks-job
workers=1
worker_spec=ecs.g6.large
worker_image=registry-vpc.cn-hangzhou.aliyuncs.com/pai-dlc/pytorch-training:1.7.1-gpu-py37-cu110-ubuntu18.04
command=echo $(date)
EOF
# Submit the job.
/home/admin/usertools/tools/dlc submit pytorchjob\
    --access_id=<your_access_key_id> \
    --access_key=<your_access_key_secret> \
    --endpoint=pai-dlc.cn-hangzhou.aliyuncs.com \
    --region=cn-hangzhou \
    --job_file=./jobfile \
    --interactive

The jobfile file describes the job information. For more information about the configuration details, see Submit commands. The following table lists the mapping between the endpoint parameter and regions.

Region	Endpoint
China (Shanghai)	pai-dlc.cn-shanghai.aliyuncs.com
China (Beijing)	pai-dlc.cn-beijing.aliyuncs.com
China (Hangzhou)	pai-dlc.cn-hangzhou.aliyuncs.com
China (Shenzhen)	pai-dlc.cn-shenzhen.aliyuncs.com
China (Hong Kong)	pai-dlc.cn-hongkong.aliyuncs.com
Singapore	pai-dlc.ap-southeast-1.aliyuncs.com
Malaysia (Kuala Lumpur)	pai-dlc.ap-southeast-3.aliyuncs.com
Germany (Frankfurt)	pai-dlc.eu-central-1.aliyun.cs.com

Run the script.
1. On the Shell node page, click the icon at the top of the page. In the Warning dialog box, click Continue to Run.
2. On the Run Parameters page, set Resource Group for Scheduling to the exclusive resource group that you created. Then, click OK.
After the run is complete, a task is created. You can view the task on the distributed training (DLC) page in the default PAI workspace.

Step 6: Set up periodic scheduling

Run the scheduled task.
1. On the Shell node page, click Scheduling Settings on the right side.
2. In the Scheduling Settings page, go to the Time Properties section and configure the Scheduling Cycle and Rerun properties.
3. In the Scheduling Dependencies section, click Use Workspace Root Node next to Parent Nodes.
4. Configure the dependencies. For more information, see Configure same-cycle scheduling dependencies.
5. Click the icon at the top of the Shell node page to save the configuration.
6. Click the icon at the top of the Shell node page to commit the scheduled task.
View instances of the scheduled task.
1. On the Shell node page, click O&M in the upper-right corner. On the Operation Center page, choose Cycle Task O&M > Cycle Instance.
2. On the instance details page, view the Scheduled Time of the automatically submitted task. In the Actions column, choose More > View Running Log to view the running log for each submission.

Platform For AI:Periodic scheduling