Platform For AI:Periodic scheduling

Benefits and use cases

Periodic scheduling for DLC jobs delivers key benefits for production machine learning workflows:

• Automated workflows: Eliminate manual job submission for recurring training tasks, reducing operational overhead and human error.

• Incremental training: Automatically retrain models when new data arrives, keeping your models current with the latest information.

• Continuous optimization: Schedule regular hyperparameter tuning experiments to continuously improve model performance.

• Production reliability: Leverage DataWorks' enterprise-grade scheduling capabilities, including dependency management, retry logic, and comprehensive monitoring.

Common use cases for periodic scheduling include:

• Daily model retraining: Retrain models daily using fresh data from the previous day's operations.

• Weekly model evaluation: Run comprehensive model evaluation and comparison jobs weekly.

• Monthly fine-tuning: Perform extensive hyperparameter tuning and model architecture experiments monthly.

• Data drift detection: Schedule regular data quality and drift detection jobs to monitor dataset changes.

Background information

You can use one of the following methods to configure periodic scheduling for a DLC job:

Choose your method

DLC periodic scheduling offers two implementation methods. Choose the approach that best matches your requirements:

Recommendation: Start with Method 1 if you're new to periodic scheduling or have simple requirements. Use Method 2 when you need advanced customization, dynamic parameter generation, or integration with existing automation scripts.

Available methods

• Method 1: Load a DLC job on a PAI DLC node in the DataWorks console and configure job scheduling

• Method 2: Create a script task and configure job scheduling

Method 1: Using PAI DLC nodes for periodic scheduling

When to use this method

This method is ideal for:

• Simple, static training jobs that don't require dynamic parameter changes

• Users who prefer GUI-based configuration over command-line operations

• Quick setup and deployment without custom scripting

• Teams leveraging existing DLC job configurations

Step 1: Create a DLC job

Log on to the Platform for AI (PAI) console. Go to the Distributed Training Jobs page, and create a DLC job. In this topic, a PyTorch-based DLC job is used as an example. For information about how to create a PyTorch-based DLC job, see Submit a standalone training job that uses PyTorch.

Step 2: Create a PAI DLC node

1. Go to the DataStudio page.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and O&M > Data Development. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.

2. Right-click the target workflow and choose Create Node > Machine Learning > PAI DLC.

3. In the Create Node dialog box, configure the Name parameter and click Confirm. Then, you can use the node to develop tasks and configure task scheduling properties.

4. On the DLC node tab, search for the DLC job that you created from the Load PAI DLC Job by its name and load the job.

   After you load the job, the DLC node editor generates node code based on the configurations of the task in PAI. You can modify task configurations based on the code. For more information, see the "Step 2: Develop a PAI DLC task" section in the Create and use a PAI DLC node topic.

Step 3: Configure job scheduling

In the right-side pane of the node tab, click Properties. In the Properties panel, you can view configuration items such as General properties, Scheduling Parameter, Schedule, Resource Group, and Dependencies. Configure the parameters in the Schedule section. DataWorks automatically schedules and runs node tasks based on the specified scheduling cycle. For more information, see Overview.

Step 4: Debug the task code

To check whether the node is configured as expected, perform the following operations.

1. (Optional) Select a resource group for the run and assign values to custom parameters.

   • Click the icon in the toolbar. In the Parameters dialog box, select a resource group for debugging.

   • If your code uses variables, you can assign test values here for debugging. For more information about the parameter assignment logic, see What are the differences in parameter assignment among Run, Advanced Run, and smoke testing in the development environment?.

2. Save and run the code.

   Click the icon in the toolbar to save your code, and then click the icon to run the task.

3. (Optional) Perform a smoke test.

   To check how the scheduled task will execute, you can perform a smoke test in the development environment. You can run this test when you commit the node or anytime after. For more information, see Perform smoke testing.

Step 5: Submit the task

After you configure the node, you must submit and deploy it. After deploying, the node runs periodically according to its schedule.

1. Click the icon in the toolbar to save the task.

2. Click the icon in the toolbar to submit the task.

   In the Submit dialog box, enter a Change description. You can also choose whether to perform a code review after the node is submitted.

   Note

   • You must configure the rerun properties and dependencies for the node before you can submit it.

   • Code reviews help ensure code quality and prevent errors from being deployed to the production environment. If you enable code review, a reviewer must approve the submitted code before it can be deployed. For more information, see Code review.

If you are using a workspace in standard mode, you must click Deploy in the upper-right corner of the node editor page after the node is submitted. This deploys the task to the production environment. For more information, see Publish tasks.

Step 6: View operations logs

Once the node is committed and published, it runs periodically according to its schedule. You can click Operation Center where you can view the scheduled task's status. For more information, see Manage scheduled tasks.

Best practices and considerations

• Job naming: Use descriptive job names that include the scheduling frequency (e.g., "daily-model-retraining", "weekly-evaluation-job") to easily identify scheduled jobs in the PAI console.

• Scheduling cycle: Choose appropriate scheduling intervals based on your data update frequency and computational requirements. Avoid overly frequent scheduling that may cause resource contention.

• Resource allocation: Ensure sufficient quota is available for your scheduled jobs, especially for high-frequency or resource-intensive tasks.

• Monitoring and alerts: Configure monitoring and alerting for your scheduled jobs to quickly detect and respond to failures or performance issues.

• Version control: Track changes to your DLC job configurations, as these will be automatically applied to all future scheduled runs.

Method 2: Using script tasks for periodic scheduling

When to use this method

This method is ideal for:

• Complex workflows requiring dynamic job configuration based on runtime conditions

• Integration with existing automation scripts and CI/CD pipelines

• Advanced parameter generation from external sources or databases

• Custom error handling, logging, and notification requirements

• Teams with DevOps expertise preferring infrastructure-as-code approaches

Best practices and considerations

• Script validation: Always test your script thoroughly in a non-production environment before scheduling. Use the "Run" button to perform smoke tests.

• Error handling: Implement robust error handling in your scripts to gracefully handle failures and provide meaningful error messages.

• Resource management: Ensure your exclusive resource group has sufficient capacity for all scheduled jobs, especially during peak execution times.

• Security: Store sensitive credentials (access_id, access_key) securely using DataWorks' credential management features instead of hardcoding them in scripts.

• Logging and monitoring: Implement comprehensive logging in your scripts and configure DataWorks monitoring to track job execution and performance.

• Version control: Maintain version control for your scheduling scripts to track changes and enable rollback when needed.

Troubleshooting

Common issues and solutions when configuring periodic scheduling for DLC jobs:

Common issues

• Permission denied errors: Ensure you have the required permissions to use DLC and DataWorks. Contact your administrator to grant the necessary RAM roles and permissions.

• Resource group not available: Verify that your exclusive resource group is properly created and associated with your workspace. Ensure the resource group has sufficient capacity for your scheduled jobs.

• DLC package installation failed: Ensure you have the required permissions to install packages on the resource group. Contact technical support if installation continues to fail.

• Job submission fails: Verify your job configuration file syntax and ensure all required parameters (access_id, access_key, endpoint, region) are correctly specified.

• Scheduled jobs not running: Check the scheduling cycle configuration in the DataWorks Properties panel. Verify that parent node dependencies are properly configured and that the job has been committed successfully.

Debugging tips

• Check operation logs: Use the Operation Center in DataWorks to view detailed execution logs for each scheduled instance. Look for error messages and stack traces.

• Test with manual run: Before scheduling, manually run your job or script to verify it works correctly. This helps isolate scheduling issues from job configuration issues.

• Monitor resource usage: Check CPU, memory, and network usage during job execution to identify resource bottlenecks or configuration issues.

• Verify dependencies: Ensure all parent nodes and dependencies are properly configured and complete successfully before your scheduled job runs.

Performance optimization

• Optimize scheduling frequency: Avoid overly frequent scheduling that may cause resource contention. Consider your data update patterns and business requirements when setting scheduling intervals.

• Right-size resources: Choose appropriate instance types and quantities based on your workload requirements. Monitor actual resource usage and adjust as needed.

• Leverage parallel execution: For independent tasks, configure parallel execution in DataWorks to reduce overall pipeline duration.

• Implement caching strategies: Cache intermediate results and datasets to avoid redundant computation in subsequent scheduled runs.

References

You can view and manage DLC jobs that are submitted in the PAI console.

• View training details

• Manage training jobs