SchedulerX lets you create DataWorks jobs and orchestrate them with other jobs in a single workflow to process data.
Prerequisites
Before you begin, make sure that you have:
The Enterprise Edition of DataWorks activated in the DataWorks console
An Alibaba Cloud account with operation permissions on DataWorks Enterprise Edition
The SchedulerX client upgraded to V1.3.4 or later (if using SDK dependencies)
Connect to DataWorks
SchedulerX supports two access methods for connecting to DataWorks. Choose the method that fits your deployment model.
Option 1: Use SDK dependencies
Add the following properties to the SchedulerX client startup configuration to authenticate with DataWorks:
# Credentials for DataWorks access
spring.schedulerx2.aliyunAccessKey=<your-access-key-id>
spring.schedulerx2.aliyunSecretKey=<your-access-key-secret>| Placeholder | Description |
|---|---|
<your-access-key-id> | AccessKey ID of your Alibaba Cloud account |
<your-access-key-secret> | AccessKey secret of your Alibaba Cloud account |
Option 2: Use an agent
Deploy the SchedulerX agent and configure it to access DataWorks. For full agent setup instructions, see Use the SchedulerX agent to connect an application to SchedulerX (Script or HTTP jobs).
Installation package
Download the agent installation package, then add your credentials to the agent.properties file:
# Credentials for DataWorks access
aliyunAccessKey=<your-access-key-id>
aliyunSecretKey=<your-access-key-secret>Docker images
Select a Docker image that matches your network and CPU architecture:
| Network | x86_64 | arm64 |
|---|---|---|
| China (Hangzhou) (Internet) | schedulerx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx2/agent:latest-amd64 | schedulerx-registry.cn-hangzhou.cr.aliyuncs.com/schedulerx2/agent:latest-arm64 |
| Singapore (Internet) | schedulerx-registry.ap-southeast-1.cr.aliyuncs.com/schedulerx2/agent:latest-amd64 | schedulerx-registry.ap-southeast-1.cr.aliyuncs.com/schedulerx2/agent:latest-arm64 |
When starting the Docker container, set the following environment variables with the AccessKey ID and AccessKey secret of your Alibaba Cloud account:
| Environment variable | Description |
|---|---|
ALIYUN_ACCESS_KEY | AccessKey ID |
ALIYUN_SECRET_KEY | AccessKey secret |
Create and schedule DataWorks jobs
Setting up DataWorks jobs in SchedulerX involves two stages: preparing nodes in the DataWorks console, then building the workflow in the SchedulerX console.
Stage 1: Prepare nodes in DataWorks
Create a manually triggered workflow. For details, see Create a manually triggered workflow.
Add a node to the workflow. Do not configure dependencies at this stage. For details, see Create nodes and configure node dependencies.
Submit the workflow to make the node available in SchedulerX. For details, see Commit a workflow.
Stage 2: Build and schedule the workflow in SchedulerX
Create a workflow and add the DataWorks nodes as jobs. For details, see Create a workflow.
Define dependencies between jobs by dragging the connection point from one job node to another in the workflow editor.
Configure scheduled triggering (cron) for the workflow. For details, see Cron.
Verify the results
After the workflow triggers, open the workflow instance list to check the progress and status of each node. Right-click any node to view details about the execution result.
What to do next
View job details: In the SchedulerX job instance list, check execution details for individual job nodes, terminate a job, or rerun a job.
Monitor DataWorks nodes: In DataWorks Operation Center, query instance information for the scheduled DataWorks nodes.