Realtime Compute for Apache Flink:Manage workflows

Prerequisites

Before you begin, ensure that:

• The batch jobs you want to schedule are deployed

• You are working in a supported region (see Limitations)

Steps

1. Log on to the Realtime Compute for Apache Flink console.

2. In the Actions column of the target workspace, click Console.

3. In the left navigation pane, choose Operation Center > Task Orchestration.

4. Click Create Workflow and configure the following parameters.

   Parameter	Description
   Workflow name	Must be unique within the current project.
   Variable configuration	Defines variables for data computation based on preset values. Each variable has a name (for example, ${date}) and a value, which can be a static date, a time format, or an expression. The following system time variables are built in: <br>- system.biz.date / ${system.biz.date}: The day before the scheduled time of a daily scheduling instance. Format: yyyyMMdd. <br>- system.biz.curdate / ${system.biz.curdate}: The scheduled date of a daily scheduling instance. Format: yyyyMMdd. <br>- system.datetime / ${system.datetime}: The scheduled time of a daily scheduling instance. Format: yyyyMMddHHmmss. <br><br> Important Variable configuration is not required for materialized table workflows. Workflow-level configurations take precedence over job-level configurations.
   Scheduling type	Controls how the workflow is triggered. Choose based on your use case: <br>- Manual Trigger: Run the workflow on demand by clicking Run. Use this for temporary tests or one-time processing tasks where you control the timing. <br>- Recurring Schedule: Trigger the workflow automatically on a defined schedule (by minute, hour, or day). Use this for production pipelines that must run at regular intervals. <br><br> Important Workflows that include materialized table nodes must use Recurring Schedule.
   Scheduling cycle	Required for Recurring Schedule only. Use cron expressions to define the schedule. Examples: <br>- 0 0 */4 ? * * — every 4 hours <br>- 0 0 2 ? * * — daily at 02:00 <br>- 0 0 5,17 ? * MON-FRI — at 05:00 and 17:00, Monday through Friday <br><br>For cron expression syntax, see Rules for writing cron expressions.
   Scheduling start time	Required for Recurring Schedule only. Set this to a future time. If the start time is in the past, the workflow may miss its first scheduling window and result in a dry run or failure. <br><br> Important After creating a recurring workflow, enable its Scheduling State to activate it.
   Failure retries	The number of times to retry a failed node. By default, failed nodes are not retried. Node-level retry settings override this workflow-level setting.
   Failure notification	The email address that receives alerts when a workflow node fails. To receive alerts via DingTalk or SMS, configure notifications through Cloud Monitor. For details, see Configure monitoring and alerting.
   Resource queue	The default deployment target for all nodes in the workflow. For details, see Manage resource queues. Note This setting does not change the deployment target of corresponding deployed batch jobs. Node-level resource queue settings override this workflow-level setting.
   Tags	Optional tag name and value for organizing workflows.

5. Click Create. The workflow node editor opens.

6. Configure the initial node. The editor includes an initial node by default. Click the node, configure the parameters in the Edit Node panel, and click Save.

   After you create a materialized table node, a dialog box prompts you to build descendant nodes using data lineage. Descendant nodes must be partitioned tables created in VVR 11.0 or later with stream refresh mode and a freshness of less than 30 minutes.

   Job

   Parameter	Description
   Job	Select a deployed batch job from the current project. Fuzzy search is supported.
   Node name	The display name of the node within the workflow.
   Upstream nodes	Other nodes in the workflow that this node depends on. The initial node has no upstream dependencies.
   Failure retries	The number of retries for this node. If set, this overrides the workflow-level retry count.
   Status subscription	Notification settings for this node. Subscriptions are supported for Start and Fail statuses.
   Timeout	Maximum allowed run time for the node. If exceeded, the node is marked as failed.
   Resource queue	Deployment target for this node. If not set, defaults to the workflow-level resource queue. Note This does not change the deployment target of the corresponding deployed batch job.
   Tags	Optional tag name and value for this node.

   Materialized table

   Parameter	Description
   Materialized table	Select a partitioned table created in Ververica Runtime (VVR) 11.0 or later with stream refresh mode.
   Node name	The display name of the node within the workflow.
   Time partition	The partition field and format (for example, yyyyMMdd) for the materialized table.
   Resource configuration	Resource allocation for scheduled backfills. Enable Auto-infer to let the system determine the appropriate concurrency automatically.
   Upstream nodes	Other nodes in the workflow that this node depends on. The initial node has no upstream dependencies. After you create a materialized table node, descendant nodes are inferred automatically from data lineage.
   Failure retries	The number of retries for this node. If set, this overrides the workflow-level retry count.
   Status subscription	Notification settings for this node. Subscriptions are supported for Start and Fail statuses.
   Timeout	Maximum allowed run time for the node. If exceeded, the node is marked as failed.
   Resource queue	Deployment target for this node. If not set, defaults to the workflow-level resource queue. Note This does not change the deployment target of the corresponding materialized table.
   Tags	Optional tag name and value for this node.

7. (Optional) Click Add Node to add more nodes.

8. In the upper-right corner, click Save, then click OK in the confirmation dialog box.

Perform a data backfill

1. Log on to the Realtime Compute for Apache Flink console.

2. In the Actions column of the target workspace, click Console.

3. In the left navigation pane, choose Operation Center > Task Orchestration.

4. In the Actions column of the target workflow, click Run.

5. In the Run dialog box, select Data Backfill as the scheduling method and configure the following parameters.

   Parameter	Description
   Time interval	The time period to backfill. The interval is passed to the workflow's time variables to refresh data in the corresponding partitions.
   Resource queue	The queue where the backfill task runs. Defaults to default-queue.

   

6. Click OK.

Manage data backfill instances

Data backfill instances are managed the same way as regular workflow instances. Click the workflow name to go to its instance page.

On the Workflow Instance List and Details tab, view the backfill instances and their details, including run times and statuses.

For more information, see Manage workflow instances and node instances.