After you submit or publish tasks from Data Integration and Data Development in both development and production environments, the Operation Center provides unified management through five modules: operation overview, task operation, instance operation, monitoring management, and system configuration.
Scenarios
-
Global Perspective Control: View instance statistics for offline and real-time instances, including running trends, failure rankings, alert rankings, and duration metrics. Quickly identify and respond to anomalies across projects.
-
Resource Cost Savings: The resource dashboard compares allocated versus actual CPU and memory usage at global and task levels. Use these insights to optimize resource allocation and reduce costs without affecting task stability.
-
Task Operation Management: Manage code tasks from Data Integration, Modeling R&D, Coding R&D, and Data Distilling. Monitor node status and manage upstream and downstream dependencies.
-
Running Resource Control: Set throttling rules to handle compute engine bottlenecks, insufficient resources, or task scheduling conflicts. Throttling ensures stability and prioritizes critical data outputs.
-
Abnormal Alerts: Configure alert rules for physical tasks and logical table fields through baseline operations. The system sends alerts via phone, text message, DingTalk, or email when anomalies occur.
Features
After you submit or publish nodes to the production environment, the Operation Center lets you perform data backfill for recurring tasks, run one-time tasks, monitor status, configure alerts, view statistics, and set timeout or failure policies.
The following table describes each module:
|
Functional Module |
Description |
|
Operation Overview |
|
|
Task Operation |
Categorizes nodes as auto-triggered, real-time, or one-time based on scheduling type. Auto-triggered nodes include script, detail/aggregate table, and extraction nodes. Manage nodes, view DAGs and instances, backfill data, and change owners in batch. |
|
Instance Operation |
Covers baseline, recurring, data backfill, one-time, and real-time instances. View DAGs, nodes, and logs, or rerun instances in batch. |
|
Monitoring Management |
Includes baseline and offline task monitoring.
|
|
System Configuration |
Includes throttling and runtime settings.
|
Task Instance Generation Logic
The Operation Center manages recurring, one-time, and real-time tasks, triggered by schedule or manually. Scheduling intervals range from minutes to years, and tasks can be triggered for data backfill, manual execution, or real-time operation.
-
By default, tasks in the development environment do not run automatically; manual triggering is required.
-
Recurring tasks begin normal scheduling upon publication to the production environment.
-
Recurring Instance Generation Logic
When you submit or publish a node with a schedule type of recurring task, it appears in the recurring task list. Recurring tasks produce two instance types:
Instance Type
Instance Generation Time
Instance Running Logic
Instance Running Conditions
Recurring Instance
Auto-triggered nodes create next-day instances at 23:00 each night (T+1 generation).
-
Nodes submitted and published before 23:00 generate instances for the next day.
-
Nodes submitted and published after 23:00 generate instances on the third calendar day.
NoteSchedule resource group changes only affect new instances. To update an existing instance's resource group, modify the node's resource configuration and publish before 23:00, or modify the resource setting on an instance that has not started.
After snapshot generation, recurring instances run automatically based on the node's scheduling properties.
Recurring instances must meet the following conditions to start:
-
All dependent parent node instances have a successful running status.
-
The instance node has reached its scheduled running time.
-
The scheduled resources accommodate the operation of the instance.
-
The instance and its associated auto-triggered nodes are not in a paused state.
Running status is covered in Instance Running Diagnosis.
Data Backfill Instance
Manually generated through data backfill operations on current auto-triggered nodes.
Executes backfills based on the configured data timestamp.
NoteIn production, data backfill validates that auto-triggered nodes from the development environment operate correctly and ensure data production.
-
-
One-time Instance Generation Logic
When you submit or publish a node with a schedule type of one-time business, it appears in the one-time task list. Select Run to execute the task. View execution details on the One-time Instance page.
-
Real-time Instance Generation Logic
After you submit or publish real-time tasks, you can initiate and adjust resource configurations from the real-time task list in the Operation Center. In both Basic and Dev-Prod modes within Prod, submitted real-time tasks automatically create instances with an initial Stopped status. Real-time tasks are categorized into real-time computing and real-time integration tasks.
Access the Operation Center
Quick Entry (Recommended)
On the Dataphin home page, select Operation Scheduling.
Regular Entry
-
On the Dataphin home page, click R&D in the top menu bar.
-
On the Data Development page, select O&M in the top menu bar.
