Dataphin:Manage real-time instances

Area

Description

① Filter and search area

You can search for a real-time instance by entering its object name or ID.

• My stopped instances: Instances that you own and are in the Stopped status.

• Instances I operated today: Instances that you operated on today.

• Project: Filters instances by their project. You can select multiple projects for which you have O&M - Access Directory permissions.

• Expand filters: Click Filter to show all filter options:

  • Sub-type: Includes Flink SQL, Flink Template SQL, Flink DataStream, and real-time integration.

  • Status: Filters by one or more statuses, including Succeeded, Starting, Startup Failed, Running, Failed, Stopping, and Stopped.

  • Last operator: Filter by the user who last operated on the instance.

  • Owner: Filters by the operations owner of the instance.

  • Engine type: Filters by engine type, including Ververica Flink, Blink, Flink, and FusionInsight Flink.

  • Cluster: The cluster ID for the Ververica Flink real-time compute engine.

  • Engine version: The engine version used by the instance.

  • Operation date: The date an operation was performed on the instance.

• Reset: After expanding the filters, click Reset to clear all selected filter conditions.

② List operations

• View instance statistics: Navigates to the instance statistics page. For details, see View instance statistics.

• Export instance list: Exports the currently displayed instance list to a .csv file.

• Customize list header: Click the icon to select which columns to display and reorder them.

  Note

  • The first column and the Actions column cannot be hidden or reordered.

  • Header settings revert to their defaults if you clear your browser cache or use a different browser.

• Refresh list: Click the icon to refresh the current list.

③ Real-time instance list

The real-time instance list displays Instance Object, Status, Last Operator, Last Update Time, Owner, and Project. Column widths are adjustable and saved locally in your browser. Settings reset if you clear your browser history and cookies.

• Instance object: When you submit a real-time task for operations, the system generates a real-time instance with an initial status of Stopped. You can start it manually.

• Status: The status of the instance, such as Starting, Running, Stopping, Stopped, Failed, or Startup Failed. For more information, see Real-time instance status reference.

• Consumer lag (s): The task's consumer lag, which is the delay in consuming data from the source.

• Last operator: The user who last operated on the instance.

• Last update time: The time of the last operation.

• Owner: The operations owner of the instance.

• Project: The project to which the instance belongs, displayed in the format Project English name (Project Chinese name).

• Engine type: The engine type used by the real-time instance.

• Cluster: The cluster on which the real-time instance is running. This applies only to the Ververica Flink real-time engine; this field is empty for other engines.

• Resource queue: The resource queue where the real-time instance is running. This applies only to Blink and Flink real-time engines; the field is empty for other engines.

• Engine version: The engine version on which the real-time instance is running.

The Actions column lists the available operations for each real-time instance. For details on individual operations, see Individual operations.

④ Batch operations

Perform batch operations such as start, stop, and decommission on multiple real-time instances.

For details on supported batch operations, see Batch operations.

Icon

Status

Description

Succeeded

The instance has finished processing all data in a bounded stream.

Starting

The instance is preparing to start but has not yet begun executing.

Running

The instance is currently running.

Stopping

The system is stopping the instance.

Stopped

The instance has stopped.

Failed

The instance failed during execution.

Startup Failed

The instance failed to start.

Actions

Description

Start

In the Actions column of the target real-time instance, click the icon to start the instance. For details, see Start a real-time instance.

Stop

In the Actions column of a Running instance, click the icon. You can choose to stop without savepoint or stop with savepoint.

• Stop without savepoint: Stops the job without creating a savepoint. The next time the job starts, it will not resume from a saved state.

  Important

  Stopping without a savepoint clears any previously saved savepoint but does not clear checkpoints.

• Stop with savepoint: Creates a savepoint before stopping the job. You can then choose to resume from this savepoint the next time the job starts.

Decommission

In the Actions column of the target real-time instance, click the icon to decommission it from the production environment. This action is available for instances with a status of Stopped, Failed, or Startup Failed.

View runtime analysis

In the Actions column, click the icon to view the instance's runtime information, data curves, failover parameters, and more.

The compute engine is Apache Flink. For more information, see View runtime analysis.

If the real-time instance uses the Ververica Flink or Blink engine, see View the details of a job.

View real-time task

In the Actions column, click the icon and select View Real-time Task to navigate to the task page for the current instance and view its details.

View instance code

In the Actions column, click the icon and select View Instance Code to view the task code for the current instance.

View instance parameters

Click the icon in the Actions column of the target real-time instance, and select View Instance Parameters to view the runtime parameters of the instance. For example, parameters such as metrics.reporter.promappmgr.port.

View synchronization object

In the Actions column, click the icon and select View Synchronization Object to open the Synchronization Object tab of the instance details page. This tab shows details for both incremental synchronization and full synchronization.

• Incremental Synchronization: This section displays the Status, Start Time, Lag, and Latest Read Position. You can start, stop, view runtime logs, and view node code.

  • Status: The status of the original real-time integration task, which can be Succeeded, Starting, Startup Failed, Running, Failed, Stopping, or Stopped. If it fails, the system displays the cause of the error. If the real-time task is paused, its status is still Running, but the status appears as Running (Paused).

  • Start Time: The start time of the incremental synchronization.

  • Lag: The processing lag of the incremental task.

  • Latest Read Position: The timestamp of the most recently read data.

• Full Synchronization: This section displays the Overall Synchronization Status, Synchronization Progress, Source Object, Target Object, Status, Synchronization Time, Duration, Speed, and Data Volume. You can search by source or target object name and filter by synchronization status.

  • Overall synchronization status: The overall status of the full synchronization for all source objects. Possible statuses are Not Started, Synchronizing, Synchronization Succeeded, or Synchronization Failed.

    • Not Started: All source objects are waiting to be synchronized. Progress is not displayed in this state.

    • Synchronizing: At least one source object is currently being synchronized.

    • Synchronization Succeeded: All source objects have a status of Synchronization Succeeded, Ignored, or Stopped.

    • Synchronization Failed: At least one source object has failed to synchronize.

  • Synchronization progress: The overall progress of the full synchronization, displayed as Completed count/Total object count.

  • Source Object, Destination Object: Displayed as schema.Source/Destination table name or Source/Destination data source name.

  • Status: Can be Not Started, Stopping, Stopped, Ignored, Synchronizing, Synchronization Failed, or Synchronization Succeeded.

    • Not Started: Not yet running. Waiting for the incremental task to pause before starting.

    • Stopping: A user manually stopped the last ad-hoc synchronization task, but it has not yet finished stopping.

    • Stopped: The last ad-hoc synchronization task was manually stopped.

    • Ignored: The status changes to Ignored if you click the Ignore action when the task has not started or has failed.

    • Synchronizing: The last ad-hoc task is currently running.

    • Synchronization Failed: The last ad-hoc task failed.

    • Synchronization Succeeded: The last ad-hoc task succeeded.

  • Sync Time: The start and end time of a temporary task, displayed as Start Time ~ End Time, or - if the task has not started.

  • Duration: The total runtime of the ad-hoc task.

  • Speed: The synchronization speed. The formula is Total data synchronized by the last completed ad hoc task / Time consumed.

  • Data volume: The total data volume synchronized by the last ad-hoc task.

  • Actions: When the incremental synchronization status is Running, you can View Logs, Run, Stop, and Ignore.

    • View Logs: Opens the log viewer for the last ad-hoc task.

    • Run, Batch Run: Available for objects with a status of Ignored, Stopped, Synchronization Failed, or Synchronization Succeeded. Running a full synchronization clears the data in the target table. You can also adjust the scheduling resource group for the task.

    • Stop, Batch Stop: Available for objects that are currently synchronizing. Stopping an object does not block the main task.

    • Ignore, Batch Ignore: Available for objects with a status of Not Started or Synchronization Failed. Ignoring an object does not block the main task.

Refresh synchronization object

In the Actions column, click the icon and select Refresh Synchronization Object. Click OK in the Refresh Synchronization Object dialog box to run the instance with the new synchronization scope.

• Only the synchronization object scope changes:

  • New synchronization object added: The system pauses incremental synchronization (task status remains Running), and runs an ad-hoc full synchronization task for the new object. The system does not rerun full synchronization tasks that have already succeeded. After all new ad-hoc full tasks are complete, the incremental synchronization resumes.

  • Synchronization object deleted: The system removes the object, and the incremental task continues with a reduced scope.

• Other changes are made (e.g., task configuration or target table adjustments): The system stops the incremental synchronization. If the synchronization scope also changed, the system first synchronizes the new objects and then restarts the incremental task. You can manually manage existing full synchronization tasks on the synchronization object page as needed.

Actions

Description

Start

Start real-time development instances:

1. Select one or more real-time development instances, or click Select All on This Page, then click Start and choose Start Real-time Development Instance.

2. In the Batch Start Real-time Development Instances dialog box, configure the startup options.

   • Start n instances: Displays the number and names of the instances to be started, separated by commas.

   • Start mode: You can start without state or from a valid state.

     • Start without State: Starts the task without any initial state.

     • Start from a Valid State: Starts the task from an existing, valid state.

       Note

       You can only use the stateless start option if all selected instances are stateless.

   • Read data from: Setting a start time causes all source tables that support startTime (such as Log Service, Kafka, MySQL, MongoDB, and OceanBase) to begin reading data from that time.

     Note

     Code parameter settings in an SQL job take precedence over this specified start time.

   • Business date: Assign a value to time variables by setting a business date. The system then calculates these variables based on the specified date, which defaults to the current day.

     Note

     Code parameter settings in an SQL job take precedence over this specified business date.

3. Click OK to start the instances.

Stop

Stop real-time development instances:

1. Select one or more real-time development instances, or select Select All on This Page, then click Stop and choose Stop Real-time Development Instance.

2. In the Batch Stop Real-time Development Instances dialog box, configure the options.

   • Stop n instances: Displays the number and names of the instances to be stopped, separated by commas.

   • Stop mode:

     • Stop without savepoint: Stops the task without retaining its current state. The task will start without state next time.

     • Stop with savepoint: Retains the current state. The next time you start the task, you can choose to resume from this saved state.

Decommission

Decommissions the selected real-time instances from the production environment.

Area

Description

① Node information pane

Displays summary information for the selected node. Click View node details for full details.

Ververica Flink real-time engine: Includes Node Details, Synchronization Object (only for instances from incremental + full real-time integration tasks), Runtime Log, Operation Log, Node Code, and Task Parameters (only for compute task instances).

• Node details: Contains basic and runtime information. Basic information includes node name, ID, instance ID, and project. Runtime information includes status, runtime duration, and more.

• Synchronization object: Includes source and target objects, showing the status of full and incremental synchronization. When the incremental synchronization task is running, you can manually Run, Stop, or Ignore the full synchronization.

  When running, you also need to configure the scheduling resource group, Memory, Throttling, Concurrency, and Partition Key (only available when running a single synchronization object).

  • Scheduling resource group: The scheduling resources used for the full synchronization task. Defaults to the project's default resource group.

  • Memory: Channel configuration. Enter a positive integer. The maximum value is 16384 MB (16 GB). The default value is 4096 MB. If left empty, it defaults to 1024 MB.

  • Throttling: Limits the synchronization speed by data volume. Enter a positive integer. The default is 10 M/s. Leaving this empty means no limit.

  • Concurrency: Channel configuration. Sets the maximum desired concurrency for full synchronization tasks. Enter a positive integer. The default is 10. If left empty, the concurrency is 3.

  • Partition key: Used with concurrency to partition data for parallel reading. If left empty, the system automatically identifies a field. You can specify a column from the source table as the partition key. This field must be an integer or datetime column to ensure accurate partitioning. For best performance, we recommend using a primary key or an indexed column as the partition key.

• Runtime log: Displays the runtime logs of the current node. If a failure generates dirty data, you can download the dirty data file.

• Operation log: Displays the operation logs for the current node, including the time, operator, and specific action.

• Runtime analysis: Uses a dashboard to display various runtime metrics (such as failure counts and backpressure data) over a selected time range.

• Logs: Displays startup logs, runtime logs, and exception information separately.

• Node code: Displays the specific code for the current node.

• Task parameters: Displays the task parameters for the current node.

② Node search and filter area

Set the number of upstream and downstream levels to expand from the main node. You can also search for a node name to locate it in the DAG.

③ Scheduling dependency graph

Displays the scheduling dependency graph. Expand to show more upstream and downstream nodes and perform operations on them. Hover over a node to view its name, type, scheduling cycle, owner, and description.

④ Canvas adjustment area

Adjust the DAG display scale: set a specific zoom level (default 100%), zoom in (max 200%), zoom out (min 20%), fit to canvas, or enter full-screen mode. Also displays the node ID and name of the main node.

Actions

Description

Expand parent nodes

Expands the DAG to show more levels of parent or child nodes.

Expand child nodes

View real-time task

Navigates to the task node DAG for the current instance to view task details, upstream and downstream nodes, and perform task operations. For more information, see View and manage real-time tasks.

View node code

Views the task code that generated the current instance.

View runtime log

Views the runtime log of the instance node. If the task generates dirty data, you can click Download Dirty Data File to download it for local inspection. The file includes the Time, Error content, and Error cause. By default, the system retains these files for 7 days.

View task parameters

View the task parameters of the current instance node, such as execution.checkpointing.interval.

Edit development node

Navigates to the task editing page for the current instance in the development project. This is applicable only in the Dev-Prod development mode.

Edit node

Navigates to the task editing page that generated the current instance node. This is applicable only in Basic mode.

View production node

Views the configuration of the task in the Prod project that generated the current instance node.

View operation log

Views the operation log for the current instance, including Operation Time, Operator, and Action.

View runtime analysis

Views the instance's runtime information, data curves, failover parameters, and more.

If the real-time instance uses the Ververica Flink or Alibaba Blink engine, see View the details of a job.

If the real-time instance uses the Flink real-time engine, see View runtime analysis.

View synchronization object

In the Synchronization Object tab of the current real-time instance's details, view details for incremental synchronization and full synchronization. Field details are the same as described in the View Synchronization Object section under Individual operations.

Refresh synchronization object

Click Refresh Synchronization Object and then click OK in the Refresh Synchronization Object dialog box. This runs the current real-time instance based on the new synchronization scope. For details about each field, see Refresh synchronization object in Manage real-time instances - Single operations.

Start

Starts the current real-time instance. For details, see Start a real-time instance.

Stop

Stops a real-time instance that is Running or Succeeded. For instances with a status of Succeeded, only stop without savepoint is supported. You can choose to stop without savepoint or stop with savepoint.

• Stop without savepoint: Stops the job without creating a savepoint. The job will start without any saved state on its next run.

• Stop with savepoint: Creates a savepoint before stopping the job. The next time you start the job, you can choose to resume from this saved state.

Decommission

Decommissions a real-time instance from the production environment. This action is available only for instances in the Stopped state.