Dataphin:Real-time instances

Section

Description

①Filter and search area

You can search for a real-time instance by its name or ID.

• My stopped instances: Instances that you own with a Stopped status.

• Instances I operated today: Instances that you own and have operated on today.

• Project: Filter instances by project. You can select multiple projects for which you have the O&M - Access Directory permission.

• Expand Filters: Click Filter to show all filter options. The filter options include:

  • Subtype: Includes Flink SQL, Flink Template SQL, Flink DataStream, Real-time Integration.

  • Running status: Filter instances by one or more statuses. Options include Succeeded, Starting, Start Failed, Running, Failed, Stopping, and Stopped.

  • Last operator: Filter instances by the user who last operated on them.

  • Owner: Filter instances by their O&M owner.

  • Engine type: Filter instances by engine type. Options include Ververica Flink, Blink, Flink, and FusionInsight Flink.

  • Cluster: The cluster ID.

  • Engine version: The engine version of the instance.

  • Operation date: The date an operation was performed on the instance.

• Reset: After expanding the filter options, click Reset to clear all selected filter criteria.

②List operations

• View instance statistics: Click to view detailed statistics for the instance. For more information, see View Instance Statistics.

• Export instance list: Click to export the filtered or searched instances. The list is exported in .csv format.

• Customize columns: Click the icon to select which columns to display and to adjust their order.

  Note

  • The first column and the Actions column cannot be hidden or reordered.

  • If you clear your browser's cache or use a different browser, the browser resets the column settings to their defaults.

• Refresh list: Click the icon to refresh the current list.

③Real-time instance list

The real-time instance list displays real-time instances. The list includes information such as the instance, running status, last operator, last updated time, owner, and project. You can resize each column. Your column width settings are saved locally in your browser. Clearing your browser's history and cookies removes these settings.

• Instance: The system creates a real-time instance after a real-time task is submitted to the O&M center. Its initial running status is Stopped. You can start it manually.

• Running status: The current status of the instance. Possible statuses include Succeeded, Starting, Running, Stopping, Stopped, Failed, and Start Failed. For more information, see Real-time instance statuses.

• Consumption latency (s): The delay when consuming data from the data source.

• Last operator: The account that last operated on the instance.

• Last updated time: The time the last operation was performed on the instance.

• Owner: The O&M owner of the instance.

• Project: The project to which the instance belongs. The display format is Project English Name (Project Chinese Name).

• Engine type: The engine type used by the real-time instance.

• Cluster: The cluster on which the real-time instance runs. This field applies only to the Ververica Flink real-time engine and is empty for other engines.

• Resource queue: The resource queue in which the real-time instance runs. This field applies only to Blink and Flink real-time engines and is empty for other engines.

• Engine version: The engine version of the instance.

The Actions column lists the O&M operations available for real-time instances. For information about individual operations, see Individual Operations.

④Batch operations

You can use the Start, Stop, and Take Offline operations in the batch operations area to manage multiple real-time instances at once.

For information about batch O&M operations on real-time instances, see Batch Operations.

Icon

Status

Description

Succeeded

Applies to instances that process bounded stream data and have completed execution.

Starting

The instance is starting but not yet running.

Running

The instance is currently running.

Stopping

The system is stopping the instance.

Stopped

The instance is stopped.

Failed

The instance failed during execution.

Start Failed

The instance failed to start.

Operation

Description

Start

In the Actions column for the real-time instance, click the icon to start the instance. For more information, see Start a real-time instance.

Stop

In the Actions column for a Running real-time instance, click the icon to stop it. Two stop modes are available: stateless stop and stop with savepoint.

• Stateless stop: Does not create a savepoint. The job starts without any state the next time it runs.

  Important

  A stateless stop clears any saved savepoints but does not affect checkpoints.

• Stop with savepoint: Creates a savepoint to retain the current state. You can choose to start the job from this state the next time it runs.

Take offline

In the Actions column for the real-time instance, click the icon to take the instance offline from the production environment. You can take an instance offline if it is in the Stopped, Failed, or Start Failed state.

View Run Analytics

In the Actions column for the real-time instance, click the icon to view its operational information, data curves, failover details, and other parameters.

This operation applies to instances that use the Apache Flink compute engine. For more information, see View Run Analytics.

If the real-time instance uses the Ververica Flink or Blink compute engine, see View Job Details.

View Real-time Task

In the Actions column for the real-time instance, click the icon and select View Real-time Task to open the page for the job that generated the instance and view its details.

View Instance Code

In the Actions column for the real-time instance, click the icon and select View Instance Code to view the code of the job that generated the instance.

View Instance Parameters

In the Actions column for the real-time instance, click the icon and select View Instance Parameters to view the runtime parameters for the instance, such as metrics.reporter.promappmgr.port.

View Synchronization Object

In the Actions column for the real-time instance, click the icon and select View Synchronization Object to open the Synchronization Object tab of the instance details page. This tab provides details for both incremental synchronization and full synchronization.

• Incremental Synchronization: Displays the running status, start time, latency, and latest read position. You can start, stop, view run logs, and view node code.

  • Running Status: The status of the original real-time integration job. Possible statuses include Completed, Starting, Start Failed, Running, Failed, Stopping, and Stopped. If a job fails, you can view the error details. Although the job status remains Running when a real-time job is paused, the running status on this page is displayed as Running (Paused).

  • Start Time: The start time of the real-time incremental synchronization.

  • Latency: The latency of the incremental job.

  • Latest Read Position: The timestamp of the most recently read data.

• Full Synchronization: Displays information such as the overall synchronization status, progress, source object, destination object, status, synchronization time, time consumed, speed, and data volume. You can search by source or destination object name and filter by synchronization status.

  • Overall Synchronization Status: The overall status of the full synchronization for all source objects. Possible statuses are Not Started, Synchronizing, Synchronization Succeeded, and Synchronization Failed.

    • Not Started: The synchronization for all source objects has not started. In this state, progress is not displayed.

    • Synchronizing: At least one source object is being synchronized.

    • Synchronization Succeeded: All source objects have a status of Synchronization Succeeded, Ignored, or Stopped.

    • Synchronization Failed: At least one source object failed to synchronize.

  • Synchronization Progress: The overall progress of the full synchronization, displayed as Completed objects/Total objects.

  • Source Object and Destination Object: Displayed as schema.source/destination_table_name and source/destination_data_source_name.

  • Status: Can be Not Started, Stopping, Stopped, Ignored, Synchronizing, Synchronization Failed, or Synchronization Succeeded.

    • Not Started: Not running. The object is waiting for the incremental job to be paused before starting synchronization.

    • Stopping: The most recent temporary synchronization job was manually stopped and is still stopping.

    • Stopped: The most recent temporary synchronization job was manually stopped.

    • Ignored: The status changes to Ignored if you click Ignore when the job has not started or has failed.

    • Synchronizing: The last temporary job is running.

    • Synchronization Failed: The last temporary job failed.

    • Synchronization Succeeded: The last temporary job succeeded.

  • Synchronization Time: The start and end time of the temporary job, displayed as Start time - End time. If the job has not started or finished, this is shown as -.

  • Time Consumed: The total runtime of the temporary job.

  • Speed: The synchronization speed, calculated as Total data synchronized in the last temporary job / Time consumed.

  • Data Volume: The total amount of data synchronized by the last temporary job.

  • Actions: Operations such as View Logs, Run, Stop, and Ignore are available only when the incremental synchronization status is Running.

    • View Logs: Click to open the log page for the last temporary job.

    • Run, Batch Run: This operation is available only when the status is Ignored, Stopped, Synchronization Failed, or Synchronization Succeeded. Running a full synchronization for an object clears the data in the corresponding destination table. You can also adjust the scheduling resource group for the job.

    • Stop, Batch Stop: This operation is available only for objects that are currently synchronizing. Stopping an object does not block the main job.

    • Ignore, Batch Ignore: This operation is available only for objects that are Not Started or have failed. Ignoring an object does not block the main job.

Refresh Synchronization Object

Click the icon in the Actions column of the target real-time instance, select Refresh Synchronization Object, and then click OK in the Refresh Synchronization Object dialog box. The instance will then run based on the new synchronization scope.

• If only the synchronization scope changed:

  • Adding synchronization objects: The incremental synchronization is paused (the job status remains Running). Full synchronization jobs run for the new objects, while already completed full synchronizations are not re-run. After all new full synchronization jobs finish, incremental synchronization resumes.

  • Removing synchronization objects: The specified objects are removed from the scope, and the incremental synchronization job continues with a reduced scope.

• If there are other changes in addition to the scope (such as adjustments to job configuration or destination tables): The incremental synchronization is stopped. If the scope also changed, the full synchronization for new objects completes first, and then the incremental synchronization job restarts. You can manually manage existing full synchronization jobs for each object on the synchronization object page as needed.

Operation

Description

Start

Start Real-time Development Instances:

1. Select one or more real-time development instances, or click Select All on This Page, then click Start and choose Start Real-time Development Instances.

2. In the Batch Start Real-time Development Instances dialog box, configure the start settings.

   • Start n Instances: Shows the number and names of the selected instances, separated by commas (,).

   • Start Mode: You can choose between Stateless Start and Start from a Valid State.

     • Stateless Start: Starts the job without any initial state.

     • Start from a Valid State: Starts the job from an existing valid state.

       Note

       If your selection includes stateless instances, you can only choose Stateless Start for the batch operation.

   • Data Read Point-in-Time: Setting a specific start time causes all source tables that support startTime (such as Log Service, Kafka, MySQL, MongoDB, and OceanBase) to begin reading data from that time.

     Note

     Parameter settings in the SQL job code take precedence over this specified start time.

   • Business Date: You can set the business date to assign values to time variables. Time-based variables are calculated based on this business date, which defaults to the current day.

     Note

     Parameter settings in the SQL job code take precedence over this specified business date.

3. Click OK to start the instances.

Stop

Stop Real-time Development Instances:

1. Select one or more real-time development instances, or check Select All on This Page, then click Stop and choose Stop Real-time Development Instances.

2. In the Batch Stop Real-time Development Instances dialog box, configure the stop settings.

   • Stop n Instances: Shows the number and names of the selected instances, separated by commas (,).

   • Stop Mode:

     • Stateless stop: Does not retain the current state. The job starts without state the next time it runs.

     • Stop with savepoint: Retains the current state, allowing you to start from this state the next time the job runs.

Take offline

Takes the selected real-time instances offline from the production environment.

Section

Description

①Node Information

Displays summary information for the currently selected node. You can also click View Node Details for more information.

Ververica Flink real-time engine: The available tabs include Node Details, Sync Object (only for real-time instances generated from real-time integration tasks with both incremental and full synchronization), Runtime Log, Operation Log, Node Code, and Task Parameters (only for compute task instances).

Flink real-time engine: The available tabs include Node Details, Sync Object (only for real-time instances generated from real-time integration tasks with both incremental and full synchronization), Running Analysis, Logs, Operation Log, Node Code, and Task Parameters (only for compute task instances).

• Node Details: Displays basic and runtime information for the current node. Basic information includes the node name, ID, instance ID, and project. Runtime information includes the running status and runtime duration.

• Sync Object: Displays the source and destination objects, and the status of both full and incremental synchronization. When an incremental synchronization task is running, you can manually Run, Stop, or Ignore the full synchronization.

  When running a full synchronization, you also need to configure the Scheduling Resource Group, Memory, Rate Limiting, Concurrency, and Split Key (only available when running a single sync object).

  • Scheduling Resource Group: The scheduling resource used for the full synchronization task. It defaults to the project's default resource group.

  • Memory: The memory allocated to the data transfer channel. Only positive integers are allowed. The maximum memory is 16384 MB (16 GB), and the default value is 4096 MB. If left empty, it defaults to 1024 MB.

  • Rate Limiting: The data transfer rate limit. Only positive integers are allowed. The default is 10 M/s. If left empty, no rate limit is applied.

  • Concurrency: The maximum number of concurrent threads for the full synchronization task. Only positive integers are allowed. The default is 10. If left empty, the default is 3.

  • Split Key: A column used to partition data for concurrent reads. By default, this is empty, and the system automatically selects a suitable column. You can specify a column from the source table as the split key. This field must be of an integer or datetime type; otherwise, partitioning may be inaccurate. We recommend using a primary key or an indexed column as the split key for optimal performance.

• Runtime Log: Displays the runtime logs for the current node. If a run fails and produces dirty data, you can download the dirty data file.

• Operation Log: Shows a history of operations performed on the node, including the operation time, operator, and the operation performed.

• Running Analysis: Displays a dashboard with various runtime metrics over a selected time period, such as the number of failures and backpressure data.

• Logs: Shows the startup log, runtime log, and exception information separately.

• Node Code: Displays the specific code for the current node.

• Task Parameters: Displays the parameters for the current node's task.

②Node Search and Filter

Allows you to quickly set the number of upstream and downstream levels to display around the main node. If the graph becomes too crowded, you can search for a node by name to quickly locate it within the current view.

③Scheduling Dependency Graph

Displays the instance's scheduling dependency graph. You can expand it to show more upstream and downstream nodes and perform O&M operations on them. Hover over a DAG node to view its name, type, scheduling cycle, O&M owner, and description.

④Canvas Adjustment

This section provides controls to adjust the DAG's display scale, including setting a specific zoom level (default is 100%), zooming in (up to 200%), zooming out (down to 20%), fitting to the canvas, and full-screen mode. This area also displays the node ID and name of the main node in the current DAG.

Operation

Description

Expand Parent Nodes

Expands dependency nodes at different levels relative to the main node in the DAG.

Expand Child Nodes

View Real-time Task

Opens the DAG of the task that generated the current instance node, where you can view task node details, upstream and downstream information, and manage the task node. For more information, see View and manage real-time tasks.

View Node Code

Views the task code that generated the current instance.

View Runtime Log

Views the runtime log of the instance node. If the task generates dirty data, you can click Download Dirty Data File to download and inspect it locally. The file includes the time the dirty data was generated, error message, and cause of the error. The file is retained for 7 days by default.

View Task Parameters

Views the task parameters for the current instance node, such as execution.checkpointing.interval.

Edit Development Node

This operation applies only to the Dev-Prod development model.

Edit Node

This operation applies only in Basic mode.

View Generation Node

Views the configuration of the task in the Prod project that generated the current instance node.

View Operation Log

Views the operation log for the current instance, including the operation time, operator, and the operation performed.

View Running Analysis

Views the runtime information, data curves, failover details, and other parameters for the current real-time instance.

If the real-time instance uses the Ververica Flink or Alibaba Blink real-time engine, see View Job Details.

If the real-time instance uses the Flink real-time engine, see View Running Analysis.

View Sync Object

Displays details for both incremental synchronization and full synchronization in the Sync Object tab. For field descriptions, see the View Sync Object operation for individual instances.

Refresh Sync Object

Click Refresh Synchronization Object, and then click OK in the Refresh Synchronization Object dialog box. The current real-time instance will then run based on the new synchronization scope. For details about the fields, see Refresh Synchronization Object in Manage a real-time instance - Single operation.

Start

Starts the current real-time instance. For specific steps, see Start a real-time instance.

Stop

Stops the current real-time instance, which can be in the Running or Completed state (For instances in the Completed state, only stateless stop is supported). Two stop modes are available: stateless stop and stop with savepoint.

• Stateless stop: Does not create a savepoint. The task starts without any state on the next run.

• Stop with savepoint: Creates a savepoint to retain the current state. You can choose to start the task from this state on the next run.

Decommission

Decommissions the real-time instance from the production environment. Only instances in the Stopped state can be decommissioned.

Parameter

Description

Use cases

dataphin.dev.k8s.TerminationGracePeriodSeconds

The termination grace period for a Kubernetes Pod. The default value is 60 seconds.

Increase this value if log uploads cannot be completed within 60 seconds.

dataphin.dev.k8s.sidecar.env.LOG_ARCHIVE_DELAY_S

The delay before log collection starts. The default value is 5 seconds.

If the TaskManager or JobManager process takes a long time to exit, increase this value to ensure all logs are written to disk.

dataphin.dev.k8s.sidecar.env.JVM_MAX_HEAP_MB

The maximum heap size (the JVM -Xmx setting) for the log collection process. The default value is 128 MB.

If an OutOfMemory (OOM) error occurs during log collection, increase this value.

dataphin.dev.k8s.sidecar.env.LOG_FILE_PATTERN

Specifies the filename pattern for log uploads. By default, this parameter is empty, which means all files are uploaded. Files are uploaded if their names contain the pattern.

If you have a large number of log files or the files are very large, configure this parameter to select which files to upload.

dataphin.dev.k8s.sidecar.env.GRINGOTTS_SO_TIMEOUT_MS

The timeout for the upload API. The default value is 30 seconds.

This value typically does not need to be adjusted.

dataphin.dev.k8s.sidecar.env.MAX_LOG_BUFFER_MB

The size of the memory buffer for logs. The default value is 5 MB.

If an OOM error occurs during a log upload, decrease this value.

dataphin.dev.k8s.sidecar.env.LOG_UPLOAD_RETRY_TIMES

The number of retries after an upload API timeout. By default, the upload is retried once, resulting in a total of two attempts.

Configure as needed.