Realtime Compute for Apache Flink:Configure job resources

You can use the following formulas to calculate resource requirements:

• Number of CUs = MAX(Total CPU for JobManager and TaskManagers, Total memory for JobManager and TaskManagers / 4)

• Actual number of TaskManagers = ceil(Parallelism / Slots per TaskManager)

• Actual slots per TaskManager = Parallelism / Actual number of TaskManagers

For example, assume you set the parallelism to 12 and the slots per TaskManager to 4.

In this example, JobManager CPU is 2 cores, JobManager Memory is 4 GiB, TaskManager CPU is 2 cores, and TaskManager Memory is 4 GiB.

In the Realtime Compute for Apache Flink console, the actual number of TaskManagers is 3, and each TaskManager has 4 slots.

The actual number of TaskManagers and slots per TaskManager are calculated as follows:

1. Actual number of TaskManagers = ceil(Configured parallelism / Configured slots per TaskManager) = ceil(12 / 4) = 3.

2. Actual slots per TaskManager = Parallelism / Actual number of TaskManagers = 12 / 3 = 4.

Configure basic resources

Configure slot resources

1. In expert mode, click Get Plan Now to fetch the resource plan graph.

2. Click the Edit icon on a slot box. The generated resource plan graph displays multiple slot boxes, each containing VERTEX operator information and a PARALLELISM value.

3. Modify the slot configuration. In the dialog box, you can configure the CPU, heap memory, off-heap memory, and parallelism parameters.

   The parallelism you set here applies to all operators within this Slot Sharing Group. After you save the configuration, the system automatically:

   • Sets the same parallelism for all operators in this Slot Sharing Group.

   • Allocates the required memory for the state backend, Python, and operators based on the job's computation logic. This allocation is automatic.

   Note

   • For a Source node, we recommend setting a parallelism that is proportional to its partition count. In other words, the parallelism should be a divisor of the partition count. For example, if a Kafka topic has 16 partitions, set the parallelism to 16, 8, or 4 to avoid data skew.

   • Setting the parallelism of a Source node too low can cause a bottleneck, as one Source may read too much data and reduce job throughput.

   • For other nodes, set the parallelism according to their data traffic, assigning higher parallelism to nodes with more traffic.

   Note

   In expert mode, the slot configuration dialog box exposes CPU, Heap Memory, and Off-Heap Memory for manual configuration. Other memory components — including JVM Overhead, network buffers, and framework memory — are automatically allocated by the system at their default proportions. These components are pre-allocated when the TaskManager starts and are not dynamically adjusted during job execution.

   If you encounter insufficient memory errors caused by these automatically allocated components, configure the following parameters in the Other Configurations field under Running Parameters Configuration:

   • taskmanager.memory.jvm-overhead.fraction: Size of JVM Overhead as a fraction of total TaskManager memory. Default: 0.1 (10%).

   • taskmanager.memory.jvm-overhead.max: Maximum size of JVM Overhead.

   • taskmanager.memory.jvm-overhead.min: Minimum size of JVM Overhead.

4. Click OK.

Configure operator resources

By default, all operators share a single Slot Sharing Group, which prevents you from configuring their resources individually. To configure resources for a specific operator, enable Multiple SSG mode. This mode assigns an independent slot to each operator, allowing you to configure its resources on that slot.

1. On the Configuration tab, click Edit in the Resources section, and set Resource mode to expert mode.

2. (Optional) If no resource plan is displayed, click Get Plan Now.

   By default, the generated resource plan graph shows all operators within a single slot box.

3. Turn on the Multiple SSG mode switch and then click Re-fetch.

   This action splits the operators in the sharing group into individual slots.

4. Click the Edit icon on the slot box that corresponds to the target operator, and then modify the operator resources.

   In the Modify slot dialog box, you can configure the CPU, heap memory, off-heap memory, and parallelism parameters.

5. Click OK.

Operator parallelism, chaining strategy, and State TTL

You can configure the parallelism, chaining strategy, and State TTL for individual operators.

1. Click the Expand icon on the target VERTEX box.

   After expansion, the VERTEX box displays each operator node, its PARALLELISM value, and an Edit icon next to each operator.

   Note

   You can click the Edit icon on a VERTEX to set the parallelism for all operators within that VERTEX in a batch.

2. Click the Edit icon for the operator.

3. Configure the operator resources.

   The following table describes the parameters.

   Parameter	Description
   Parallelism	The parallelism for the operator.
   Chaining strategy	Chaining connects multiple operators into a single task, improving performance by reducing data transfer and serialization overhead. However, you can break a chain to gain finer control over the execution flow. The following strategies are available: ALWAYS (Default): The operator can always be chained with upstream and downstream operators. HEAD: The current operator acts as the head of a chain. It is not chained with upstream operators but remains chained with downstream operators. NEVER: The current operator is not chained with any upstream or downstream operators.
   State TTL	You can set the expiration time in seconds, minutes, hours, or days. By default, the operator inherits the job's state expiration time, which defaults to 1.5 days. To configure the job-level expiration time, see Configure running parameters. Note This feature is supported only in Ververica Runtime (VVR) 8.0.7 and later. TTL configuration is supported only for stateful operators. State expiration is an approximate cleanup mechanism. The system does not guarantee that expired state is removed immediately after the TTL elapses. The actual cleanup time depends on background state access patterns and cleanup policies.

4. Click OK.

Does setting parallelism equal consuming the same number of CUs?

No. Parallelism refers to the number of concurrent tasks running in a job. CU (Compute Unit) is the billing and resource unit used by Realtime Compute for Apache Flink. They are not in a 1:1 relationship.

Total CU consumption is calculated with the following formula:

Total CU consumption = Parallelism × CU per task

The CU consumed by each task is determined by the resource specification per slot configured for the job. Increasing parallelism increases total CU consumption, but the increase is proportional to the per-task CU specification, not a simple 1:1 ratio.

Why does SET 'parallelism.default' not take effect in SQL jobs?

Using SET 'parallelism.default' = 'N'; directly in SQL job statements is not effective in Realtime Compute for Apache Flink. To configure parallelism, use one of the following methods:

1. Set parallelism in the WITH clause of the relevant SQL statement.

2. Modify the parallelism in the resource configuration editing interface on the O&M > Deployments page.

What should I do when a Print Sink or Join operator causes insufficient resources or poor performance?

The appropriate action depends on the specific symptom:

• Print Sink causes TaskManager resource shortage — This issue cannot be resolved by parameter tuning alone. Evaluate your resource requirements based on actual data volume, verify that source table field types are correctly defined (for example, use BIGINT instead of STRING where appropriate), and allocate sufficient TaskManager resources to support the debugging overhead of the Print Sink.

• Join performance is poor despite low memory utilization — Increase the number of CUs. However, if the root cause is data skew, adding more resources may not resolve the issue. Resolve the data skew problem first, then evaluate whether additional resources are needed.

• Job restarts frequently with high end-to-end latency — If the job is in a full-state synchronization phase following a stateless restart, try setting the number of slots to 1, configuring appropriate parallelism separately, and using a 1-core 4 GiB TaskManager specification with moderate concurrency (for example, 10) to test and optimize performance.