Realtime Compute for Apache Flink:Configure job resources

You can use the following formulas for calculation:

• Number of CUs configured for the job = MAX(Total CPU of JM and TMs, Total memory of JM and TMs / 4)

• Actual number of TMs (if the configured TM CPU or memory is less than or equal to the default maximum) = Configured concurrency / Configured slots per TaskManager

• Actual number of TMs (if the configured TM CPU or memory is greater than the default maximum) = MAX(⌈Total TM CPU / 16 cores⌉, ⌈Total TM memory / 64 GiB⌉)

  • Total TM CPU = Configured concurrency / Configured slots per TaskManager × Configured CPU per TM

  • Total TM memory = Configured concurrency / Configured slots per TaskManager × Configured memory per TM

  • The default maximum CPU per TM is 16 cores.

  • The default maximum memory per TM is 64 GiB.

• Actual number of allocatable slots per TM = ⌈Configured concurrency / Actual number of TMs⌉

For example, if you set Concurrency to 12 and Slots per TaskManager to 4, the configuration is shown in the following figure.

In the Flink development console, you will see that the actual number of TMs is 3 and the number of slots per TaskManager is 4.

The calculation process for the actual number of TMs and slots per TaskManager is as follows:

1. Actual number of TMs = [Configured concurrency / Configured slots per TaskManager] = [12/4] = 3

2. Actual number of slots per TaskManager = ⌈Concurrency / Actual number of TMs⌉ = ⌈12/3⌉ = 4

Configure basic resources

Configure slot resources

1. In Expert Mode, click Get Plan Now to view the resource plan graph.

   

2. Click the icon on the SLOT box.

3. Modify the slot configuration information.

   The concurrency set here is the uniform concurrency for all operators within this slot sharing group. After you complete the settings, the system automatically performs the following operations:

   • The system automatically sets the same concurrency for all operators in this slot sharing group.

   • The system automatically generates the required memory for the state backend, Python, and operators based on the job's computation logic. You do not need to configure this manually.

   Note

   • We recommend that the concurrency of a Source node is proportional to the number of partitions. This means the number of partitions should be divisible by the concurrency. For example, if a Kafka topic has 16 partitions, we recommend setting the concurrency to 16, 8, or 4 to avoid data skew. Also, do not set the concurrency of a Source node too low. A low concurrency can cause a single Source to read an excessive amount of data, which can lead to an input bottleneck and affect job throughput.

   • We recommend that you configure the concurrency for nodes other than the Source node as needed. For nodes with high traffic, set a higher concurrency. For nodes with low traffic, set a lower concurrency.

   • We recommend that you adjust the Heap Memory and Off-heap Memory sizes only when there is a clear exception or requirement, such as an out-of-memory (OOM) error or severe garbage collection (GC). This is because adjusting these memory sizes does not significantly change the job throughput during normal operation.

4. Click OK.

Configure operator resources

By default, all operators are placed in a single slot sharing group, so you cannot modify the resource configuration for individual operators. To set resources for individual operators, enable Multiple SSG Mode. This mode assigns a dedicated slot to each operator, allowing you to configure its resources individually. The steps to set operator resources are as follows:

1. On the Deployment Details tab for the job, in the Resource Configuration section, click Edit and set Resource Mode to Expert Mode.

2. (Optional) If no resource plan is available, you can click Get Plan Now.

   

3. Turn on the Multiple SSG Mode switch. Then, click Regenerate.

   The operators within the sharing group are now split into individual slots.

   

4. Click the icon on the SLOT box corresponding to the target operator, and then modify the operator resources.

   

5. Click OK.

Configure operator concurrency, chain strategy, and TTL

You can configure the concurrency, chaining strategy, and state time-to-live (TTL) for a single operator.

1. Click the icon on the target VERTEX box to expand the vertex.

   

   Note

   You can click the icon on the target VERTEX to set the concurrency for all operators under that vertex in a single operation.

2. Click the icon of the operator.

   

3. Configure the operator resources.

   

   The following describes the parameters:

   Parameter	Description
   Concurrency	The concurrency of the corresponding operator.
   Chaining Strategy	A chain is a logical computation chain formed by connecting multiple operators. It can improve job execution efficiency and performance, and reduce the overhead of data transmission and serialization between operators. However, you may sometimes need to break the chain to better control the job's execution flow and performance. The following strategies are supported: ALWAYS (default): The operator can always be chained with upstream and downstream operators. HEAD: The current operator acts as the head of the chain. It only breaks the chain with upstream operators. Downstream nodes remain chained with the current operator. NEVER: The current operator is not chained with any upstream or downstream operators.
   Operator State Time-to-Live (TTL) Settings	You can set the expiration time in seconds, minutes, hours, or days. The default is the job's expiration time. For jobs without a configured expiration time, the default is 1.5 days. For more information about configuring a job's expiration time, see Runtime parameter configuration. Note Only supported in real-time computing engine VVR 8.0.7 and later versions. Only stateful operators support TTL configuration.

4. Click OK.