Dataphin:Alibaba Blink custom resources for real-time mode

Alibaba Blink custom resource configuration entry

1. On the Dataphin home page, single click the top menu bar Development to navigate to the Development page.

2. On the Development page, use the following figure as a guide to reach the custom resource configuration page for BLINK_SQL tasks.

   

Configuration description

The resource configuration page displays a topology where each block represents a computing task with configurable resources. Grouping nodes on the same machine minimizes cross-network data transfers and improves performance.

Configure group runtime parameters

1. Single click the in the upper right corner of the desired group to open the Customize Group Runtime Parameters dialog box and set the parameters.

   Parameter	Description
   core	Typically set to 0.25, meaning one CPU supports up to four simultaneous threads. Maximum value: 1.
   heap_memory	The heap memory size for a Java application, in MB. You can adjust heap_memory and its components using JVM command-line parameters. Allocate additional heap memory for program caches and other overheads based on the program's scale. Blink programs typically require extra heap memory overhead, such as memory for program caching. Adjust the size according to the program's scale.
   parallel	The number of concurrent threads. A higher value demands more resources and may not always improve performance. Typically, a compute node can process two to four thousand data pieces per second. Important If the source is tt, the queue size of tt caps the concurrency. Exceeding this limit results in an error.
   direct_memory	The direct memory size in MB, allocated outside the JVM heap. Required for tasks that use igraph or swift. Recommended value: 16 to 32 MB. Direct memory improves read and write efficiency by avoiding data copying between the Java heap and native heap. Java NIO uses channels and buffers for data operations. By setting direct_memory, you allocate non-heap memory through native functions. A direct ByteBuffer object enables direct reads without transferring data between the Java and native heaps, improving I/O performance.
   native_memory	The native memory size in MB. The actual usage varies based on OS processes, bytecode length, thread count, garbage collection data, and third-party packages. For example, a 32-bit OS may support up to 3-4 GB of native memory. Native memory stores: Java object status for garbage collection and heap management. JNI call information. JIT compilation data, including Java bytecode and machine code. Direct buffer details.

2. Once parameters are set, single click OK.

Configure operator runtime parameters

1. Access the Customize Operator Runtime Parameters dialog box and configure the parameters.

   For details on the core, heap_memory, parallel, direct_memory, and native_memory parameters within the Customize Operator Runtime Parameters dialog box, refer to Configure group runtime parameters and . The table below provides descriptions for only the state_size and chain_strategy parameters.

   Parameter	Description
   state_size	The default state data size used during job execution, typically set to 0.
   chain_strategy	The node chaining policy. Valid values: Always: The default setting, indicating all nodes are on the same machine unless specified otherwise. Never: Each node is deployed independently on a separate machine. Head: The node can share a machine with others but only as the head node in a group. Important Head and Never are rarely used. In most cases, use Always.

2. After setting the parameters, single click OK.