Resource queues in Realtime Compute for Apache Flink isolate and allocate compute resources across deployments within a namespace. Each namespace includes a default queue named default-queue. Create custom queues to prevent a single deployment or user from consuming all available resources and causing failures for others.
Usage notes
Queues can run only in exclusive mode. Queues cannot run in preemption mode. If a queue runs in preemption mode, the resources in the queue cannot be preempted.
After specific resources in a queue are released, the resources can be used by a new deployment.
If no custom queue exists, all deployments run in
default-queue.
Assign a queue to a workload
| Workload type | Where to set | Parameter | Reference |
|---|---|---|---|
| Deployment | Deploy draft dialog box | Deployment Target | Create a deployment |
| Session cluster | Standard section of the Create Session Cluster page | Deployment Target | Create a session cluster |
| Workflow | Create Workflow panel | Resource Queue | Manage workflows |
Common console steps
The Add, Edit, and Delete procedures all begin with the same navigation steps:
Log on to the Realtime Compute for Apache Flink console.
Find the target workspace and click Console in the Actions column.
In the left-side navigation pane, choose O&M > Queue Management.
Add a queue
Queue name rules
| Rule | Detail |
|---|---|
| Uniqueness | Must be unique within the namespace |
| Length | Maximum 32 characters |
| Characters | Lowercase letters, digits, and hyphens (-) only |
| Mutability | Cannot be changed after creation |
Procedure
Complete the common console steps.
In the upper-left corner of the Queue Management page, click Add queue.
Configure the queue parameters. The parameters differ depending on whether elastic resources are allocated to the namespace.
Namespaces without elastic resources
Applies when the workspace billing method is pay-as-you-go, subscription, or Hybrid billing and no elastic resources are allocated to the namespace.
Parameter Description Queue Name Enter a name that follows the queue name rules. Resources (CU) Number of CUs available to the queue. The maximum value equals the remaining unallocated CUs in the namespace. If not enough CUs are available, reconfigure CUs in other queues (see Edit a queue) or increase namespace resources (see Reconfigure resources). Namespaces with elastic resources
Applies when the workspace billing method is Hybrid billing and elastic resources are allocated to the namespace.
Parameter Description Queue Name Enter a name that follows the queue name rules. Fixed Resources (CUs) Number of fixed CUs available to the queue. If not enough fixed CUs are available, adjust resources using the subscription workspace method. See Reconfigure resources. Elastic Resources (CUs) Number of elastic CUs available to the queue. The Queue Management page shows the maximum allocatable elastic CUs. If not enough elastic CUs are available, adjust resources using the pay-as-you-go workspace method. See Reconfigure resources. Click OK.
Edit a queue
Adjust CU allocation for an existing queue when resource needs change.
Complete the common console steps.
Find the target queue and click Edit in the Actions column.
In the Edit Queue dialog box, update the resource values. For parameter descriptions, see Add a queue.
ImportantThe Resources (CU) value cannot be set lower than the number of CUs currently in use by the queue.
Click OK.
Delete a queue
The default queue (
default-queue) cannot be deleted.A queue with running deployments cannot be deleted.
Deleting a queue that a deployment references causes an error when that deployment restarts.
Complete the common console steps.
Find the target queue, click Delete in the Actions column, and then click OK.
Use cases
Isolate batch workloads: Assign batch deployments to dedicated queues so they do not starve other jobs of resources.
Prioritize critical jobs: Combine queues with the Workflows (public preview) feature to set scheduling priorities for batch drafts.
References
Reconfigure resources -- Adjust namespace resources or quota limits.
Create a deployment -- Create SQL, JAR, or Python deployments.
Create a session cluster -- Set up a session cluster for debugging.
Workflows (public preview) -- Schedule and prioritize batch deployments.