Associate fully managed Flink resources with a PAI workspace to run large-scale distributed model training jobs using Alibaba Cloud Realtime Compute for Apache Flink — an end-to-end real-time big data analytics platform built on Apache Flink that can process data with sub-second response times.
Prerequisites
Before you begin, make sure you have:
You have an Alibaba Cloud account. If you do not have one, create one first.
The required permissions for your account type (see Permissions below)
Permissions
An Alibaba Cloud account can complete all operations without additional authorization. RAM users require specific permissions depending on the operation.
| Operation | Permission required |
|---|---|
| Purchase fully managed Flink resources | AliyunStreamFullAccess policy — see Grant permissions to a RAM user |
| Submit a job to Flink | Owner role in the Flink console namespace — see Authorize an account to perform operations in a namespace |
| Associate fully managed Flink resources with a workspace | Administrator role in the workspace |
| Run model training in Machine Learning Designer | Algorithm developer role in the workspace |
For workspace role management, see Manage members of a workspace.
Purchase fully managed Flink resources
Log on to the PAI console.
In the left-side navigation pane, choose AI Computing Resources > Resource Quota. On the Resource Quota page, click the Fully Managed Flink Resources tab.
(First-time activation only) Click Activate and complete the purchase process. For setup instructions, see Activate fully managed Flink.
NoteSkip this step if you have already activated Realtime Compute for Apache Flink.
On the Fully Managed Flink tab, click Resources.
In the Realtime Compute for Apache Flink console, click Purchase. For purchase options, see Activate fully managed Flink.
After purchasing, the Fully Managed Flink Resources tab displays your resource details.
Associate fully managed Flink resources with a workspace
Associate Flink resources with a workspace using either of the following methods:
During workspace creation: Add the resource group when creating the workspace. See Create and manage a workspace.
For an existing workspace: For details, see Manage the computing resources of a workspace.
Log on to the PAI console.
In the left-side navigation pane, click Workspaces. On the workspace list page, click the target workspace name.
On the right side of the workspace details page, choose Configure Workspace > Configure Computing Resource. On the Fully Managed Flink Resources tab, associate the resource.
Train models in Machine Learning Designer
Supported components
The following component types run on fully managed Flink resources:
| Component type | Details |
|---|---|
| Alink framework components | All components except those in the Beta Algorithm folder. Each Alink component is marked by a purple dot. |
| Custom algorithm components | See PyAlink Script. |
Run a training pipeline
Go to the workspace associated with fully managed Flink resources. On the Visualized Modeling (Designer) page, create a blank pipeline. See Create a custom pipeline.
Drag the supported Alink or custom components onto the canvas.
On the Pipeline properties tab in the right-side pane, set Default Resource Preferred by Alink or FlinkML to Flink.
ImportantTo run Alink components as a group, set Default Resource Type Preferred by Alink to Flink. If this parameter is not set, the Alink group uses its own default resource type. See Alink components.
Run components using one of the following approaches:
Approach Description Reference Single component Run one component that uses fully managed Flink resources, such as a PyAlink Script component. PyAlink Script Mixed resource types Combine Flink-based and non-Flink components in a single pipeline. For example, a Factorization Machine (FM) recommendation model can include FM Train and FM Prediction components (Flink) alongside a Binary Classification Evaluation component (MaxCompute). Create an FM recommendation model based on the Alink framework Multiple Flink components Run several Flink-based components at once. Alink components After a component finishes, right-click it on the canvas and select View Log to inspect the output. On the Log tab, click a Ververica Platform (VVP) link to view the component's computing details.
