Platform for AI (PAI) allows you to associate fully managed Flink resources with PAI workspaces and use the resources for the large-scale distributed training of models. This topic describes how to purchase fully managed Flink resources and use the resources to train models.
Fully-managed Flink resources
Alibaba Cloud Realtime Compute for Apache Flink is an end-to-end real-time big data analytics platform that is built on Apache Flink, and can process data with sub-second response times. For more information, see What is Alibaba Cloud Realtime Compute for Apache Flink?
Prerequisites
Before you perform the operations described in this topic, make sure that the following requirements are met:
You have an Alibaba Cloud account. If you do not have an Alibaba Cloud account, create one first.
Account and permission requirements
Alibaba Cloud account (recommended): You can use an Alibaba Cloud account to complete all operations without additional authorization.
RAM user:
If you want to use a RAM user to purchase fully managed Flink resources, you must grant the
AliyunStreamFullAccesspermissions to the RAM user. For more information, see Grant permissions to a RAM user.If you want to submit a task to Flink, you must assign the owner role to the RAM user in the namespace in the Flink console. For more information, see Authorize an account to perform operations in a namespace.
If you want to use a RAM user to associate fully managed Flink resources with a workspace, you must assign the administrator role to the RAM user in the workspace. If you want to use a RAM user to perform model training by using fully managed Flink resources in Machine Learning Designer, you must assign the algorithm developer role to the RAM user in the workspace. For more information, see Manage members of the workspace.
Purchase fully managed Flink resources
Log on to the PAI console.
In the left-side navigation pane, choose . On the Resource Quota page, click Fully Managed Flink Resources.
(Optional) On the Fully Managed Flink Resources tab, click Activate.
NoteThe first time you use fully managed Flink resources, you need to perform this operation and purchase fully managed Flink resources. For more information, see Activate fully managed Flink.
If you have activated Realtime Compute for Apache Flink and want to purchase more fully managed Flink resources, go to the next step.
On the Fully Managed Flink tab, click Resources.
In the Realtime Compute for Apache Flink console, click Purchase. For more information about how to purchase fully managed Flink resources, see Activate fully managed Flink.
After you purchase fully managed Flink resources, you can view the resources and other information on the Fully Managed Flink Resources tab.
Associate fully managed Flink resources with a workspace
If you want to use fully managed Flink resources for computing in a workspace, you need to associate the resources with the workspace by using one of the following methods:
Add a resource group to a workspace when you create the workspace. For more information, see Create a workspace.
Associate fully managed Flink resources with an existing workspace by performing the following steps:
Log on to the PAI console.
In the left-side navigation pane, click Workspaces. On the Workspace list page, find the workspace with which you want to associate fully managed Flink resources and click the workspace name.
On the right side of the Workspace Details page, click Resources and follow the on-screen instructions to associate fully managed Flink resources with the workspace. For more information, see the "Manage computing resources of the workspace" section in the Manage workspaces topic.
Train models in Machine Learning Designer by using fully managed Flink resources
Go to the workspace that is associated with fully managed Flink resources. Create a blank pipeline on the Visualized Modeling (Designer) page. For more information, see Prepare data.
You can drag the following components that can run on fully managed Flink resources to the canvas.
All components of the Alink framework, except those in the Beta Algorithm folder. Each Alink component is marked by a purple dot.
Custom algorithm components. For more information, see PyAlink Script.
On the Pipeline properties tab in the right-side pane, set the Default Resource Type Preferred by Alink parameter to Flink.
ImportantIf you want to run Alink components in a group, you must set the Default Resource Type Preferred by Alink parameter to Flink. Otherwise, resources of the default type for the Alink group are used. For more information about how to set the execution resource type for Alink groups, see Advanced feature: Alink components run in groups.
Run components by using one of the following methods:
Run a single component that depends on fully managed Flink resources. For example, you can run a PyAlink Script component. For more information, see PyAlink Script.
Concatenate and run components that depend on fully managed Flink resources and components that depend on resources of other types. For example, you can create a Factorization Machine (FM) recommendation model that includes the FM Train, FM Prediction, and Binary Classification Evaluation components. The FM Train and FM Prediction components depend on fully managed Flink resources. The Binary Classification Evaluation component depends on MaxCompute resources. For more information, see Create an FM recommendation model based on the Alink framework.
Run multiple components that depend on fully managed Flink resources at a time. For more information, see Advanced feature: Alink components run in groups.
After you run a component, right-click the component on the canvas and select View Log to view the logs of the component.
On the Log tab, you can also click a Ververica Platform (VVP) link in the log content to view the computing details of the component.