Machine Learning Platform for AI (PAI) allows you to associate fully managed Flink resources with PAI workspaces and use the resources for the large-scale distributed training of models. This topic describes how to purchase fully managed Flink resources and use the resources to train models.
Fully-managed Flink resources
Alibaba Cloud Realtime Compute for Apache Flink is an end-to-end real-time big data analytics platform that is built on Apache Flink, and is capable of processing data with sub-second response times. For more information, see What is Alibaba Cloud Realtime Compute for Apache Flink?
Prerequisites
Before you perform the operations described in this topic, make sure that the following requirements are met:
You have an Alibaba Cloud account. If you do not have an Alibaba Cloud account, create one first.
Account and permission requirements
Alibaba Cloud account (recommended): You can use an Alibaba Cloud account to complete all operations without additional authorization.
RAM user:
If you want to use a RAM user to purchase fully managed Flink resources, you must grant the
AliyunStreamFullAccesspermissions to the RAM user. For more information, see Grant permissions to a RAM user.If you want to use a RAM user to associate fully managed Flink resources with a workspace, you must assign the administrator role to the RAM user in the workspace. For more information, see Manage the members of a workspace.
If you want to use a RAM user to train models by using fully managed Flink resources in Machine Learning Designer, you must assign the algorithm development role to the RAM user in the workspace. For more information, see Manage the members of a workspace.
Purchase fully managed Flink resources
Log on to the PAI console.
Follow the instructions shown in the following figure to go to the Fully Managed Flink Resources tab.
Optional. On the Fully Managed Flink Resources page, click Activate.
NoteThis step is required if you have not activated Realtime Compute for Apache. For more information about how to activate Realtime Compute for Apache Flink and purchase fully managed Flink resources, see Activate fully managed Flink.
If you have activated Realtime Compute for Apache Flink and want to purchase more fully managed Flink resources, go to the next step.
On the Flink tab, click Resource Management. You are navigated to the Realtime Compute for Apache Flink console.
In the Realtime Compute for Apache Flink console, click Purchase to purchase fully managed Flink resources. For more information, see Activate fully managed Flink.
After you complete the purchase, you can view the fully managed resources that you purchased and key information about the resources.
Associate fully managed Flink resources with a workspace
If you want to use fully managed Flink resources for computing in a workspace, you need to associate the resources with the workspace by using one of the following methods:
Add a resource group to a workspace when you create the workspace. For more information, see Create a workspace.
Associate fully managed Flink resources with an existing workspace by performing the following steps:
Log on to the PAI console.
In the left-side navigation pane, click Workspaces. On the Workspace list page, find the workspace with which you want to associate fully managed Flink resources and click the workspace name.
On the workspace details page, follow the instructions shown in the following figure to associate fully managed Flink resources with the workspace.
Train models in Machine Learning Designer by using fully managed Flink resources
Go to the workspace that is associated with fully managed Flink resources. Create a blank pipeline on the Visualized Modeling (Machine Learning Designer) page. For more information, see Prepare data.
You can drag the following components that can run on fully managed Flink resources to the canvas.
All components of the Alink framework, except those in the Beta Algorithm folder. Each Alink component is marked by a purple dot.
Custom algorithm components. For more information, see PyAlink Script.
On the Pipeline properties tab in the right-side pane, set the Default Resource Type Preferred by Alink parameter to Flink.
ImportantIf you want to run Alink components in a group, you must set the Default Resource Type Preferred by Alink parameter to Flink. Otherwise, resources of the default type for the Alink group are used. For more information about how to set the execution resource type for Alink groups, see Advanced feature: Alink components run in groups.
Run components by using one of the following methods:
Run a single component that depends on fully managed Flink resources. For example, you can run a PyAlink Script component. For more information, see PyAlink Script.
Concatenate and run components that depend on fully managed Flink resources and components that depend on resources of other types. For example, you can create a Factorization Machine (FM) recommendation model that includes the FM Train, FM Prediction, and Binary Classification Evaluation components. The FM Train and FM Prediction components depend on fully managed Flink resources, whereas the Binary Classification Evaluation component depends on MaxCompute resources. For more information, see Create an FM recommendation model based on the Alink framework.
Run multiple components that depend on fully managed Flink resources at a time. For more information, see Advanced feature: Alink components run in groups.
After you run a component, right-click the component on the canvas and select View Log to view the logs of the component.
On the Log tab, you can also click a Ververica Platform (VVP) link in the log content to view the computing details of the component.