【DSW Gallery】Using Python SDK to manage DSW instances

overview
PAI-DSW (Data Science Workshop) is a cloud machine learning interactive development IDE tailored for data science teams and algorithm teams, making reading data, developing, training and deploying models easier, more efficient and safer.
While using DSW on PAI's web console, you can also use OpenAPI and the corresponding Python SDK to use DSW to manage your DSW instance based on code. This article will introduce how to use the DSW Python SDK (the bottom layer encapsulates the call to OpenAPI) to create, stop, start, delete instances, and save the environment (mirror).
You can refer to PAI-DSW Quick Start to get more functions and best practices related to DSW and PAI.
prerequisite
• Activate PAI-DSW and complete the authorization. For details, please refer to DSW activation and authorization.
• [Optional] Purchase a dedicated resource group, which is currently in the testing period, please contact us if necessary.
• You have obtained the AccessKey ID and AccessKey Secret of your Alibaba Cloud account. For details, see Obtaining an AccessKey.
Step 1: Install the Python SDK
• Install the AI Workspace SDK
• Install PAI-DSW SDK
• Install the SDK for dedicated resource group management (optional)
!pip install alibabacloud-aiworkspace20210204 -U -q
!pip install alibabacloud_pai_dsw20220101 -U -q
# Install the SDK related to the functions of the exclusive resource group. Before the official release to pypi, please use the following address:
!pip install https://sdk-portal-us-prod.oss-accelerate.aliyuncs.com/downloads/u-b8602de7-c468-436c-8a02-2eca4a30d376-python-paistudio.zip -U -q
Step 2: Initialize the Client object in the Python SDK
PAI's Python SDK is the standard Alibaba Cloud SDK, and each sub-product has its own installation package and package; however, the usage of each package is the same:
• Import the Client class corresponding to the sub-product (Aliyun service) from the client Package
• Import various request/response classes that need to be used from the modelsl Package
• Use Alibaba Cloud AK to initialize the Client object and specify the Region
• Call various methods in the Client object, each method corresponds to an OpenAPI, simple parameters are python atomic types, and complex parameters are various request/response objects in the models Package
# Import various Client and Models classes
from alibabacloud_pai_dsw20220101.client import Client as DSWClient
from alibabacloud_aiworkspace20210204.client import Client as AIWorkspaceClient
from alibabacloud_aiworkspace20210204.models import (ListWorkspacesRequest,
ListImagesRequest,
ListDatasetsRequest,
ListResourcesRequest)
from alibabacloud_pai_dsw20220101.models import (ListInstancesRequest,
ListEcsSpecsRequest,
CreateInstanceRequest,
StopInstanceRequest,
CreateInstanceShutdownTimerRequest,
CreateInstanceRequestRequestedResource,
CreateInstanceSnapshotRequest,
ListInstanceSnapshotRequest,
GetInstanceMetricsRequest,
CreateIdleInstanceCullerRequest)
from alibabacloud_tea_openapi.models import Config as AliyunConfig
from alibabacloud_tea_openapi.client import TeaException
from alibabacloud_paistudio20220112.models import (ListResourceGroupMachineGroupsRequest)
from alibabacloud_paistudio20220112.client import Client as StudioClient
# Construct the Client object of each sub-product (that is, the Service of different sub-products)
# Prepare the AK of the Region and Alibaba Cloud account
region_id = 'cn-beijing'
access_key_id = '**your access_key_id**'
access_key_secret = '**your access_key_secret**'
import os.path
config_path="/mnt/data/pai.config"
if os.path.isfile(config_path):
with open(config_path) as f:
access_key_id = f.readline().strip(' ')
access_key_secret = f.readline().strip(' ')
# create client
workspace_client = AIWorkspaceClient(
AliyunConfig(access_key_id=access_key_id,
access_key_secret=access_key_secret,
region_id=region_id,
endpoint='aiworkspace.{}.aliyuncs.com'.format(region_id)))
dsw_client = DSWClient(
AliyunConfig(access_key_id=access_key_id,
access_key_secret=access_key_secret,
region_id=region_id,
endpoint='pai-dsw.{}.aliyuncs.com'.format(region_id)))
# Studio Client is only required for prepaid resource groups
studio_client = StudioClient(AliyunConfig(access_key_id=access_key_id,
access_key_secret=access_key_secret,
region_id = region_id))
# Define a Helper function to display DSW instance information
def show_instance(instance_id):
instance = dsw_client.get_instance(instance_id=instance_id).body
print(instance.status, instance.instance_name, instance.ecs_spec, instance.accumulated_running_time_in_ms)
Step 3: Getting Ready - AI Assets in the Workspace
The use of PAI-DSW often involves information such as OSS, NAS, CPFS, container mirroring, and proprietary resource groups. In order to facilitate management, the PAI platform uniformly calls these resources "AI assets" and organizes them using workspaces to facilitate reuse and authority management. Workspace-related functions can be queried and manipulated using the workspace_client object.
Prepare AI workspace (required)
Workspace is the top-level concept of PAI, which provides unified computing resource management and personnel authority management capabilities for the team. The purpose is to provide AI developers with full-process development tools and AI asset management capabilities that support team collaboration.
When the PAI platform is activated, it will automatically create a default workspace for the user
# Query the properties and ID of an existing workspace (Workspace)
workspace_name = '**Existing AI workspace name**'
# Get a list of workspaces.
workspaces = workspace_client.list_workspaces(ListWorkspacesRequest(
page_number=1,
page_size=10,
workspace_name=workspace_name, # Fuzzy matching, if no name is specified, all workspaces will be returned
))
if len(workspaces. body. workspaces) == 0:
raise RuntimeError('Please specify the correct workspace_name')
for workspace in workspaces.body.workspaces:
print(workspace.workspace_name,
workspace.workspace_id,
workspace.status, workspace.creator)
# Use the first result of the query as the workspace for subsequent operations, you can switch as needed or directly specify a string-type ID
workspace_id = workspaces.body.workspaces[0].workspace_id
Prepare to mirror (required)
When creating a DSW instance, you need to specify a certain image as the AI development environment. PAI-DSW supports the selection of different types of images:
• Community mirrors: Standard mirrors provided by the community. For details about different mirrors, see Community Mirror Version Details.
• PAI platform image: various official images provided by Alibaba Cloud PAI products, supporting different resource types, Python #image_id='', #You can specify the ID of the image in the workspace, but only one parameter can be specified with image_url
image_url=image_uri)
try:
ins_resp = dsw_client.create_instance(request)
except TeaException as t:
print('Create instance failed, error message:' + t.message)
else:
instance_id = ins_resp.body.instance_id
print("Created Instance ID:", instance_id)
show_instance(instance_id)