All Products
Search
Document Center

DSW2.0 Introduction

Last Updated: Jun 03, 2020

PAI-DSW 2.0 is released on May 7, 2020. This new version can provide you a more flexible and open AI development environment and a better smooth development experience. PAI-DSW 2.0 is an integrated development environment that integrates computing resources and development platforms. It helps to reduce the workloads on operation and maintenance, and enables advanced AI developers to do custom development. PAI_DSW 2.0 further reduces the costs and improves the efficiency. Currently, PAI-DSW 2.0 is available in the Hangzhou region and Shenzhen region.

Overview

PAI-DSW 2.0 uses the underlying computing resources Alibaba Cloud ECS and cloud disk storage. Based on Alibaba cloud native technologies such as Container Registry Docker and Container Service for Kubernetes, PAI-DSW 2.0 builds a fully integrated machine learning development environment. PAI-DSW 2.0 provides the following features:

  1. New methods of purchasing PAI services, rich computing resources and convenient purchasing experience.
  2. Efficient and secure instance management. You can stop or start a DSW instance as needed, and one-click save an image. PAI-DSW 2.0 can quickly restore the instance development environment and the VPC access.
  3. An integrated AI development environment.
  • Built-in big data development packages and algorithm packages and the sudo permission to install the third-party libraries.
  • Built-in plug-ins of JupyterLab, such as Git, and Tensorboard to improve development efficiency.
  • Official images, which covers multiple versions of mainstream computing frameworks, such as TensorFlow and PyTorch.
  • Embedded WebIDE, which supports users to install plug-ins.
  1. It provides built-in PAI basic capabilities, the vision algorithm package EasyVision, the automatic parameter adjustment tool AutoML, the compilation optimization Tensor Accelerator and Optimizer (TAO), and the direct reading MaxCompute CommonIO.

image.png

Create an instance

Log on to the PAI-DSW console, and click Create Instance. On the purchase page, select the services such as the region and availability zone, instance resource, and image version, to complete the purchase and creation of a DSW instance. Steps:

Configure an instance

Instance name: It must start with an uppercase letter, a lowercase letter, a Chinese character or a digit, and can contain “-“ or “_”. The length is 1 to 27 characters.
Instance version: DSW 2.0.
Regions and availability zones: The current available regions are Hangzhou and Shenzhen. Select the appropriate region. For more information, see Regions and zones.
Billing method: supports the pay-as-you-go method.
Network configuration: The private network VPC, the security groups and switches are created by the PAI-DSW platform.
Instance resource : The applicable instance family includes ecs.g5, ecs.c5, ecs.gn6v, and ecs.gn5. For more information, see ECS instance family.
Instance image: includes various versions of Python, TensorFlow, and PytTorch.

Name Type
py36_cuda100_tf2.0_ubuntu GPU
py36_cpu_tf1.15_ubuntu CPU
py36_cpu_tf2.0_ubuntu CPU
py36_cuda90_tf1.12_ubuntu GPU
py36_cpu_tf1.12_ubuntu CPU
py36_cuda100_tf1.15_ubuntu GPU
py27_cpu_tf1.12_ubuntu CPU
py27_cuda90_tf1.12_ubuntu GPU
py36_cuda101_torch1.3_ubuntu GPU
py36_cpu_torch1.3_ubuntu CPU
py36_cuda101_tf2.1_torch1.4_ubuntu GPU
py36_cpu_tf2.1_torch1.4_ubuntu CPU

Storage: supports the system disk to install the basic image of the instance. You can specify the type and capacity of the system disk. The minimum size is 40 GB.
After you complete the above configurations, click Confirm to enter the confirmation page. You cannot modify the configurations except for the instance name after a DSW instance is created.

Confirm Order

On the confirmation page, you can view the selected configurations of the instance and the dependency inspection status. Select PAI-DSW Product Agreement, and click Create Instance to complete the creation.
Verification: If you are not a verified user, you must follow the instruction to complete the verification.
Authorization: If the authorization fails, visit Cloud resource access authorization. For more information, see the description of DSW role permission.
ESS status: To create a DSW instance, you need to activate ESS. The product is free to use. You can go to the ESS console to activate the service.
Account status: A DSW instance uses the ECS service, so you need to ensure that the account balance is greater than or equal to 100 yuan, including cash and coupons.
Click Retest to refresh
Select Product agreement, and click Create Instance to complete the instance creation. You are redirected to the PAI-DSW console. In the PAI-DSW console, you can find the newly created DSW instance. The bill is charged after the DSW instance is created. For more information about the billing method, see the PAI-DSW billing method.

Instance management

Start a DSW instance

If the instance is stopped or failed to start, you can click the start button to restart the instance. After the DSW instance starts, the system automatically loads the image saved last time and restores the development environment. The bill is charged after the DSW instance is created.

Stop a DSW instance

After the instance is stopped, ECS and DSW stop billing. Methods to stop a DSW instance:

  • Direct stop: You can directly release the ECS instance and stop the DSW instance. The instance stops billing.
  • Save the system and then stop: After you save the image, stop the ECS instance. You do not need to pay for an ECS instance after it is stopped. When you restart the DSW instance again, the system restores the development environment. If you choose this way of stop, you need a little longer time to stop a DSW instance than the direct stop.


Delete a DSW instance

You can delete the DSW instance and the ECS instance in the PAI-DSW console. The VPC, switches, and security groups created by default are remained.

Instance environment

Multiple development environments

PAI-DSW 2.0 provides a built-in JupyterLab environment for interactive development of machine learning, the TensorBoard visualization tool, and Git plug-ins. It helps users to understand, debug, optimize TensorFlow programs and manage codes. PAI-DSW 2.0 also supports WebIDE. You can manage each ipynb file of JupyterLab in WebIDE. You can directly debug the and write the. py codes in the browser, and track the running of the programme. You can also install plug-ins as needed. If you are accustomed to command-line programming, you can better use the optimized terminal and have a focused development experience.

The sudo permission

To meet the needs of users to install various dependent packages and provide a local development experience, PAI-DSW 2.0 opens the sudo permission and supports users to install plug-ins.

PAI basic capabilities

PAI-DSW 2.0 provides built-in basic capabilities of PAI. You can use the vision algorithm package EasyVision to perform image classification and prediction in a DSW instance. You can use automatic tuning of algorithm hyperparameters through automatic tuning of AutoML. You can use the optimized algorithm during model training that are provided by the algorithm component TAO. PAI-DSW 2.0 also supports the algorithm component CommonIO to read MaxCompute table data directly. It supports standard interfaces such as TableRecordDataSet, TableReader, and TableWriter, which helps the training program directly submitted to the distributed training cluster. For more information about PAI, see the built-in demo files in DSW.

Write ODPS SQL using dswmagic

Reading and write MaxCompute using pyodps

Getting started with PAIAutoML hyperparameter tuning

Advanced PAIAutoML hyperparameter tuning

Advanced PAIAutoml hyperparameter tuning - PAITF

Access the data of OSS Bucket in DSW

Read the data of OSS using tensorflow_io.oss

Evaluate the image classification and predict the model using easy-vision

A Word2Vec case

Resource Access Management (RAM) authorization

Before you create a DSW instance, you need to grant/authorize the following role permissions.

Name Description
AliyunPAIDSWDefaultRole PAI-DSW uses this role to access your cloud resources.
AliyunCSDefaultRole Container Service uses this role to access your cloud resources (during cluster operations).
AliyunCSManagedLogRole Container Service uses this role to access your cloud resources (during cluster operations).
AliyunCSManagedCmsRole Container Service uses this role to access your cloud resources (during cluster operations).
AliyunCSClusterRole Container Service uses this role to access your cloud resources.
AliyunCSKubernetesAuditRole Container Service for Kubernetes uses this role to access your cloud resources.
AliyunCSManagedNetworkRole Container Service uses this role to access your cloud resources.
AliyunCSManagedKubernetesRole Container Service for Managed Kubernetes uses this role to access cloud resources.
AliyunCSKubernetesAuditRole Container Service for Kubernetes uses this role to access your cloud resources.
AliyunESSDefaultRole ESS uses this role to access your cloud resources.