Data Science Workshop (DSW) is a cloud-based integrated development environment (IDE) provided by Platform for AI (PAI). DSW consolidates a variety of tools to facilitate your development at different stages and for different scenarios. This topic describes the features and common operations supported by DSW.
DSW integrates open source JupyterLab and provides plug-ins for customized development. You can directly start Notebook in DSW to write, debug, and run Python code without any configurations. DSW also provides various computing resources and supports multiple data sources. You can use EASCMD to deploy models that are trained in DSW as online services in Elastic Algorithm Service (EAS).
Features
Supports real-time resource monitoring. The CPU or GPU utilization can be monitored during algorithm development.
Supports a variety of data sources, such as MaxCompute, Object Storage Service (OSS), and Apsara File Storage NAS (NAS).
Allows you to write and execute SQL statements.
Supports a variety of resource types, including those that belong to the public and dedicated resource group types. If you select the public resource group type, a variety of CPUs and GPU models are provided.
Allows you to switch between different resource types to reduce resource usage costs.
Provides built-in big data development packages and algorithm libraries, and allows you to install third-party libraries.
Procedure
Step 1: Preparations
Prepare a resource group
Before you create a DSW instance, you need to prepare the general computing resources or intelligent computing LINGJUN resources that are required by the training jobs. The general computing resources are packed in resource groups, including a public resource group and dedicated resource groups. To prepare the general computing resources, you must complete the authorization for Deep Learning Containers (DLC). After the authorization is completed, the system automatically creates the general resource group. You do not need to manually add a public resource group. For more information about DLC authorization, see Grant the permissions that are required to use DLC. Before you use a dedicated resource group, you must purchase and configure a resource group. For more information, see Create a resource group and purchase general computing resources and Lingjun resource quotas.
Prepare a dataset
Create a DSW instance by using the public resource group:
PAI provides a free disk quota for persistent storage. The amount of free storage space is limited by the quota. You can mount datasets to obtain a larger storage space.
Create a DSW instance by using a dedicated resource group:
DSW instances use the built-in system disk for temporary storage. You can mount datasets to persist data.
You can mount OSS, NAS, and Cloud Parallel File Storage (CPFS) datasets. For information about how to create a dataset, see Create and manage datasets.
Prepare an image
Before you create a DSW instance, you need to prepare an image for the instance. The DSW instance builds an environment based on the specified image. The following types of images are supported:
Official images
DSW provides various types of preset official images, such as PyTorch, TensorFlow, and ModelScope. PAI provides multiple image versions to meet your requirements for a specific framework version. This facilitates model development, training, and deployment.
Custom images
You can also use a custom image to create a DSW instance to meet your development requirements in specific scenarios. For information about how to create a custom image, see View and add images.
Step 2: Create and use a DSW instance
After you complete the preceding preparations, you can create a DSW instance. For more information, see Create and manage DSW instances.
The following topics provide descriptions on how to use DSW instances in specific scenarios:
Step 3: Prepare data files for development
DSW supports uploading data to and downloading data from multiple data sources, including OSS, NAS, and MaxCompute. You can import data files that are required for development from different data sources to a DSW instance and export the processed data to the specified data source. You can also use a DSW instance to upload and download small-sized data files. For more information about the data transmission feature, see Work with DSW.
Step 4: Use DSW to debug and train models
DSW provides an interactive development environment that allows you to track the model training process and debug the code. You can train models on the DSW instance page. You can also manage third-party libraries in a DSW instance to meet your development requirements. For more information, see Manage third-party libraries. For information about the best practices on how to use DSW, see DSW use cases.
What to do next
You can deploy a trained model as an EAS online service to perform model inference. For more information, see Deploy model services by using EASCMD or DSW.