DSW Gallery

DSW Gallery provides a wealth of cases and solutions in AI R&D scenarios, covering such as: PAI/DSW function introduction, how to use Alibaba Cloud products (including SDK or CLI tools), data science, deep learning, and industry solutions, It supports quick start in DSW, helping you to quickly familiarize yourself with the AI R&D process under Cloud Native, and use PAI, Alibaba Cloud and other products to improve the efficiency and quality of R&D.
prerequisite
• Aliyun account
• Activate PAI
• At least one instance of DSW running
• OSS, MaxCompute, NAS, Dataworks (open as needed)
⌨️ How to use
The cases provided by DSW Gallery consist of at least 3 parts
1. Summary: title, description, tags
2. Documents: case use documents (detailed step descriptions and related codes, diagrams, etc.)
3. Resources: ipynb and related codes, which can be loaded and executed by DSW
Summary
How to quickly find the case you need?
• DSW Gallery can conduct fuzzy queries based on keywords in case summaries. You can use keywords in the title or description, or select tags of interest, see ① ② ③ in Fig.1.
• ① Fuzzy search by case title and description
• ② Display cases by type
• ③ Search by tags, click on a tag to retrieve all cases containing the tag
• DSW Gallery supports multiple keyword combination queries, and the weights are sorted from high to low. For example: When two conditions are combined to query: first search for cases under "Alibaba Cloud Products", and then continue to search for cases satisfying "name or description odp" in the query results of the previous condition.
Fig.1 - Search Sample in Gallery
document
The document shows the usage method or usage details of the case, and roughly includes three parts:
1. Prerequisites: Necessary conditions for use cases (MaxCompute, HDFS, Dataworks, EAS, DLC, PAIFlow, NNI, Automl, Blade, etc.)
2. Operating environment: DSW instance specification (CPU/MEM/GPU), Docker image version
3. Detailed steps: the code of each step of the actual operation, commands, and output pictures, charts, etc. after execution
resource
After browsing the case document, if you are very interested in it, you can load the case into the DSW instance with one click. After clicking "Open in DSW" and selecting "AI Workspace", the running DSW instance under the current "AI Workspace" will be selected by default. Take Fig.2 as an example:
• ① "Open in DSW" button in the upper right corner of the document
• ② If there is no workspace you need, please click the link to create it
• ③ If you do not have any running DSW instances in this workspace, please click the link to create
• ④ After the DSW instance is selected, click OK to go to DSW.
Fig.2 - Open Sample in DSW
Warning
1. The DSW instance specification should not be lower than the instance specification required by the operating environment in the case document, and the image version should be consistent. See Fig.3
2. If you do not have a workspace and are an account administrator. You need to contact the account administrator to add you to any workspace and grant the role of algorithm development. The specific operation method can refer to the document
3. If the case has been opened in the DSW instance before, after clicking Open in DSW of the case again, DSW will prompt the path where the case is located. You have two options, see Fig.4
• According to the prompt, go to the path where the case is located
• Re-download the case to another path
Fig.3 - Check the instance specification and image version of DSW
Fig.4 - The case already exists, prompting to download again
📚Case
Product Features
DSW is a cloud machine development IDE, the following documents can help you get started with DSW faster.
• DSW activation and authorization PAI-DSW is a cloud-based machine learning interactive development IDE tailored for AI developers. It can open Notebook anytime, anywhere to quickly read data, develop algorithms, train and deploy models. This article is used to help you quickly open DSW and authorize sub-accounts for use.
• DSW instance management This article briefly describes how to manage DSW instances in the DSW console.
• Data mounting (OSS, NAS) This article briefly describes how to manage DSW data, including (NAS, OSS) and other data mounting.
• Image management PAI-DSW relies on container technology to provide an out-of-the-box machine learning development environment. Each DSW instance uses an image to start the container. Jupyterlab, VSCode, and Terminal run in the container. After the user enters the container, use Jupyterlab to start the Kernel or Start the machine learning task by using the shell command to start the process on the Terminal.
• VPC network configuration The PAI-DSW instance runs in PAI's managed VPC, and is isolated from the user's VPC network by default. This article takes you through how to access VPC services in the PAI-DSW instance, such as reading data in the VPC RDS, code in git deployed in the clone VPC, and so on.
• Enterprise resource management: Instance sharing and authority management PAI-DSW is based on AI workspace and provides rich instance resource management and authority management capabilities to help enterprises and teams realize flexible collaborative development of multiple people.
• Payment mode PAI-DSW sales methods currently have two forms: pay-as-you-go (personal version) and pay-by-resource group (in public beta). This article takes you through the payment model of PAI-DSW.
• Introduction to DSW IDE PAI-DSW integrates the open source JupyterLab, and conducts in-depth customized development in the form of plug-ins. You can write notebooks, debug and run Python code without any operation and maintenance configuration. This article introduces how to use (JupyterLab, WebIDE, Terminal).
Alibaba Cloud product documentation
Alibaba Cloud products (such as: Alibaba Cloud Object Storage OSS, Alibaba Cloud Data Warehouse Service MaxCompute, Alibaba Cloud PAI cloud-native AI basic platform PAI-DLC, online prediction PAI-EAS, etc.) provide SDK and CLI tools, and DSW is combined with these products Use to help you speed up model development and deployment
• MaxCompute (ODPS) is an enterprise-level SaaS (Software as a Service) model cloud data warehouse suitable for data analysis scenarios. It provides access to offline and streaming data, supports large-scale data computing and query acceleration capabilities, and provides you with Data warehouse solutions and analytical modeling > services for various computing scenarios. PAI-DSW supports the following three ways to read and write MaxCompute (ODPS) table data.
• PyODPS User Guide, PyODPS is the Python SDK of MaxCompute, which provides a simple and convenient Python programming interface. PyODPS supports fast, flexible and expressive data structures like Pandas. You can use the data result processing function of Pandas through the DataFrame API provided by PyODPS. This article is used to help you quickly start using PyODPS, and it can be used in actual projects.
• PAIIO usage guide, PAIIO is a module specially developed for TensorFlow tasks to read MaxCompute Table data. It provides TableRecordDataset dataset, and you can easily use TableRecordDataset to build TF tasks.
• COMMON_IO usage guide. COMMON_IO provides two simple and easy-to-use interfaces, TableReader and TableWriter. You can use COMMON_IO to read and write MaxCompute Table data conveniently. If you want to read MaxCompute Table data to build PyTorch tasks, COMMON_IO is also recommended.
• OSS Usage Guide Alibaba Cloud Object Storage OSS (Object Storage Service) is a massive, secure, low-cost, and highly reliable cloud storage service. You can use the API, SDK interface or OSS migration tool provided by Alibaba Cloud to easily move massive data into or out of Alibaba Cloud OSS.
deep learning
PAI provides python sdk for deep learning models in various scenarios, such as: EasyVision (visual intelligence enhancement algorithm package), EasyTransfer natural language processing (NLP), EasyASR (speech intelligence enhancement algorithm package), EasyCompression (model compression).
• EasyCV image classification This article will introduce how to quickly use Resnet50 to train and infer image classification models based on EasyCV in pai-dsw.
• Use EasyVision for object detection EasyVision (Vision Intelligence Enhancement Algorithm Package) provides training and prediction functions for various models, aiming to help computer vision application developers quickly and easily build vision models and apply them to production. This article takes target detection as an example to introduce how to use EasyVision in PAI-DSW.
• EasyNLP text classification This article will introduce how to quickly use BERT to train and infer text classification models based on EasyNLP in pai-dsw.
• Introduction of Hugging Face Hugging Face (HF for short, official website address) was originally a large open source community focusing on NLP technology. The open source natural language processing pre-training model library Transformers on github has been downloaded more than one million times, and more than 64,000 on github stars. This article describes how to access HF using the Python SDK.
data science
• House Price Prediction of Kaggle Competition This article shows how to use a data set that contains numeric types and non-numeric types to do feature engineering, and finally achieve a better regression effect. It is a very good example of data analysis, which involves Panda and SKLearn applications.
• scikit-learn cookbook This article introduces the application of sklearn in machine learning
other
• Tensorflow2 And Keras Tensorflow 2 is a deep learning framework developed by Google based on Tensorflow 1. This article introduces the common APIs of TF2.0.