DataWorks:Personal development environment

Features

In DataStudio, you can create personal development environment instances at the account level. When connected to the network, you can easily access NAS storage and Git repositories, clone code from Git repositories, and develop and debug Python and Notebook code online. You can also commit the code to a project workspace for scheduling.

A personal development environment instance:

• Supports the SQL, AI Notebook, and Python ecosystems. You can locally execute SQL statements, Python scripts, and Shell commands, and develop Notebook development tasks. It also provides code debugging features.

• Supports the installation of extensions to develop specific tasks, such as DataWorks Agent tasks.

• Integrates with Git for code management. This lets you easily clone, push, and manage code.

• Supports OSS and NAS storage for easy data management and access.

• Supports DataWorks custom images. This lets you perform diverse development tasks based on the images you create.

With these features, DataStudio provides an efficient, flexible, and powerful development environment. This helps you process and analyze data and train models, which improves development efficiency and code quality.

Billing

When you create a personal development environment instance, you must specify a resource group and compute units (CUs). You are charged for the CUs of the resource group based on their runtime. For more information, see Serverless resource group billing.

Scope

• Workspace limits: Personal development environments are supported only in workspaces where the Use Data Studio (New Version) option is enabled.

• Resource group limits: Only Serverless resource groups are supported.

• Quantity limits: Each member can create a maximum of 10 personal development environment instances in a workspace.

• Storage limits: Each personal development environment instance provides 30 GiB of free disk space (instance-specific storage). This storage is reclaimed 15 days after the instance is stopped. Save your personal code files in a timely manner.

• Access control: DataStudio manages permissions for personal development environment instances differently based on roles, such as tenant administrator, workspace administrator, or instance owner.

  • Tenant administrators and workspace administrators

    • Can view and manage all personal development environment instances in the current workspace. .

    • Can only use personal development environment instances that they own.

  • Personal development environment instance owners

    • Can only view, use, and manage personal development environment instances that they own. .

      Management operations includeChange Configuration, Stop, Start, Delete, Create Image, Scheduled Shutdown, View Auto-shutdown Policy, and other operations.

Create a personal development environment instance

1. Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose Shortcuts > Data Studio in the Actions column.

2. In the top navigation bar, click Personal development environment. Select an existing personal development environment instance or create a new one as needed.

   Parameter		Description
   Instance Name		The name of the personal development environment instance. You can customize the name.
   Instance Owner		By default, the current user who creates the workspace is set as the instance owner. .
   Resource Details	Resource Group	Select the Serverless resource group for the personal development environment instance.
   	Resource Type	You can configure CPU and GPU resource types.
   	Resource Quota	Select the resource specifications for the personal development environment instance as needed. After you select the specifications, the following information is displayed: Specification Name, CPU, Memory, Bandwidth, and Resource Quota (the number of CUs deducted from the resource group for the selected specification). For GPU instances, GPU Specification and GPU Memory are also displayed. You can allocate the Maximum CUs or Guaranteed CUs that the current personal development environment instance can use in the Allocate CU Quota for a Task section of the Serverless resource group. Important Note the Resource Quota information. When the personal development environment instance is running, if it uses a pay-as-you-go resource group, you are charged based on the formula: Resource quota × Instance runtime. If the instance uses a subscription resource group, the available quota of that resource group is consumed.
   Select Image		Select a system-provided image or a custom image as needed. System-provided images: For more information, see DataWorks official images. Custom images: In addition to system-provided images, you can select a custom image that you have created. A custom image lets you pre-install specific third-party packages to meet custom development needs. For information about how to create a custom image, see Create a DataWorks image in a personal development environment.
   Storage	System Disks	Each personal development environment instance provides 30 GiB of free disk space (the instance's own storage). This storage is reclaimed 15 days after the instance is stopped. Save your personal code files promptly. The default path for the instance's own storage is /mnt/workspace.
   	Datasets	To read and write data stored on OSS or NAS in the personal development environment, you can add a dataset to the environment. Dataset: You can select a dataset of the file storage type (OSS or NAS) created in the current workspace. Note the following: When you select a dataset of the NAS file storage type, if the network of the current DataWorks resource group is not connected to the NAS mount target, you must adjust the VPC network to ensure connectivity. Note The network is connected if the VPC bound to the DataWorks resource group is the same as the VPC bound to the NAS mount target. Mount Path: The default mount path is based on the dataset configuration. You can manually modify it for the current node. Advanced Configuration: When developing a node to read an OSS or NAS dataset, you can configure different managed datasets to adjust settings such as the read method and mount protocol. Read-only: If you enable this option, tasks running in the personal development environment can only read data. They cannot write data to OSS or NAS. Important Notes on deleting files in the instance storage of a personal development environment: Files deleted from the NAS space mounted by the personal development environment instance (default path: /mnt/data) are not moved to the DataStudio recycle bin. If you enable the recycle bin feature in the File Storage Console, deleted files are moved to the file system's recycle bin. Files deleted from the OSS space mounted by the personal development environment instance (default path: /mnt/data) are not moved to the DataStudio recycle bin. You can enable the scheduled backup feature to recover data. Files deleted from the instance's own storage (default path: /mnt/workspace) are not moved to the DataStudio recycle bin.
   Networking	VPC	Access resources in a VPC (such as RDS): Configure the corresponding VPC here. Access public network resources: Leave this parameter empty. The development environment supports public network access by default. Access both VPC and public network resources: After you configure a VPC, you must also configure a public gateway for the VPC to access the internet. Note To access an address that has whitelist-based access control, you must also bind a VPC that has public network capabilities.
   	Security Group	Select a created security group to attach.
   	vSwitch	Optional. If you leave this empty, the system randomly selects a vSwitch based on the selected VPC.
   Access Configuration	Enable SSH	After you enable this, you can use the native SSH method to directly log on to the personal development environment over the selected VPC. If you have configured a custom image, make sure that sshd is installed in the custom image. Important To use SSH, you must install an SSH Server in the personal development environment. For more information, see Step 4: Install an SSH Server.
   	SSH Public Key	The public key generated locally. For more information, see Step 2: Generate an SSH public key.
   	SSH Access Method	Logon over VPC: This access method is supported by default. You can access the personal development environment from other terminals (such as an ECS instance) within the VPC. Logon over Internet: Select this option to add public network access. You must also configure a NAT Gateway and an EIP.
   	VPC Internal Access Domain	This creates a built-in authoritative domain name (PrivateZone) for you. You can use this domain name within the VPC to access the SSH service or other custom services of the current instance. This avoids the inconvenience of using a changing instance IP address. Note that creating a built-in authoritative domain name incurs fees.
   	NAT Gateway	This parameter is required for public network access. Select the public NAT gateway created for the current VPC.
   	EIP	This parameter is required for public network access. Select the Elastic IP Address created for the current NAT gateway.
   Advanced Information	Instance RAM Role	An instance RAM role lets you associate a role with a personal development environment instance. This allows the instance to access other Alibaba Cloud products based on temporary credentials from Security Token Service (STS). The temporary credentials are periodically updated. This ensures the security of your Alibaba Cloud account AccessKey and enables fine-grained control and permission management through RAM. By default, the DataWorks Default Role is selected. For more information about roles, see Configure an instance RAM role for a personal development environment.

Manage personal development environment instances

Use datasets in a personal development environment

Personal development environments support using datasets to read and write data in OSS or NAS storage during node execution. For more information, see Use datasets in a personal development environment.

More information

• You can create a custom image from a personal development environment instance and then use it with other nodes to meet your business goals. For more information about how to create and use an image from a personal development environment instance, see Creating DataWorks images in personal development environments.

• For more information about how to connect a personal development environment to a Git repository to clone, push, and manage code, see Connect a personal development environment to a Git repository.