DataWorks:Personal development environment

Features

Data Studio allows you to create account-level personal development environment instances. With network connectivity, you can easily access NAS storage and Git repositories, clone code from Git repositories, and develop and debug Python and Notebook code online. You can also commit your code to a project workspace for scheduling.

A personal development environment instance:

• Supports the SQL, AI Notebook, and Python ecosystems. It allows you to run SQL statements, Python scripts, and shell commands locally, develop tasks for Notebook development, and debug your code.

• Supports installing extensions for developing corresponding tasks, such as DataWorks Agent tasks based on third-party clients.

• Supports Git integration for code management, allowing you to easily clone, push, and manage your code.

• Supports OSS and NAS for easy data management and access.

• Supports DataWorks custom images, allowing you to perform diverse development tasks based on the images that you create.

With these features, Data Studio provides an efficient, flexible, and powerful development environment. This simplifies data processing, analysis, and model training, which improves development efficiency and code quality.

Billing

When you create a personal development environment instance, you must specify a resource group and the number of CUs. You are charged for the CUs consumed by the resource group based on the instance runtime. For more information, see Serverless resource group billing.

Usage notes

• Workspace limits: This feature is available only in a workspace where Use Data Studio (New Version) is enabled.

• Resource group limits: Only serverless resource groups are supported.

• Quantity limits: Each member can create up to 10 personal development environment instances in a workspace.

• Storage limits: Each personal development environment instance provides 30 GiB of free cloud disk (the built-in storage of the instance). The storage is reclaimed 15 days after the instance is stopped. Make sure that you save your personal code files in a timely manner.

• Access control: Data Studio manages permissions for personal development environment instances differently based on identity (tenant administrator, workspace administrator, or instance owner).

  • Tenant administrator or workspace administrator

    • Can view and manage all personal development environment instances in the current workspace .

    • Can only use personal development environment instances owned by themselves.

  • Instance owner

    • Can only view, use, and manage personal development environment instances owned by themselves .

      Management operations include Change Configuration, Stop, Start, Delete, Create Image, Scheduled Shutdown, and View Automatic Shutdown Policy.

• Security controls: Tenant administrators or tenant security administrators can use the personal development environment control feature in Security Center to configure control policies for personal development environments. The following operations can be restricted:

  • File download: Administrators can specify whether to allow file downloads from directories mounted to personal development environment instances. When disabled, the download button is grayed out.

  • Extension installation: Administrators can specify whether to allow extension installation. When disabled, the installation button is grayed out.

  • Terminal usage: Administrators can specify whether to allow terminal usage. When disabled, the terminal button is grayed out.

Create a personal development environment instance

1. Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose Shortcuts > Data Studio in the Actions column.

2. Click Personal development environment at the top, and select an existing personal development environment instance or create an instance as needed.

   Parameter		Description
   Instance Name		The name of the personal development environment instance. You can specify a custom name.
   Instance Owner		By default, the current user who creates the workspace is set as the instance owner .
   Resource Details	Resource Group	Select the serverless resource group to be used by the personal development environment instance.
   	Resource Type	You can configure CPU and GPU resource types.
   	Resource Quota	Select the resource specification for the personal development environment instance. The resource quota of the selected specification must not exceed the Maximum CUs configured for the personal development environment in the resource group. Otherwise, the specification cannot be added. To adjust the quota, contact your administrator to modify the quota in the quota management section on the resource group details page. After you select a specification, the following information is displayed: Specification Name, CPU, Memory, Bandwidth, and Resource Quota (the number of CUs deducted from the resource group for the selected specification). Compared with CPU-type instances, GPU-type instances also display GPU Specification and GPU Memory. You can allocate the Maximum CUs or Minimum CUs available to the current personal development environment instance in the quota management section of the serverless resource group. Important Pay attention to the Resource Quota information here. When a personal development environment instance is running, if the instance uses a pay-as-you-go resource group, compute fees are calculated based on resource quota × instance runtime. If the instance uses a subscription resource group, it consumes the available quota of that resource group.
   Image		Select a system-provided image or a custom image as needed. System built-in images: For more information, see System built-in images. Custom images: In addition to system-provided images, you can select a custom image that you created. Custom images allow you to pre-install specific third-party packages to meet your personalized development requirements. For information about how to create a custom image, see Create a DataWorks image from a personal development environment.
   Storage Settings	System Disk	Each personal development environment instance provides 30 GiB of free cloud disk (the built-in storage of the instance). The storage is reclaimed 15 days after the instance is stopped. Make sure that you save your personal code files in a timely manner. The default path of the built-in storage of a personal development environment instance is /mnt/workspace.
   	Datasets	If you need to read data from or write data to OSS or NAS storage in a personal development environment, you can add datasets to the personal development environment. Datasets: You can select a file storage type (OSS or NAS) dataset created in the current workspace from the drop-down list. Note the following: When you select an NAS-type dataset, if the VPC of the current DataWorks resource group is not connected to the NAS mount target, you must adjust the VPC network to ensure connectivity between the resource group and the NAS mount target. Note The network is connected when the VPC associated with the DataWorks resource group is the same as the VPC associated with the NAS mount target. Mount Path: The default mount path is based on the dataset configuration. You can manually modify it for the current node. Advanced Settings: When reading OSS or NAS datasets for node development, you can configure different advanced settings to adjust the read method and mount protocol configurations for the dataset. Read Only: When read-only mode is enabled, tasks running in the personal development environment can only read data during execution and cannot write data to OSS or NAS. Important Notes on file deletion in the storage of personal development environment instances: Files deleted from the NAS space mounted to a personal development environment instance (by default located at /mnt/data) are not moved to the recycle bin in Data Studio. If the recycle bin feature is enabled in the NAS console, deleted files are moved to the file system recycle bin. Files deleted from the OSS space mounted to a personal development environment instance (by default located at /mnt/data) are not moved to the recycle bin in Data Studio. You can recover data by enabling the versioning feature. Files deleted from the built-in storage of a personal development environment instance (by default located at /mnt/workspace) are not moved to the recycle bin in Data Studio.
   Network Settings	VPC	Access VPC resources (such as RDS): Configure the corresponding Proprietary Network here. Access Internet resources: Leave this field empty. The development environment supports Internet access by default. Access both VPC and Internet resources: After you configure a Proprietary Network here, you must also configure an Internet NAT Gateway for the VPC to access the Internet. Note To access addresses protected by allowlist-based access control, you must also associate a Proprietary Network that has Internet access capability.
   	Security Group	Select an existing security group to associate.
   	VSwitch	Optional. If you do not specify a vSwitch, the system randomly selects one from the selected VPC.
   Access Configuration	Enable SSH	After SSH is enabled, you can use SSH to directly connect to the personal development environment through the selected VPC. If you use a custom image, make sure that sshd is installed in the image. Important To use SSH, you must install SSH Server in the personal development environment. For more information, see Install and configure SSH Server.
   	SSH Public Key	The public key generated locally. For more information, see Generate an SSH key pair.
   	SSH Access Method	Log on from VPC: This access method is supported by default. You can access the personal development environment from other terminals within the VPC, such as ECS instances. Log on from Internet: Select this option to enable Internet access. You must also configure an NAT Gateway and an EIP.
   	VPC Access Domain	A built-in authoritative domain name (Private Zone) is created for you. You can use this domain name within the VPC to access the SSH service or other custom services of the current instance, avoiding the inconvenience caused by changing instance IP addresses. Note that the built-in authoritative domain name incurs fees.
   	NAT Gateway	This parameter is required if you need Internet access. Select an Internet NAT Gateway created for the current VPC.
   	EIP	This parameter is required if you need Internet access. Select an EIP created for the current NAT gateway.
   Advanced Information	Instance RAM Role	An instance RAM role allows you to associate a role with a personal development environment instance. Within the instance, you can access other Alibaba Cloud services by using STS (Security Token Service) temporary credentials, which are periodically refreshed. This ensures the security of your Alibaba Cloud AccessKey and enables fine-grained access control and permission management through RAM. By default, Dataworks Default Role is selected. For more information about roles, see Configure an instance RAM role for a personal development environment.

Manage personal development environment instances

Use datasets in a personal development environment

Personal development environments support datasets, allowing you to read data from or write data to OSS or NAS storage during node execution. For more information, see Use datasets in a personal development environment.

References

• You can create a custom image from a personal development environment instance and then use the custom image during other node development to achieve your business goals. For more information about how to create and use images from a personal development environment instance, see Create and use custom images.

• For detailed instructions on connecting a personal development environment to a Git repository and cloning, pushing, and managing code, see Connect to a Git repository.

FAQ

Q: The instance fails to start with the error cu is not enough as max.

A: The resource group limits the maximum CUs for personal development environments. Go to the resource group details page and adjust the quota in the quota management section.

Q: What is the relationship between a personal development environment and the development and production environments in a standard mode workspace?

A: A personal development environment and the development and production environments in a standard mode workspace belong to different environment systems. A personal development environment is an account-level cloud-based development instance used for coding, debugging, and testing. The development and production environments in a standard mode workspace are used for scheduling and running workflows. After you write and debug your code in a personal development environment, you must submit the code to the project workspace through the deployment process before you can configure schedules in the development environment and deploy the code to the production environment for execution.

Q: How do I deploy a workflow from a personal development environment to the development environment for scheduling?

A: After you complete code development and debugging in the personal development environment, submit the code to the project workspace. Then, go to the data development page in Data Studio, open the corresponding workflow node in the development environment, configure schedule settings (including the schedule and dependencies), and deploy the workflow to the production environment.

Q: The drop-down list at the top of the personal development environment is empty and I cannot switch to the development environment or production environment.

A: The drop-down list at the top of the personal development environment is used to select and manage personal development environment instances, not to switch to the development or production environment of a standard mode workspace. If the drop-down list is empty, this is usually caused by one of the following reasons:

• The current workspace is not associated with a serverless resource group, or CU quota is not allocated for personal development environments in the resource group.

• The current user has not created any personal development environment instances. Click the drop-down list and select to create an instance.

• The workspace does not have Data Studio (new version of data development) enabled. Personal development environments are available only in Data Studio.