DataWorks Data Studio is undergoing a comprehensive architectural upgrade to keep pace with the evolution of cloud-native technologies, meet the demands of increasingly complex data processing scenarios, and address the current architecture's challenges in scalability, maintainability, and user experience. This upgrade aims to build a future-ready, high-performance, and highly available data intelligence development platform that provides users with a more efficient, intelligent, and unified experience for data development and governance.
If you encounter any issues during the upgrade, you can get technical support from the DataWorks Data Studio Upgrade Support Group.
1. Background
The original technical architecture of DataWorks was built sixteen years ago. While it met the core needs of the batch processing era, its limitations have become more apparent as technology stacks and business scenarios have evolved. This upgrade is driven by three main factors:
Technical requirements for architectural evolution
Lakehouse architecture support: The existing architecture requires an upgrade to natively support a smooth evolution from traditional data warehouses to a lakehouse architecture, unifying metadata management and data processing.
Unified offline and real-time processing: To meet real-time business demands, the new architecture must provide a unified development paradigm and metadata view, seamlessly integrating offline batch processing and real-time stream computing.
Native integration of AI and large language models (LLMs): The new architecture must natively integrate AI capabilities, such as machine learning, deep learning, and natural language processing, into the entire data development lifecycle.
Existing architecture challenges
Limitations of a monolithic, batch-centric design: The legacy architecture is based on an early monolithic design focused on batch processing. It has inherent limitations in supporting cloud-native technologies and stream computing, which restricts the platform's elasticity and performance.
Scalability and maintainability bottlenecks: Tightly coupled modules result in long iteration cycles for new features and high maintenance costs, making it difficult to meet the demands of large-scale concurrent access and custom business extensions.
Evolving user demands
Support for large-scale and complex workflows: To handle the processing needs of tens of thousands of active users and exabyte-scale data, the new architecture must deliver higher stability, throughput, and resource isolation.
Enhanced development efficiency and intelligence: Users expect a development experience as smooth as a local IDE. The new architecture introduces intelligent code assistance, performance diagnostics, and one-click deployment to significantly lower the technical barrier for data development.
2. Scope of upgrade
Module | Changes |
Data Studio | An upgrade to the user interface (UI) and user experience (UX) only. |
Other product modules, such as Operation Center and Deployment Center, are not related to this upgrade and are not included in the upgrade scope.
3. Impact of the upgrade
Impact on existing business
Item | Impact description |
Deployed production tasks | Zero impact. All online task scheduling and execution will continue to run stably without any interruption. |
Core data development features | Fully retained and enhanced. Node types, function support, core editing and execution capabilities are all fully retained and have been optimized for performance and experience in the new version. |
User asset migration
Asset type | Migration plan |
Existing nodes (such as MaxCompute SQL and Shell) | One-click migration to the new UI is supported. |
User-defined functions (UDFs) | |
Resource files (such as | |
Components | |
Ad hoc queries |
4. Feature migration details
The new Data Studio reorganizes and optimizes the features of the legacy Data Studio. The following sections describe, one by one, the new location and feature changes for each feature module of the legacy Data Studio in the new IDE:
4.1 Data development module
4.1.1. Solutions
Legacy: A standalone "Solutions" menu for organizing and managing related workflows.
New: Upgraded to "Focus Mode". You can enter Focus Mode from any file directory, providing a more immersive and flexible development experience.
4.1.2. Workflows
Legacy: A fixed directory hierarchy with workflow dashboard support.
New: You can create directories as needed and flexibly tag them as content from the legacy workflow directories.
New: The workflow dashboard has been upgraded to a directory dashboard. Any directory now supports the "View" capability.
New: If you need workflow orchestration capabilities, use the "scheduled workflow" feature in the new Data Studio.
4.1.3. Node development
Legacy: Only available node types were displayed. You could not see other node types supported by DataWorks or write development code for other node types before associating a compute engine.
New: Integrated into the project directory in Data Studio. You can see all node types and write code before associating a compute engine. The node creation interaction is being continuously optimized for a better node creation and coding experience.
4.1.3.1 Run / Run with parameters
Legacy: "Run" and "Run with Parameters" were two separate buttons. When you clicked Run with Parameters, you selected the resource group and custom parameters.
New:
The legacy "Run" and "Run with Parameters" buttons are consolidated into a single "Run" button in the new version.
The legacy "Engine Instance" is split into data source and compute resource depending on the node type:
Data source: Used for metadata suggestions in the intelligent code editor.
Compute resource: Used to determine the compute resource for code debugging.
The legacy "Resource Group" in the dialog is moved to "Resource Group" in the run configuration.
The legacy "Custom Parameters" in the "Run with Parameters" dialog is moved to "Parameters" in the run configuration.
The "Debug Configuration" panel is always visible on the right side of the node. The compute resource, resource group, and parameters required for each code run are retrieved from the run configuration.
4.1.3.2 Smoke testing / View smoke testing records
Legacy: Smoke testing could only be initiated after a successful submission.
New:
Standard mode: Smoke testing can be initiated only after a successful deployment to the development environment (equivalent to a successful submission in the legacy version).
Basic mode: Smoke testing can be initiated only after a successful deployment to the production environment (equivalent to a successful submission in the legacy version).
4.1.3.3 Code review / View code review records
Legacy: The relevant buttons were directly visible on the node toolbar.
New: During the deployment process, you can initiate a code review in the production checker step. In the left-side directory tree, click to view the code review list.
4.1.3.4 Submit / Submit and allow others to edit
Legacy:
New: The legacy "Submit" action is changed to "Deploy" in the new version. The new version allows authorized users to deploy directly. In standard mode, tasks can also be packaged and deployed through Deployment Center.
4.1.4. Table development
Legacy: Tables were created in Data Studio.
New: Upgraded to a full-featured "Data Catalog" management experience. You can create tables in Data Catalog with form-based visual creation, code-based creation, or AI-assisted creation.
4.1.5. Resource development
Legacy: Resources were created in Data Studio.
New: Consolidated into the "Resource Management" module.
4.1.6. Function development
Legacy: Functions were created in Data Studio.
New: Consolidated into the "Resource Management" module.
4.2 Component management module
Legacy:
New:
4.3 Manual task module
Legacy:
New:
4.4 Manual workflow module
Legacy:
New:
For more complete manual workflow capabilities, use: Project Directory > Workflow > Trigger-based Workflow.
4.5 Ad hoc query module
Legacy: Ad hoc query files of all users in the current workspace.
New:
Ad hoc query files of the current user across all workspaces in the current region.
Used only for code debugging. When you confirm that a query needs to be deployed as a production task, you can submit it to the project directory, configure the schedule settings, and then deploy it.
4.6 Table management module
Legacy:
New: Upgraded to a full-featured "Data Catalog" management experience. You can create tables in Data Catalog with form-based visual creation, code-based creation, or AI-assisted creation.
4.7 Public table module
Legacy:
New: Upgraded to a full-featured "Data Catalog" management experience.
4.8 Function list module
Legacy:
New: You can now ask questions about function usage directly in Copilot.
In the future product roadmap, a Function Management module will be added to manage and create engine-native functions and user-defined functions.
4.9 Pre-check module
Legacy:
New:
4.10 Run history module
Legacy:
New:
4.11 Smoke testing records module
Legacy:
New:
4.12 Compute resource module
Legacy:
New:
4.13 Settings module
Legacy:
New: Theme switching.
New: Other settings.
New: More settings to explore.
4.14 Recycle bin module
Legacy:
New:
5. Upgrade guide
5.1 Upgrade permissions
Only users with permissions equivalent to Workspace Administrator can view and perform the upgrade operation.
5.2 Upgrade procedure
5.2.1 Access the upgrade entry
Located in the top navigation bar of the DataStudio (data studio) main interface.
A blue "Upgrade to New Version" button next to the workspace selector.
Only users with the required permissions can see this button.
5.2.2 Confirm upgrade notes
After you click the "Upgrade to New Version" button, the system displays the upgrade introduction page.
Downgrade is not supported after the upgrade. Confirm the following before you proceed. If you need assistance during the upgrade, join the DataWorks Data Studio Upgrade Support Group.
If you have used the legacy DataStudio OpenAPI or need to use Migration Assistant, contact the on-duty engineer in the DataWorks Data Studio Upgrade Support Group before upgrading.
Cross-workspace task deployment between the new and legacy Data Studio is not supported between workspaces on the current version and workspaces that have been upgraded to the new version.
During the upgrade, neither the new Data Studio nor the legacy DataStudio in the current workspace can create new content or modify existing content (including UI operations and OpenAPI operations).
The upgrade takes some time. Perform the upgrade during an off-peak period for task development.
After the upgrade, the workspace name and ID remain unchanged.
After the upgrade, the legacy DataStudio becomes read-only. Code changes made in the new Data Studio are not synced back to the legacy DataStudio.
After the upgrade, in the personal directory of the new Data Studio, only the ad hoc query files migrated from the legacy DataStudio for which you are the owner are visible.
5.2.3 Perform the upgrade
When the current upgrade status is "Upgrade not started for the current workspace", click the "Start Upgrade" button to start the upgrade process.
After you click the "Start Upgrade" button, the system displays the estimated upgrade time and the number of objects to be migrated for the current workspace.
Click the "Confirm" button to officially start the upgrade.
You can view the progress and status in real time during the upgrade.
During the upgrade, click "Upgrade Details" to view the upgrade details in real time.
During the upgrade, click "Refresh" to view the latest upgrade status.
After the upgrade is complete, click "Go to Data Studio" to start using the new Data Studio.
6. Upgrade assurance
6.1 Dedicated upgrade services
Online help: Detailed operational guidance is available during the upgrade process.
Technical consultation: If you encounter any issues, contact the on-duty engineer in the DataWorks Data Studio Upgrade Support Group. On-site support can be provided if needed.
Emergency response: A rapid response mechanism is available if any issues occur during the upgrade.
Product training: Post-upgrade feature training can be provided if needed.
6.2 Rollback mechanism
If you encounter any issues during the upgrade, contact the on-duty engineer in the DataWorks Data Studio Upgrade Support Group to request a rollback to the pre-upgrade state.