The DataWorks Management Console is a visual platform for environment setup and resource management. It provides management capabilities at two levels. At the global level, you can configure tenant-level roles and alert rules. At the workspace level, you can configure dedicated compute resources, data sources, and developers for each project.
Core concepts and system architecture
Understand the core architecture and entity relationships in DataWorks before you begin.
Core concept: Workspace
A workspace is the basic logical unit in DataWorks for project management, access control, and resource isolation. All activities, such as data development, task operations and maintenance, and member management, are performed within a workspace.
Isolation: Workspaces feature strong isolation. Internal nodes, resources, and members are completely independent between different workspaces.
Container Carrier: A container that hosts functional modules, such as data development and data integration, and associates with external entities, such as data sources, compute resources, and resource groups.
Architecture and entity relationship analysis
The following figure shows the dependencies and associations between the core entities in DataWorks:
Workspace: Located at the center of the architecture diagram, the workspace is the fundamental unit for all operations. Operations related to features such as Data Development, Data Integration, and Operation Center are performed within a specific workspace.
Resource group:
NotePlease note that a resource group in DataWorks is a separately purchased package of compute resources used for task execution. This is completely different from the resource group in an Alibaba Cloud management account, which is used for resource isolation and permission segregation.
Definition: An entity that provides compute resources (CPU and memory) for task execution. It exists independently of workspaces.
Relationship: Must be attached. As shown on the right side of the figure, a resource group must be attached to one or more workspaces. After a resource group is attached, tasks in the workspace, such as data integration tasks and scheduled Data Development tasks, can obtain the compute resources required for runtime.
Compute resource:
Definition: An entity that provides the syntax parsing, compilation, and execution environment for the Data Development module, such as MaxCompute, Hologres, and EMR.
Relationship: Development dependency. As indicated by the input arrow pointing to the Data Development module in the figure, you must attach a compute resource to a workspace. Then, developers can write, debug, and commit code, such as SQL and Spark, in the Data Development module.
Data source:
Definition: A set of connection information used to access external data storage, such as MySQL, Oracle, and OSS.
Relationship: Integration dependency. As indicated by the input arrow pointing to the Data Integration module in the figure, a data integration task depends on the data sources preconfigured in the workspace for its source and destination.
Custom image:
Definition: A packaged environment that contains a specific operating system, runtime environment, and third-party libraries.
Relationship: Attach as needed. As shown on the left side of the figure, you can attach a custom image when a personal development environment (Studio) or a specific type of Data Development node, such as a PyODPS node, requires dependencies beyond the standard environment, such as a specific version of a Python library. This is an advanced feature.
Cross-workspace global services:
As shown at the top of the figure, modules such as Data Governance and Data Map operate above the workspace level. They provide tenant-level global data views and management capabilities. Their permission systems are independent of individual workspaces.
Notes
Access control: If you cannot view or access certain features, contact your Alibaba Cloud account owner or an administrator to ensure that you have the required permissions. For more information, see Member Permission Management.
Region isolation: DataWorks workspaces and resources are isolated by region. Before you perform any operation, make sure that you have selected the correct region for your target resources in the top navigation bar of the console.
Getting started
If you are configuring DataWorks for the first time, follow the standard workflow below to ensure that the environment is complete and available.
Ensure that you are using an Alibaba Cloud account or a Resource Access Management (RAM) user that has the AliyunDataWorksFullAccess permission. Otherwise, contact an administrator to grant permissions to a RAM user.
1. Create a workspace
Log on to the DataWorks console. In the top navigation bar, switch to the target region. In the navigation pane on the left, select Workspace and click Create Workspace. Select a workspace mode based on your requirements. Standard mode is recommended because it supports complete isolation of development, testing, and production environments.
For more information, see Create a workspace.
2. Configure and attach a resource group
Create/Purchase a Resource Group: In the navigation pane on the left, go to Resource Group and purchase a resource group that is suitable for your task payload requirements, such as a Serverless Resource Group.
Attach to a workspace: Attach the purchased resource group to your workspace. This step is a prerequisite to ensure that core features, such as data integration, task scheduling, and data services, run properly.
3. Add members to a workspace
Switch to the workspace list, and click Details in the Actions column for the target workspace to open its details page.
Add project team members (RAM users) and grant them appropriate workspace-level roles, such as Workspace Administrator, Developer, and O&M personnel, to enable collaborative development and access control.
For more information, see Add members to a workspace.
4. Attach a compute resource
Go to the Computing Resource list on the workspace details page, and attach an existing compute resource instance, such as a MaxCompute project, to the workspace. This step is a prerequisite for data development.
5. Add a data source
Go to the Data Sources page on the workspace details page, click Add Data Source, configure the connection information for the source or destination database for data synchronization, and then test the connection.
After you complete these five steps, your DataWorks workspace is ready for basic development and operations. You can start data integration and data development.
Management console feature overview
Overview and navigation
The left-side navigation pane of the console is the entry point for all management features. It includes the workspace list, resource group list, image management, and purchased resources and services. When you log on to the DataWorks console, the overview page appears by default. This page provides quick access to the core scenarios of DataWorks, your frequently used workspaces, product updates, and other information.
Global configuration
The following configurations are at the tenant level and take effect for all workspaces in the current region.
In the navigation bar, choose , or click the management button in the Actions column of the workspace list to open the Management Center.
You can view the alert configuration, tenant members and roles, and extensions configured for the tenant.
Functional module
Description and core operations
Alert configuration
View and configure information about DataWorks alert contacts and set alert limits.
• Alert resources: View the usage of alert resources (text messages and phone calls) and set the maximum number of alerts that can be sent per day. After the limit is reached, sending stops.
For more information, see View and set alert contacts.
Tenant members and roles
• Tenant members: Configure members with permissions for global modules, such as Data Security Guard and Data Map.
• Tenant roles: View or customize global roles and their permissions.For more information, see Global module access control.
Extensions
Enable or configure extensions from the Open Platform to validate and control development and O&M flows.
For more information, see Extensions.
Workspace details configuration
The following configurations apply only to the current workspace.
Feature module | Description and core operations |
Workspace configuration | Manage the basic properties and security settings of the workspace, such as whether to allow downloads of For more information, see Configure a workspace. |
Data Source | Centrally manage the source and destination connection information for data synchronization in the current workspace. For more information, see Data Source Management. |
Computing Resource | Attach and manage the data development compute resources that the workspace uses, such as MaxCompute, Hologres, and AnalyticDB. For more information, see Compute Resource Management. |
Cluster management | Register and manage self-built CDH/CDP and EMR clusters. Configure authentication files and account mappings. This is visible only for workspaces that do not use the new Data Studio. |
Workspace Members and Roles | View permissions for preset roles, or create custom roles and configure their feature permissions and role mappings to compute resources. Add or remove members from the workspace and grant them preset or custom workspace-level roles. For more information, see Workspace-level module permission control. |
Workspace Management
In the navigation pane on the left of the console, click Workspace. You can view all workspaces of the current tenant in the specified region, manage the DataWorks workspaces that you have joined, or create new workspaces.
Resource Access Management (RAM) users can view only the workspaces that they have joined. To allow a RAM user to view a workspace, you must add the RAM user as a member to that workspace. For more information, see Add a member to a workspace.
Area | Feature | Description | References |
1 | Select region | Switch the DataWorks region. The list of workspaces changes based on the selected region. | - |
2 | Create workspace | Create a new DataWorks workspace. | |
3 | View basic information about a workspace | View basic information about the workspace, such as its mode and administrator. The information includes:
|
|
4 | Related operations | View key metrics for the workspace, quickly navigate to specific modules, or perform other operations on the workspace.
| |
5 | Upgrade version | Upgrade the DataWorks version that you are currently using.
| For more information about the differences between product editions, see Features of different DataWorks editions. |
Purchase exclusive resource group | Purchase a serverless resource group. After the purchase, click Resource Groups in the navigation pane on the left to view the details of the resource group. |
Resource group list
In the navigation pane on the left, click Resource Group. You can purchase the resources you need, view the details of purchased resources in a specific region, and manage them.
Area | Feature | Description | References |
1 | Create resource group | Creates a new Serverless resource group. | |
2 | View basic information of a resource group | View basic information about a resource group, such as its running status, expiration time, and usage.
| |
3 | |||
4 | Basic operations for a resource group | View detailed information about a resource group and perform modification operations.
Note Modifying a resource group takes some time. Wait for the process to complete. |
to perform operations such as scale-out, scale-in, renewal, unsubscribe, quota management, billing method conversion, and modifying the upper limit for concurrent data scheduling.Large model service
In the navigation pane on the left of the console, click Large Model Management to open the Large model service page. The Large model service offers a one-stop solution for efficient deployment, secure communication, and convenient invocation. You can easily deploy models using DataWorks Serverless resource groups and directly invoke large models from data integration and data development tasks. For more information, see Large Model Service Management.
Image management
In the navigation pane on the left, click Image Management. On this page, you can view the official DataWorks images. If tasks that run on a Serverless resource group require a specific development environment, such as third-party dependencies, you can create a custom image that integrates the required development packages and dependencies. Then, when you run the tasks, you can specify the Serverless resource group as the execution resource and the custom image as the runtime environment. For more information, see Custom Image.
Purchased resources and services
In the navigation pane on the left of the console, click Purchased Resources and Services. On this page, you can view the details of your subscription and pay-as-you-go DataWorks services and resource groups, and their corresponding bills and billing rules. You can also renew, upgrade, downgrade, or unsubscribe from services. For more information, see View bill details and Billing overview.