DataWorks datasets let you manage unstructured data such as images and documents. You can create datasets backed by Object Storage (OSS or COS), File Storage NAS (general-purpose or extreme), or DLF Lance tables, and track data versions.
Overview
Datasets let you read and write data stored in OSS and NAS from DataWorks. You can create multiple dataset versions, track changes, and revert to a previous version if needed.
Billing
The DataWorks dataset feature is free. OSS and NAS storage incurs separate storage and network fees (OSS billing, NAS billing).
Create a dataset
-
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. On the page that appears, click Go to Data Map.
-
On the Data Map page, click Data Catalog (
) in the left-side navigation pane. Then, under Catalogs, click DataSet. -
Click your target workspace. On the dataset list page, click Create Dataset and configure the settings described below.
If an administrator has configured custom attributes for the dataset type, you can set values such as business domain and data sensitivity level when creating the dataset. Custom attributes can inherit workspace-level values.
When you create a new version, custom attribute values are inherited from the dataset itself and cannot be modified on the version creation page. To change them, edit the dataset's custom attributes first.
In the Dataset Configuration step of the creation form, after you select a Storage Type, the Import Configuration step displays fields specific to that type. The available storage types are listed below; refer to the matching subsection for the field details.
Storage type |
Typical use case |
Availability |
Object Storage (OSS) |
Mount a folder in an OSS bucket so that data development nodes can read and write unstructured data. |
Available by default. |
Object Storage (COS) |
Mount a directory in Tencent Cloud COS, reusing an already-registered COS data source. |
Available only when the current workspace has at least one COS data source registered. |
File Storage NAS (General-purpose / Extreme) |
Mount a sub-path of an Alibaba Cloud NAS file system. Suitable for scenarios that need shared access or concurrent reads and writes across multiple tasks. |
Available by default. General-purpose and Extreme NAS share the same form; the actual file system selection determines the variant. |
DLF Lance |
Mount a Lance table that resides in a Paimon catalog under a DLF data source. Registration is at table granularity. |
Requires a DLF data source in the current workspace with at least one Paimon catalog. |
Storage type: Object Storage (OSS)
-
Dataset Configuration:
Configuration item
Description
Storage type
Object Storage (OSS).
Content type
Optional. Select the data type. Default: General.
-
Import Configuration:
Configuration item
Description
OSS path
Path of the OSS folder to mount. The bucket and endpoint are parsed from the path automatically; no separate fields are required.
NoteMake sure you have the required OSS Bucket permissions.
Default mount path
Mount path for the OSS folder in DataWorks. Default:
/mnt/data/. You can change the mount path; a custom path must start with/mnt/.
Storage type: Object Storage (COS)
This option appears in the Storage Type dropdown only when the current workspace has at least one COS data source registered. If COS is not listed, register and bind a COS data source to the workspace first.
-
Dataset Configuration:
Configuration item
Description
Storage type
Object Storage (COS).
Content type
Optional. Select the data type. Default: General.
-
Import Configuration:
Configuration item
Description
COS data source
Required. Select a COS data source that is already registered in the current workspace. The dropdown supports fuzzy search by data source name and displays the data source ID for each entry. If no data source is available, use the Create Data Source link in the dropdown to create one.
COS path
Required. Manually enter the COS directory path to mount. Make sure the account has read and write permissions for the path.
NoteCOS datasets do not support a custom default mount path. The Application Configuration step is therefore not displayed in the creation form.
Storage type: File Storage NAS (General-purpose / Extreme)
General-purpose NAS and Extreme NAS share the same form; only the file systems shown in the File system dropdown differ:
When you select File Storage (General-purpose NAS) as the Storage Type, the File system dropdown lists only General-purpose NAS systems and the default File system path is the root directory
/.When you select File Storage (Extreme NAS) as the Storage Type, the File system dropdown lists only Extreme NAS systems and the default File system path is
/share/.
-
Dataset Configuration:
Configuration item
Description
Storage type
Select File Storage (General-purpose NAS) or File Storage (Extreme NAS file systems).
Content type
Optional. Select the data type. Default: General.
-
Import Configuration:
Configuration item
Description
File system
Required. Select a NAS file system in the current region under your Alibaba Cloud account. The dropdown is automatically filtered by the selected Storage Type (General-purpose or Extreme). File systems that use the SMB protocol are not supported.
File system mount target
Required. Configure a mount target for the NAS file system.
ImportantThe mount target VPC must connect to the resource group VPC:
-
Use the same VPC for both the NAS mount target and the resource group.
-
For cross-VPC scenarios, follow the Overview to connect the NAS mount target VPC to the resource group VPC.
File system path
Required. Path of the NAS folder to mount. The default value depends on the Storage Type: General-purpose NAS defaults to
/, Extreme NAS defaults to/share/. The path must exist in the NAS file system; otherwise, the dataset fails when used.Default mount path
Mount path for the NAS folder in DataWorks. Default:
/mnt/data/. You can change the mount path; a custom path must start with/mnt/. -
Storage type: DLF Lance
DLF Lance datasets are registered at table granularity. Before you start, register a DLF data source in the current workspace and make sure the DLF contains at least one Paimon catalog and a Lance table. This storage type does not display the Import format field (import is per table; no directory or file format selection is needed).
-
Dataset Configuration:
Configuration item
Description
Storage type
DLF Lance.
Content type
Optional. Select the data type. Default: General.
-
Import Configuration:
Configuration item
Description
Catalog name
Required. The dropdown lists only Paimon-type catalogs under the DLF data source.
Database name
Required. The dropdown is populated based on the selected catalog and is disabled until a catalog is chosen.
Lance table name
Required. The dropdown lists only Lance-type tables under the selected database.
Default mount path
Mount path for the Lance table in DataWorks. Default:
/mnt/data/. You can change the mount path; a custom path must start with/mnt/.
View dataset details
This section is for the "I want to look up an existing dataset" scenario, describing how to enter the dataset details page and what information is available on the page. To modify a dataset (create a version, delete, and so on), see Manage datasets.
To enter the dataset details page: under , open the dataset list for the target workspace, and click Details in the Actions column of the target dataset.
The details page has two layouts. When you open the page from the Data Catalog panel (the default), the upper-right corner provides Create Version and Delete Dataset buttons. When you open it directly from a global search result or an external deep link (the standalone details route), the page header additionally provides a Manage button that takes you back to the management panel for this dataset in the Data Catalog. The tab content is the same in both layouts.
The details page contains the following tabs:
-
Properties: Displays the dataset's owner, creation time, owning workspace, description, and other core fields. If the administrator has configured custom attributes, the corresponding custom-attribute cards are also displayed (for more information, see Custom attributes; custom attribute values can be inherited from the workspace). This tab also embeds a Dataset Version block that lists all versions of the dataset and provides entries for version switching and jumping to the OSS console to view the original data (see View Dataset Data).
-
View Data: Previews sample data based on the dataset's storage type and content type, without leaving DataWorks.
NoteThis tab is visible only for datasets whose storage type is Object Storage (OSS) and whose content type is supported by the in-platform preview list. For datasets with other storage or content types, the tab is not displayed; use View Dataset Data to jump to the OSS console and inspect the original files.
-
Lineage Information: View the lineage between the dataset and upstream tables or downstream tasks.
-
Usage Notes: Maintain and read the dataset's usage notes; capability parity with the table details Notes tab.
Manage datasets
This section is for the "I want to change an existing dataset" scenario and is organized by user task. All operations are triggered either from the upper-right corner of the dataset details page (see View dataset details) or from the Dataset Version block in the Properties tab.
-
Create Version (update what the dataset points to):
In the upper-right corner of the dataset details page, click Create Version to open the version creation page. In the Version Configuration area, you can re-enter the OSS Path or NAS file system configuration (the fields shown depend on the dataset's storage type) and reset the Default Mount Path. The Dataset Configuration area (name, storage type, content type, custom attributes, and so on) is read-only when you create a new version; to change those values, edit the dataset itself first.
-
View dataset data (jump to an external console):
Available only for datasets whose storage type is Object Storage (OSS). In the Properties tab, locate the Dataset Version block, select the target version, and click the View in OSS button in the operations area. You are redirected to the OSS console and taken directly to the storage path of that version. If the dataset's content type supports in-platform preview, the operations area instead shows a Data View button that opens the Data View tab so you can preview the data without leaving DataWorks.
-
Delete version (revoke a single version):
In the Properties tab, in the upper-right corner of the Dataset Version block, select the target version from the dropdown and click Delete. Only the selected version is deleted; the dataset itself is preserved and its other versions remain accessible.
-
Delete dataset (remove the entire dataset):
In the upper-right corner of the dataset details page, click Delete to delete the dataset and all of its versions.
Deleting a dataset or a version only removes the record from the DataWorks dataset feature; it does not delete the original files in OSS, NAS, or other underlying storage. However, the deletion cannot be undone in DataWorks, so proceed with caution.
Use a dataset
Use datasets in Data Studio with Shell nodes, Python nodes, Notebook development, or in your personal development environment.