DataWorks:Use datasets

Prerequisites

Before you begin, make sure that you have:

• An OSS bucket or NAS file system. Create a bucket or create a file system.

• A dataset created in DataWorks. Create a dataset. This example uses an OSS dataset named datasets-oss, created in the China (Shanghai) region, with the bucket path oss://datasets-oss.oss-cn-shanghai.aliyuncs.com/dataset01/v1/ mounted to /mnt/data/dataset01.

Step 1: Configure the dataset for a node

Open a Shell or Python node and add the datasets-oss dataset under Debug Configurations.

Step 2: Write data from the node

After attaching the dataset, interact with OSS data the same way you interact with local files. This example uses the default ossfs 2.0 tool to write file01.txt to the mount path /mnt/data/dataset01.

echo "Hello World" > /mnt/data/dataset01/file01.txt
ls -tl /mnt/data/dataset01

Step 3: Verify data in OSS

After the node runs, the file is automatically written to the OSS path that corresponds to the dataset mount path. In this example, /mnt/data/dataset01 maps to oss://datasets-oss.oss-cn-shanghai.aliyuncs.com/dataset01/v1/. Navigate to that OSS path to confirm the file was written.

Prerequisites

Before you begin, make sure that you have:

• An OSS bucket or NAS file system. Create a bucket or create a file system.

• A dataset created in DataWorks. Create a dataset. This example uses a NAS dataset named datasets-nas, created in China (Shanghai), with the path nas://****.cn-shanghai.nas.aliyuncs.com/mnt/dataset02/v1/ mounted to /mnt/data/dataset02.

Step 1: Configure the dataset for the personal development environment

Create a personal development environment instance and select the datasets-nas NAS dataset.

Step 2: Read and write the dataset in a Notebook

1. At the top of the Data Development page, switch to your personal development environment instance and create a Notebook.

2. Write data to the dataset:

   import os
   
   # Define the destination path and file name.
   file_path = "/mnt/data/dataset02/file02.txt"
   
   # Create the folder if it doesn't exist.
   os.makedirs(os.path.dirname(file_path), exist_ok=True)
   
   # Write the content.
   content = "Hello World!"
   try:
       with open(file_path, "w", encoding="utf-8") as file:
           file.write(content)
       print(f"The file has been successfully written to: {file_path}")
   except Exception as e:
       print(f"Write failed: {str(e)}")

3. Read data from the dataset:

   file_path = "/mnt/data/dataset02/file02.txt"
   with open(file_path, "r") as file:
       content = file.read()
   content

4. Run the two code blocks separately.

   Note

   Confirm the Python kernel when prompted. This example uses Python 3.11.9.

   

Step 3: Configure scheduling

On the right side of the Notebook node, click Scheduling Settings and add the dataset options. The parameters must match the dataset settings you configured when creating the personal development environment instance.

Use ossfs 2.0 (default for OSS)

ossfs 2.0 provides high-performance sequential read and write access to OSS by mounting. It is the default tool for OSS datasets and suits AI training, data processing, and other workloads with sequential I/O patterns.

Set advanced parameters under Dataset > Advanced Configuration. Separate multiple options with a comma. For the full parameter list, see ossfs 2.0 mount options.

Stable data source — If the files being read are not modified during the task, set a long cache time to reduce metadata requests. Typical use case: read a batch of existing files and produce new output files.

{"mountOssType":"ossfs", "attr_timeout": "7200"}

Fast read and write — Use a short metadata cache time to balance cache efficiency with data freshness.

{"mountOssType":"ossfs", "attr_timeout": "3", "negative_timeout":"0"}

Consistent view across distributed nodes — By default, ossfs updates file data based on the metadata cache. Use this configuration to get a synchronized view across multiple nodes.

{"mountOssType":"ossfs", "negative_timeout": "0", "close_to_open":"false"}

High-concurrency tasks with OOM risk — If many files are opened concurrently and cause out-of-memory (OOM) errors, reduce memory pressure with this configuration.

{"mountOssType":"ossfs", "readdirplus": "false", "inode_cache_eviction_threshold":"300000"}

Use ossfs 1.0

ossfs 1.0 mounts an OSS bucket as a local file system on Linux and provides broader file operation support than ossfs 2.0. If you encounter file operation compatibility issues with ossfs 2.0, switch to ossfs 1.0.

For mount parameters, see ossfs 1.0 mount options.

Use JindoFuse

JindoFuse mounts an OSS dataset to a specified path in a container and is suited for:

• Reading OSS data as if it were a local dataset, especially when the dataset is small enough to benefit from JindoFuse's local cache.

• Writing data to OSS.

Set advanced parameters under Dataset > Advanced Configuration. Separate multiple configurations with a comma.

{
  "mountOssType":"jindofuse",
  "fs.oss.download.thread.concurrency": "2 × number of CPU cores",
  "fs.oss.upload.thread.concurrency": "2 × number of CPU cores",
  "attr_timeout": 3,
  "entry_timeout": 0,
  "negative_timeout": 0
}

For parameter descriptions and additional options, see JindoFuse user guide and Using JindoFuse to mount and access data.

Use a NAS dataset

For NAS datasets, specify the NFS mount parameters using the nasOptions parameter.

Default configuration:

{"nasOptions":"vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"}

To customize parameter values, see Manually mount an NFS file system.