DataWorks:Use datasets

Preparations: Create a dataset

1. Create a bucket or create a file system.

   This example uses an OSS dataset. A bucket named datasets-oss is created in the China (Shanghai) region, and the /dataset01/v1 folder is created.

2. Create a dataset.

   In this example, an OSS dataset named datasets-oss is created. The OSS path oss://datasets-oss.oss-cn-shanghai.aliyuncs.com/dataset01/v1/ is mounted to /mnt/data/dataset01.

1. Configure a dataset for a node

Configure the datasets-oss dataset in the Debug Configurations of a Shell or Python node.

2. Use the dataset in the node

This section uses a Shell node as an example. After you attach an OSS dataset to a Shell node, you can manage the OSS data in the Shell node code in the same way that you manage local files. The following example shows how to use the default ossfs 2.0 tool to upload the file01.txt file to the mount path /mnt/data/dataset01 of the datasets-oss dataset and write data to the file.

Sample code:

echo "Hello World" > /mnt/data/dataset01/file01.txt
ls -tl /mnt/data/dataset01

3. Verify that data is written in OSS

After the code in 2. Use the dataset in the node is executed, the file is automatically written to the OSS storage path that corresponds to the dataset mount path. You can navigate to the OSS storage path to view the file written from the DataWorks node. In this example, the storage path that corresponds to the mount path /mnt/data/dataset01 of the OSS dataset datasets-oss is oss://datasets-oss.oss-cn-shanghai.aliyuncs.com/dataset01/v1/. The following figure shows an example of the data written to the OSS path.

Preparations: Create a dataset

1. Create a bucket or create a file system.

2. Create a dataset.

   This example uses a NAS dataset. A NAS dataset named datasets-nas is created in the China (Shanghai) region. The NAS path nas://****.cn-shanghai.nas.aliyuncs.com/mnt/dataset02/v1/ is mounted to /mnt/data/dataset02.

1. Configure a dataset for the personal development environment

Create a personal development environment instance and select the datasets-nas NAS dataset that you created.

{"nasOptions":"vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"}

2. Read the dataset in a Notebook

1. At the top of the Data Development page, switch to the personal development environment instance and create a Notebook.

2. Configure the following content in the Notebook.

   1. Write data to the specified path in the dataset.

      import os
      
      # Define the destination path and file name.
      file_path = "/mnt/data/dataset02/file02.txt"
      
      # Make sure that the folder exists. If the folder does not exist, create it.
      os.makedirs(os.path.dirname(file_path),exist_ok=True)
      
      # Write the content.
      content = "Hello World!"
      try:
          with open(file_path, "w", encoding="utf-8") as file:
              file.write(content)
          print(f"The file has been successfully written to: {file_path}")
      except Exception as e:
          print(f"Write failed: {str(e)}")

   2. Read data from the specified path in the dataset.

      file_path = "/mnt/data/dataset02/file02.txt"
      with open(file_path, "r") as file:
          content = file.read()
      content

3. Run the two Python code blocks separately.

   Note

   During runtime, you must confirm the Python kernel. This example uses Python 3.11.9.

   

3. Configure scheduling

On the right side of the Notebook node, click Scheduling Settings and add the dataset-related options. The parameter configuration must be the same as the dataset parameters that you set when you created the personal developer environment instance.

Use ossfs 2.0 to mount and access OSS

ossfs 2.0 is a client designed for high-performance access to OSS by mounting. It provides excellent sequential read and write capabilities and can fully leverage the high bandwidth of OSS. It is suitable for scenarios that require high sequential read and write performance, such as AI training and compute-intensive applications such as data processing. These workload scenarios mainly involve sequential and random reads, sequential writes (append-only), and do not require full POSIX semantics.

In Dataset > Advanced Configuration, you can set advanced parameters. Separate multiple options with a comma (,). For more information about advanced parameters and other options, see ossfs 2.0 mount options. The following are examples of common scenarios:

• The data source remains unchanged during the task: If none of the files to be read are modified during the read process, you can configure a long cache time to reduce the number of metadata requests. A typical scenario is reading a batch of existing files and generating a batch of new files after processing.

  {"mountOssType":"ossfs", "attr_timeout": "7200"}

• Fast read and write: Use a short metadata cache time to balance cache efficiency and file data timeliness.

  {"mountOssType":"ossfs", "attr_timeout": "3", "negative_timeout":"0"}

• Read and write consistency for distributed tasks: By default, ossfs updates file data based on metadata cache. Use the following configuration to achieve a synchronized view across multiple nodes.

  { "mountOssType":"ossfs","negative_timeout": "0", "close_to_open":"false"}

• Too many files are opened at the same time, causing an OOM error: If the task concurrency is high and many files are opened at the same time, an out-of-memory (OOM) issue may occur. You can use the following configuration to relieve memory pressure.

  {"mountOssType":"ossfs","readdirplus": "false", "inode_cache_eviction_threshold":"300000"}

Use ossfs 1.0 to mount and access OSS

ossfs 1.0 lets you mount an OSS bucket as a local file system on a Linux system. Compared with ossfs 2.0, ossfs 1.0 provides more comprehensive file operation support. If you encounter file operation incompatibility issues after you mount a directory with ossfs 2.0, you can try using ossfs 1.0.

For more information about the parameters required for mounting with ossfs 1.0, see ossfs 1.0 mount options.

Use JindoFuse to mount and access OSS

You can use the JindoFuse component to mount an OSS dataset to a specified path in a container. This tool is suitable for the following scenarios:

• You want to read OSS data as if it were a local dataset, or the dataset is small and can effectively use the local cache of JindoFuse for acceleration.

• You need to write data to OSS.

In Dataset > Advanced Configuration, you can set advanced parameters. Use a comma (,) to separate multiple configurations. The following is an example configuration. For parameter descriptions and more parameter configurations, see JindoFuse User Guide and Using JindoFuse to mount and access data.

{ 
  "mountOssType":"jindofuse",
  "fs.oss.download.thread.concurrency": "2 × number of CPU cores",
  "fs.oss.upload.thread.concurrency": "2 × number of CPU cores",
  "attr_timeout": 3,
  "entry_timeout": 0,
  "negative_timeout": 0
}

Use a NAS dataset

If the dataset you added is a NAS dataset, you can specify the configurations for accessing the NAS file system that uses the NFS protocol (the nasOptions parameter). The following code shows the default configurations. To customize parameter values, see Manually mount an NFS file system.

{"nasOptions":"vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"}