Cloud Parallel File Storage:Dataflow for CPFS for Lingjun

Features

CPFS for Lingjun supports these dataflow features:

• Account-level dataflow

  Supports same-account and cross-account dataflows with OSS buckets.

• Directory-level dataflow

  Map any subdirectory of a CPFS for Lingjun file system to any prefix in an OSS bucket for granular permission control and flexible data transfer.

• Data import and export

  • Create batch or streaming tasks to import and export data between a CPFS for Lingjun file system and OSS. Batch tasks preload an entire dataset before a compute job starts. Streaming tasks handle on-demand writeback and preload of individual files, such as model checkpoints, during training.

    Warning

    CPFS for Lingjun exports file modification timestamps to OSS custom metadata named x-oss-meta-alihbr-sync-mtime. Do not delete or modify this metadata, or the file system timestamps will be incorrect.

  • If a task fails, check the task report to identify the cause.

    Important

    The task report is for reference only. The destination data is the definitive version. Verify data consistency between source and destination.

Dataflow task types

CPFS for Lingjun provides two dataflow task types: batch tasks and streaming tasks.

Batch task

A batch task imports or exports all files in a directory at once, ideal for preloading entire datasets before AI training.

Streaming task

A streaming task transfers individual files on demand, such as model checkpoints during training. Submit a subtask through an API operation to trigger each transfer. The system does not automatically synchronize file changes.

Task descriptions

Four task types are available: import, export, stream import, and stream export.

Limitations

• Dataflow

  • Same-account dataflows require CPFS for Lingjun 2.4.0 or later. Cross-account dataflows require 2.6.0 or later.

  • Maximum 10 dataflows per CPFS for Lingjun file system.

  • Each file system path can link to only one OSS bucket.

  • Dataflows cannot span different regions.

• Restrictions on paths, filenames, and directory names

  • Renaming a non-empty directory in a dataflow-associated path returns a Permission Denied or directory-not-empty error.

  • Use special characters in directory and file names with caution.

    • Supported characters include uppercase and lowercase letters, numbers, exclamation points (!), hyphens (-), underscores (_), periods (.), asterisks (*), and parentheses (()).

    • The following characters are not supported and may cause task failures:

      • A subdirectory or file name cannot be "..".

      • A path cannot contain backslashes (\) or consecutive backslashes (\\).

      • A subdirectory or file name cannot contain forward slashes (/).

  • Maximum path length: 1,023 characters.

• Dataflow task limitations

  • Streaming tasks require CPFS for Lingjun 2.6.0 or later and are API-only.

  • Maximum four concurrent batch tasks per dataflow. Concurrent streaming tasks are unlimited.

  • Import limitations

    • Importing a symbolic link converts it into a regular data file. The symbolic link information is lost.

    • If an OSS bucket has versioning enabled, only the latest version of an object is copied.

    • Filenames or subdirectory names cannot exceed 255 bytes.

  • Export limitations

    • Symbolic links are exported as regular, empty objects. The linked file is not synchronized.

    • Hard links are exported as regular files.

    • Socket, Device, and Pipe files are exported as regular, empty objects.

    • Directory paths cannot exceed 1,023 characters.

Performance metrics

Billing

Dataflow for CPFS for Lingjun is in public preview and free of charge.

Procedure

1. Create a dataflow.

   • Same-account: Create a same-account dataflow.

   • Cross-account: Create a cross-account dataflow.

2. Create a batch task or a streaming task.

   • Streaming: Manage streaming tasks (OpenAPI).

   • Batch: Manage batch tasks.

3. Verify the data.

   After a task completes, verify data accuracy at the destination.

   Warning

   If you delete source data without verifying destination data, you are responsible for any resulting data loss.