Container Service for Kubernetes:Fluid overview for data acceleration

Features

Fluid provides features through Dataset and Runtime resources, as shown in the following figure.

• Native support for dataset abstraction: Provides the fundamental support required by data-intensive applications. This enables efficient data access and reduces multi-dimensional management costs.

• Extensible data engine plug-ins: Provides a unified access interface for easy connection to third-party storage. Fluid uses different Runtimes to perform data operations.

• Automated data operations: Provides multiple operation modes that integrate with automated operations and maintenance (O&M) systems.

• Data elasticity and scheduling: Improves data access performance by combining data caching technology with elastic scaling and data affinity scheduling.

• Runtime platform independence: Supports diverse environments, such as native, edge, serverless, and multi-cluster Kubernetes. This makes it suitable for hybrid cloud scenarios.

Important concepts

• Dataset: A set of logically related data that is used by a computing engine. For example, Spark uses datasets in big data scenarios, and TensorFlow uses datasets in AI scenarios. Dataset management involves multiple dimensions, such as security, version management, and data acceleration.

• Runtime: The execution engine that provides capabilities such as dataset security, version management, and data acceleration. It defines a series of lifecycle interfaces. You can implement these interfaces to support dataset management and acceleration.

• AlluxioRuntime: An execution engine from the Alluxio community that supports Dataset data management and caching. It accelerates access to persistent volume claims (PVCs), Ceph, and Cloud Parallel File System (CPFS), and effectively supports hybrid cloud scenarios.

• JuiceFSRuntime: A distributed cache acceleration engine based on JuiceFS. JuiceFSRuntime supports scenario-specific data caching and acceleration. For more information about JuiceFS, see Introduction to JuiceFS. For more information about how to use JuiceFS in Fluid, see Use JuiceFS in Fluid.

• JindoRuntime: A C++-based execution engine from the Alibaba Cloud E-MapReduce (EMR) team's JindoFS that supports Dataset data management and caching. It can accelerate data access to Object Storage Service (OSS), OSS-HDFS, and Hadoop Distributed File System (HDFS).

• ThinRuntime: An extensible implementation for general-purpose storage systems. It lets you connect to various storage systems with minimal code and reuse the core capabilities that Fluid provides for data orchestration, management, and runtime platform access.