JindoData is a data lake storage acceleration suite for big data and AI ecosystems. It provides comprehensive access acceleration for data lake storage systems on Alibaba Cloud and other platforms.
Built on a unified architecture and kernel, the JindoData suite includes the JindoFS storage system (formerly JindoFS Block mode), the JindoFSx storage acceleration system (formerly JindoFS Cache mode), the JindoSDK for big data, and fully compatible ecosystem tools such as JindoFuse and JindoDistCp.
Notes
JindoData is applicable to EMR 5.14.0 and earlier versions, and EMR 3.48.0 and earlier versions.
JindoData is not supported in EMR 5.15.0 and later versions or EMR 3.49.0 and later versions. Use JindoCache for caching and DLF-Auth for authentication.
JindoFS storage system
JindoFS is a cloud-native storage system based on Alibaba Cloud OSS. It is binary-compatible with Apache HDFS and provides a similar feature set, which simplifies HDFS usage and migration. JindoFS is a major upgrade to the former JindoFS Block mode.
The Alibaba Cloud OSS-HDFS service (JindoFS service) is the managed version of JindoFS on Alibaba Cloud. Deeply integrated with Alibaba Cloud OSS, it is available out of the box and eliminates the need to deploy and maintain JindoFS on self-managed clusters.
For more information about the OSS-HDFS service, see What is the OSS-HDFS service?.
JindoFSx storage acceleration system
JindoFSx, also known as the JindoData service, is a major upgrade to the former JindoFS Cache mode. This cloud-native acceleration system speeds up access to cloud storage for big data and AI workloads through data caching, metadata caching, and P2P acceleration. JindoFSx manages multiple backend storage systems under a unified namespace while remaining compatible with each system's native access protocols and providing unified permission management. It is natively optimized for Alibaba Cloud OSS and the Alibaba Cloud OSS-HDFS service, and also supports multicloud object storage such as Amazon S3, Apache HDFS, and NAS.
Ecosystem support and tools
-
JindoSDK is supported.
JindoSDK provides a Hadoop SDK and cloud-based HDFS interfaces with built-in optimizations for Alibaba Cloud OSS, delivering significantly better performance than the community Hadoop edition. It also supports the JindoFS storage system and service, the JindoFSx storage acceleration system, and multicloud object storage.
-
The JindoShell CLI is supported.
JindoShell is a command-line interface (CLI) that extends HDFS shell commands with additional features and performance optimizations for data access operations.
-
JindoFuse POSIX support
JindoData provides POSIX support for Alibaba Cloud OSS, the JindoFS storage system and service, and the JindoFSx storage acceleration system.
-
You can use JindoDistCp for data migration.
JindoDistCp migrates data from on-premises HDFS to the cloud and across cloud providers. It supports migration from various storage systems to Alibaba Cloud OSS and the JindoFS service, and its usage is similar to Hadoop DistCp.
-
JindoTable is supported.
JindoTable works with compute engines such as Spark, Hive, and Presto to provide management features for tabular data.
-
Ecosystem plug-ins
Beyond JindoSDK's default Hadoop support, JindoData provides additional plug-ins such as the Flink Connector.