SmartData is a core, proprietary component of E-MapReduce (EMR) that provides unified storage optimization, cache optimization, compute acceleration, and various storage feature extensions, including data access, data governance, and data security.
The following figure shows where SmartData fits in the EMR architecture.
SmartData includes the following components:
JindoFS core subsystem: Provides caching and cache acceleration for various remote storage systems. For more information, see JindoFS overview and usage.
JindoTable core subsystem: Provides table-level and partition-level optimization and administration for tabular data sources, such as Hive data warehouses. For more information, see Use JindoTable.
JindoManager: Provides a management page for JindoFS and JindoTable services, where you can perform tasks such as viewing statistical metrics for files and tables in the cache.
JindoSDK: Provides a unified SDK for open source compute engines in EMR. JindoSDK supports Java, C, C++, and Python, with access methods including Hadoop Compatible File System (HCFS) interfaces, POSIX-based APIs, and Table interfaces.
Toolsets: Includes tools such as the Jindo tool and the Jindo DistCp migration tool.
Various connectors: Includes connectors for Hadoop, Flink, and TensorFlow, with additional support for Kite SDK, Apache Beam, Flume, Sqoop, and Kafka.
SmartData currently supports the following data sources through JindoFS and JindoTable: Alibaba Cloud OSS, Apache Hadoop Distributed File System (HDFS), Hive data warehouses, and Alibaba Cloud MaxCompute.
SmartData is developed and released independently. For version-specific details, see Distributions.
For version-specific SmartData documentation, see the following: