The SmartData component is the storage layer of the EMR Jindo engine. It provides unified storage, caching, compute optimization, and feature extensions for various EMR compute engines. The SmartData component consists of JindoFS, JindoTable, and related toolsets. This topic describes the updates in SmartData 3.6.x.
JindoFS
The following table describes the new features of JindoFS in this version.
| Feature | Description |
| JindoFS supports multicloud, the S3 protocol, and cache acceleration | JindoFS supports the S3 protocol. This lets you access Amazon S3 and other S3-compatible systems. It also supports cache acceleration for more efficient access. |
| JindoFS supports cache acceleration for HDFS | JindoFS adds support for cache acceleration when accessing the Hadoop Distributed File System (HDFS). This improves access efficiency. |
| JindoFS supports transactions for MetaSync and Data Cache | JindoFS provides an option for transactional loading for metadata caching. This ensures that the entire preloading task is transactional and prevents incomplete intermediate states from appearing in the metadata view during loading. |
| JindoFS optimizes the cache preloading mechanism |
|
JindoSDK
The following table describes the new features of JindoSDK in this version.
| Feature | Description |
| JindoSDK supports the local cache policy | JindoSDK supports a local cache policy. This allows local data caching even without a deployed SmartData service, which improves the efficiency of accessing OSS data. |
| JindoSDK supports the Object Store API, which is on par with the FileSystem API | In addition to its existing support for FileSystem semantics, JindoSDK now supports Object Store API semantics. The Object Store API more closely corresponds to object storage systems such as OSS, which makes it easier to use. The Object Store API also optimizes operations such as Copy and Rename to improve execution efficiency. |
| JindoSDK supports server-side cache optimization for OSS | JindoSDK supports the OSS accelerator feature. When this feature is enabled, you can customize the access bandwidth for OSS based on the accelerator's capacity. |
| JindoSDK supports second-level domain Endpoints for OSS | JindoSDK introduces a new feature that lets you use a second-level domain or an IP address to access the OSS service in specific environments. Access using second-level domain Endpoints is disabled by default. |
JindoTable
The following table describes the new features of JindoTable in this version.
| Feature | Description |
| JindoTable supports data tiering for HDFS and archiving to OSS | JindoTable adds new commands to move tables or partitions to OSS in batches and synchronize metadata. You can select partitions using filter conditions and specify a storage policy for copying data to OSS. For data that is already in OSS, batch archiving is supported. |
| JindoTable supports restoring and retrieving archived data from OSS | JindoTable provides new commands to restore and retrieve data stored in OSS in batches. |
| JindoTable supports Presto query acceleration for Parquet data | The JindoTable NativeEngine query acceleration engine supports scenarios that use Presto and Parquet. This feature significantly improves the efficiency of querying Parquet data with Presto. |
| JindoTable supports Spark 3 query acceleration for Parquet and ORC data | The JindoTable NativeEngine query acceleration engine now supports the Spark 3 compute engine and data in Parquet and ORC formats. This comprehensively improves the query efficiency of Spark 3. |
| JindoTable supports query acceleration for data stored in HDFS | The JindoTable NativeEngine query acceleration engine now supports HDFS storage. This allows Spark and Presto to efficiently read HDFS files through the NativeEngine. |
| JindoTable supports analyzing OSS access logs | You can analyze OSS access logs using SQL statements. |
JindoFuse
The following table describes the new features of JindoFuse in this version.
| Feature | Description |
| JindoFuse improves support for training and online scenarios | JindoFuse adds support for mounting by specifying a JindoFS namespace. You can also mount by specifying an OSS folder in SDK mode. |