E-MapReduce:Introduction to SmartData 3.0.x

JindoFS storage optimizations

• The standalone configuration for the Jindo Namespace service is improved. The service can now update and write metadata to Tablestore asynchronously in standalone mode.
• The option to use Tablestore as the metadata backend for the Jindo Namespace service is removed. The Tablestore-based high availability (HA) solution is no longer supported.
• Support for Archive Storage is added. File data can be stored as the OSS Archive storage class to reduce costs.
• The `Archive`, `Unarchive`, and `Status` commands are added for JindoFS tiered storage. These commands archive data to a specified folder and check the progress and status of archive operations.
• The JindoFS `ls2` command is added to view file information.
• Offline export and analytical queries of fsimage for the JindoFS storage system are now supported.
• Cross-cluster access to the JindoFS storage system is now supported.

For more information, see Tiered storage command usage instructions.

JindoFS cache optimizations

• The organization of cached data on disks is improved. This change removes the dependency on the system disk, makes data disks fully independent, and enhances disk offline operations.
• The caching service is improved to enhance node fault tolerance and node offline operations.
• The selection policy for writing cache blocks to disk is improved. Round Robin is now the default policy.
• The read and write flows are improved and fault tolerance is enhanced.
• The `Cache`, `Uncache`, and `Status` commands are added for JindoFS tiered storage. These commands cache data to a specified folder, support data preloading, and check caching progress and status.
• The way small files use cache space is optimized and more accurate statistics are provided for related metrics.

JindoTable computing optimizations

• The `JindoTable Optimize` command is added. This command optimizes Hive table operations, such as merging small files in a partition.
• The `JindoTable Archive`, `Unarchive`, and `Status` commands are added. These commands archive data to specified tables and partitions and check the progress and status of archive operations.
• Support for the `JindoTable Cache`, `Uncache`, and `Status` commands is added. These commands cache data to specified tables and partitions, support data preloading, and check caching progress and status.
• Exporting MaxCompute tables to the JindoFS caching system is now supported. This provides a preloading mechanism for structured data before machine learning training.

For more information, see JindoTable usage instructions.

JindoFS OSS extensions and support

• Ranger permission integration on the client is now supported. This feature captures various OSS operations and uses JindoFS service records to perform Ranger permission checks.
• Operation audits on the client are now supported. This feature captures various OSS operations and logs them through the JindoFS service for auditing.
• The Hadoop Credentials Provider framework is now supported for specifying the OSS AccessKey configuration in the standard Hadoop way.
• The Flink connector is now supported. The Flink engine can now use OSS for source, sink, and checkpoint storage.
• A lightweight (lite) version of the JindoFS OSS SDK (Hadoop Connector) is provided. This version is designed for non-standard environments, such as a user's Internet Data Center (IDC) cluster.

JindoManager system administration

JindoManager now provides a user interface (UI) for viewing the system status, file statistics on the JindoFS storage system, and cache metrics on the caching system.

JindoTools toolset

The distribution mechanism for the Jindo DistCp tool is improved. Different distribution packages are now provided for use within EMR clusters and in non-EMR cluster environments.

A lightweight (lite) version of Jindo DistCp is now available. It is designed for non-standard environments, such as a user's IDC cluster.