All Products
Search

This Product

    E-MapReduce:Introduction to SmartData 3.1.x

    Document Center

    E-MapReduce:Introduction to SmartData 3.1.x

    Last Updated:Sep 17, 2026

    The SmartData component is the storage layer of the EMR Jindo engine. It provides unified storage, caching, compute optimization, and feature extensions for various EMR compute engines. The SmartData component includes JindoFS, JindoTable, and an associated toolset. This topic describes the updates in SmartData 3.1.x.

    Background information

    The following limitations apply when you use SmartData 3.1.x:
    • JindoFS in cache mode supports metadata caching, which you can enable by modifying the meta-cache switch. This feature is intended only for training scenarios. Do not use it for analysis scenarios, because improper configuration can cause synchronization issues with other write paths.
    • JindoFS namespace names can contain only letters, digits, and hyphens (-).
    • The maximum file size that Jindo DistCp currently supports is 78 GB.
    • Although JindoFS in block mode supports the checksum feature, Jindo DistCp does not currently support it.

    Feature changes

    JindoFS storage optimization

    • Support for the file checksum feature. This feature aligns with the open source Hadoop Distributed File System (HDFS) checksum-related interfaces and supports two algorithms: MD5MD5CRC and COMPOSITE_CRC. For the MD5MD5CRC algorithm, an extended interface is implemented that lets you pass the block size. Corresponding Shell commands are also supported. This provides better support for comparing files with HDFS files.
    • The transparent file compression feature lets you set a compression policy for a directory. New file blocks written to the directory are compressed before they are stored in the OSS backend. For data with a high compression ratio, this can significantly save storage space and reduce the amount of data read and written.
    • Support for flush semantics for file writing. After the flush interface is called, data is persisted to the current position and the file becomes readable.
    • Fixed an issue where the hadoop fs -ls -R command fails to execute for directories with deep hierarchies and many subdirectories because threads enter a waiting state.
    • Enhanced the hadoop fs -stat command to display information such as atime and privilege.
    • Added a path rewrite feature for the Jindo HDFS client to reduce the workload of modifying paths during cluster migration.

      For more information, see Rewrite Jindo HDFS client paths.

    JindoFS cache optimization

    • Small file cache optimization is provided for machine learning training scenarios. This significantly improves cache efficiency and read performance for many small files.
    • The cache command is provided to preload small file directories, which significantly improves preloading efficiency.
    • Support for automatically triggering data caching. You can set a target directory to track and a time interval. At each interval, the system automatically discovers new files in the directory and triggers a cache operation.

    JindoTable compute optimization

    • JindoTable Dump in TF format now supports two-dimensional arrays.
    • Jindo mc dump now supports Gzip compression using the -c parameter.

    JindoManager system administration

    The JindoManager service is a new addition that centralizes operations management, status monitoring, and other features for the Jindo system. It provides a web UI to view the status of various Jindo system components.

    JindoTools toolset

    • Optimized the Job Committer logic for small files in the Jindo DistCp tool. This significantly reduces the number of requests to OSS and improves the performance of DistCp when handling many small files.
    • Optimized file batching in the Jindo DistCp tool by implementing a more effective batching policy, which improves overall performance.

    JindoFS ecosystem support

    • Flink streaming jobs can now be written to JindoFS with recoverability. Both block and cache modes are supported. When combined with a replayable data source, such as Kafka, you can achieve Exactly-Once semantics.
    • Flink now implements the entropy injection feature. When streaming jobs write data to OSS or JindoFS in either block or cache mode, you can use entropy injection for write paths. This feature lets you replace a specific part of a path with a random string to improve write efficiency.
    • Support for the JindoFS Tensorflow Connector. By implementing the Tensorflow Filesystem, the connector supports native Tensorflow I/O interfaces. It supports Tensorflow 1.15 and later, and Tensorflow 2.3 and later.