E-MapReduce:SmartData user guide (EMR 3.22.0 to 3.25.1)

Overview

JindoFS supports the block storage mode and cache mode.

JindoFS adopts a heterogeneous multi-backup mechanism. Storage Service provides the data storage capability. Data is stored in OSS to ensure high reliability. Redundant backups are stored in the local cluster to accelerate read operations. Namespace Service manages metadata of JindoFS. In this case, metadata is queried from Namespace Service instead of OSS, which improves query performance. This query method of JindoFS is similar to that of Hadoop Distributed File System (HDFS).

Prepare the environment

• Create a cluster

  Select EMR 3.22.0 or later and select SmartData in the optional services. For more information, see Create a cluster.

• Configure the cluster

  JindoFS uses OSS as its storage backend. Before you can use JindoFS, you must configure OSS-related parameters. You can do this in two ways. The first method is to modify Bigboot parameters after the cluster is created. This change requires restarting the SmartData service to take effect. The second method is to add custom configurations during cluster creation, which allows the services to start with your custom parameters once the cluster is ready.

  • Initialize parameters after cluster creation

    You can configure all parameters related to JindoFS in Bigboot, as shown in the following figures.

    1. On the Service Configuration page, click the bigboot tab.

       Set the value of jfs.namespaces to test.

    2. Click Custom Configuration. In the Add Configuration Item dialog box, add the following configuration items: Set jfs.namespaces.test.uri to oss://oss-bucket/oss-dir, set jfs.namespaces.test.mode to block, configure the corresponding OSS AccessKey ID and AccessKey Secret, and then click OK.

    Note

    • The listed parameters are required.

    • JindoFS supports multiple namespaces. This topic uses test as an example namespace.

    Parameter	Description	Example
    jfs.namespaces	The namespaces supported by the current JindoFS instance. Use commas to separate multiple namespaces.	test
    jfs.namespaces.test.uri	The URI of the storage backend for the test namespace.	oss://oss-bucket/oss-dir Note You can configure this to a specific directory within an OSS bucket. The namespace then uses this directory as its root directory for read and write operations.
    jfs.namespaces.test.mode	The storage mode for the test namespace.	block Note JindoFS supports block and cache storage modes.
    jfs.namespaces.test.oss.access.key	The AccessKey ID for OSS.	xxxx Note For optimal performance and stability, we recommend using an OSS bucket that is under the same account and in the same region as your EMR cluster. This allows the cluster to have password-free access to OSS, eliminating the need to configure an AccessKey ID and AccessKey Secret.
    jfs.namespaces.test.oss.access.secret	The AccessKey Secret for OSS.

    Save and deploy the JindoFS configuration. Restart all components in SmartData to use JindoFS.

  • Add custom configurations during cluster creation

    You can add custom settings when you create an EMR cluster. For example, to enable password-free access to an OSS bucket in the same region, select Custom Software Configuration and add the following configuration for the test namespace.

    [
    {
    "ServiceName":"BIGBOOT",
    "FileName":"bigboot",
    "ConfigKey":"jfs.namespaces","ConfigValue":"test"
    },{
    "ServiceName":"BIGBOOT",
    "FileName":"bigboot",
    "ConfigKey":"jfs.namespaces.test.uri",
    "ConfigValue":"oss://oss-bucket/oss-dir"
    },{
    "ServiceName":"BIGBOOT",
    "FileName":"bigboot",
    "ConfigKey":"jfs.namespaces.test.mode",
    "ConfigValue":"block"
    }
    ]

Use JindoFS

The use of JindoFS is similar to that of HDFS. JindoFS also provides a prefix. To use JindoFS, you only need to replace the hdfs prefix with the jfs prefix.

JindoFS supports most of the computing components in the EMR cluster, including Hadoop, Hive, Spark, Flink, Presto, and Impala.

Examples:

• Shell commands

  hadoop fs -ls jfs://your-namespace/ 
  hadoop fs -mkdir jfs://your-namespace/test-dir
  hadoop fs -put test.log jfs://your-namespace/test-dir/
  hadoop fs -get jfs://your-namespace/test-dir/test.log ./

• MapReduce job

  hadoop jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar teragen -Dmapred.map.tasks=1000 10737418240 jfs://your-namespace/terasort/input
  hadoop jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar terasort -Dmapred.reduce.tasks=1000 jfs://your-namespace/terasort/input jfs://your-namespace/terasort/output

• Spark SQL

  CREATE EXTERNAL TABLE IF NOT EXISTS src_jfs (key INT, value STRING) location 'jfs://your-namespace/Spark_sql_test/';

Disk space watermark control

JindoFS uses OSS as its backend for massive storage capacity, but local disk space is limited. To manage local disk usage, JindoFS automatically evicts cold data backups based on two watermark parameters: node.data-dirs.watermark.high.ratio and node.data-dirs.watermark.low.ratio. Both parameters are set as a ratio (a decimal from 0 to 1) of total disk capacity. When disk usage reaches the high watermark, JindoFS evicts data until the usage level drops to the low watermark. The value for the low watermark must be lower than the high watermark.

Configure the storage policy

JindoFS provides multiple storage policies to meet different storage needs. The following table lists four available storage policies for a directory.

JindoFS provides a command-line tool Admin to configure the storage policy of a directory. The default storage policy is WARM. New files are stored based on the storage policy configured for the parent directory. Run the following command to configure the storage policy:

jindo dfsadmin -R -setStoragePolicy [path] [policy]

Run the following command to obtain the storage policy configured for a directory:

jindo dfsadmin -getStoragePolicy [path]

Use the Admin tool