E-MapReduce:Accelerate OSS with JindoCache transparent caching

Prerequisites

• You have created an E-MapReduce (EMR) cluster and selected the JindoCache service. For more information, see Create a cluster.

• You have activated Object Storage Service (OSS). For more information, see Activate OSS.

Limitations

Files are stored as objects in OSS.

Procedure

JindoCache uses CacheSets to manage caching policies. You can define different caching policies for different paths, and a cluster can have one or more CacheSets.

1. Select a caching policy.

   1. Log on to the cluster. For more information, see Log on to a cluster.

   2. Create a cacheset.xml file.

      In this example, the cacheset.xml file is in the /path directory.

      <?xml version="1.0" encoding="UTF-8"?>
      <cachesets>
          <cacheset>
              <name>name1</name>
              <path>oss://emr-test/dir1</path>
              <cacheStrategy>DISTRIBUTED</cacheStrategy>
              <metaPolicy>
                  <type>ALWAYS</type>
              </metaPolicy>
              <readPolicy>CACHE_ASIDE</readPolicy>
              <writePolicy>WRITE_AROUND</writePolicy>
          </cacheset>
          <cacheset>
              <name>name2</name>
              <path>oss://emr-test/dir2</path>
              <cacheStrategy>DHT</cacheStrategy>
              <metaPolicy>
                  <type>ONCE</type>
              </metaPolicy>
              <readPolicy>CACHE_ASIDE</readPolicy>
              <writePolicy>WRITE_AROUND</writePolicy>
          </cacheset>
      </cachesets>

      Parameter	Description	Example
      name	The name of the CacheSet. The name must be unique. Specifying a duplicate name overwrites the existing CacheSet.	name1
      path	The parent path to which the caching policy applies. All subpaths inherit the policy of this CacheSet.	oss://emr-test/dir1
      cacheStrategy	The caching strategy. Valid values: DISTRIBUTED and DHT (Distributed Hash Table). The DHT strategy is a read-only acceleration policy suitable for small files. Select a strategy based on your requirements.	DISTRIBUTED
      metaPolicy	The metadata caching policy. Valid values: ALWAYS and ONCE. ALWAYS: Metadata is not cached. All metadata operations directly access the remote storage. ONCE: Metadata is cached. The metadata is read from the remote storage only for the first time. Subsequent reads use the local cache. Select a policy based on your requirements. Note If cacheStrategy is set to DHT, metaPolicy must be set to ONCE.	ALWAYS
      readPolicy	The read policy for files. Only CACHE_ASIDE is supported. This policy prioritizes reading data from the cache.	CACHE_ASIDE
      writePolicy	The write policy for files. The following policies are supported: WRITE_AROUND: Data is written directly to the remote storage. CACHE_ONLY: Data is written to a staging area. Note The CACHE_ONLY policy must be used with the ONCE policy. WRITE_THROUGH: Data is written to the cache.	WRITE_AROUND

   3. Run the following command to refresh the CacheSet configuration in the JindoCache system.

      jindocache -refreshCacheSet -path /path/cacheset.xml

      A successful command returns the message Successfully refresh cacheset !!!. For more information about JindoCache commands, see JindoCache CLI usage instructions.

   4. You can use the listCacheSet command to view the CacheSet information in the system.

      jindocache -listCacheSet

2. Configure JindoSDK.

   Configure the JindoCache OSS implementation class in Hadoop-Common. In the EMR console, go to the configuration page of the Hadoop-Common service, click the core-site.xml tab, and set the following parameter. For more information, see Manage configuration items.

   Parameter	Description
   fs.xengine	The value must be jindocache. If you leave this parameter empty, the client does not use the cache and communicates directly with the backend.

   Note

   This is a client-side configuration. You do not need to restart the JindoCache service.

   After you complete the configuration, jobs that access OSS automatically use caching. Because JindoCache caches transparently, no job modifications are required. When a job reads data from OSS, the data is cached in JindoCache. Subsequent reads of the same data hit the cache, which improves read performance.

FAQ

Configure OSS credentials

JindoCache supports password-free access to OSS. For cross-account access, you must configure credentials, including an AccessKey ID, AccessKey secret, and endpoint.

1. Go to the common tab of the JindoCache service.

   1. Log on to the E-MapReduce console.

   2. In the top navigation bar, select the region and resource group.

   3. On the Clusters page, find the target cluster and click Services in the Actions column.

   4. In the JindoCache service section, click Configure.

   5. Click the common tab.

2. Add and apply the configuration items.

   1. Click Add Configuration Item.

   2. In the Add Configuration Item dialog box, add the following configuration items.

      For more information about how to add and apply configuration items, see Manage configuration items.

      Note

      Replace XXX with the name of the OSS bucket.

      Parameter	Description
      jindocache.oss.bucket.XXX.accessKeyId	The AccessKey ID for the OSS bucket.
      jindocache.oss.bucket.XXX.accessKeySecret	The AccessKey secret for the OSS bucket.
      jindocache.oss.bucket.XXX.endpoint	The endpoint of the OSS bucket.