E-MapReduce:Accelerate OSS-HDFS access with JindoCache

Prerequisites

• You have created a cluster and selected the JindoCache service. For more information, see Create a cluster.

• You have enabled OSS-HDFS and granted the required permissions. For more information, see Enable OSS-HDFS.

Procedure

1. Define a caching policy.

   JindoCache uses CacheSets to manage caching policies. You can define one or more CacheSets to apply different policies to different paths.

   1. Log on to the cluster. For more information, see Log on to a cluster.

   2. Create a cacheset.xml file.

      In this example, the cacheset.xml file is in the /path directory.

      <?xml version="1.0" encoding="UTF-8"?>
      <cachesets>
          <cacheset>
              <name>name1</name>
              <path>oss://emr-test/dir1</path>
              <cacheStrategy>DISTRIBUTED</cacheStrategy>
              <metaPolicy>
                  <type>ALWAYS</type>
              </metaPolicy>
              <readPolicy>CACHE_ASIDE</readPolicy>
              <writePolicy>WRITE_AROUND</writePolicy>
          </cacheset>
          <cacheset>
              <name>name2</name>
              <path>oss://emr-test/dir2</path>
              <cacheStrategy>DHT</cacheStrategy>
              <metaPolicy>
                  <type>ONCE</type>
              </metaPolicy>
              <readPolicy>CACHE_ASIDE</readPolicy>
              <writePolicy>WRITE_AROUND</writePolicy>
          </cacheset>
      </cachesets>

      Parameter	Description	Example
      name	The name of the CacheSet. This name must be unique. A duplicate name overwrites the existing CacheSet.	name1
      path	The parent path for the policy. The policy in this CacheSet is applied to all its subpaths.	oss://emr-test/dir1
      cacheStrategy	The caching policy. Valid values are DISTRIBUTED and DHT (Distributed Hash Table). The DHT policy is ideal for accelerating read-only access to small files.	DISTRIBUTED
      metaPolicy	The metadata caching policy. Valid values are ALWAYS and ONCE. ALWAYS: Does not cache metadata. All metadata operations read directly from the remote source. ONCE: Caches metadata. After the initial read from the remote source, subsequent metadata requests are served from the local cache. Note If cacheStrategy is set to DHT, metaPolicy must be set to ONCE.	ALWAYS
      readPolicy	The read policy. Only CACHE_ASIDE is currently supported, which means data is first read from the cache.	CACHE_ASIDE
      writePolicy	The write policy. Supported values include: WRITE_AROUND: Writes data directly to the remote storage, bypassing the cache. CACHE_ONLY: Writes data only to the cache. Note If you use the CACHE_ONLY policy, you must also set metaPolicy to ONCE. WRITE_THROUGH: Writes data to both the cache and the remote storage.	WRITE_AROUND

   3. Run the following command to refresh the CacheSets in the JindoCache system.

      jindocache -refreshCacheSet -path /path/cacheset.xml

      If the command is successful, the output includes the message Successfully refresh cacheset !!!. For more information about JindoCache commands, see JindoCache CLI User Guide.

   4. Use the listCacheSet command to view information about the CacheSets in the system.

      jindocache -listCacheSet

2. Configure the JindoSDK.

   Configure the JindoCache implementation class for OSS-HDFS in Hadoop-Common. In the EMR console, go to the configuration page for the Hadoop-Common service, select the core-site.xml tab, and modify the configuration item. For detailed steps, see Manage configuration items.

   Parameter	Description
   fs.xengine	The value must be jindocache. If you leave this parameter empty, the client does not use the cache and communicates directly with the backend.

   Note

   This is a client-side configuration. You do not need to restart the JindoCache service.

   After you complete these configurations, jobs that access OSS-HDFS can use the caching feature. JindoCache provides transparent caching, so you do not need to modify your jobs. When a job reads data from OSS-HDFS, the data is automatically cached in the JindoCache system. Subsequent requests for the same data are served from the cache, improving read performance.

FAQ

Configure an AccessKey for OSS-HDFS

JindoCache supports credential-free access to OSS-HDFS. However, if you need to access OSS-HDFS across different accounts, you must configure credentials, including an AccessKey ID, AccessKey secret, and an endpoint.

1. Go to the common tab for the JindoCache service.

   1. Log on to the E-MapReduce console.

   2. In the top navigation bar, select a region and a resource group as needed.

   3. On the Clusters page, find the target cluster and click Services in the Actions column.

   4. In the JindoCache service section, click Configure.

   5. Click the common tab.

2. Add and apply the configuration items.

   1. Click Add Configuration Item.

   2. In the Add Configuration Item dialog box, add the following configuration items.

      For detailed steps, see Manage configuration items.

      Note

      Replace YYY with the name of your OSS-HDFS bucket.

      Parameter	Description
      jindocache.oss.bucket.YYY.accessKeyId	The AccessKey ID for OSS-HDFS access.
      jindocache.oss.bucket.YYY.accessKeySecret	The AccessKey secret for OSS-HDFS access.
      jindocache.oss.bucket.YYY.endpoint	The endpoint for OSS-HDFS. Example: cn-hangzhou.oss-dls.aliyuncs.com.
      jindocache.oss.bucket.YYY.data.lake.storage.enable	The value must be true.