JindoFS in block storage mode supports file encryption. The encryption mechanism and usage method are similar to those of encryption zones for Apache HDFS. Keys are managed by Key Management Service (KMS). You can configure encryption policies for a directory that contains sensitive data. This way, you can transparently encrypt the data that you want to write to the directory and transparently decrypt the data that you want to read from the directory. In this process, you do not need to change your code.

Prerequisites

  • An E-MapReduce (EMR) cluster is created. For more information, see Create a cluster.
  • KMS is activated. For more information, see Activate KMS.

Background information

The following figure shows the architecture of JindoFS in block storage mode with file encryption supported.File encryption supported by JindoFS in block storage mode

Configure KMS-related parameters for JindoFS

  1. Go to the SmartData service.
    1. Log on to the Alibaba Cloud EMR console.
    2. In the top navigation bar, select the region where your cluster resides. Select the resource group as required. By default, all resources of the account appear.
    3. Click the Cluster Management tab.
    4. On the Cluster Management page that appears, find the target cluster and click Details in the Actions column.
    5. In the left-side navigation pane, click Cluster Service and then SmartData.
  2. Go to the namespace tab for the SmartData service.
    1. Click the Configure tab.
    2. In the Service Configuration section, click the namespace tab.
  3. Add configuration information.
    1. On the namespace tab for the SmartData service, click Custom Configuration in the upper-right corner.
    2. In the Add Configuration Item dialog box, add the parameters described in the following table.
      Parameter Description
      crypto.provider.type The type of the provider. Set this parameter to ALIYUN.
      crypto.provider.endpoint The public endpoint of KMS. For more information, see Request syntax.
      crypto.provider.kms.accessKeyId The AccessKey ID used to access KMS.
      crypto.provider.kms.accessKeySecret The AccessKey secret used to access KMS.
  4. Save the configurations.
    1. In the upper-right corner of the Service Configuration section, click Save.
    2. In the Confirm Changes dialog box, specify Description and turn on Auto-update Configuration.
    3. Click OK.
  5. Restart Namespace Service.
    1. Choose Actions > Restart Jindo Namespace Service in the upper-right corner.
    2. In the Cluster Activities dialog box, specify the related parameters.
    3. Click OK.
    4. In the Confirm message, click OK.

Use JindoFS KeyProvider

JindoFS KeyProvider is used to connect JindoFS to KMS. The keys for JindoFS are stored in KMS. You can use JindoFS KeyProvider to create, query, and rotate keys based on KMS.
  • Create a key: You can run the following command to create a key:
    jindo key -create -keyIdName <keyIdName>
    Note In this topic, <keyIdName> in commands indicates the name of the key that you created.
    For example, run the following command to create a key named policy_test:
    jindo key -create -keyIdName policy_test
    After the policy_test key is created, it is displayed on the Secrets page of the KMS console. policy_test
  • Query keys: You can run the following command to query the names of all existing keys for JindoFS:
    jindo key -list
    The following information is returned:
    Listing Keys:
            policy_test
            policy_test2
  • Rotate a key: You can run the following command to periodically rotate a key based on its ID. After the key is rotated, the key version changes. If a file is encrypted after the rotation, the latest key version is used. If a file is decrypted after the rotation, the key version that corresponds to the file is used.
    jindo key -roll -keyIdName <keyIdName>
    For example, run the following command to rotate the policy_test key:
    jindo key -roll -keyIdName policy_test
    After the policy_test key is rotated, the version of the key is updated. The original version is in the ACSPrevious state and the new version is in the ACSCurrent state. policy_test

Manage encryption policies for JindoFS

You can run the following commands to configure and query encryption policies.
  • Configure an encryption policy:
    jindo jfs -setCryptoPolicy -keyIdName <keyIdName> <path>
    Note <path> in the preceding example indicates the path of a file on JindoFS, for example, jfs://test/.
  • Query an encryption policy:
    jindo jfs -getCryptoPolicy <path>
Example:
  1. Query the encryption policy of the jfs://test/ path:
    jindo jfs -getCryptoPolicy jfs://test/

    {NONE} is returned.

  2. Configure an encryption policy for the jfs://test/ path:
    jindo jfs -setCryptoPolicy -keyIdName policy_test jfs://test/
  3. Go to the bigboot directory and query the encryption policy of the jfs://test/ path again:
    jindo jfs -getCryptoPolicy jfs://test/
    The following information is returned:
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/opt/apps/ecm/service/b2jindosdk/3.4.0-hadoop3.1/package/b2jindosdk-3.4.0-hadoop3.1/lib/jindo-distcp-3.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/opt/apps/ecm/service/hadoop/3.2.1-1.0.1/package/hadoop-3.2.1-1.0.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    21/03/12 13:52:34 WARN: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    21/03/12 13:52:35 INFO: Jboot log name is /var/log/bigboot/jboot-20210312-135234-12953.LOG
    21/03/12 13:52:35 INFO: Write buffer size 1048576, logic block size 134217728
    21/03/12 13:52:35 INFO: cmd=getFileStatus, src=jfs://test/, dst=null, size=0, parameter=null, time-in-ms=7, version=3.4.0
    21/03/12 13:52:35 INFO: cmd=getCryptoPolicy, src=jfs://test/, dst=null, size=0, parameter=, time-in-ms=2, version=3.4.0
    The crypto policy of path: jfs://test/ is {cipherSuite: AES_CTR_NOPADDING_256, keyIdName: policy_test2, keyIdVersion: null, edek: , iv: }
    21/03/12 13:52:35 INFO: Read total statistics: oss read average <none>, cache read average <none>, read oss percent <none>
                                    
    After you configure the encryption policy, you can read data from or write data to the files under this path.
    • Copy a local file to HDFS:
      hadoop fs -put test.log jfs://test/
    • Display the content of a file:
      hadoop fs -cat jfs://test/test.log