JindoFS in block storage mode supports file encryption. The encryption mechanism and usage method are similar to those of encryption zones for Apache HDFS. Keys are managed by Key Management Service (KMS). You can configure encryption policies for a directory that contains sensitive data. This way, you can transparently encrypt the data that you want to write to the directory and transparently decrypt the data that you want to read from the directory. In this process, you do not need to change your code.
Prerequisites
- An E-MapReduce (EMR) cluster is created. For more information, see Create a cluster.
- KMS is activated. For more information, see Activate KMS.
Background information
The following figure shows the architecture of JindoFS in block storage mode with
file encryption supported.
Configure KMS-related parameters for JindoFS
- Go to the SmartData service.
- Log on to the Alibaba Cloud EMR console.
- In the top navigation bar, select the region where your cluster resides. Select the resource group as required. By default, all resources of the account appear.
- Click the Cluster Management tab.
- On the Cluster Management page that appears, find the target cluster and click Details in the Actions column.
- In the left-side navigation pane, click Cluster Service and then SmartData.
- Go to the namespace tab for the SmartData service.
- Click the Configure tab.
- In the Service Configuration section, click the namespace tab.
- Add configuration information.
- Save the configurations.
- In the upper-right corner of the Service Configuration section, click Save.
- In the Confirm Changes dialog box, specify Description and turn on Auto-update Configuration.
- Click OK.
- Restart Namespace Service.
- Choose in the upper-right corner.
- In the Cluster Activities dialog box, specify the related parameters.
- Click OK.
- In the Confirm message, click OK.
Use JindoFS KeyProvider
JindoFS KeyProvider is used to connect JindoFS to KMS. The keys for JindoFS are stored
in KMS. You can use JindoFS KeyProvider to create, query, and rotate keys based on
KMS.
- Create a key: You can run the following command to create a key:
jindo key -create -keyIdName <keyIdName>
Note In this topic, <keyIdName> in commands indicates the name of the key that you created.For example, run the following command to create a key named policy_test:jindo key -create -keyIdName policy_test
After the policy_test key is created, it is displayed on the Secrets page of the KMS console. - Query keys: You can run the following command to query the names of all existing keys
for JindoFS:
jindo key -list
The following information is returned:Listing Keys: policy_test policy_test2
- Rotate a key: You can run the following command to periodically rotate a key based
on its ID. After the key is rotated, the key version changes. If a file is encrypted
after the rotation, the latest key version is used. If a file is decrypted after the
rotation, the key version that corresponds to the file is used.
jindo key -roll -keyIdName <keyIdName>
For example, run the following command to rotate the policy_test key:jindo key -roll -keyIdName policy_test
After the policy_test key is rotated, the version of the key is updated. The original version is in the ACSPrevious state and the new version is in the ACSCurrent state.
Manage encryption policies for JindoFS
You can run the following commands to configure and query encryption policies.
- Configure an encryption policy:
jindo jfs -setCryptoPolicy -keyIdName <keyIdName> <path>
Note<path>
in the preceding example indicates the path of a file on JindoFS, for example, jfs://test/. - Query an encryption policy:
jindo jfs -getCryptoPolicy <path>
Example:
- Query the encryption policy of the jfs://test/ path:
jindo jfs -getCryptoPolicy jfs://test/
{NONE}
is returned. - Configure an encryption policy for the jfs://test/ path:
jindo jfs -setCryptoPolicy -keyIdName policy_test jfs://test/
- Go to the bigboot directory and query the encryption policy of the jfs://test/ path again:
jindo jfs -getCryptoPolicy jfs://test/
The following information is returned:SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apps/ecm/service/b2jindosdk/3.4.0-hadoop3.1/package/b2jindosdk-3.4.0-hadoop3.1/lib/jindo-distcp-3.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apps/ecm/service/hadoop/3.2.1-1.0.1/package/hadoop-3.2.1-1.0.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/03/12 13:52:34 WARN: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/03/12 13:52:35 INFO: Jboot log name is /var/log/bigboot/jboot-20210312-135234-12953.LOG 21/03/12 13:52:35 INFO: Write buffer size 1048576, logic block size 134217728 21/03/12 13:52:35 INFO: cmd=getFileStatus, src=jfs://test/, dst=null, size=0, parameter=null, time-in-ms=7, version=3.4.0 21/03/12 13:52:35 INFO: cmd=getCryptoPolicy, src=jfs://test/, dst=null, size=0, parameter=, time-in-ms=2, version=3.4.0 The crypto policy of path: jfs://test/ is {cipherSuite: AES_CTR_NOPADDING_256, keyIdName: policy_test2, keyIdVersion: null, edek: , iv: } 21/03/12 13:52:35 INFO: Read total statistics: oss read average <none>, cache read average <none>, read oss percent <none>
After you configure the encryption policy, you can read data from or write data to the files under this path.- Copy a local file to HDFS:
hadoop fs -put test.log jfs://test/
- Display the content of a file:
hadoop fs -cat jfs://test/test.log
- Copy a local file to HDFS: