Use Key Management Service (KMS) instances together with user-defined functions (UDFs) to encrypt and decrypt field-level data in MaxCompute SQL. This document provides complete Java and Python implementations using envelope encryption.
How it works
MaxCompute UDFs run with high concurrency. Calling KMS directly to encrypt each field would generate too many API calls and degrade performance. Envelope encryption solves this: KMS generates a data key once per UDF initialization, and that key encrypts field values locally.
Encryption flow:
During UDF initialization, the KMS instance client initializes and generates a data key.
During data processing, the data key encrypts each field value (envelope encryption).
Decryption flow:
During UDF initialization, the KMS instance client, a thread pool, and an LRU cache initialize.
During data processing, KMS decrypts the data key ciphertext, and the plaintext data key decrypts the field value.
Encrypted field format:
Encrypted fields are stored as strings with the following layout:
| Segment | Length | Description |
|---|---|---|
| Data key length | 4 bytes (fixed) | Length of the data key ciphertext — used by the decryption UDF to locate the boundary between the data key ciphertext and the field ciphertext |
| Data key ciphertext | Variable | The encrypted data key, decrypted by KMS during decryption |
| Field ciphertext | Variable | The encrypted field value |
Envelope encryption supports only Galois/Counter Mode (GCM). When creating the customer master key (CMK), set Key Type to Symmetric Key.
Choose a UDF
SynchronousKeyEncryptUDF | EncryptUDF | |
|---|---|---|
| Use when | Large datasets, high concurrency | Small tables, low concurrency |
| Data key reuse | Keys are stored in a KMS secret and reused across runs — reduces API call volume and prevents key proliferation | A new data key is generated each time the UDF initializes |
| Performance | Higher — fewer GenerateDataKey calls | Lower — one GenerateDataKey call per UDF initialization |
| Additional setup | Requires a generic secret and a RAM secret | None |
Without SynchronousKeyEncryptUDF, each UDF worker generates its own data key. At high concurrency, this produces a large number of GenerateDataKey API calls and accumulates many distinct data keys in KMS. SynchronousKeyEncryptUDF avoids this by storing the data key as a KMS secret that all workers share.
Prerequisites
Before you begin, ensure that you have:
A KMS instance of the software key management type or hardware key management type. See Purchase and enable a KMS instance
A CMK with Key Type set to Symmetric Key (GCM mode required). See Key types and specifications
An Application Access Point (AAP) created, a client key saved, and the KMS instance CA certificate obtained. See Create an AAP and Obtain the instance CA certificate
Sample code in Java
Step 4: Create an encryption UDF
SynchronousKeyEncryptUDF (recommended for large datasets)
SynchronousKeyEncryptUDF stores data keys as KMS secrets. Multiple UDF workers look up the same key version instead of each generating a new data key. This keeps the number of data keys bounded and reduces GenerateDataKey API calls.
The following diagram shows the data key generation flow:
Before writing the UDF class, create two secrets:
Create a generic secret named `xxxxSynchronousSecret` Create a generic secret in the KMS console or by calling KMS API operations. The secret stores your data key. Set the secret name to
xxxxSynchronousSecretand use a random value as the initial secret value. Specify the secret name via theudf.synchronous.secret.nameparameter in the UDF. See Manage and use generic secrets.Create a RAM secret named `xxxxRamSecret` Create a RAM secret in the KMS console or by calling KMS API operations. The secret stores the AccessKey pair of the KMS account used to call the shared gateway API — the UDF retrieves it for identity authentication at runtime. Specify the secret name via the
udf.ram.secret.nameparameter in the UDF. See Manage and use RAM secrets and Create an AccessKey pair. Secret value format:{ "AccessKeyId": "LTAI****************", "AccessKeySecret": "yourAccessKeySecret" }
Define the SynchronousKeyEncryptUDF.java class:
This example loads credentials from a properties file for demonstration. In production, store sensitive values (client key content, passwords) outside the JAR and control access via KMS instance permissions.
package com.aliyun.kms.sample;
import com.aliyun.kms.kms20160120.TransferClient;
import com.aliyun.kms.kms20160120.model.KmsConfig;
import com.aliyun.kms.kms20160120.model.KmsRuntimeOptions;
import com.aliyun.kms20160120.Client;
import com.aliyun.kms20160120.models.*;
import com.aliyun.odps.udf.ExecutionContext;
import com.aliyun.odps.udf.UDF;
import com.aliyun.odps.udf.UDFException;
import com.aliyun.tea.TeaException;
import com.aliyun.tea.utils.StringUtils;
import com.google.gson.Gson;
import org.apache.commons.codec.binary.Base64;
import javax.crypto.Cipher;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.IOException;
import java.io.InputStream;
import java.io.Serializable;
import java.nio.charset.StandardCharsets;
import java.security.SecureRandom;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ConcurrentHashMap;
public class SynchronousKeyEncryptUDF extends UDF {
private static Gson gson = new Gson();
private static Client instanceClient;
private static com.aliyun.kms.kms20160120.Client shareClient;
private static Map<String, DataKeyObject> dataKeyObjectMap;
private static Base64 base64 = new Base64();
private String runTaskIdentify;
private String keySecretName;
private String keyId;
private Properties properties = new Properties();
@Override
public void setup(ExecutionContext ctx) throws UDFException, IOException {
super.setup(ctx);
try {
InputStream in = ctx.readResourceFileAsStream(UDFConstant.KMS_UDF_CONF_NAME);
properties.load(in);
// Key dimension: use getRunningProject() to scope keys by project,
// or getTableInfo() to scope keys by table.
runTaskIdentify = ctx.getRunningProject();
dataKeyObjectMap = new ConcurrentHashMap<>();
keySecretName = properties.getProperty(UDFConstant.KMS_SYNCHRONOUS_SECRET_NAME_KEY);
keyId = properties.getProperty(UDFConstant.UDF_ENCRYPT_KEY_ID_KEY);
if (StringUtils.isEmpty(keySecretName)) {
throw new UDFException("the secret name is null");
}
buildInstanceClient(ctx);
checkSecretExist();
buildShareClient();
} catch (Exception e) {
throw new UDFException(e);
}
}
public String evaluate(String keyIdentify, String data) {
if (UDFUtils.isNull(data)) {
return data;
}
byte[] dataBytes = data.getBytes(StandardCharsets.UTF_8);
byte[] iv = null;
byte[] cipherTextBytes = null;
try {
String keyVersion = UDFUtils.getSynchronousKeyVersion(runTaskIdentify, keyIdentify);
if (!dataKeyObjectMap.containsKey(keyVersion)) {
dataKeyObjectMap.put(keyVersion, getDataKeyObject(keyVersion));
}
DataKeyObject dataKeyObject = dataKeyObjectMap.get(keyVersion);
iv = new byte[UDFConstant.GCM_IV_LENGTH];
SecureRandom random = new SecureRandom();
random.nextBytes(iv);
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
SecretKeySpec keySpec = new SecretKeySpec(dataKeyObject.plaintextBytes, "AES");
GCMParameterSpec gcmParameterSpec = new GCMParameterSpec(UDFConstant.GCM_TAG_LENGTH * 8, iv);
cipher.init(Cipher.ENCRYPT_MODE, keySpec, gcmParameterSpec);
cipherTextBytes = cipher.doFinal(dataBytes);
// Output format: [4-byte key length][data key ciphertext][base64(iv)][base64(ciphertext)]
return dataKeyObject.encryptedDataKeyPart + base64.encodeAsString(iv) + base64.encodeAsString(cipherTextBytes);
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
private DataKeyObject getDataKeyObject(String keyVersion) throws Exception {
try {
return getDataKeyObjectByGetSecretValue(keyVersion);
} catch (TeaException e) {
if ("Forbidden.ResourceNotFound".equals(e.code)) {
return buildDataKeyObject(keyVersion);
}
}
return null;
}
private DataKeyObject getDataKeyObjectByGetSecretValue(String keyVersion) throws Exception {
String secretData = getSecretValue(keyVersion);
DataKeyPersistentObject dataKeyPersistentObject = gson.fromJson(secretData, DataKeyPersistentObject.class);
byte[] plaintextBytes = base64.decode(dataKeyPersistentObject.plaintext);
return new DataKeyObject(plaintextBytes, dataKeyPersistentObject.encryptedDataKeyPart);
}
private DataKeyObject buildDataKeyObject(String keyVersion) throws Exception {
GenerateDataKeyRequest request = new GenerateDataKeyRequest();
request.setKeyId(keyId);
KmsRuntimeOptions runtimeOptions = new KmsRuntimeOptions();
GenerateDataKeyResponse response = instanceClient.generateDataKeyWithOptions(request, runtimeOptions);
String plaintext = response.getBody().plaintext;
byte[] plaintextBytes = base64.decode(plaintext);
String encryptedDataKeyPart = String.format(UDFConstant.ENCRYPT_DATA_KEY_FORMAT, response.getBody().ciphertextBlob.length()) + response.getBody().ciphertextBlob;
DataKeyObject dataKeyObject = new DataKeyObject(plaintextBytes, encryptedDataKeyPart);
DataKeyPersistentObject dataKeyPersistentObject = new DataKeyPersistentObject(plaintext, encryptedDataKeyPart);
PutSecretValueRequest putSecretValueRequest = new PutSecretValueRequest();
putSecretValueRequest.secretData = dataKeyPersistentObject.toJsonString();
putSecretValueRequest.secretName = keySecretName;
putSecretValueRequest.versionId = keyVersion;
try {
shareClient.putSecretValue(putSecretValueRequest);
} catch (TeaException e) {
if ("Rejected.ResourceExist".equals(e.code)) {
return getDataKeyObjectByGetSecretValue(keyVersion);
}
}
return dataKeyObject;
}
@Override
public void close() throws UDFException, IOException {
super.close();
}
private void buildInstanceClient(ExecutionContext ctx) throws Exception {
try {
KmsConfig config = new KmsConfig();
String clientKeyFileName = properties.getProperty(UDFConstant.KMS_CLIENT_KEY_FILE_KEY);
byte[] clientKeyFileContent = ctx.readResourceFile(clientKeyFileName);
config.setClientKeyContent(new String(clientKeyFileContent, StandardCharsets.UTF_8));
String caFileName = properties.getProperty(UDFConstant.KMS_INSTANCE_CA_FILE_KEY);
byte[] caFileContent = ctx.readResourceFile(caFileName);
config.setCa(new String(caFileContent, StandardCharsets.UTF_8));
config.setPassword(properties.getProperty(UDFConstant.KMS_CLIENT_KEY_PASSWORD_KEY));
config.setEndpoint(properties.getProperty(UDFConstant.KMS_INSTANCE_ENDPOINT_KEY));
instanceClient = new TransferClient(config);
} catch (Exception e) {
throw new UDFException(e);
}
}
private void checkSecretExist() throws Exception {
getSecretValue(null);
}
private String getSecretValue(String versionId) throws Exception {
GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest();
getSecretValueRequest.setSecretName(keySecretName);
if (!StringUtils.isEmpty(versionId)) {
getSecretValueRequest.setVersionId(versionId);
}
KmsRuntimeOptions runtimeOptions = new KmsRuntimeOptions();
GetSecretValueResponse getSecretValueResponse = instanceClient.getSecretValueWithOptions(getSecretValueRequest, runtimeOptions);
return getSecretValueResponse.body.secretData;
}
private void buildShareClient() throws Exception {
String ramSecretName = properties.getProperty(UDFConstant.KMS_RAM_SECRET_NAME_KEY);
GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest();
getSecretValueRequest.setSecretName(ramSecretName);
KmsRuntimeOptions runtimeOptions = new KmsRuntimeOptions();
GetSecretValueResponse getSecretValueResponse = instanceClient.getSecretValueWithOptions(getSecretValueRequest, runtimeOptions);
AccessKeyObject accessKeyObject = gson.fromJson(getSecretValueResponse.body.secretData, AccessKeyObject.class);
com.aliyun.teaopenapi.models.Config shareConfig = new com.aliyun.teaopenapi.models.Config()
.setAccessKeyId(accessKeyObject.AccessKeyId)
.setAccessKeySecret(accessKeyObject.AccessKeySecret)
.setEndpoint(properties.getProperty(UDFConstant.KMS_ENDPOINT_KEY));
shareClient = new com.aliyun.kms.kms20160120.Client(shareConfig);
}
private static class AccessKeyObject implements Serializable {
private String AccessKeyId;
private String AccessKeySecret;
}
private final static class DataKeyObject {
private byte[] plaintextBytes;
private String encryptedDataKeyPart;
public DataKeyObject() {
}
public DataKeyObject(byte[] plaintextBytes, String encryptedDataKeyPart) {
this.plaintextBytes = plaintextBytes;
this.encryptedDataKeyPart = encryptedDataKeyPart;
}
}
private final static class DataKeyPersistentObject implements Serializable {
private String plaintext;
private String encryptedDataKeyPart;
public DataKeyPersistentObject() {
}
public DataKeyPersistentObject(String plaintext, String encryptedDataKeyPart) {
this.plaintext = plaintext;
this.encryptedDataKeyPart = encryptedDataKeyPart;
}
public String toJsonString() {
return gson.toJson(this);
}
}
}Key methods:
| Method | Description |
|---|---|
setup | Reads the configuration file, initializes the KMS instance SDK and Alibaba Cloud SDK, and checks whether the initial secret exists |
buildInstanceClient | Initializes the KMS instance SDK |
buildShareClient | Retrieves the AccessKey pair from the RAM secret and initializes the Alibaba Cloud SDK |
getDataKeyObjectByGetSecretValue | Retrieves a data key stored in the generic secret |
buildDataKeyObject | Generates a new data key and stores it in the secret |
evaluate | Encrypts a field value and returns the encrypted string |
KMS API operations used:
KMS instance API: GenerateDataKey (generate a data key) and GetSecretValue (retrieve a secret value)
KMS API: PutSecretValue (store a secret value)
For full API and SDK documentation, see API references and SDK references.
EncryptUDF (for small tables with low concurrency)
Define the EncryptUDF.java class:
package com.aliyun.kms.sample;
import com.aliyun.kms.kms20160120.TransferClient;
import com.aliyun.kms.kms20160120.model.KmsConfig;
import com.aliyun.kms.kms20160120.model.KmsRuntimeOptions;
import com.aliyun.kms20160120.Client;
import com.aliyun.kms20160120.models.GenerateDataKeyRequest;
import com.aliyun.kms20160120.models.GenerateDataKeyResponse;
import com.aliyun.odps.udf.ExecutionContext;
import com.aliyun.odps.udf.UDF;
import com.aliyun.odps.udf.UDFException;
import org.apache.commons.codec.binary.Base64;
import javax.crypto.Cipher;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.security.SecureRandom;
import java.util.Properties;
public class EncryptUDF extends UDF {
private static Client client;
private static String plaintext;
private static byte[] plaintextBytes;
private static String encryptedDataKeyPart;
private static Base64 base64 = new Base64();
@Override
public void setup(ExecutionContext ctx) throws UDFException, IOException {
super.setup(ctx);
try {
InputStream in = ctx.readResourceFileAsStream(UDFConstant.KMS_UDF_CONF_NAME);
Properties properties = new Properties();
properties.load(in);
KmsConfig config = new KmsConfig();
String clientKeyFileName = properties.getProperty(UDFConstant.KMS_CLIENT_KEY_FILE_KEY);
byte[] clientKeyFileContent = ctx.readResourceFile(clientKeyFileName);
config.setClientKeyContent(new String(clientKeyFileContent, StandardCharsets.UTF_8));
String caFileName = properties.getProperty(UDFConstant.KMS_INSTANCE_CA_FILE_KEY);
byte[] caFileContent = ctx.readResourceFile(caFileName);
config.setCa(new String(caFileContent, StandardCharsets.UTF_8));
config.setPassword(properties.getProperty(UDFConstant.KMS_CLIENT_KEY_PASSWORD_KEY));
config.setEndpoint(properties.getProperty(UDFConstant.KMS_INSTANCE_ENDPOINT_KEY));
String keyId = properties.getProperty(UDFConstant.UDF_ENCRYPT_KEY_ID_KEY);
client = new TransferClient(config);
GenerateDataKeyRequest request = new GenerateDataKeyRequest();
request.setKeyId(keyId);
KmsRuntimeOptions runtimeOptions = new KmsRuntimeOptions();
GenerateDataKeyResponse response = client.generateDataKeyWithOptions(request, runtimeOptions);
plaintext = response.getBody().plaintext;
plaintextBytes = base64.decode(plaintext);
// Output format: [4-byte key length][data key ciphertext]
encryptedDataKeyPart = String.format(UDFConstant.ENCRYPT_DATA_KEY_FORMAT, response.getBody().ciphertextBlob.length()) + response.getBody().ciphertextBlob;
} catch (Exception e) {
throw new UDFException(e);
}
}
public String evaluate(String data) {
if (UDFUtils.isNull(data)) {
return data;
}
byte[] dataBytes = data.getBytes(StandardCharsets.UTF_8);
byte[] iv = null;
byte[] cipherTextBytes = null;
try {
iv = new byte[UDFConstant.GCM_IV_LENGTH];
SecureRandom random = new SecureRandom();
random.nextBytes(iv);
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
SecretKeySpec keySpec = new SecretKeySpec(plaintextBytes, "AES");
GCMParameterSpec gcmParameterSpec = new GCMParameterSpec(UDFConstant.GCM_TAG_LENGTH * 8, iv);
cipher.init(Cipher.ENCRYPT_MODE, keySpec, gcmParameterSpec);
cipherTextBytes = cipher.doFinal(dataBytes);
// Output format: [4-byte key length][data key ciphertext][base64(iv)][base64(ciphertext)]
return encryptedDataKeyPart + base64.encodeAsString(iv) + base64.encodeAsString(cipherTextBytes);
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
@Override
public void close() throws UDFException, IOException {
super.close();
}
}Key methods:
| Method | Description |
|---|---|
setup | Reads the configuration file, initializes the KMS instance SDK, and generates the data key ciphertext |
evaluate | Encrypts a field value and returns the encrypted string |
KMS API operations used:
KMS instance API: GenerateDataKey (generate a data key)
For full API and SDK documentation, see API references and SDK references.