PolarDB:Transparent data encryption (TDE)

Glossary

How it works

• Key management module

  • Key structure

    TDE uses a two-layer key structure that consists of a key encryption key (KEK) and a data encryption key. The data encryption key encrypts the database data, and the KEK encrypts the data encryption key.

    • Key encryption key (KEK) and its verification value (HMACK): The system runs the command specified by the polar_cluster_passphrase_command parameter and calculates the SHA-512 hash of the output to obtain 64 bytes of data. The first 32 bytes are the KEK, and the last 32 bytes are the HMACK.

    • Table data encryption key (TDEK) and WAL data encryption key (WDEK): These keys are generated by a secure random number generator and are the actual keys used for data and WAL log encryption. Verification values are generated using the HMAC algorithm. These values are used for KEK verification and are saved on shared storage.

    The KEK and HMACK are obtained from an external source each time. For example, you can retrieve them from KMS. For testing purposes, you can run echo passphrase to obtain them. The ENCMDEK and KEK_HMAC must be saved on shared storage. This ensures that the primary and read-only nodes can read the file and obtain the actual data encryption key at the next startup. The data structure is as follows:

    typedef struct KmgrFileData
    {
        /* version for kmgr file */
        uint32      kmgr_version_no;
    
        /* Are data pages encrypted? Zero if encryption is disabled */
        uint32      data_encryption_cipher;
    
        /*
         * Wrapped Key information for data encryption.
         */
        WrappedEncKeyWithHmac tde_rdek;
        WrappedEncKeyWithHmac tde_wdek;
    
        /* CRC of all above ... MUST BE LAST! */
        pg_crc32c   crc;
    } KmgrFileData;

    This file is generated during database initialization (initdb). This ensures that a standby node can obtain the file using pg_basebackup.

    When a cluster is running, TDE-related control information is stored in the memory of a process. The structure is as follows:

    static keydata_t keyEncKey[TDE_KEK_SIZE];
    static keydata_t relEncKey[TDE_MAX_DEK_SIZE];
    static keydata_t walEncKey[TDE_MAX_DEK_SIZE];
    char *polar_cluster_passphrase_command = NULL;
    extern int data_encryption_cipher;

  • Key encryption

    Keys are generated during database initialization. The process is shown in the following figure:

    1. The system runs the command specified by polar_cluster_passphrase_command to obtain the 32-byte KEK and the 32-byte HMACK.

    2. The system calls the random number generation algorithm in OpenSSL to generate an MDEK.

    3. The system uses the MDEK to call the HKDF algorithm in OpenSSL to generate a TDEK.

    4. The system uses the MDEK to call the HKDF algorithm in OpenSSL to generate a WDEK.

    5. The system uses the KEK to encrypt the MDEK and generate an ENCMDEK.

    6. The system generates a KEK_HMAC from the ENCMDEK and HMACK using the HMAC algorithm. The KEK_HMAC is used as verification information when the key is reverted.

    7. The system writes the ENCMDEK, KEK_HMAC, and other information from the KmgrFileData structure to the global/kmgr file.

  • Key decryption

    When a database crashes or restarts, the data encryption keys must be recovered from the stored ciphertext. The process is as follows:

    1. The system reads the global/kmgr file to obtain the ENCMDEK and KEK_HMAC.

    2. The system runs the command specified by polar_cluster_passphrase_command to obtain the KEK and HMACK.

    3. The system generates a new KEK_HMAC (KEK_HMAC') from the ENCMDEK and HMACK using the HMAC algorithm. It then compares KEK_HMAC' with the stored KEK_HMAC. If they match, the process continues. If they do not match, an error is returned.

    4. The system uses the KEK to decrypt the ENCMDEK and generate an MDEK.

    5. The system uses the MDEK to call the HKDF algorithm in OpenSSL to generate the TDEK. Because the input information is deterministic, the same TDEK is generated.

    6. The system uses the MDEK to call the HKDF algorithm in OpenSSL to generate the WDEK. Because the input information is deterministic, the same WDEK is generated.

  • Key rotation

    Key rotation is the process of re-encrypting the MDEK with a new KEK and generating a new kmgr file. This involves decrypting the MDEK with the old KEK and then re-encrypting it with the new KEK. The process is shown in the following figure:

    1. The system reads the global/kmgr file to obtain the ENCMDEK and KEK_HMAC.

    2. The polar_cluster_passphrase_command returns a 64 byte KEK + HMAC.

    3. The system generates a new KEK_HMAC (KEK_HMAC') from the ENCMDEK and HMACK using the HMAC algorithm. It then compares KEK_HMAC' with the stored KEK_HMAC. If they match, the process continues. If they do not match, an error is returned.

    4. The system uses the KEK to decrypt the ENCMDEK and generate an MDEK.

    5. The system runs the command specified by polar_cluster_passphrase_command again to obtain a new KEK (new_KEK) and a new HMACK (new_HMACK).

    6. The system uses the new_KEK to encrypt the MDEK and generate a new ENCMDEK (new_ENCMDEK).

    7. The system generates a new KEK_HMAC (new_KEK_HMAC) from the new_ENCMDEK and new_HMACK using the HMAC algorithm. The new_KEK_HMAC is used as verification information for future key reversions.

    8. The system writes the new_ENCMDEK, new_KEK_HMAC, and other information from the KmgrFileData structure to the global/kmgr file.

• Encryption module

  All user data is encrypted at the page level using the AES-128 or AES-256 encryption algorithm. By default, AES-256 is used. The (page LSN, page number) pair is used as the initialization vector (IV) for the encryption of each data page. An IV ensures that encrypting the same plaintext multiple times produces different ciphertext.

  The header data structure of each page is as follows:

  typedef struct PageHeaderData
  {
      /* XXX LSN is member of *any* block, not only page-organized ones */
      PageXLogRecPtr pd_lsn;      /* LSN: next byte after last byte of xlog
                                   * record for last change to this page */
      uint16      pd_checksum;    /* checksum */
      uint16      pd_flags;       /* flag bits, see below */
      LocationIndex pd_lower;     /* offset to start of free space */
      LocationIndex pd_upper;     /* offset to end of free space */
      LocationIndex pd_special;   /* offset to start of special space */
      uint16      pd_pagesize_version;
      TransactionId pd_prune_xid; /* oldest prunable XID, or zero if none */
      ItemIdData  pd_linp[FLEXIBLE_ARRAY_MEMBER]; /* line pointer array */
  } PageHeaderData;

  Note

  Where:

  • Encryption is not supported for pd_lsn because decryption requires an initialization vector (IV).

  • The pd_flags field includes the 0x8000 flag to indicate whether the page is encrypted. This flag is not encrypted. This provides backward compatibility for reading plaintext pages and lets you enable TDE for existing clusters.

  • pd_checksum is not encrypted. This allows the page checksum to be verified against the ciphertext.

  • Encrypted files

    Files that contain user data are encrypted. For example, files in the following subdirectories of the data directory are encrypted:

    • base/

    • global/

    • pg_tblspc/

    • pg_replslot/

    • pg_stat/

    • pg_stat_tmp/

  • When to encrypt

    Data is organized into pages and encrypted at the page level. Before a page is written to disk, its checksum is calculated. Even if checksums are disabled, checksum-related functions such as PageSetChecksumCopy or PageSetChecksumInplace are still called. Therefore, the encryption process occurs just before the checksum is calculated to ensure that all user data on the storage medium is encrypted.

• Decryption module

  When a page is read from storage into memory, its checksum is verified. Even if checksums are disabled, the PageIsVerified function is still called. Therefore, the decryption process occurs immediately after the checksum is verified to ensure that the data in memory is in plaintext.