All Products
Search

This Product

    Data Lake Formation:Row filters and column masking

    Document Center

    Data Lake Formation:Row filters and column masking

    Last Updated:Sep 22, 2026

    This topic describes row filters and column masking in DLF, including configuration, limitations, and rule-conflict handling.

    Limitations

    Row filter limitations

    • Engine version: Only EMR Serverless Spark and Realtime Compute for Apche Flink are supported:

      • Flink requires vvr-11.6-jdk11-flink-1.20 or later.

      • Serverless Spark requires one of the following versions: esr-5.2.0 (Spark 4.0.1, Scala 2.13) or later, esr-4.8.0 (Spark 3.5.2, Scala 2.12) or later, or esr-3.7.0 (Spark 3.4.4, Scala 2.12) or later

    • Single rule limit: For the same table, a user can configure only one composite expression.

    • Write permission conflict: For the same user and table, you cannot configure a row filter together with write permissions (ALL or UPDATE).

    • Schema change impact: If you rename or delete a column, the original filter expression becomes invalid and you must reconfigure it.

    • Multi-policy merging: When filters based on role and user are configured simultaneously, the system merges them using an AND logical operation.

    Column masking limitations

    • Engine version: You configure column masking rules in the DLF console. When you query data, the rules are pushed down to the compute engine for execution. Serverless Spark and Flink are supported.

      • For Serverless Spark, use esr-5.2.0 (Spark 4.0.1, Scala 2.13), esr-4.8.0 (Spark 3.5.2, Scala 2.12), or esr-3.7.0 (Spark 3.4.4, Scala 2.12), or a later version.

      • For Flink, use vvr-11.7-jdk11-flink-1.20 or later.

    • Write permission conflict: For the same user and table, you cannot grant a column masking rule together with write permissions (ALL or UPDATE).

    • Schema change restriction: You cannot rename, delete, or change the type of a masked column or its nested fields. Revoke the masking rule for the column before making those changes.

    Row filters

    Row filter expressions restrict the data range of query results. They fall into two categories: simple expressions and composite expressions.

    Simple expressions

    A simple expression defines basic comparison logic for a single column.

    Syntax:

    <Column Name> <Comparison Operator> <Value>

    Parameters:

    • Column Name: The column to filter. Supported data types:

      • Boolean: BOOLEAN

      • String: CHAR, VARCHAR

      • Numeric: DECIMAL, TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE

      • Temporal: TIME, TIMESTAMP

    • Comparison Operator: The comparison method. For supported operators, see the comparison table below.

    • Value: A constant for comparison. The type must match the column data type.

    Composite expressions

    Composite expressions combine multiple simple expressions with logical operators for more complex filtering.

    Syntax:

    (<Simple Expression>) <AND/OR> (<Simple Expression>)

    Composite expressions support nesting and can be further combined using the logical operators AND or OR.

    Operator comparison table

    The following table lists the supported operators and their corresponding predicate implementations in Apache Paimon.

    Operator

    Description

    Paimon Predicate

    Example

    AND

    Logical AND

    PredicateBuilder.and

    year > 2010 AND country <> 'US'

    OR

    Logical OR

    PredicateBuilder.or

    (year > 2010) OR (month < 8)

    IS NULL

    Is null

    PredicateBuilder.isNull

    country IS NULL

    IS NOT NULL

    Is not null

    PredicateBuilder.isNotNull

    country IS NOT NULL

    IN

    Set inclusion

    PredicateBuilder.in

    id IN (1, 2, 3)

    NOT IN

    Set exclusion

    PredicateBuilder.notIn

    id NOT IN (1, 2, 3)

    =

    Equals

    PredicateBuilder.equal

    year = 2010

    <>

    Does not equal

    PredicateBuilder.notEqual

    year <> 2010

    <

    Less than

    PredicateBuilder.lessThan

    year < 2010

    <=

    Less than or equal to

    PredicateBuilder.lessOrEqual

    year <= 2010

    >

    Greater than

    PredicateBuilder.greaterThan

    year > 2010

    >=

    Greater than or equal to

    PredicateBuilder.greaterOrEqual

    year >= 2010

    BETWEEN

    Range

    PredicateBuilder.between

    year BETWEEN 2010 AND 2025

    Column masking

    Column masking lets specified users or roles see masked values instead of source data when they query a table. For example, you can return null values for a phone-number column or replace an ID-number column with fixed text.

    Configure column masking

    1. In the navigation pane, choose Catalogs, then select the target catalog, database, and table. Click the Permissions tab and click Grant Permissions.

    2. For Principal, select a RAM user/RAM role or DLF role. Then select the principal in Select DLF User. We recommend using DLF roles to grant the same masking rules to multiple users.

    3. In Table, select Select. You cannot enable column masking rules unless this permission is selected.

    4. Select Enable Column Masking Rules, then choose a masking method for each column to mask.

      Masking method

      Effect

      Supported column types

      Set to null

      Returns null values for the entire column.

      Any type.

      Fixed-text replacement

      Replaces the entire column with a specified fixed string.

      String columns only; the replacement text can contain up to 1,024 bytes.

    5. Click OK to save. You can view or revoke saved rules in the Column Masking Rules section on the Permissions tab.

    Notes

    • When updating a rule, include all columns that must remain masked for that principal and table. Columns omitted from the update lose masking.

    • You can configure rules for the same column for multiple principals. Set to null can coexist with any method. Other masking methods and their settings must be identical.