All Products
Search

This Product

    E-MapReduce:Delta Lake parameters

    Document Center

    E-MapReduce:Delta Lake parameters

    Last Updated:Mar 26, 2026

    Delta Lake supports three categories of parameters, each with a different configuration scope and prefix.

    CategoryHow to setPrefix
    Spark SQL parametersUsed to execute SQL statementsspark.sql.
    Runtime parametersSet dynamically per sessionspark.databricks.delta.
    Non-runtime parametersSet as global defaults in the Spark configuration file, or per-table in TBLPROPERTIESGlobal: spark.databricks.delta.properties.defaults. Table: delta.

    For non-runtime parameters, table properties take precedence over global parameters. To set a table property, use the TBLPROPERTIES clause in a CREATE TABLE statement.

    -- Set a global default
SET spark.databricks.delta.properties.defaults.deletedFileRetentionDuration = interval 2 weeks;

-- Set a table property (applies to this table only)
CREATE TABLE my_table (id INT, data STRING)
TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 2 weeks');

    Parameter reference

    The following tables describe the available parameters grouped by function.

    Data retention

    ParameterData typeDefaultDescription
    spark.databricks.delta.properties.defaults.deletedFileRetentionDuration or delta.deletedFileRetentionDurationCalendarIntervalinterval 1 weekHow long Delta Lake retains tombstones (markers for deleted files) before physically removing them. If retentionDurationCheck.enabled is true, an exception is thrown when you run VACUUM against tombstones still within this period. The retention period must be at least 1 hour.
    spark.databricks.delta.properties.defaults.logRetentionDuration or delta.logRetentionDurationCalendarIntervalinterval 30 daysHow long Delta log files remain valid. A log file expires when its corresponding data file is compacted, or when this duration elapses. Delta checks for and deletes expired log files each time it generates a checkpoint, preventing unbounded log growth.
    spark.databricks.delta.retentionDurationCheck.enabledBooleantrueWhether Delta enforces the retention period check when deleting tombstones. Set to false only to remove recently merged small files, but we recommend that you use the default value. Otherwise, data generated recently may be deleted, which causes data read/write failures.

    Schema evolution

    ParameterData typeDefaultDescription
    spark.databricks.delta.schema.autoMerge.enabledBooleanfalseWhether Delta automatically merges the schema of incoming data with the destination table schema on write. We recommend that you do not set this parameter to true. Instead, pass the mergeSchema option explicitly when writing data to control schema evolution per write operation.

    Performance tuning

    ParameterData typeDefaultDescription
    spark.databricks.delta.snapshotPartitionsInt10Number of partitions used when reading Delta log metadata. Increase this value for large Delta logs to improve parsing performance; decrease it for small Delta logs. This parameter has a significant effect on Delta table parsing speed.
    spark.sql.sources.parallelPartitionDiscovery.parallelismInt1000Number of parallel tasks Delta uses to scan files. This parameter applies only to the VACUUM command. Decrease it if the number of files is small to avoid unnecessary overhead. This is a Spark SQL parameter.