Hologres V5.0.0 and later support Blob Storage. Use the blob_columns table property to store large values in TEXT and BYTEA columns separately. This reduces point query I/O and Compaction overhead without changing the table schema or query semantics. This topic describes how Blob Storage works, how to use it, and its limits.
Background
In Hologres column-oriented tables, very large values such as long text, document content, and wide log fields cause the following problems:
-
More files: Large values take up so much space that each data file holds far fewer rows, which produces a large number of small files.
-
Point query I/O amplification: A point query that returns a single row still reads every large value in the data block that contains the row.
-
Compaction write amplification: Compaction repeatedly reads, rewrites, and moves large values.
Blob Storage stores values that exceed a size threshold separately and keeps only a reference to each value in the original column. This improves how large values are stored, read, and compacted, without changing the table schema or query semantics.
Use cases
Blob Storage suits workloads that contain large-value columns and consist mainly of writes and point queries, such as:
-
Storing large text or document content, such as LLM corpora and conversation content.
-
Storing very wide log columns, such as log fields of tens of KB or more per row.
-
Retrieving entire rows, including large-value columns, by primary key or a highly selective filter.
Do not enable Blob Storage for OLAP workloads that frequently run full scans on large-value columns. For more information, see Limits.
How it works
-
Per-row adaptive storage: For each column with Blob Storage enabled, the system checks the value size row by row. Values that reach the threshold are compressed and stored separately, and the original column keeps only a reference. Smaller values stay in the original column. A single column can contain rows stored either way.
-
On-demand reads: The system reads a large value only when the query actually returns it. Rows filtered out by the WHERE clause and columns that the query does not reference never trigger a read of separately stored large values.
-
No rewrite during Compaction: By default, Compaction moves only the references when it merges files and does not read or rewrite the separately stored large values. Large values are reorganized only when the system reclaims deleted data, or when the proportion of invalid data or file fragmentation becomes too high.
Companion feature: Blob Column Lazy Seek
For highly selective point queries such as WHERE k = x, Lazy Seek waits until filtering identifies the matching rows, and then reads only the large values of those rows. It does not read and decompress every large value in the row group.
-
Parameter: Controlled by the Grand Unified Configuration (GUC) parameter
hg_experimental_enable_blob_column_lazy_seek. Default value:on. -
Conditions: Lazy Seek takes effect only when exactly one row in a row group matches and the Blob Storage column is a direct output column. Otherwise, the system falls back to reading the entire column. Query results are not affected.
Benefits
-
Much lower point query I/O: A point query reads only the large values of the matching rows instead of all large values in the row group. With Lazy Seek, the amount of data read and decompressed drops significantly.
-
Much lower Compaction write amplification: By default, Compaction moves only references and does not rewrite large values, so it finishes much faster.
-
Fewer files: With large values stored separately, data files are smaller and each file holds more rows, so fewer small files are created.
-
Transparent to applications: You can enable Blob Storage on existing TEXT and BYTEA columns directly. No special column type or SQL change is required.
Blob Storage is not designed to reduce total storage. Separately stored large values still consume storage. Measured by pg_relation_size, tables with and without Blob Storage use roughly the same amount of storage after Full Compaction runs on both tables. The benefits of Blob Storage are lower point query I/O, more efficient Compaction, and fewer files, not storage savings.
Limits
-
Blob Storage is supported only in Hologres V5.0.0 and later.
-
Only TEXT and BYTEA columns are supported. Blob Storage is available for column-oriented tables and row-column hybrid tables, but not for row-oriented tables.
-
Columns with Blob Storage enabled do not support dictionary encoding or bitmap indexes. If you set
dictionary_encoding_columnsorbitmap_columnsfor such a column, the system ignores the setting and treats it as disabled. Other indexes, including full-text inverted indexes, work as usual. Avoid enabling Blob Storage on low-cardinality columns that you filter by equality. Without the bitmap index, equality filters fall back to full table scans that must read the separately stored large values, which makes queries significantly slower. -
OLAP queries that fully scan large-value columns become slower: A full scan must read and decompress each large value one by one, which is slower than without Blob Storage. If your workload frequently runs full table scans on large-value columns, evaluate the impact before you enable Blob Storage.
-
Storage usage depends on your data and does not necessarily decrease: For small, highly repetitive (low-cardinality) values, storage usage may increase. For large values with little repetition, storage usage may decrease. Run Full Compaction before you compare storage usage.
-
Partitioned tables: Child partitions inherit
blob_columnsfrom the parent table.ATTACH PARTITIONfails unless the parent table and the child table have the sameblob_columnssetting. -
In a column that stores raw binary data exactly 32 bytes long, a value has a very small chance of being misidentified as a large-value reference. If you need to store such data, disable Blob Storage for that column.
Enable Blob Storage
Call set_table_property to set the blob_columns table property, which specifies the columns that use Blob Storage. Syntax:
CALL set_table_property('<TABLE_NAME>', 'blob_columns', '<COLUMN_NAME>:{on|auto|off}[,...]');
The following table describes the parameters.
|
Parameter |
Description |
|
TABLE_NAME |
The name of the table. |
|
COLUMN_NAME |
The name of the column. Only TEXT and BYTEA columns are supported. Separate multiple columns with commas (,), for example, The default value of |
|
on |
Forces Blob Storage on for the column. Large values are stored separately. |
|
auto |
The system decides, based on the size threshold, whether to store each value separately. |
|
off |
Disables Blob Storage for the column. |
You can change blob_columns at any time. A change does not rewrite existing files immediately. Like bitmap_columns and dictionary_encoding_columns, it takes effect gradually as background Compaction runs.
-- Force Blob Storage on for the payload column
CALL set_table_property('<TABLE_NAME>', 'blob_columns', 'payload:on');
-- Let the system decide based on the size threshold
CALL set_table_property('<TABLE_NAME>', 'blob_columns', 'payload:auto');
Verify that Blob Storage takes effect
-
Check the configuration: Run the following statement to view the table DDL. If the output contains the
blob_columnsproperty, the configuration is applied.SELECT hg_dump_script('<TABLE_NAME>'); -
Compare point query I/O: Enable execution statistics and clear the cache. Then run
EXPLAIN ANALYZEon the same highly selective point query and comparedecompress_out_bytesandtotal_read_bytesinse_read_statistic.SET hg_experimental_show_execution_statistics_in_explain = on; SELECT hg_admin_command('freecache', 'table_name=<TABLE_NAME>'); EXPLAIN ANALYZE SELECT id, payload FROM <TABLE_NAME> WHERE k = 5000;With Blob Storage enabled and Lazy Seek set to
on, a point query that matches a single row decompresses only that row's data. With Lazy Seek disabled or without Blob Storage, the query decompresses the entire row group. -
Check storage usage: Run Full Compaction first, and then query storage usage with
pg_relation_sizeorhologres.hg_relation_size. Without Full Compaction, unmerged fragment files skew the comparison.SELECT hologres.hg_full_compact_table('<SCHEMA_NAME>.<TABLE_NAME>'); SELECT pg_relation_size('<TABLE_NAME>'); -- Alternatively SELECT hologres.hg_relation_size('<SCHEMA_NAME>.<TABLE_NAME>', 'data');
Upgrade and downgrade compatibility
-
Upgrade: Compatible. Upgrading an instance to Hologres V5.0.0 or later has no impact.
-
Downgrade: Not supported. Tables with Blob Storage enabled cannot be downgraded to a version earlier than Hologres V5.0.0. Earlier versions do not recognize separately stored large values and return the references in the original column as regular values, which produces incorrect data.
-
Before you downgrade: Remove all separately stored large values by using one of the following methods:
-
Drop the tables that have Blob Storage enabled.
-
Set the relevant columns in
blob_columnstooff, runVACUUMorhologres.hg_full_compact_tableon the table, and wait for Compaction to complete. Then run the following statement to confirm that no separately stored large values remain. If it returnsf(false), you can safely downgrade.SELECT EXISTS ( SELECT 1 FROM <TABLE_NAME> WHERE hg_file_meta IS NOT NULL AND COALESCE((hg_file_meta ->> 'blob_file_count')::bigint, 0) > 0 ) AS has_blob_file;
-
Example
The following example creates a column-oriented table named docs, forces Blob Storage on for the large-value column payload, and then runs a point query and checks storage usage.
-- Create a table and force Blob Storage on for the large-value column payload
BEGIN;
CREATE TABLE docs (
id INT PRIMARY KEY,
k INT,
payload TEXT
)
WITH (orientation = 'column');
CALL set_table_property('docs', 'clustering_key', 'k:asc');
CALL set_table_property('docs', 'blob_columns', 'payload:on');
COMMIT;
-- Write large values and flush them to disk
INSERT INTO docs SELECT g, g, repeat(md5(g::text), 3200) FROM generate_series(1, 10000) g;
SELECT hg_admin_command('flush', 'table_name=docs');
-- Highly selective point query: matches a single row, and Lazy Seek reads only that row's large value
SELECT id, payload FROM docs WHERE k = 5000;
-- Check storage usage (always run Full Compaction before you compare)
SELECT hologres.hg_full_compact_table('public.docs');
SELECT pg_size_pretty(pg_relation_size('docs'));