Lindorm:Common causes for unexpected query results

Data not written yet, or query ran before write completed

In big data pipelines, a failure or delay anywhere in the write path means data may not reach LindormTable when you expect it. If a query returns no results, the data may not have been written yet.

Diagnose: Add a hint to your query to return the timestamp alongside results. Compare that timestamp to the time you expect the write to have completed.

For details on adding hints to retrieve timestamps, see Use hints to implement data versioning.

If no timestamp is specified when writing a row, the returned timestamp reflects the time the row was actually written to the table.

STRING field contains stop characters or invisible characters

If a STRING column value contains invisible characters — appended by a program bug, for example — an exact-match query fails because the stored value differs from the queried value.

Example: A value stored as 1000<invisible character> does not match where orderID = "1000".

Diagnose: Run a range query to inspect the actual stored value:

SELECT * FROM <table> WHERE orderID > "1000" LIMIT 1;

If the row is returned, the value likely has a trailing invisible character. Fix the write path to strip such characters before writing.

Stop characters in the middle of a string: LindormTable does not support stop characters in the middle of a STRING value. Writing a string such as "1000\<stop character>\1000" causes an encoding exception, and that value cannot be queried.

Column name case is wrong, or column family is missing

Two distinct configuration mistakes both produce empty results.

Case mismatch: Column names in Lindorm are case-sensitive. A query using OrderID does not match a column named orderID.

Missing column family prefix: LindormTable wide tables support multiple column families. If no column family is specified at table creation, all columns are placed in the default column family f and no prefix is needed. However, if the table has multiple column families, queries that omit the column family prefix search only in f.

Example: A column named column1 lives in a column family named meta. A query using where column1 = xxx searches in f and returns nothing. Use where meta:column1 = xxx instead.

Data expired based on the table-level TTL

LindormTable TTL is specified in seconds. Timestamps written with data are in milliseconds.

If the TTL is set correctly, data expires naturally after the configured period. For example, data written today expires tomorrow when TTL is 86,400 seconds (one day).

Two edge cases cause data to be deleted immediately after being written:

1. Timestamp too small: If you specify an earlier timestamp when you write data and the difference between the timestamp and the current time is greater than the specified TTL, the data may be deleted immediately after it is written to the table. Custom version numbers such as 1, 2, or 3 are especially prone to this.

2. Timestamp in the wrong unit: If a timestamp in microseconds or nanoseconds is written instead of milliseconds, the timestamp value is far larger than expected. The data may never expire as intended, or may be treated as written millions of seconds in the future.

Check the TTL of a table:

1. Log on to the cluster management system.

2. On the Overview page, click the table name.

3. In the Current table details section, click View table properties.

4. Check the value of TTL.

Modify the TTL:

• Use Lindorm Shell to connect to LindormTable

• ALTER TABLE

Data expired based on a cell TTL

LindormTable supports a TTL in milliseconds on individual key-value pairs — called a cell TTL. When a cell TTL is set, the actual expiration time for a key-value pair is:

min(expiration based on cell TTL, expiration based on table TTL)

Both the cell TTL and the table TTL are calculated from the key-value pair's timestamp or version number. If the timestamp is too small or too large, the pair may expire earlier or later than expected.

Example behavior:

A column has two key-value pairs, KV1 (with a cell TTL) and KV2 (no cell TTL).

• Before KV1 expires: queries may return KV1 because its timestamp is more recent.

• After KV1 expires: KV1 is cleared, and KV2 is returned instead — which may not be the value you expect.

Whether KV2 is queryable also depends on the VERSIONS attribute and major compaction. If VERSIONS is set to 1, KV2 is deleted during major compaction once KV1 is cleared, even if no cell TTL was set on KV2.

Deletion timestamp deleted more data than expected

A DELETE operation removes all data written to the target row or column before the specified timestamp or version number. If no timestamp is provided, LindormTable uses the current time — deleting all data written before now.

Two problems can occur:

1. Data written after the delete timestamp is not deleted: If a row was written after the timestamp in the DELETE request, it is not deleted and appears in subsequent queries.

2. Writes after the deletion are immediately deleted: If a large timestamp is specified in the DELETE request, subsequent writes with smaller (normal) timestamps are treated as older than the deletion marker and deleted immediately.

Timestamps cannot be specified in SQL DELETE statements.

Example: DELETE FROM sensor WHERE p1 = 10; deletes all versions of matching rows written before the current time (January 16, 2024, 16:00:00 in this example).

Write-delete race condition in the data link

In big data pipelines where writes and deletes come from separate programs or processes, a delete operation can arrive at LindormTable before the write it was meant to follow. When this happens, the write lands after the delete — the row appears briefly, then is deleted.

Flink connector issue: Older versions of the Lindorm connector for Realtime Compute for Apache Flink have a bug where a write operation can be overwritten by a concurrent delete. To work around this, set ignoreDelete=true in your Flink configuration.

For details, see Lindorm connector.

VERSIONS attribute set to 0

The VERSIONS attribute controls how many versions of a key-value pair are retained. When VERSIONS is set to 0, no data is retained — every write is immediately deleted and cannot be queried.

The default value of VERSIONS is 1 (one version retained). The MIN_VERSIONS attribute defaults to 0 and does not cause data loss.

If VERSIONS was mistakenly set to 0:

• Delete the table and recreate it with VERSIONS set to 1 or higher, or

• Modify VERSIONS to 1 or higher using Lindorm Shell or ALTER TABLE.

Check the VERSIONS attribute:

1. Log on to the cluster management system.

2. On the Overview page, click the table name.

3. In the Current table details section, click View table properties.

4. Check the value of VERSIONS.

Modify VERSIONS:

• Use Lindorm Shell to connect to LindormTable

• ALTER TABLE

For more information about data versioning, see Use hints to implement data versioning.

Data updated or deleted in an immutable table

When the MUTABILITY attribute of a table is set to IMMUTABLE, data can only be written using a single UPSERT statement. Updates and deletes are not allowed.

However, LindormTable does not strictly block update and delete operations on immutable tables. Performing such operations causes data inconsistency between the index table and the base table: the same query can match different rows depending on which table is used for lookup.

To recover, rebuild the index table and stop issuing update or delete operations against the immutable table.