After you configure a Search index for an HBase table, data that contains timestamps cannot be written to the index column in the collection. If data that contains timestamps is written to the index column, the following error message is reported: User defined timestamp is not allowed when external index is enabled.... However, data that contains timestamps can be written to non-index columns. Take note of the following scenarios:

  • By default, data that has custom timestamps cannot be written to the index column.
  • HBase tables cannot have multiple versions.
  • Support for time-to-live (TTL) of tables is limited.

1. Custom timestamps

  1. Your business requires data that contains timestamps.
  2. Data in the HBase table is synchronized to the Search index by Big Data Service (BDS), such as primary-backup synchronizations and incremental data import by ApsaraDB for RDS. Therefore, BDS may add the write timestamp to the data.

To enable custom timestamps, you can run the HBase Shell command to set the MUTABILITY property to MUTABLE_ALL. If you enable custom timestamps, a small performance loss is incurred. However, this performance loss is negligible.

# Enable custom timestamps.
hbase(main):002:0> alter 'testTable', MUTABILITY=> 'MUTABLE_ALL'
# Disable custom timestamps.
hbase(main):002:0> alter 'testTable', MUTABILITY=> 'MUTABLE_LATEST'

2. Multiple versions

After you configure a Search index for an HBase table, if the table has multiple versions, the source data in the HBase table may be inconsistent with the data that is synchronized in the Search instance. You must use HBase Shell to set the version of the HBase table to 1. You cannot specify a version of the table to delete. Delete operations that specify table versions are not allowed. Example:

    // If no column families or qualifiers are specified for the object to be deleted, the entire row of the table is deleted. This operation is supported.
    Delete delete = new Delete(Bytes.toBytes("row"));
    // Delete the f:q1 column. This operation is supported.
    delete.addColumns(Bytes.toBytes("f"), Bytes.toBytes("q1"));
    // Delete all data that have timestamps of ts1 and before ts1. This operation is supported after you set the MUTABILITY property to MUTABLE_ALL.
    delete.addColumns(Bytes.toBytes("f"), Bytes.toBytes("q1"), ts1);
    // Delete all columns whose family name is f. This operation is supported.
    delete.addFamily(Bytes.toBytes("f"));
    // Delete all data that has timestamps of ts1 and before ts1 in columns whose family name is f. This operation is supported after you set the MUTABILITY property to MUTABLE_ALL.
    delete.addFamily(Bytes.toBytes("f"), ts1);
    // Delete the latest version of the f:q1 column. This operation is not supported.
    delete.addColumn(Bytes.toBytes("f"), Bytes.toBytes("q1"));
    // Delete the data whose timestamp is ts1 in the f:q1 column. This operation is not supported.
    delete.addColumn(Bytes.toBytes("f"), Bytes.toBytes("q1"), ts1);

Note: If you delete a row in an HBase table, the document that is associated with the row is not deleted in the Search index. Instead, all fields in the document are deleted to ensure data consistency. The deleted fields do not include the ID field, which is the mapping rowkey. Generally, you want to query data in the Search index with specified conditions. Therefore, rows that meet the conditions are returned. The returned rows include the row that has the ID field. However, if only the row that has the ID field is returned, this indicates that the row is deleted from the HBase table. You must specify the conditions to filter out the row. In the future, the TTL mechanism of the Search service will be used to automatically delete rows.

3. TTL

HBase and Search support TTL, which allows the system to automatically expire table rows. However, the TTL mechanism of HBase is different from the TTL mechanism of Search. HBase expires data on a single key-value pair, whereas the Search service can only expire data based on the document that corresponds to the row in the HBase table. Furthermore, both mechanisms calculate expiration time differently. If the TTL mechanism is required in your business, you can contact customer service in the ApsaraDB for HBase Q&A DingTalk group or submit a ticket.