Elasticsearch:Searchable Snapshot

Restrictions

Enable archive data nodes

To use Searchable Snapshot, enable archive data nodes on your Alibaba Cloud Elasticsearch instance. Archive data nodes serve as the compute layer, maintaining index metadata, managing the local cache, and fetching data blocks from Alibaba Cloud OSS on demand.

1. Log on to the Alibaba Cloud Elasticsearch console.

2. Follow the appropriate steps based on your instance:

   • For new 8.17.0 instances: On the Create Instance page, select version 8.17.0. In the instance specifications section, select Archive Data Node and configure the node count and specifications.

3. Select the node specifications.

   Archive data nodes do not require large local disks because the data is stored in Alibaba Cloud OSS. We recommend configuring sufficient memory to improve the cache hit rate. Recommended configuration: 4 cores and 16 GB of memory or higher, with a disk of 500 GB or larger (Efficient Cloud Disk or ESSD).

4. Confirm the purchase and wait for the change to complete. Once the archive data nodes are ready, the Searchable Snapshot feature is enabled by default, and no further action is required.

Configure an OSS snapshot repository

Alibaba Cloud Elasticsearch includes the repository-oss plugin by default, so you can use Alibaba Cloud OSS as a snapshot repository without additional setup.

In the Kibana Dev Tools, run the following command to register an OSS snapshot repository:

PUT _snapshot/my_oss_repo
{
  "type": "oss",
  "settings": {
    "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
    "access_key_id": "<your_access_key_id>",
    "secret_access_key": "<your_secret_access_key>",
    "bucket": "<your_bucket_name>",
    "base_path": "es_snapshots",
    "compress": true,
    "chunk_size": "500mb"
  }
}

Verify the repository connection:

POST _snapshot/my_oss_repo/_verify

A successful response lists the cluster nodes, confirming that each node can connect to Alibaba Cloud OSS with the required permissions. For a complete list of OSS plugin parameters, see elasticsearch-repository-oss.

Create and mount a snapshot

PUT _snapshot/my_oss_repo/snapshot_20260227
{
  "indices": "logs-2025-*",
  "ignore_unavailable": true,
  "include_global_state": false
}

GET _snapshot/my_oss_repo/snapshot_20260227/_status

POST logs-2025-01-01/_doc
{ "message": "test log data 01", "level": "info", "timestamp": "2025-01-01T10:00:00" }

POST logs-2025-01-02/_doc
{ "message": "test log data 02", "level": "warn", "timestamp": "2025-01-02T10:00:00" }

POST logs-2025-01-03/_doc
{ "message": "test log data 03", "level": "error", "timestamp": "2025-01-03T10:00:00" }

POST logs-2025-01-04/_doc
{ "message": "test log data 04", "level": "info", "timestamp": "2025-01-04T10:00:00" }

POST logs-2025-01-05/_doc
{ "message": "test log data 05", "level": "debug", "timestamp": "2025-01-05T10:00:00" }

POST _snapshot/my_oss_repo/snapshot_20260227/_mount?storage=shared_cache&wait_for_completion=true
{
  "index": ".ds-logs-2025-01-01-2026.03.03-000001",
  "renamed_index": "frozen-logs-2025-01-01",
  "index_settings": {
    "index.number_of_replicas": 0
  },
  "ignore_index_settings": ["index.refresh_interval"]
}

GET frozen-logs-2025-01-01/_search
{
  "query": {
    "match": {
      "message": "error"
    }
  }
}

GET logs-2025-*,frozen-logs-2025-*/_search
{
  "query": {
    "range": {
      "timestamp": {
        "gte": "2025-01-01",
        "lt": "2025-02-01"
      }
    }
  }
}

DELETE frozen-logs-2025-01-01

POST _snapshot/my_oss_repo/snapshot_20260227/_restore
{
  "indices": ".ds-logs-2025-01-01-2026.03.03-000001",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1"
}

Automate the data lifecycle with ILM

You can use an Index Lifecycle Management (ILM) policy to automatically migrate indices to the Frozen Tier. The following policy implements a Hot → Warm → Frozen → Delete lifecycle. When an index enters the Frozen phase, ILM automatically creates a snapshot in Alibaba Cloud OSS and mounts it in partially mounted mode, without manual intervention:

PUT _ilm/policy/logs_lifecycle_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my_oss_repo"
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Lifecycle phase overview:

How it works

Searchable Snapshot uses a storage and compute separation architecture, where archive data nodes (the compute layer) and Alibaba Cloud OSS (the storage layer) work together:

• Alibaba Cloud OSS serves as the persistent storage layer and snapshot repository. All snapshot data is stored in Alibaba Cloud OSS, which uses built-in redundancy to ensure data reliability. This eliminates the need for replica shards in Elasticsearch.

• The archive data node is the compute layer, responsible for receiving query requests and coordinating data access. Each node contains three key components:

  • Metadata: Stores the index structure and shard mapping information, maintaining a logical view of the Searchable Snapshot index.

  • Shared Cache: Caches frequently accessed data blocks. This cache is shared by all shards of Searchable Snapshot indices on the same node.

  • Query Engine: Receives query requests, checks for a cache hit, and fetches data blocks from Alibaba Cloud OSS on demand if a miss occurs.

The data flow is divided into write and query paths:

• Write Path: Indices are snapshotted to Alibaba Cloud OSS through manual actions or ILM policies. When mounted with the storage=shared_cache parameter, an index enters partially mounted mode: data remains in Alibaba Cloud OSS while the archive data node loads only the metadata.

• Query Path: The archive data node first checks the local Shared Cache for the required data blocks. On a cache hit, it returns the result directly, with performance similar to local data. On a cache miss, it fetches the required data blocks from Alibaba Cloud OSS, returns the result, and then caches the data for future queries.

The Shared Cache defaults to 90% of the node's total disk space (or the total space minus 100 GB, whichever is smaller). It uses a Least Recently Used (LRU) policy to evict cold data blocks.

Elasticsearch categorizes data into multiple tiers based on access frequency. Searchable Snapshot serves the most cost-effective of these, the Frozen Tier:

Searchable Snapshot reduces costs through the following mechanisms:

• Elimination of replica shards: Alibaba Cloud OSS provides data reliability through its built-in redundancy, eliminating the need for replica shards. This reduces storage requirements by 50% and removes cross-availability zone traffic costs.

• Storage and compute separation: All data is stored in Alibaba Cloud OSS, which has a much lower unit price than cloud disk storage (often by a factor of ten or more). Archive data nodes only store metadata and a hot data cache.

• Extremely high data density ratio: Archive data nodes can achieve a memory-to-data ratio of 1:1500. A node with 64 GB of memory can manage approximately 100 TB of archived data, whereas the same resources in the Warm Tier could only support about 10 TB.

Use cases

• Log archiving: Migrate older logs to the Frozen Tier to reduce storage costs while retaining search capabilities.

• Compliance and auditing: Store data long-term for compliance purposes at a low cost, with on-demand retrieval.

• Historical data analysis: Analyze historical data that is accessed infrequently without occupying expensive, high-performance storage.

Related documentation

• Elasticsearch Searchable Snapshots official documentation

• Mount Snapshot API

• ILM Searchable Snapshot Action

• Alibaba Cloud OSS Snapshot Repository Plugin

• Node configuration for Alibaba Cloud Elasticsearch instances