The apack plug-in is developed by the Alibaba Cloud Elasticsearch team. This plug-in provides the physical replication feature. It greatly reduces CPU overheads and improves write performance in scenarios such as logging and time series analysis. In these scenarios, replica shards are configured for indexes, large amounts of data are written, and data visibility is latency-insensitive. This topic describes how to use the physical replication feature of the apack plug-in.

Prerequisites

An Alibaba Cloud Elasticsearch V6.7.0 cluster is created. The kernel version of the cluster is 1.2.0 or later. For more information, see Create an Elasticsearch cluster.

Background information

Basic principle of the physical replication feature:

If the feature is disabled, the system writes index data to a primary shard when the node where the primary shard resides receives a write request. Then, the system synchronizes the request to the nodes where the replica shards of the primary shard reside and writes the index data to the replica shards. This process is the same as that of open-source Elasticsearch. In this process, index data is written to not only the primary shard and its replica shards but also their translogs. After the feature is enabled, index data is written only to the primary shard, its translog, and the translogs of its replica shards. Each time the primary shard is refreshed, incremental index data is copied to its replica shards over the network. This feature delays data visibility but significantly improves the write performance of a cluster. It also ensures data reliability and consistency.

Performance testing of the physical replication feature:
  • Test environment
    • Node configuration: 5 data nodes (each with 8 vCPUs and 32 GiB of memory) and one 2-TiB standard SSD
    • Dataset: 74-GiB nyc_taixs of Rally provided by open-source Elasticsearch
    • Index configuration: five primary shards and one replica shard for each primary shard (default configuration)
  • Test result
    Service Write speed (document/s)
    Open-source Elasticsearch 6.7.0 127,305
    Alibaba Cloud Elasticsearch V6.7.0 (with the physical replication feature enabled) 184,592
  • Test conclusion

    Alibaba Cloud Elasticsearch with the physical replication feature enabled delivers a write performance 45% better than open-source Elasticsearch.

Note You can run the commands provided in this topic in the Kibana console. For more information, see Log on to the Kibana console.

Precautions

  • In addition to the physical replication feature, the apack plug-in provides the vector retrieval feature. For information about how to use the vector retrieval feature, see Use the aliyun-knn plug-in for vector search.
  • Only Alibaba Cloud Elasticsearch V6.7.0 clusters support the apack plug-in. If you purchase a cluster after the plug-in is launched, this plug-in is already installed on your cluster. If your cluster is purchased before the plug-in is launched and the kernel version of the cluster is 1.2.0 or later, you can manually install the plug-in on the cluster. For information about how to install the plug-in, see Install and remove a built-in plug-in. If the kernel version of your cluster is earlier than 1.2.0, you need to first update the kernel. For more information, see Update the kernel of a cluster. During the update, the system automatically installs the plug-in for your cluster. The plug-in cannot be removed after installation.
  • The physical replication feature of the apack plug-in works on indexes but is disabled by default. You must manually enable it before you can use it.
  • You can disable the physical replication feature for an index. However, before you disable it, disable the index.
  • Before you enable the physical replication feature for an index, disable the index and set the number of replica shards to 0.

Enable the physical replication feature for a new index

When you create an index, use the settings configuration to enable the physical replication feature for the index. 
PUT index-1
{
"settings": {
 "index.replication.type" : "segment"
 }
}

Disable the physical replication feature for an index

  1. Disable the index.
    POST index-1/_close
  2. Update the settings configuration to disable the physical replication feature.
    PUT index-1/_settings
{
"index.replication.type" : null
}
  3. Enable the index.
    POST index-1/_open

Enable the physical replication feature for an existing index

  1. Set the number of replica shards to 0.
    PUT index-1/_settings
{
  "index.number_of_replicas": 0
}
  2. Disable the index.
    POST index-1/_close
  3. Update the settings configuration to enable the physical replication feature.
    PUT index-1/_settings
{
"index.replication.type" : "segment"
}
  4. Enable the index.
    POST  index-1/_open
  5. Set the number of replica shards to 1.
    PUT index-1/_settings
{
  "index.number_of_replicas": 1
}