Add, delete, modify, and query index data in Elasticsearch - Elasticsearch

This topic describes how to create an Alibaba Cloud Elasticsearch cluster and access the cluster. It also describes how to call Elasticsearch RESTful APIs to create an index, create a document and insert data into the document, search for data, and delete the index.

Prerequisites

An Alibaba Cloud account is created. To create an Alibaba Cloud account, visit the account registration page.

Scenarios

A finance service enterprise uses an online platform to manage its wealth management products and uses conventional databases to provide the search feature for customers. The wealth management products provided by the enterprise offer satisfactory returns, and the customer base of the enterprise grows rapidly. The expansion of its business system and the increase in customer information cause the inherent issues of conventional databases to become noticeable. These issues include slow search responses, low accuracy, and the low performance of data service devices. To resolve these issues and improve customer satisfaction, the enterprise purchases the Alibaba Cloud Elasticsearch service. This topic uses the preceding scenario to describe how to use Alibaba Cloud Elasticsearch to create a cluster and search for data.

For example, the enterprise provides the following wealth management products:

{
  "products":[
    {"productName":"Wealth Management Product A","annual_rate":"3.2200%","describe":"180-day wealth management product that requires the minimum investment of USD 20,000, enables low-risk investment, and allows you to select whether to receive push messages for returns."}
    {"productName":"Wealth Management Product B","annual_rate":"3.1100%","describe":"90-day wealth management product that requires the minimum investment of USD 10,000 and daily pushes messages for returns credited to your account."}
    {"productName":"Wealth Management Product C","annual_rate":"3.3500%","describe":"270-day wealth management product that requires the minimum investment of USD 40,000 and daily pushes messages for returns immediately credited to your account."}
    {"productName":"Wealth Management Product D","annual_rate":"3.1200%","describe":"90-day wealth management product that requires the minimum investment of USD 12,000 and daily pushes messages for returns credited to your account."}
    {"productName":"Wealth Management Product E","annual_rate":"3.0100%","describe":"30-day wealth management product that requires the minimum investment of USD 8,000 and daily pushes messages for returns."}
    {"productName":"Wealth Management Product F","annual_rate":"2.7500%","describe":"3-day popular wealth management product that does not require service fees, requires the minimum investment of USD 500, and pushes messages for returns."}
  ]
}

Procedure

Step 1: Create a cluster
Create an Alibaba Cloud Elasticsearch cluster.
Step 2: Access the cluster by using Kibana
After the state of the cluster becomes Normal, log on to the Kibana console of the cluster to access the cluster.
Step 3: Create an index
Call a RESTful API to create an index.
Step 4: Create documents and insert data into the documents
Call a RESTful API to create documents and insert data into the documents.
Step 5: Search for data
Call a RESTful API to perform a full-text search or search for data by condition.
Step 6: (Optional) Delete the index
Call a RESTful API to delete the index to save resources if you no longer require the index.
Step 7: (Optional) Release the cluster
Release the cluster if you no longer require the cluster. After a cluster is released, data stored in the cluster cannot be recovered. We recommend that you back up data before you release a cluster.

Video tutorial

Step 1: Create a cluster

Go to the Elasticsearch buy page.

On the buy page, configure the parameters.

Parameter	Description
Billing Method	Subscription: If you plan to use an Elasticsearch cluster for an extended period of time, we recommend that you select the subscription billing method. You need to pay an upfront fee for a subscription Elasticsearch cluster. The subscription billing method is more cost-effective than the pay-as-you-go billing method. Pay-as-you-go: If you want to use an Elasticsearch cluster for a short period of time or for program development or functional tests, we recommend that you select the pay-as-you-go billing method. You are charged for a pay-as-you-go Elasticsearch cluster on an hourly basis. You can release a pay-as-you-go Elasticsearch cluster at any time. After you release the cluster, the billing of the cluster immediately stops. You can create a pay-as-you-go Elasticsearch cluster first. After you confirm that the created Elasticsearch cluster meets your business requirements, you can switch the billing method of the Elasticsearch cluster to subscription. Note For more information, see Billing methods.
Region and Zone	Select a region based on your business requirements. After an Elasticsearch cluster is created, you cannot change the region of the cluster. A region that is closer to the geographical location of your users provides lower latency and higher access speed. If you want an Elasticsearch cluster to communicate with other Alibaba Cloud services over an internal network, deploy the cluster in the same region as the services.
Number of Zones	1-AZ: This deployment method is the default deployment method. It is used to handle non-critical workloads. 2-AZ: This deployment method implements cross-zone disaster recovery. It is used to handle production workloads. 3-AZ: This deployment method provides high availability. It is used to handle production workloads that require high service availability. Note When you deploy a cluster across zones, you do not need to specify each zone. The system automatically selects the zones. For more information, see Deploy and use a multi-zone Elasticsearch cluster.
Network Type	The value of this parameter can be only VPC.
VPC	Select a virtual private cloud (VPC) in the specified region. Note If you want an Elasticsearch cluster to communicate with other Alibaba Cloud services over an internal network, we recommend that you deploy the cluster in the same VPC as the services. If no VPC meets your business requirements, create a VPC first. For more information, see Create a VPC and a vSwitch.
VSwitch	Select a vSwitch in the specified VPC. Only available vSwitches in the specified zone are displayed. Note If no vSwitch meets your business requirements, create a vSwitch first. For more information, see Create a VPC and a vSwitch.
Instance Type	Kernel-enhanced Edition: In addition to open source Elasticsearch features, Elasticsearch clusters of this edition provide more than 10 enhanced features based on the optimized kernel AliES. Standard Edition: Elasticsearch clusters of this edition are fully compatible with all open source Elasticsearch features and provide all advanced features of the X-Pack plug-in for free. For more information, see Version features and Features supported by Kernel-enhanced Edition.
Version	Kernel-enhanced Edition: 7.10, 7.16, and 6.7 Standard Edition: 8.13, 8.9, 8.5, 7.7, 6.8, 6.3, 5.6, and 5.5 For more information, see Version features and Release Notes. Note The versions supported in different regions and zones may vary. You can view the supported versions on the buy page.
Parameters for different types of nodes	Configure nodes based on your business requirements. Note If you do not know the specifications of the nodes that you want to purchase, you need to first evaluate specifications and storage capacity for your Elasticsearch cluster. For more information, see Evaluate specifications and storage capacity. For more information about how to configure each type of node, see Configure nodes for an Elasticsearch cluster.
Resource Group	Select a resource group. You can use resource groups to manage Elasticsearch clusters. For more information, see Classify resources into resource groups and grant permissions on the resource groups.
Username	The value of this parameter can be only elastic. The elastic account is used to access the Elasticsearch cluster and log on to the Kibana console of the cluster. Note If you change the password of the elastic account, you cannot use the elastic account to access the cluster before the new password takes effect. Therefore, we recommend that you do not use the elastic account in your business. You can log on to the Kibana console of your Elasticsearch cluster and create a user with the required role to access the cluster. For more information, see Use the role-based access control (RBAC) mechanism provided by Elasticsearch X-Pack to implement access control.
Password	The password of the elastic account. This parameter is required.
Duration	You need to configure this parameter if you set Billing Method to Subscription. We recommend that you select Auto-renewal to prevent your business from being affected due to cluster suspension or release after the cluster expires. Monthly subscription: The auto-renewal period is one month. Yearly subscription: The auto-renewal period is one year.

Click Buy Now. The Confirm Order page appears. Confirm the order, read the terms of service, select the check box for Terms of Service, and then click Activate Now.
In the top navigation bar of the Elasticsearch Clusters page, select the region where the newly created Elasticsearch cluster resides. Then, view the cluster.
A period of time is required before an Elasticsearch cluster can provide services after it is purchased. The period of time varies based on the specifications, data structure, and data volume of the cluster. In most cases, a few hours are required. You can view the status of the cluster in the Status column.

Step 2: Access the cluster by using Kibana

After the state of the cluster changes to Normal, you can perform the following steps to use Kibana to access the cluster:

Note

You can also use a cURL command or a client to access the cluster. For more information, see Access an Elasticsearch cluster.

On the Elasticsearch Clusters page, find the newly created cluster and click its ID.
In the left-side navigation pane of the page that appears, choose Configuration and Management > Data Visualization.
(Optional) In the Kibana section of the page that appears, click Modify Configuration. In the Network Access Configuration section of the Kibana Configuration page, configure a public IP address whitelist for Kibana. You need to add the IP address of your client to the whitelist. For more information, see Configure a public or private IP address whitelist for Kibana.
If this is the first time you log on to the Kibana console of your cluster over the Internet and you have not modified the configuration for Kibana access over the Internet, you need to modify the configuration as prompted.
Return to the Data Visualization page. In the Kibana section, click Access over Internet.
On the Kibana logon page, enter the username and password and click Log in.
The username is elastic. The password is the one that is specified when you create the cluster. If you forget the password, you can reset it. For more information, see Reset the access password for an Elasticsearch cluster.
On the page that appears, click Explore on my own.
Click the icon in the upper-left corner of the page that appears and choose Management > Dev Tools.

On the Console tab, run the following command to access the cluster:

GET /

If the access is successful, the following result is returned:

{
  "name": "es-cn-wuf3pxg630005****-data-h-0",
  "cluster_name": "es-cn-wuf3pxg63000****",
  "cluster_uuid": "1kaVmF46T2yOapAA6M****",
  "version": {
    "number": "8.9.1",
    "build_flavor": "default",
    "build_type": "docker",
    "build_hash": "652eb44fa71049c1fcdc08c967ef9f751d98****",
    "build_date": "2023-08-18T06:51:36.666651565Z",
    "build_snapshot": false,
    "lucene_version": "9.7.0",
    "minimum_wire_compatibility_version": "7.17.0",
    "minimum_index_compatibility_version": "7.0.0"
  },
  "tagline": "You Know, for Search"
}

Step 3: Create an index

Run one of the following commands to create an index named product_info:

V7.0 or later

PUT /product_info
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
      "properties": {
        "productName": {
          "type": "text",
          "analyzer": "ik_smart"
        },
        "annual_rate":{
          "type":"keyword"
        },
        "describe": {
          "type": "text",
          "analyzer": "ik_smart"
        }
    }
  }
}

Earlier than V7.0

PUT /product_info
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
    "products": {
      "properties": {
        "productName": {
          "type": "text",
          "analyzer": "ik_smart"
        },
        "annual_rate":{
          "type":"keyword"
        },
        "describe": {
         "type": "text",
         "analyzer": "ik_smart"
        }
      }
    }
  }
}

Important

Mapping types are deprecated in open source Elasticsearch 7.0 and later. However, these mapping types are still supported in versions earlier than Elasticsearch 7.0. If mapping types are used in Alibaba Cloud Elasticsearch V7.0 and later, the system displays the error message "type": "mapper_parsing_exception". For more information about the error details and solution, see Removal of mapping types.

In the preceding example, an index named product_info is created. In a version earlier than Elasticsearch V7.0, the index is of the products type. In Elasticsearch V7.0 or later, the index is of the _doc type. The index contains the productName, annual_rate, and describe fields.

If the command is successfully run, the following result is returned:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "product_info"
}

Step 4: Create documents and insert data into the documents

Call the _bulk API to insert multiple data records at a time.

Earlier than V7.0

POST /product_info/products/_bulk
{"index":{}}
{"productName":"Wealth Management Product A","annual_rate":"3.2200%","describe":"180-day wealth management product that requires the minimum investment of USD 20,000, enables low-risk investment, and allows you to select whether to receive push messages for returns."}
{"index":{}}
{"productName":"Wealth Management Product B","annual_rate":"3.1100%","describe":"90-day wealth management product that requires the minimum investment of USD 10,000 and daily pushes messages for returns credited to your account."}
{"index":{}}
{"productName":"Wealth Management Product C","annual_rate":"3.3500%","describe":"270-day wealth management product that requires the minimum investment of USD 40,000 and daily pushes messages for returns immediately credited to your account."}
{"index":{}}
{"productName":"Wealth Management Product D","annual_rate":"3.1200%","describe":"90-day wealth management product that requires the minimum investment of USD 12,000 and daily pushes messages for returns credited to your account."}
{"index":{}}
{"productName":"Wealth Management Product E","annual_rate":"3.0100%","describe":"30-day wealth management product that requires the minimum investment of USD 8,000 and daily pushes messages for returns."}
{"index":{}}
{"productName":"Wealth Management Product F","annual_rate":"2.7500%","describe":"3-day popular wealth management product that does not require service fees, requires the minimum investment of USD 500, and pushes messages for returns."}

V7.X

POST /product_info/_doc/_bulk
{"index":{}}
{"productName":"Wealth Management Product A","annual_rate":"3.2200%","describe":"180-day wealth management product that requires the minimum investment of USD 20,000, enables low-risk investment, and allows you to select whether to receive push messages for returns."}
{"index":{}}
{"productName":"Wealth Management Product B","annual_rate":"3.1100%","describe":"90-day wealth management product that requires the minimum investment of USD 10,000 and daily pushes messages for returns credited to your account."}
{"index":{}}
{"productName":"Wealth Management Product C","annual_rate":"3.3500%","describe":"270-day wealth management product that requires the minimum investment of USD 40,000 and daily pushes messages for returns immediately credited to your account."}
{"index":{}}
{"productName":"Wealth Management Product D","annual_rate":"3.1200%","describe":"90-day wealth management product that requires the minimum investment of USD 12,000 and daily pushes messages for returns credited to your account."}
{"index":{}}
{"productName":"Wealth Management Product E","annual_rate":"3.0100%","describe":"30-day wealth management product that requires the minimum investment of USD 8,000 and daily pushes messages for returns."}
{"index":{}}
{"productName":"Wealth Management Product F","annual_rate":"2.7500%","describe":"3-day popular wealth management product that does not require service fees, requires the minimum investment of USD 500, and pushes messages for returns."}

V8.X

POST /product_info/_bulk
{"index":{}}
{"productName":"Wealth Management Product A","annual_rate":"3.2200%","describe":"180-day wealth management product that requires the minimum investment of USD 20,000, enables low-risk investment, and allows you to select whether to receive push messages for returns."}
{"index":{}}
{"productName":"Wealth Management Product B","annual_rate":"3.1100%","describe":"90-day wealth management product that requires the minimum investment of USD 10,000 and daily pushes messages for returns credited to your account."}
{"index":{}}
{"productName":"Wealth Management Product C","annual_rate":"3.3500%","describe":"270-day wealth management product that requires the minimum investment of USD 40,000 and daily pushes messages for returns immediately credited to your account."}
{"index":{}}
{"productName":"Wealth Management Product D","annual_rate":"3.1200%","describe":"90-day wealth management product that requires the minimum investment of USD 12,000 and daily pushes messages for returns credited to your account."}
{"index":{}}
{"productName":"Wealth Management Product E","annual_rate":"3.0100%","describe":"30-day wealth management product that requires the minimum investment of USD 8,000 and daily pushes messages for returns."}
{"index":{}}
{"productName":"Wealth Management Product F","annual_rate":"2.7500%","describe":"3-day popular wealth management product that does not require service fees, requires the minimum investment of USD 500, and pushes messages for returns."}

If "errors" : false is returned, data is inserted into the created documents.

Step 5: Search for data

Important

The following command can be used only for Elasticsearch clusters of a version earlier than V7.0. Before you run the command for an Elasticsearch V7.X cluster, you must change products in the command to _doc. Before you run the command for an Elasticsearch V8.X cluster, you must remove products from the command.

Full-text search

Run the following command to search for products whose descriptions contain daily pushes messages for returns credited to your account:

GET /product_info/products/_search
{
  "query": {
    "match": {
      "describe": "daily pushes messages for returns credited to your account"
    }
  }
}

If the command is successfully run, the following result is returned:

{
  "took" : 21,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 6,
    "max_score" : 1.3968885,
    "hits" : [
      {
        "_index" : "product_info",
        "_type" : "products",
        "_id" : "WLvWYXAB8Rql5AUxLqUU",
        "_score" : 1.3968885,
        "_source" : {
          "productName" : "Wealth Management Product B",
          "annual_rate" : "3.1100%",
          "describe" : "90-day wealth management product that requires the minimum investment of USD 10,000 and daily pushes messages for returns credited to your account."
        }
      },
      {
        "_index" : "product_info",
        "_type" : "products",
        "_id" : "WrvWYXAB8Rql5AUxLqUU",
        "_score" : 1.3968885,
        "_source" : {
          "productName" : "Wealth Management Product D",
          "annual_rate" : "3.1200%",
          "describe" : "90-day wealth management product that requires the minimum investment of USD 12,000 and daily pushes messages for returns credited to your account."
        }
      },
      {
        "_index" : "product_info",
        "_type" : "products",
        "_id" : "WbvWYXAB8Rql5AUxLqUU",
        "_score" : 1.3547361,
        "_source" : {
          "productName" : "Wealth Management Product C",
          "annual_rate" : "3.3500%",
          "describe" : "270-day wealth management product that requires the minimum investment of USD 40,000 and daily pushes messages for returns immediately credited to your account."
        }
      },
      {
        "_index" : "product_info",
        "_type" : "products",
        "_id" : "W7vWYXAB8Rql5AUxLqUU",
        "_score" : 1.1507283,
        "_source" : {
          "productName" : "Wealth Management Product E",
          "annual_rate" : "3.0100%",
          "describe" : "30-day wealth management product that requires the minimum investment of USD 8,000 and daily pushes messages for returns."
        }
      },
      {
        "_index" : "product_info",
        "_type" : "products",
        "_id" : "XLvWYXAB8Rql5AUxLqUU",
        "_score" : 0.5753642,
        "_source" : {
          "productName" : "Wealth Management Product F",
          "annual_rate" : "2.7500%",
          "describe" : "3-day popular wealth management product that does not require service fees, requires the minimum investment of USD 500, and pushes messages for returns."
        }
      },
      {
        "_index" : "product_info",
        "_type" : "products",
        "_id" : "V7vWYXAB8Rql5AUxLqUU",
        "_score" : 0.31854028,
        "_source" : {
          "productName" : "Wealth Management Product A",
          "annual_rate" : "3.2200%",
          "describe" : "180-day wealth management product that requires the minimum investment of USD 20,000, enables low-risk investment, and allows you to select whether to receive push messages for returns."
        }
      }
    ]
  }
}

Note

Alibaba Cloud Elasticsearch allows you to use a tokenizer to search for data. It also allows you to sort the records in search results by score. In the preceding result, the descriptions of the first two products contain daily pushes messages for returns credited to your account, and the descriptions of the last two products contain only pushes messages. The higher the ranking of a product in the search result, the higher the matching degree and score of the product.

Search by condition

Run the following command to search for products with an annualized rate of 3.0000% to 3.1300%:

GET /product_info/products/_search
{
  "query": {
    "range": {
      "annual_rate": {
        "gte": "3.0000%",
        "lte": "3.1300%"
      }
    }
  }
}

If the command is successfully run, the following result is returned:

{
  "took" : 10,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "product_info",
        "_type" : "products",
        "_id" : "WLvWYXAB8Rql5AUxLqUU",
        "_score" : 1.0,
        "_source" : {
          "productName" : "Wealth Management Product B",
          "annual_rate" : "3.1100%",
          "describe" : "90-day wealth management product that requires the minimum investment of USD 10,000 and daily pushes messages for returns credited to your account."
        }
      },
      {
        "_index" : "product_info",
        "_type" : "products",
        "_id" : "WrvWYXAB8Rql5AUxLqUU",
        "_score" : 1.0,
        "_source" : {
          "productName" : "Wealth Management Product D",
          "annual_rate" : "3.1200%",
          "describe" : "90-day wealth management product that requires the minimum investment of USD 12,000 and daily pushes messages for returns credited to your account."
        }
      }
    ]
  }
}

Note

The system finds products that meet your requirements based on the search conditions and displays the products in ascending order.

For more information about search methods, see Query DSL.

Step 6: (Optional) Delete the index

After you are familiar with Alibaba Cloud Elasticsearch, you can delete the index by running the following command to avoid wasting resources:

Warning

A deleted index cannot be recovered. Proceed with caution.

DELETE /product_info

If the index is deleted, the following result is returned:

{
  "acknowledged" : true
}

Step 7: (Optional) Release the cluster

If you no longer require a pay-as-you-go cluster, you can release it. After the cluster is released, you are no longer charged for the cluster. In addition, the data stored in the cluster is deleted and cannot be recovered.

Warning

After a cluster is released, data stored in the cluster cannot be recovered. We recommend that you back up data before you release a cluster. For more information, see Data backup overview.

On the Elasticsearch Clusters page, find the cluster that you want to release, move your pointer over the icon in the Actions column, and then select Release Cluster.
In the message that appears, click OK.

Note

For information about how to unsubscribe from a subscription Elasticsearch cluster that you no longer require, see Unsubscription and refund policies.

References

Learn how to access and configure an Alibaba Cloud Elasticsearch cluster: Access and configure an Elasticsearch cluster
Learn how to migrate data from a third-party Elasticsearch source to an Alibaba Cloud Elasticsearch cluster: Select a data migration solution
Learn how to synchronize data from an ApsaraDB RDS for MySQL database to an Alibaba Cloud Elasticsearch cluster: Select a synchronization method
Learn FAQ about Alibaba Cloud Elasticsearch: FAQ