Elasticsearch:Migrate documents from a Solr cluster to Alibaba Cloud Elasticsearch

Prerequisites

Before you begin, make sure you have:

• An Alibaba Cloud Elasticsearch V6.X cluster with Auto Indexing enabled. This example uses V6.3.2. For setup instructions, see Create an Alibaba Cloud Elasticsearch cluster and Configure the YML file.

• An Elastic Compute Service (ECS) instance running in the same region, zone, and virtual private cloud (VPC) as the Elasticsearch cluster. This example uses CentOS 7.3. For setup instructions, see Step 1: Create an ECS instance.

• Solr installed on the ECS instance. This example uses Solr 5.0.0. For more information, see the official Solr documentation.

• Python 3.0 or later installed on the ECS instance. This example uses Python 3.6.2.

• pysolr 3.3.3 or later (but earlier than 4.0) installed on the ECS instance.

Install solr-to-es

1. Connect to the ECS instance. For instructions, see Connect to a Linux instance by using a password or key.

   This example uses a common user account.

2. Download solr-to-es and navigate to the directory where setup.py is stored.

3. Install solr-to-es:

   sudo python setup.py install

Migrate documents

All migration commands follow the same structure: provide the Solr query URL, the Elasticsearch endpoint with credentials, the target index name, and the index type.

Run the following command to migrate documents from a Solr Core to an Elasticsearch index:

sudo python __main__.py <solr_url>:8983/solr/<my_core>/select http://<username>:<password>@<elasticsearch_url>:9200 <elasticsearch_index> <doc_type>

Replace the placeholders with your actual values:

If you use a different version of solr-to-es, try this alternative command syntax instead. For details, see the solr-to-es documentation.

sudo solr-to-es [-h] [--solr-query SOLR_QUERY] [--solr-fields COMMA_SEP_FIELDS]
                 [--rows-per-page ROWS_PER_PAGE] [--es-timeout ES_TIMEOUT]
                 solr_url elasticsearch_url elasticsearch_index doc_type

Running this command in the environment described in this topic returns the following error: -bash: solr-to-es.py: command not found

Example

This example queries all documents in the my_core Solr Core and writes them to the elasticsearch_index index in your Elasticsearch cluster.

1. On the ECS instance, navigate to the solr-to-es-master/solr_to_es directory.

2. Run the following command:

   sudo python __main__.py 'http://116.62.**.**:8983/solr/my_core/select?q=*%3A*&wt=json&indent=true' 'http://elastic:Your password@es-cn-so4lwf40ubsrf****.public.elasticsearch.aliyuncs.com:9200' elasticsearch_index doc_type

   The Solr query URL in this command uses the following parameters:

   Parameter	Description
   q	Required. The query expression using Solr's standard query syntax. *%3A* matches all documents.
   wt	Response format. Supported values: json, xml, python, ruby, csv.
   indent	Whether to indent the response for readability. Default: false.

   For the remaining parameters, see the Migrate documents section.

3. Log on to the Kibana console of your Elasticsearch cluster. For instructions, see Log on to the Kibana console.

   This example uses the Kibana console of an Elasticsearch V6.7 cluster. Steps may vary for other versions.

4. In the left-side navigation pane, click Dev Tools, then click Go to work.

5. On the Console tab, run the following command to confirm that the elasticsearch_index index was created:

   GET _cat/indices?v

6. Run the following command to view the migrated documents:

   GET /elasticsearch_index/doc_type/_search

   If successful, the response looks similar to this:

   {
     "took" : 12,
     "timed_out" : false,
     "_shards" : {
       "total" : 5,
       "successful" : 5,
       "skipped" : 0,
       "failed" : 0
     },
     "hits" : {
       "total" : 2,
       "max_score" : 1.0,
       "hits" : [
         {
           "_index" : "elasticsearch_index",
           "_type" : "doc_type",
           "_id" : "Tz8WNW4BwRjcQciJ****",
           "_score" : 1.0,
           "_source" : {
             "id" : "2",
             "title" : [
               "test"
             ],
             "_version_" : 1648195017403006976
           }
         },
         {
           "_index" : "elasticsearch_index",
           "_type" : "doc_type",
           "_id" : "Tj8WNW4BwRjcQciJ****",
           "_score" : 1.0,
           "_source" : {
             "id" : "1",
             "title" : [
               "change.me"
             ],
             "_version_" : 1648195007391203328
           }
         }
       ]
     }
   }

What's next

After confirming that the documents have been migrated successfully:

1. Update your application's connection settings to point to the Elasticsearch endpoint.

2. Validate that search queries return the expected results.

3. If Auto Indexing is no longer needed, disable it to avoid unintended index creation. For instructions, see Configure the YML file.