Lindorm:Use sharding (aliases)

Background information

• In the business logic, Table A is used to store business data. To use Table B to store the data, you can modify the configuration.

• Sharding

  In most business scenarios, only data generated in the previous week is frequently accessed. You can create a table each week to store the latest data. This approach lets you query hot data efficiently. However, you must manually create and delete tables, which increases the complexity of your business logic.

Scenarios

• Manage time series data

  If your business data is time-sensitive, you can create multiple indexes based on time. This reduces the size of a single index and improves query performance. During this process, you do not need to manually create or delete indexes.

• Rebuild an index

  You can rebuild indexes without affecting existing index-based data queries. After the indexes are rebuilt, the new indexes are used to query data. During this process, you do not need to modify existing code.

Create an alias for an existing collection

curl "http://solrhost:8983/solr/admin/collections?action=CREATEALIAS&name=your_alias_name&collections=your_collection_name_A"        

The preceding URL indicates: Create an alias named your_allias_name that points to a collection your_collection_name_A. In your business logic, you can set to access only your_alias_name, and the kernel will automatically forward requests to the actual collection.

To change the collection name to your_collection_name_B, execute the command to change the alias.

curl "http://solrhost:8983/solr/admin/collections?
action=ALIASPROP&name=your_alias_name&collections=your_collection_name_B"            

You can access the new collection without making any changes to your business code.

Automatic table sharding

LindormSearch can automatically create collections based on values in the time field. This helps you simplify your business logic. The following example shows how to configure the system to create a collection each week and automatically delete expired collections.

curl "http://solrhost:8983/solr/admin/collections?action=CREATEALIAS&name=test_router_alias&router.start=NOW-30DAYS/DAY&router.autoDeleteAge=/DAY-90DAYS&router.field=your_timestamp_l&router.name=time&router.interval=%2B7DAY&router.maxFutureMs=8640000000&create-collection.collection.configName=_indexer_default&create-collection.numShards=2"            

The preceding business logic means: Starting from 30 days ago (assuming today is March 7), create an index every 7 days using your_timestamp_l field. The field value must be within 100 days of the current time, and the system will periodically delete indexes that expired 90 days ago.

The following figures show the result:

Delete an alias

To delete a common alias, run the following command:

curl "http://solrhost:8983/solr/admin/collections?action=DELETEALIAS&name=your_alias_name"           

To delete an alias with automatic table sharding, you also need to actively delete the associated collections.

1. Obtain the list of collections associated with the alias.

   curl "http://solrhost:8983/solr/admin/collections?action=LIST"                    

   Collections with names starting with test_router_alias are all associated with this alias.

2. Delete the alias.

   curl "http://solrhost:8983/solr/admin/collections?action=DELETEALIAS&name=test_router_alias"                    

3. Delete all collections.

   curl "http://solrhost:8983/solr/admin/collections?action=DELETE&name=collection_name"                    

References

CREATEALIAS: CREATEALIAS

LIST: LIST

How to query using LONG type time

import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.impl.ClusterStateProvider;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.util.StrUtils;

import java.time.Instant;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeFormatterBuilder;
import java.time.temporal.ChronoField;
import java.util.AbstractMap;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Optional;

public class SolrDemo {
  private static final DateTimeFormatter DATE_TIME_FORMATTER = new DateTimeFormatterBuilder()
      .append(DateTimeFormatter.ISO_LOCAL_DATE).appendPattern("[_HH[_mm[_ss]]]")
      .parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
      .parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
      .parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
      .toFormatter(Locale.ROOT).withZone(ZoneOffset.UTC);

  private static final String zkHost = "localhost:2181/solr";
  private CloudSolrClient cloudSolrClient;
  private ClusterStateProvider clusterStateProvider;

  public SolrDemo() {
    cloudSolrClient = new CloudSolrClient.Builder(
        Collections.singletonList(zkHost), Optional.empty()).build();
    cloudSolrClient.connect();
    clusterStateProvider = cloudSolrClient.getClusterStateProvider();
  }

  public void close() throws Exception {
    if (null != cloudSolrClient) {
      cloudSolrClient.close();
    }
  }

  private List<String> findCollection(String aliasName, long start, long end) {
    List<String> collections = new ArrayList<>();
    if (start > end) {
      return collections;
    }
    // Find suitable collections based on [start, end]
    if (clusterStateProvider.getState(aliasName) == null) {
      // Get all collection lists corresponding to the current aliasName
      // test_router_alias_2020-03-04, test_router_alias_2020-02-26, test_router_alias_2020-02-19, test_router_alias_2020-02-12, test_router_alias_2020-02-05
      List<String> aliasedCollections = clusterStateProvider.resolveAlias(aliasName);

      // Extract date and time from collection names
      // 2020-03-04T00:00:00Z=test_router_alias_2020-03-04,
      // 2020-02-26T00:00:00Z=test_router_alias_2020-02-26,
      // 2020-02-19T00:00:00Z=test_router_alias_2020-02-19,
      // 2020-02-12T00:00:00Z=test_router_alias_2020-02-12,
      // 2020-02-05T00:00:00Z=test_router_alias_2020-02-05
      List<Map.Entry<Instant, String>> collectionsInstant = new ArrayList<>(aliasedCollections.size());
      for (String collectionName : aliasedCollections) {
        String dateTimePart = collectionName.substring(aliasName.length() + 1);
        Instant instant = DATE_TIME_FORMATTER.parse(dateTimePart, Instant::from);
        collectionsInstant.add(new AbstractMap.SimpleImmutableEntry<>(instant, collectionName));
      }

      // Find appropriate collections based on query time
      Instant startI = Instant.ofEpochMilli(start);
      Instant endI = Instant.ofEpochMilli(end);
      for (Map.Entry<Instant, String> entry : collectionsInstant) {
        Instant colStartTime = entry.getKey();
        if (!endI.isBefore(colStartTime)) {
          collections.add(entry.getValue());
          System.out.println("find collection: " + entry.getValue());
          if (!startI.isBefore(colStartTime)) {
            break;
          }
        }
      }
    } else {
      collections.add(aliasName);
    }
    System.out.println("query " + collections);
    return collections;
  }

  public void run() throws Exception {
    try {
      // [2020-03-07 2020-03-10]
      long start = 1583538686312L;
      long end = 1583797886000L;
      String aliasName = "test_router_alias";
      String collections = StrUtils.join(findCollection(aliasName, start, end), ',');
      QueryResponse res = cloudSolrClient.query(collections, new SolrQuery("*:*"));
      for (SolrDocument sd : res.getResults()) {
        System.out.println(sd.get("id") + " " + sd.get("gmtCreate_l"));
      }
    } finally {
      cloudSolrClient.close();
    }
  }

  public static void main(String[] args) throws Exception {
    SolrDemo solrDemo = new SolrDemo();
    solrDemo.run();
    solrDemo.close();
  }
}