Faiss-Server uses gRPC to implement vector similarity search based on open source Faiss.

Supported formats of vector files

File name extensions are used to identify file formats. The following file formats are supported:
  • .fvecs
  • .svm (Libsvm file)
    Example of a Libsvm file:
    21187279 1:0.2663046344870018 2:0.36652042181588923 3:1.0686633708024278 4:0.48038935355720136 5:0.25852895390543884 6:0.3548201486996048 7:0.5256999599627583 8:0.6950687558969181 9:0.678305894615746 10:0.22776047265501018
    21264640 1:0.3939700179792862 2:0.2754414668803289 3:0.9439534329739017 4:0.40100161008614665 5:0.4995636031244379 6:0.013824724791385053 7:0.5587276328436032 8:0.5785080421720411 9:0.2784678162956546 10:0.5387681136371216

    The content in the file must be in the format of tagid 1:xx 2:xx 3:xx. tagid can be any character string. In most cases, it is an item ID (itemid). The digit on the left of each colon (:) is a vector dimension number. In most cases, it starts from 1. xx indicates the value of a dimension.

  • .index (index file generated by Faiss)

    If you directly load a Libsvm file that contains a large volume of data, it takes a long time to build an index. You can first generate an index file based on the Libsvm file in offline mode. Then, use Faiss-Server to load the index file. This way, you can build an index in a short period of time.

Note When you name a file, make sure that the file format and file name extension match. In some scenarios, when an OSS file is generated, OSS adds random characters to the file name extension. In this case, change the file name extension to .index before you load the file.

Load a local vector file

Sample file: article_word2vec.svm

Sample code for loading the file:
docker run -it -p  9000:9000  -v /home/bruceding.jing:/index registry.cn-beijing.aliyuncs.com/pai-recommend/faiss-server:0.0.2 --index_source=/index/article_word2vec.svm --index_params=Flat
//The following content is the Docker output:
2020-05-01 15:18:01,134 INFO src/faiss_parameter.cc:39 index source:/index/article_word2vec.svm,index params:Flat
2020-05-01 15:18:01,134 INFO src/faiss_index.cc:26 init index
2020-05-01 15:18:01,377 INFO src/faiss_index.cc:119 size:10907,dimension:10
2020-05-01 15:18:01,377 INFO src/faiss_index.cc:120 index_params=Flat
2020-05-01 15:18:01,377 INFO src/main.cc:55 create index success
2020-05-01 15:18:01,377 INFO src/faiss_server.cc:23 Server listening on 0.0.0.0:9000

The output shows that the gRPC server is started and it listens on port 9000.

The following table describes the parameters in the sample code.
Parameter Description
-p The -p parameter of Docker is used for port mapping.

In this example, port 9000 is used. It is the default listening port of Faiss-Server.

registry.cn-beijing.aliyuncs.com/pai-recommend/faiss-server:0.0.2 The Docker image address provided by Faiss-Server.
--index_source The path of the vector file.
--index_params The mode in which a Faiss index is built. In this example, the index is built in Flat mode.

Faiss-Server uses the index_factory function of Faiss to build indexes. For more information about the --index_params parameter, see Faiss indexes.

Load a vector file in OSS

In most cases, a vector file is generated in OSS. Faiss-Server allows you to build an index based on a vector file in OSS.
docker run  -it -p 9000:9000  registry.cn-beijing.aliyuncs.com/pai-recommend/faiss-server:0.0.2  --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxx --accesskey=xxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server --object_name=demo_data/article_word2vec.svm
The following table describes the parameters in the sample code.
Parameter Description
--index_params IVF256,Flat indicates that a clustered index is built, and the indexed data is divided into 256 pieces.
--search_params In this example, Faiss-Server searches for the first 128 clusters whose centroids are closest to the vector. For more information about this parameter, see Index IO, cloning and hyper parameter tuning.
--bucket_name The name of the OSS bucket.
--object_name The path of the vector file in OSS.
--endpoint The endpoint of OSS. In this example, a public endpoint is specified. If a VPC is used, you must specify the internal endpoint of OSS to minimize traffic fees and accelerate the data read speed.
Note To read data from OSS, you must specify the AccessKey ID and AccessKey secret of your Alibaba Cloud account.
You can run the following command to view the parameters supported by Faiss-Server:
docker run registry.cn-beijing.aliyuncs.com/pai-recommend/faiss-server:0.0.2 --help

Load the version file in OSS

Faiss-Server is an online service. It may take a long time to directly load a vector file to Faiss-Server. To save time, you can generate an index file based on the vector file and load the index file to Faiss-Server.

In some scenarios, vector files need to be loaded on a regular basis.

A vector file may have multiple versions, and its latest version is automatically recorded in a version file. Faiss-Server asynchronously checks the content in the version file. When Faiss-Server finds that the version file is updated, it automatically loads the latest index and idxmap files.

Flowchart
Procedure:
  • 1. Use a vector recall algorithm to generate a vector file for an item and store the file in an OSS bucket.
  • 2 and 3. Faiss-Server reads the vector file from OSS and builds an index.
    The version file contains the locations of the index and idxmap files.
    • The index file is the generated Faiss index file.
    • The idxmap file is a mapping table that maps tag IDs (tagid) to specific vector locations in an index.
  • 4. Faiss-Server is always in the started state and listens to the version file.
  1. Generate an index file.
    Run the following command to build an index. After the index is built, the process exits. The gRPC service is not involved in this operation. You do not need to set the -p parameter of Docker in the command.
    docker run  -it registry.cn-beijing.aliyuncs.com/pai-recommend/faiss-server:0.0.2  --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxx --accesskey=xxxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server --object_name=demo_data/article_word2vec.svm --version_name=demo_data/versions --action=build_index
    Parameters in the command:
    • --version_name: specifies the location of the version file in OSS. If the file does not exist, the system automatically generates one.
    • --action: Set this parameter to build_index.
    After the code is successfully executed, three files are generated.Index
    If the Libsvm vector file is stored on your computer, run the following command to place the generated files into OSS:
    docker run  -it -v /home/bruceding.jing:/index  registry.cn-beijing.aliyuncs.com/pai-recommend/faiss-server:0.0.2  --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxx --accesskey=xxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server --index_source=/index/article_word2vec.svm --version_name=demo_data/versions --action=build_index
    Note The -v parameter in the command is used to transfer the Libsvm file to Docker.
  2. Start Faiss-Server.
    Open another terminal window and run the following command to start Faiss-Server:
    docker run  -it  -p 9000:9000  registry.cn-beijing.aliyuncs.com/pai-recommend/faiss-server:0.0.2  --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxxx --accesskey=xxxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server  --version_name=demo_data/versions  --check_version=true
    Parameters in the command:
    • --index_params and --search_params: Set these parameters to the same values as those specified when you build the index.
    • --version_name: specifies the location of the version file in OSS. If the file does not exist, the system automatically generates one.
    • --check_version: If you set this parameter to true, the version check feature is enabled. Only after you enable the version check feature, Faiss-Server automatically loads the latest index when the version file is updated.
    After Faiss-Server is started, go back to the previous terminal window and build a new index. If information similar to the following output is displayed, the new index is loaded.
    2020-05-01 17:09:37,478 INFO src/faiss_index.cc:197 start to load
    2020-05-01 17:09:37,567 INFO src/oss_util.cc:30 GetObjectToFile success62
    2020-05-01 17:09:37,568 INFO src/faiss_index.cc:219 localPath=/tmp/20200501170924.index
    2020-05-01 17:09:37,703 INFO src/oss_util.cc:30 GetObjectToFile success535963
    2020-05-01 17:09:37,815 INFO src/oss_util.cc:30 GetObjectToFile success98159
    2020-05-01 17:09:37,817 INFO src/faiss_index.cc:245 load index success
  3. Append a vector file.
    After an entire index file is loaded, you can run the following command to append another Libsvm file to Faiss-Server:
    docker run  -it  -p 9000:9000  registry.cn-beijing.aliyuncs.com/pai-recommend/faiss-server:0.0.2  --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxx --accesskey=xxxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server  --version_name=demo_data/versions  --check_version=true --append_dir=demo_data/append

    --append_dir: specifies the directory where the Libsvm file you want to append is stored.

    If you have placed the Libsvm file in this directory, information similar to the following output is displayed:
    2020-05-01 17:24:53,161 INFO src/oss_util.cc:30 GetObjectToFile success2428471
    2020-05-01 17:24:53,357 INFO src/faiss_index.cc:383 append index, file:article_word2vec.svm, size=10907, idx size=10907

Test Faiss-Server

After you start Faiss-Server, use the following tool to test its vector similarity search function:

faiss-client-go.tar.gz

./faiss-client-go --host "11.158.166.161:9000" --vector "1:0.8254435895816826 2:0.3546776922906237 3:0.14982954432756015 4:0.4796270382425792 5:0.5478646494350313 6:0.3771617042371068 7:0.5107318036421319 8:0.7005006312566686 9:0.7030542773195687 10:0.692132791047145"  --k 20
The following table describes the parameters in the preceding example.
Parameter Description
--host Specifies the IP address and port number of Faiss-Server.
--vector Specifies the requested vector data in the same format as the Libsvm file. The dimensions must also be consistent with those in the Libsvm file. In this example, 10-dimensional data is used.
--k Specifies the number of the search results to be returned.

The default value is 10. In this example, this parameter is set to 20.

The following information is returned:Output
Note tagid list: the tag IDs in the Libsvm file. In most cases, tag IDs are item IDs (itemid).

Deploy Faiss-Server in EMR

  1. Create an EMR Data Science cluster. For more information, see Overview.
    The Faiss-Server service is supported by default.data_science
    Note The registry.cn-beijing.aliyuncs.com/pai-recommend/faiss-server:0.0.2 image is integrated into EMR. The image name is faiss-server:1.0.0.
  2. Create a Shell job to load a vector file and start the gRPC service.
    sudo docker run -d --restart unless-stopped -p 9001:9000  faiss-server:1.0.0  --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxxx --accesskey=xxx --endpoint=oss-cn-huhehaote-internal.aliyuncs.com --bucket_name=faiss-test  --object_name=article_word2vec.svm