Hologres long memory service achieves 96.82% overall accuracy on LoCoMo (Long Conversation Memory), a public benchmark that measures memory retrieval across long, multi-turn conversations. This topic describes the dataset, the evaluation method, and the per-category results.
Background
LoCoMo is a long-conversation memory benchmark introduced by Snap Research at ACL 2024. It measures how well an AI system stores and retrieves information across long, multi-turn conversations. Each conversation contains about 300 turns and 9,000 tokens, spread over up to 35 separate sessions. The dataset therefore simulates weeks of ongoing interaction between a user and an AI assistant.
Hologres long memory service achieves an overall accuracy of 96.82% on LoCoMo. The test code and raw results are open source. For more information, see hologres-openmemory-locomo-benchmark.
Dataset
The following table lists the key parameters of the LoCoMo dataset.
Key parameters of the LoCoMo dataset
|
Parameter |
Description |
|
Conversations |
10 long conversations |
|
Turns per conversation |
About 300 |
|
Tokens per conversation |
About 9,000 |
|
Sessions per conversation |
Up to 35 |
|
Evaluation questions per run |
1,540 |
|
Test runs |
5 |
|
Total questions evaluated |
7,700 (5 × 1,540) |
The evaluation questions cover the following four categories of retrieval and reasoning capabilities.
Question categories and examples
|
Category |
Description |
Example |
|
Single-hop |
Requires retrieving a single fact that is stated directly in the conversation. |
"Where did Caroline move from four years ago?" |
|
Temporal reasoning |
Requires reasoning about dates, times, and the order of events. |
"When did Caroline attend the LGBTQ support group?" |
|
Multi-hop reasoning |
Requires linking evidence across several parts of the conversation. |
"What fields has Caroline studied?" |
|
Open-domain |
Requires combining conversation context with general world knowledge. |
"What education-related paths might Caroline pursue?" |
Evaluation method
Procedure
-
Ingest the data: Write the 10 LoCoMo conversations to the long memory service session by session. The service automatically stores and indexes the extracted memories.
-
Run the queries: For each conversation, send its evaluation questions to the service. The service answers each question based on the memories that it has stored.
-
Score the answers: GPT-4.1 serves as the judge model (LLM-as-a-judge). It compares each answer with the ground truth and marks the answer as correct or incorrect. Semantically equivalent wording counts as correct. For example, a date in a different format that refers to the same day counts as correct.
-
Aggregate the results: Calculate the accuracy for each category and the overall accuracy.
Configuration
-
Judge model: GPT-4.1 (temperature=0.1)
-
Scoring: binary (correct=1, incorrect=0)
Results
The following table lists the accuracy of the Hologres long memory service by category.
LoCoMo benchmark results
|
Scenario |
Single-hop |
Temporal reasoning |
Multi-hop reasoning |
Open-domain |
Overall accuracy |
|
Score |
97.38% |
98.44% |
96.81% |
86.46% |
96.82% |
The preceding results were measured on the public LoCoMo dataset by using the configuration described in this topic, and are provided for reference only. Actual retrieval accuracy varies depending on your conversation content, model versions, and parameter settings. We recommend that you benchmark the service against your own workloads before you rely on these figures.