Context Recall

Context Recall Metric

Context Recall measures the number of relevant documents or pieces of information that were successfully retrieved. It emphasizes the importance of capturing all important results. A higher recall score indicates that fewer relevant documents were overlooked.

$$\text{context recall} = {|\text{GT claims that can be attributed to context}| \over |\text{Number of claims in GT}|}$$

Key Points

• Focus: Context Recall is primarily concerned with ensuring that no significant information is missed.

• Interpretation: Higher recall means that more relevant documents have been retrieved, while lower recall indicates that important documents were left out.

• Comparison Reference: Calculating context recall requires a reference set of relevant documents to compare against, ensuring an accurate assessment of what has been retrieved.

In summary, context recall is crucial for applications where catching all relevant information is essential.

Example Code: Context Recall Evaluation

This example demonstrates how to compute the Context Recall metric using the ContextRecallEvaluator with the OpenAI language model.

import logging
import sys
from dotenv import find_dotenv, load_dotenv
from dynamiq.evaluations.metrics import ContextRecallEvaluator
from dynamiq.nodes.llms import OpenAI

# Load environment variables for the OpenAI API
load_dotenv(find_dotenv())

# Configure logging level
logging.basicConfig(stream=sys.stdout, level=logging.INFO)

# Initialize the OpenAI language model
llm = OpenAI(model="gpt-4o-mini")

# Sample data
questions = ["What can you tell me about Albert Einstein?"]
contexts = [
    (
        "Albert Einstein (14 March 1879 - 18 April 1955) was a German-born "
        "theoretical physicist, widely held to be one of the greatest and most "
        "influential scientists of all time. Best known for developing the "
        "theory of relativity, he also made important contributions to quantum "
        "mechanics. His mass-energy equivalence formula E = mc^2 has been called "
        "'the world's most famous equation'. He received the 1921 Nobel Prize in "
        "Physics for his services to theoretical physics."
    )
]
answers = [
    (
        "Albert Einstein, born on 14 March 1879, was a German-born theoretical "
        "physicist, widely held to be one of the greatest and most influential "
        "scientists of all time. He received the 1921 Nobel Prize in Physics for "
        "his services to theoretical physics."
    )
]

# Initialize evaluator and evaluate
evaluator = ContextRecallEvaluator(llm=llm)
recall_scores = evaluator.run(
    questions=questions, 
    contexts=contexts, 
    answers=answers, 
    verbose=True  # Set to False to disable verbose logging
)

# Print the results
for idx, score in enumerate(recall_scores):
    print(f"Question: {questions[idx]}")
    print(f"Context Recall Score: {score}")
    print("-" * 50)

print("Context Recall Scores:")
print(recall_scores)