Context Precision

Context Precision Metric

Context Precision measures the proportion of relevant chunks in the retrieved contexts. It is computed as the average of precision@k across all chunks in the context.

Definitions

• Precision@k: The ratio of relevant chunks (true positives) at rank k to the total number of chunks

$$\text{Precision@k} = {\text{true positives@k} \over (\text{true positives@k} + \text{false positives@k})}$$

• Context Precision is calculated using the formula:

  $$\text{Context Precision@K} = \frac{\sum_{k=1}^{K} \left( \text{Precision@k} \times v_k \right)}{\text{Total number of relevant items in the top } K \text{ results}}$$

Where:

• K is the total number of chunks in the retrieved contexts.

• ( v_k \in {0, 1} ) is the relevance indicator at rank k (1 if relevant, 0 if not).

Summary

Context Precision provides insight into the effectiveness of the retrieved contexts by quantifying how many relevant chunks are identified among the top results.

Example Code: Context Precision Evaluation

This example demonstrates how to compute the Context Precision metric using the ContextPrecisionEvaluator with the OpenAI language model.

import logging
import sys
from dotenv import find_dotenv, load_dotenv
from dynamiq.evaluations.metrics import ContextPrecisionEvaluator
from dynamiq.nodes.llms import OpenAI

# Load environment variables for the OpenAI API
load_dotenv(find_dotenv())

# Configure logging level
logging.basicConfig(stream=sys.stdout, level=logging.INFO)

# Initialize the OpenAI language model
llm = OpenAI(model="gpt-4o-mini")

# Sample questions, answers, and context lists
questions = [
    "What can you tell me about Albert Einstein?",
    "Who won the 2020 ICC World Cup?",
    "What is the tallest mountain in the world?",
]

answers = [
    (
        "Albert Einstein, born on 14 March 1879, was a German-born theoretical physicist, "
        "widely held to be one of the greatest and most influential scientists of all time. "
        "He received the 1921 Nobel Prize in Physics for his services to theoretical physics."
    ),
    "England",
    "Mount Everest.",
]

contexts_list = [
    [
        # Contexts for the first question
        "Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist, "
        "widely held to be one of the greatest and most influential scientists of all time. "
        "Best known for developing the theory of relativity, he also made important contributions to quantum mechanics.",
        "Einstein's mass-energy equivalence formula E = mc^2 has been called 'the world's most famous equation'."
    ],
    [
        # Contexts for the second question
        "The 2022 ICC Men's T20 World Cup, held from October 16 to November 13, 2022, in Australia, "
        "was the eighth edition of the tournament. England emerged victorious, defeating Pakistan by five wickets.",
        "The 2016 ICC World Twenty20 was held in India, where the West Indies won."
    ],
    [
        # Contexts for the third question
        "The Andes is the longest mountain range in the world. It features many high peaks.",
        "Mount Kilimanjaro is the highest mountain in Africa, standing at 5,895 meters."
    ],
]

# Initialize the Context Precision Evaluator
evaluator = ContextPrecisionEvaluator(llm=llm)
correctness_scores = evaluator.run(
    questions=questions,
    answers=answers,
    contexts_list=contexts_list,
    verbose=False,  # Set to True for detailed logging
)

# Print the evaluation results
for idx, score in enumerate(correctness_scores):
    print(f"Question: {questions[idx]}")
    print(f"Context Precision Score: {score}")
    print("-" * 50)

print("All Context Precision Scores:")
print(correctness_scores)