Context Precision measures the proportion of relevant chunks in the retrieved contexts. It is computed as the average of precision@k across all chunks in the context.
Definitions
Precision@k: The ratio of relevant chunks (true positives) at rank k to the total number of chunks
Context Precision is calculated using the formula:
Where:
K is the total number of chunks in the retrieved contexts.
( v_k \in {0, 1} ) is the relevance indicator at rank k (1 if relevant, 0 if not).
Summary
Context Precision provides insight into the effectiveness of the retrieved contexts by quantifying how many relevant chunks are identified among the top results.
Example Code: Context Precision Evaluation
This example demonstrates how to compute the Context Precision metric using the ContextPrecisionEvaluator with the OpenAI language model.
import loggingimport sysfrom dotenv import find_dotenv, load_dotenvfrom dynamiq.evaluations.metrics import ContextPrecisionEvaluatorfrom dynamiq.nodes.llms import OpenAI# Load environment variables for the OpenAI APIload_dotenv(find_dotenv())# Configure logging levellogging.basicConfig(stream=sys.stdout, level=logging.INFO)# Initialize the OpenAI language modelllm =OpenAI(model="gpt-4o-mini")# Sample questions, answers, and context listsquestions = ["What can you tell me about Albert Einstein?","Who won the 2020 ICC World Cup?","What is the tallest mountain in the world?",]answers = [ ("Albert Einstein, born on 14 March 1879, was a German-born theoretical physicist, ""widely held to be one of the greatest and most influential scientists of all time. ""He received the 1921 Nobel Prize in Physics for his services to theoretical physics." ),"England","Mount Everest.",]contexts_list = [ [# Contexts for the first question"Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist, ""widely held to be one of the greatest and most influential scientists of all time. ""Best known for developing the theory of relativity, he also made important contributions to quantum mechanics.","Einstein's mass-energy equivalence formula E = mc^2 has been called 'the world's most famous equation'." ], [# Contexts for the second question"The 2022 ICC Men's T20 World Cup, held from October 16 to November 13, 2022, in Australia, ""was the eighth edition of the tournament. England emerged victorious, defeating Pakistan by five wickets.","The 2016 ICC World Twenty20 was held in India, where the West Indies won." ], [# Contexts for the third question"The Andes is the longest mountain range in the world. It features many high peaks.","Mount Kilimanjaro is the highest mountain in Africa, standing at 5,895 meters." ],]# Initialize the Context Precision Evaluatorevaluator =ContextPrecisionEvaluator(llm=llm)correctness_scores = evaluator.run( questions=questions, answers=answers, contexts_list=contexts_list, verbose=False, # Set to True for detailed logging)# Print the evaluation resultsfor idx, score inenumerate(correctness_scores):print(f"Question: {questions[idx]}")print(f"Context Precision Score: {score}")print("-"*50)print("All Context Precision Scores:")print(correctness_scores)