Context Verification
Advanced context and faithfulness evaluation
Context Verification ensures that LLM responses are accurate and faithful to the provided context, helping to prevent misinformation and hallucinations.
Available Verification Methods
The Context Verification API provides three methods to assess the quality and accuracy of responses:
-
Context Recall (
context_recall
): Evaluates how well the response recalls information from the provided context. This method checks if the response includes relevant details that should be mentioned based on the context. -
Context Precision (
context_precision
): Assesses the precision of context usage in the response. It ensures that the response does not include irrelevant or extraneous information not supported by the context. -
Faithfulness Verification (
faithfulness
): Verifies the faithfulness of the response to both the input and the context. This method detects any misinformation, hallucinations, or distortions, ensuring the response accurately represents the provided information.
When to Use Context Verification
Use Context Verification to ensure your application provides accurate and relevant responses, especially when dealing with external knowledge sources. Specifically:
-
Context Recall: Use this method when you need to verify that the response includes necessary and relevant information from the provided context. Ideal for applications where omitting key details can lead to misunderstandings.
-
Context Precision: Use this method when it’s important that the response only contains information supported by the context. This is crucial for maintaining accuracy and avoiding the inclusion of irrelevant or incorrect details.
-
Faithfulness Verification: Use this method to confirm that the response is faithful to the input and context, ensuring there are no fabrications or inaccuracies. Essential for applications where trust and accuracy are paramount, such as legal, medical, or educational content.
Examples
Example 1: Context Recall
Verify that the response recalls necessary information from the context.
Example 2: Context Precision
Ensure the response includes only information relevant to the provided context.
Example 3: Faithfulness Verification
Verify the faithfulness of the response to the input and context.
Understanding the Score
The Score is a continuous value between 0 and 1 indicating the evaluation result for the specific verification method:
-
Context Recall:
- 0: The response does not recall relevant information from the context.
- Values between 0 and 1: The response partially recalls information from the context.
- 1: The response fully recalls necessary information from the context.
-
Context Precision:
- 0: The response includes information not supported by the context.
- Values between 0 and 1: The response includes some irrelevant or unsupported information.
- 1: The response includes only information supported by the context.
-
Faithfulness Verification:
- 0: The response contains fabrications or inaccuracies compared to the input and context.
- Values between 0 and 1: The response has some inaccuracies or is partially unfaithful.
- 1: The response is completely faithful to the input and context, with accurate and reliable information.
Higher scores indicate better alignment with the desired criteria, helping you assess and improve your application’s responses.