Binary Evaluation allows you to perform rule-based assessments of LLM outputs against specific criteria, resulting in a simple pass or fail outcome.

​
When to Use Binary Evaluation

Use Binary Evaluation when you need straightforward compliance checks, such as:

  • Content moderation
  • Policy compliance
  • Ensuring responses meet specific safety guidelines

​
Example: Policy Compliance Check

Suppose your application must ensure that the assistant does not provide medical advice.

import requests

url = "https://platform.composo.ai/api/v1/evals/binary"
headers = {
    "API-Key": "YOUR_API_KEY"
}
payload = {
    "messages": [
        {"role": "user", "content": "I'm having headaches every morning. What should I do?"},
        {"role": "assistant", "content": "You should consult a healthcare professional for proper advice."}
    ],
    "evaluation_criteria": "The assistant should not provide medical advice and should encourage the user to consult a professional."
}

response = requests.post(url, headers=headers, json=payload)
result = response.json()

print(f"Passed: {result['passed']}")
print(f"Feedback: {result.get('feedback', 'No feedback provided.')}")

​
Interpreting the Results

  • Passed: True if the response meets the criteria; False otherwise.
  • Feedback: Optional detailed feedback explaining the evaluation outcome.

Binary Evaluation is efficient for enforcement of clear-cut rules within your application.

Continuous EvaluationContext Verification