Composo x Metabase

​

Introduction

​

What is Metabase?

​

Your Data in Composo

​

Your Evaluation Database

• Request ID: Unique identifier for each evaluation request (UUID)
• Agent Instance ID: Identifier for the specific agent instance being evaluated (null for response/tool evaluations)
• Eval Type: Type of evaluation - response (LLM responses), tool (tool usage), agent (multi-agent traces), or chatsession (chat-based agent evaluations)
• Score Type: How the score should be interpreted - reward (continuous 0-1 score) or binary (pass/fail converted to 1.0/0.0)
• Name: Agent name for multi-agent evaluations (null for response/tool evaluations)
• Criteria: Full evaluation criteria text (starts with prefixes like “Reward responses”, “Agent passes if”, etc.)
• Score: Numerical result (0-1 scale, where higher is better; null if criteria not applicable)
• Explanation: Detailed reasoning and analysis behind the score
• Subject: JSON data containing what was evaluated:
  • For response/tool evaluations: {messages, tools, system} - the conversation and available tools
  • For agent evaluations: The specific agent instance interactions being evaluated
• Email: User who ran the evaluation
• Model Class: The evaluation model used (e.g., “align-lightning”)
• Created At: Timestamp when the evaluation was performed

​

Viewing Individual Evaluations

​

Collections

• Your personal collection: Private workspace for your analyses
• Team collections: Shared dashboards and queries (e.g., “Acme Corp Collection”)

​

Creating Your First Query

​

Basic Query: Finding Red Flags

1. Click + New → Question
2. Select your Evaluations table
3. Click Filter → Score → Less than → enter 0.5
4. Click Filter again → Created At → select your time range
5. Click Visualize

​

Visualizing Your Data

​

Choosing a Visualization

• Line charts: Track score trends over time
• Bar charts: Compare different agents or evaluation types
• Tables: See detailed row-by-row data
• Numbers: Display single metrics like average score or red flag rate

​

Summarizing Data

​

Aggregations and Grouping

1. Click Summarize
2. Choose a metric: Count of rows, Average of Score, etc.
3. Add Group by: Created At (for time series) or Name (to compare evaluations)

• Average Score by Created At → See quality trends over time
• Count by Name → Which evaluations run most frequently
• Average Score by Agent Instance ID → Compare agent performance

​

Custom Expressions: Red Flag Rate

1. Click Summarize → Custom Expression
2. Enter:

CountIf([Score] < 0.5) / 
  (CountIf([Score] > 0.5) + CountIf([Score] < 0.5))

3. Name it “red_flag_rate”
4. Group by Created At: Minute (or Hour/Day)

​

Building Dashboards

​

Creating a Dashboard

1. After creating a query, click Save and give it a descriptive name
2. Click + New → Dashboard
3. Name your dashboard (e.g., “Production Quality Monitor”)
4. Click Add a saved question and select your queries
5. Resize and arrange charts as needed

​

Dashboard Features

• Tabs: Organize related metrics (e.g., “Quality By Agent” vs “Red Flags”)
• Dashboard filters: Add filters that apply to multiple charts simultaneously
• Auto-refresh: Set dashboards to update automatically every few minutes
• Sharing: Click the sharing icon to share with teammates or generate public links

​

Advanced Filtering

• Score ranges: Score is between 0.3 and 0.7
• Text search: Criteria contains “hallucination”
• Multiple time ranges: Created At is Previous 7 days AND Created At Hour of day is between 9 and 17
• Specific agents: Agent Instance ID is one of [list of IDs]

​

SQL Queries (Advanced)

1. Click + New → Question → Native query
2. Write your SQL against the evaluations table
3. Use variables with {{variable_name}} to make queries reusable

SELECT 
  date_trunc('hour', created_at) as hour,
  name,
  avg(score) as avg_score,
  count(*) as eval_count
FROM "external"."evaluations"
WHERE created_at > current_date - interval '7 days'
  AND score < 0.5
GROUP BY 1, 2
ORDER BY 1 DESC

​

Getting Help

​

Metabase Resources

• Documentation
• Learning Center
• Video Tutorials

​

Composo Support

• Data questions: Contact your Composo account team
• Technical support: [email protected]
• Evaluation schema: See reference below