Skip to main content
LLM evaluators use large language models to evaluate the quality of AI-generated responses based on custom criteria. They’re ideal for qualitative evaluations like coherence, relevance, faithfulness, and tone.

​
Set up an LLM provider

LLM evaluators call a model through an AI provider configured in your HoneyHive workspace. Configure at least one provider before creating an LLM evaluator.
1

Open AI Providers settings

Go to Settings > Workspace > AI Providers.If you can view AI provider settings, you can also open this page from the LLM evaluator editor by clicking Configure AI Providers ↗ above the Provider dropdown.
2

Save provider credentials

Find the provider you want to use, click the pencil icon, enter the required credentials, and click Save. The provider’s status changes to Configured.See Supported providers for the required credentials for each provider.
3

Choose a provider and model

In the LLM evaluator editor, select the configured Provider and Model.
See Provider Keys for credentials, workspace scoping, and permissions.

​
Creating an LLM Evaluator

  1. Navigate to the Evaluators tab in the HoneyHive console.
  2. Click Add Evaluator and select LLM Evaluator.
  3. Write the evaluator prompt, configure the return type and filters, and click Create.
LLM evaluator editor showing event filters, AI provider selection (OpenAI/gpt-4o), prompt editor with template syntax, and configuration options

​
Event Schema

LLM evaluators operate on event objects from your traces. Use {{ }} syntax to reference event properties in your prompt.
PropertyDescriptionExample
event_typeType of event: model, tool, chain, or session{{ event_type }}
event_nameName of the event or session{{ event_name }}
inputsInput data (prompt, query, context, etc.){{ inputs.question }}
outputsOutput data (completion, response, etc.){{ outputs.content }}
feedbackUser feedback and ground truth{{ feedback.ground_truth }}
Click Show Schema in the evaluator console to explore all available event properties for your project.
For detailed event schema documentation and tracing setup, see Configuring Tracing for Server-Side Evaluators.

​
Evaluation Prompt

Define your evaluation prompt using the {{ }} syntax to inject event data: 
[Instruction]
Evaluate the AI assistant's answer based on:
1. Relevance to the question
2. Accuracy of information
3. Clarity and coherence
4. Completeness of the answer

Provide a brief explanation and rate the response on a scale of 1 to 5.

[Question]
{{ inputs.question }}

[Context]
{{ inputs.context }}

[AI Assistant's Answer]
{{ outputs.content }}

[Evaluation]
Explanation:
Rating: [[X]]
Use the [[X]] pattern for ratings. The evaluator automatically extracts the value inside the brackets.
Looking for ready-made examples? Check out our LLM Evaluator Templates.

​
Advanced Template Syntax

Beyond basic {{ field }} references, LLM evaluator prompts support Jinja2 conditionals, loops, and filters. This is most useful for multi-turn conversations and agent traces, where you often need to evaluate only user turns, assistant replies, or tool calls.

​
Score a subset of a conversation

A chat trace captures every role (system, user, assistant, tool). To judge user satisfaction, filter the conversation to user turns with selectattr. This surfaces signals like repeated questions, “that didn’t help”, or “thanks, that works” without sending the full transcript. 
[Instruction]
Below are only the user's messages from a support conversation, in order.
Rate the user's overall satisfaction from 1 (frustrated, unresolved) to 5 (clearly satisfied).

[User messages]
{%- for message in inputs.chat_history | selectattr("role", "equalto", "user") %}
{{ loop.index }}. {{ message.content | truncate(300) }}
{%- endfor %}

[Evaluation]
Explanation:
Rating: [[X]]
This example assumes the evaluated event has inputs.chat_history as a conversation array. Some integrations, including OpenAI and OpenAI Agents SDK, capture chat messages on individual model events while the root session event may not include a rolled-up inputs.chat_history by default. Confirm the field exists in Show Schema before using this pattern, and adapt the path if your event uses another array such as inputs.messages or outputs.chat_history. For online evaluation, target events that already contain the conversation array, or use session-level filters only when the session event has that array at evaluation time. For ready-to-use versions of this and related multi-turn judges (frustration, resolution, tool trajectory, repetition loops, and more), see Conversation Evaluator Templates.

​
Adapt to varying event shapes

Conditionals and fallbacks let one prompt handle events that don’t always carry the same fields. This RAG faithfulness prompt includes context only when present and falls back gracefully when a field is missing: 
[Instruction]
Rate how well the answer uses the provided context. Score 1-5.

{%- if inputs.context %}
[Context]
{{ inputs.context | truncate(2000) }}
{%- endif %}

[Question]
{{ inputs.question | default("N/A") }}

[Answer]
{{ outputs.content | default("N/A") }}

[Evaluation]
Explanation:
Rating: [[X]]

​
Common patterns

PatternExample
Filter by role{% for m in inputs.chat_history | selectattr("role", "equalto", "user") %}
Drop noiseinputs.chat_history | rejectattr("role", "equalto", "tool")
Compact transcriptinputs.chat_history | map(attribute="content") | join("\n")
Alternate message pathReplace inputs.chat_history with inputs.messages if that is what Show Schema displays
Optional field{% if inputs.context %}...{% endif %}
Fallback{{ outputs.content | default("N/A") }}
Truncate{{ inputs.context | truncate(2000) }}
First / last turn{{ inputs.chat_history[0].content }}, {{ inputs.chat_history[-1].content }}
Loops and filters require an actual array field. Use Show Schema to confirm where conversation history is available for the event type your evaluator targets.
Field names are case-sensitive. Use the exact casing shown in Show Schema. For array access, prefer bracket notation (chat_history[0].content) in new prompts because it is explicit and matches JSON indexing.

​
Configuration

​
Return Type

  • Boolean: For true/false evaluations
  • Numeric: For scores or ratings (e.g., 1-5)
  • String: For categorical labels or text responses

​
Passing Range

Define the range of scores that indicate a passing evaluation. Useful for CI/CD pipelines and identifying failed test cases.

​
Enabled

Toggle to run this evaluator on all traces that match your event filters.

​
Sampling Percentage

Run your evaluator on a percentage of matching events to manage costs. New evaluators default to 10% sampling. Adjust based on event volume and cost budget - for example, set 25% to evaluate one in four matching events.
Sampling applies to all traces that match your event filters. To evaluate only a subset of events, combine sampling with specific event filters.

​
Event Filters

Use Set Up Filters to specify which events trigger this evaluator. Filters are ANDed together - an event must match all filters to be evaluated.

​
Preset Filters

Every evaluator includes two preset filters by default:
  • Event Type: Filter by model, tool, chain, or session
  • Event Name: Target a specific event name, or use “All” (e.g., “All Models”) to match any event of that type

​
Additional Filters

Click the + button to add filters on any event property. You can filter on any field available in your event schema, including nested properties using dot notation (e.g., inputs.question, metadata.model, outputs.content). Each filter consists of:
  • Field: Any property from the event schema
  • Operator: Depends on the field type (see below)
  • Value: The value to compare against
Operators by field type:
Field TypeOperators
Stringis, is not, contains, not contains, exists, not exists
Numberis, is not, greater than, less than, exists, not exists
Booleanis, exists, not exists
Datetimeis, is not, after, before, exists, not exists
Click Show Schema in the evaluator editor to browse all available event properties you can filter on.

​
Next Steps

Python Evaluators

Create code-based evaluators for programmatic checks

Evaluator Templates

Ready-to-use LLM and Python evaluator templates

Run Experiments

Use evaluators in offline experiments

Human Annotation

Set up human review workflows

Manage as Code

Check evaluators into your repo and apply them with the CLI

Use Portkey for LLM Evaluators

Route evaluator calls through Portkey to access 1,600+ models