Technical documentation for creating custom LLM evaluators in HoneyHive
Add Evaluator
and select LLM Evaluator
.gpt-4o
model. event
, which represents a span in a trace. A root event in a trace is of the type session
, while all non-root events in a trace can be of 3 core types - model
, tool
and chain
.
session
event, which being a root event does not have any parents.session
: A root event used to group together multiple model
, tool
, and chain
events into a single trace. This is achieved by having a common session_id
across all children.model
events: Used to track the execution of any LLM requests.tool
events: Used to track execution of any deterministic functions like requests to vector DBs, requests to an external API, regex parsing, document reranking, and more.chain
events: Used to group together multiple model
and tool
events into composable units that can be evaluated and monitored independently. Typical examples of chains include retrieval pipelines, post-processing pipelines, and more.Event Properties
event_type
: The type of event. Can be model
, tool
, chain
, or session
.event_name
: The name of the event or session. This can be used to identify a specific event or session you want to evaluate.inputs
: The inputs to the event or session. This can be the prompt, chat history, dynamic insertion variables in your prompt template, query for your retrieval pipeline, etc.output
: The output of the event or session. This can be a completion, a vector DB response, an API response, etc.feedback
: A JSON object presenting user feedback for the event. This object also contains Ground Truth
.Show Schema
in the evaluator console.{{}}
to reference event properties in your prompt.Boolean
: For true/false evaluationsNumeric
: For numeric scores or ratingsString
: For categorical evals or other objectssource != evaluation
when initializing the tracer.
source
is not evaluation
or playground
, i.e. typically only production or staging environments. You can not sample events when running offline evaluations.Create
in the top right corner.