HoneyHive Docs

Client-side evaluators run within your application environment, providing immediate feedback and integration with your existing infrastructure.

Evaluators can be utilized either:

online: real-time metrics for live applications
offline: controlled experimental environments

For online evaluation, HoneyHive enables you to log evaluation results directly alongside your traces at various stages of your pipeline. For offline evaluation, evaluators are most effective when used with HoneyHive’s evaluation harness, which is designed to run and manage experiments seamlessly.

Online Evaluation

Once tracing is set up for your application, performing client-side online evaluations becomes straightforward. It simply involves enriching your traces and spans with additional context using the metrics field. This field allows you to pass any custom metric, using any of the primary data types. Metrics can be logged for any type of event and at every step of your pipeline.

For example, consider a Retrieval-Augmented Generation (RAG) scenario where you have evaluator functions implemented to assess: a) retrieval quality, b) model response generation, and c) overall pipeline performance.

These metrics can be seamlessly passed alongside your traces within each relevant span, as shown below:

from honeyhive import HoneyHiveTracer, trace, enrich_span, enrich_session


HoneyHiveTracer.init(
api_key="my-api-key",
project="my-project",
)

@trace
def get_relevant_docs(query):
    medical_docs = [
        "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
        "Studies show morning exercises have better impact on blood sugar levels."
    ]
    enrich_span(metrics={"retrieval_relevance": 0.5})
    return medical_docs

@trace
def generate_response(docs, query):
    prompt = f"Question: {query}\nContext: {docs}\nAnswer:"
    response = "This is a test response."
    enrich_span(metrics={"contains_citations": True})
    return response

@trace
def rag_pipeline(query):
    docs = get_relevant_docs(query)
    response = generate_response(docs, query)
    
    
    # Add session-level metrics
    enrich_session(metrics={
        "rag_pipeline": {
            "num_retrieved_docs": len(docs),
            "query_length": len(query.split())   
        }
    })
    
    return docs, response

def main():
    query = "How does exercise affect diabetes?"
    retrieved_docs, generated_response = rag_pipeline(query)

if __name__ == "__main__":
    main()

In this example, enrich_span is being used to add metrics on particular steps: get_relevant_docs and generate_response, while enrich_session is used to set metrics that apply to the entire session or pipeline run.

You can learn more about logging external evaluation results here.

Offline Experiments

You can also use client-side evaluators as part of your experiments sessions. In an experiments settings, your evaluator will have access to outputs (as generated by the evaluated function), inputs and ground truth (as defined in your dataset).

You should define your evaluators with the appropriate parameter signature - they can accept one parameter (outputs), two parameters (outputs, inputs), or three parameters (outputs, inputs, ground_truths) depending on what data your evaluation logic requires.

def sample_evaluator(outputs, inputs, ground_truths):
    # Code here
    return random.randint(1, 5)

By default, evaluation results are stored at the session level. The return values of evaluator functions should represent meaningful evaluation metrics, such as numerical scores, booleans, or other significant measurements.

You can use your evaluators to evaluate a target function in a controlled setting with curated datasets, like this:

    from honeyhive import evaluate

    import random

    def sample_evaluator(outputs, inputs, ground_truths):
        # Code here
        return random.randint(1, 5)


    # Create function to be evaluated
    # inputs -> parameter to which datapoint or json value will be passed
    # (optional) ground_truths -> ground truth value for the input
    def function_to_evaluate(inputs, ground_truths):
        complete_prompt = f"You are an expert analyst specializing in {inputs['product_type']} market trends. Could you provide an analysis of the current market performance and consumer reception of {inputs['product_type']} in {inputs['region']}?"
        response = "This is a test response."
        return response

    if __name__ == "__main__":
        # Run experiment
        evaluate(
            function = function_to_evaluate,               # Function to be evaluated
            hh_api_key = '<HONEYHIVE_API_KEY>',
            hh_project = '<HONEYHIVE_PROJECT>',
            name = 'Sample Experiment',
            dataset_id = '<DATASET_ID>',                      # this example assumes the existence of a managed dataset in HoneyHive
            evaluators=[sample_evaluator]                 # to compute client-side metrics on each run
        )

This will run the experiment with the datapoints contained in your dataset and run the evaluation on the target function’s output for each of the datapoint.

For a complete explanation of running experiments, refer to the Experiments Quickstart Example.

Multi-step Evaluation in Experiment Runs

If your experiment involves complex, multi-step pipelines, you can log metrics either at the trace level or on a per-span level to gain more detailed insights.

In this example, we define two evaluators: consistency_evaluator for the main rag_pipeline function, and retrieval_relevance_evaluator for the document retrieval step. The first is passed directly to evaluate(), while the second is enriched within the retrieval step itself.

from honeyhive import evaluate, evaluator
from honeyhive import trace, enrich_span

def retrieval_relevance_evaluator(query, docs):
    # code here
    avg_relevance = 0.5
    return avg_relevance

@evaluator()
def consistency_evaluator(outputs, inputs, ground_truths):
    # code here
    consistency_score = 0.66
    return consistency_score


@trace
def get_relevant_docs(query):
    retrieved_docs = [
        "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
        "Studies show morning exercises have better impact on blood sugar levels."
    ]
    retrieval_relevance = retrieval_relevance_evaluator(query, retrieved_docs)
    enrich_span(metrics={"retrieval_relevance": retrieval_relevance})
    return retrieved_docs

def generate_response(docs, query):
    prompt = f"Question: {query}\nContext: {docs}\nAnswer:"
    response = "This is a test response"
    return response

def rag_pipeline(inputs, ground_truths):
    query = inputs["query"]
    docs = get_relevant_docs(query)
    response = generate_response(docs, query)
        
    return response

dataset = [
    {
        "inputs": {
            "query": "How does exercise affect diabetes?",
        },
        "ground_truths": {
            "response": "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
        }
    },
]


if __name__ == "__main__":
    # Run experiment
    evaluate(
        function = rag_pipeline,               # Function to be evaluated
        hh_api_key = '<your-api-key>',
        hh_project = '<your-project-name>',
        name = 'Multi Step Evals',
        dataset = dataset,
        evaluators=[consistency_evaluator],                 # to compute client-side metrics on each run
    )

After running this script, you should be able to see both metrics displayed in your Experiments dashboard.

Online Evaluation

These metrics can be seamlessly passed alongside your traces within each relevant span, as shown below:

from honeyhive import HoneyHiveTracer, trace, enrich_span, enrich_session


HoneyHiveTracer.init(
api_key="my-api-key",
project="my-project",
)

@trace
def get_relevant_docs(query):
    medical_docs = [
        "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
        "Studies show morning exercises have better impact on blood sugar levels."
    ]
    enrich_span(metrics={"retrieval_relevance": 0.5})
    return medical_docs

@trace
def generate_response(docs, query):
    prompt = f"Question: {query}\nContext: {docs}\nAnswer:"
    response = "This is a test response."
    enrich_span(metrics={"contains_citations": True})
    return response

@trace
def rag_pipeline(query):
    docs = get_relevant_docs(query)
    response = generate_response(docs, query)
    
    
    # Add session-level metrics
    enrich_session(metrics={
        "rag_pipeline": {
            "num_retrieved_docs": len(docs),
            "query_length": len(query.split())   
        }
    })
    
    return docs, response

def main():
    query = "How does exercise affect diabetes?"
    retrieved_docs, generated_response = rag_pipeline(query)

if __name__ == "__main__":
    main()

You can learn more about logging external evaluation results here.

Offline Experiments

def sample_evaluator(outputs, inputs, ground_truths):
    # Code here
    return random.randint(1, 5)

You can use your evaluators to evaluate a target function in a controlled setting with curated datasets, like this:

    from honeyhive import evaluate

    import random

    def sample_evaluator(outputs, inputs, ground_truths):
        # Code here
        return random.randint(1, 5)


    # Create function to be evaluated
    # inputs -> parameter to which datapoint or json value will be passed
    # (optional) ground_truths -> ground truth value for the input
    def function_to_evaluate(inputs, ground_truths):
        complete_prompt = f"You are an expert analyst specializing in {inputs['product_type']} market trends. Could you provide an analysis of the current market performance and consumer reception of {inputs['product_type']} in {inputs['region']}?"
        response = "This is a test response."
        return response

    if __name__ == "__main__":
        # Run experiment
        evaluate(
            function = function_to_evaluate,               # Function to be evaluated
            hh_api_key = '<HONEYHIVE_API_KEY>',
            hh_project = '<HONEYHIVE_PROJECT>',
            name = 'Sample Experiment',
            dataset_id = '<DATASET_ID>',                      # this example assumes the existence of a managed dataset in HoneyHive
            evaluators=[sample_evaluator]                 # to compute client-side metrics on each run
        )

This will run the experiment with the datapoints contained in your dataset and run the evaluation on the target function’s output for each of the datapoint.

For a complete explanation of running experiments, refer to the Experiments Quickstart Example.

Multi-step Evaluation in Experiment Runs

If your experiment involves complex, multi-step pipelines, you can log metrics either at the trace level or on a per-span level to gain more detailed insights.

from honeyhive import evaluate, evaluator
from honeyhive import trace, enrich_span

def retrieval_relevance_evaluator(query, docs):
    # code here
    avg_relevance = 0.5
    return avg_relevance

@evaluator()
def consistency_evaluator(outputs, inputs, ground_truths):
    # code here
    consistency_score = 0.66
    return consistency_score


@trace
def get_relevant_docs(query):
    retrieved_docs = [
        "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
        "Studies show morning exercises have better impact on blood sugar levels."
    ]
    retrieval_relevance = retrieval_relevance_evaluator(query, retrieved_docs)
    enrich_span(metrics={"retrieval_relevance": retrieval_relevance})
    return retrieved_docs

def generate_response(docs, query):
    prompt = f"Question: {query}\nContext: {docs}\nAnswer:"
    response = "This is a test response"
    return response

def rag_pipeline(inputs, ground_truths):
    query = inputs["query"]
    docs = get_relevant_docs(query)
    response = generate_response(docs, query)
        
    return response

dataset = [
    {
        "inputs": {
            "query": "How does exercise affect diabetes?",
        },
        "ground_truths": {
            "response": "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
        }
    },
]


if __name__ == "__main__":
    # Run experiment
    evaluate(
        function = rag_pipeline,               # Function to be evaluated
        hh_api_key = '<your-api-key>',
        hh_project = '<your-project-name>',
        name = 'Multi Step Evals',
        dataset = dataset,
        evaluators=[consistency_evaluator],                 # to compute client-side metrics on each run
    )

After running this script, you should be able to see both metrics displayed in your Experiments dashboard.

Online Evaluation

These metrics can be seamlessly passed alongside your traces within each relevant span, as shown below:

import { HoneyHiveTracer, traceTool, traceModel, traceChain, enrichSpan, enrichSession } from "honeyhive";

// Keep interfaces used in the functions
interface MedicalDocument {
    docs: string[];
    response: string;
}

interface RagPipelineMetrics {
    num_retrieved_docs: number;
    query_length: number;
}

// Initialize tracer 
// Ensure HH_API_KEY and HH_PROJECT are set in your environment
const tracer = await HoneyHiveTracer.init({
    sessionName: "online-client-evals",
    // apiKey and project will be picked from environment variables
});

// Define the get_relevant_docs function with traceTool
const getRelevantDocs = traceTool(function getRelevantDocs(
    query: string
): string[] {
    const medicalDocs = [
        "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
        "Studies show morning exercises have better impact on blood sugar levels."
    ];

    enrichSpan({
        metrics: { retrieval_relevance: 0.5 }
    });

    return medicalDocs;
});

// Define generateResponse with traceModel (or traceTool if not an LLM call)
const generateResponse = traceModel(function generateResponse(
    docs: string[],
    query: string
): string {
    const prompt = `Question: ${query}\nContext: ${docs}\nAnswer:`;
    const response = "This is a test response."; 

    enrichSpan({
        metrics: { contains_citations: true }
    });

    return response;
});

// Define ragPipeline with traceChain
const ragPipeline = traceChain(function ragPipeline(
    query: string
): MedicalDocument {
    const docs = getRelevantDocs(query);
    const response = generateResponse(docs, query);

    enrichSession({
        metrics: {
            rag_pipeline: {
                num_retrieved_docs: docs.length,
                query_length: query.split(" ").length
            } as RagPipelineMetrics 
        }
    });

    return { docs, response };
});

// --- Main Execution Logic ---
// Wrap the execution in tracer.trace() to establish context
await tracer.trace(async () => {
    const query = "How does exercise affect diabetes?";
    await ragPipeline(query); // Assuming ragPipeline might become async
});

// Don't forget to flush the tracer if your script exits immediately after
// await tracer.flush(); 

Previously, tracing and enrichment involved calling methods directly on the tracer instance (e.g., tracer.traceFunction(), tracer.enrichSpan()). While this pattern still works, it is now deprecated and will be removed in a future major version.

Please update your code to use the imported functions (traceTool, traceModel, traceChain, enrichSpan, enrichSession) along with the tracer.trace() wrapper as shown in the example above. This new approach simplifies usage within nested functions by not requiring the tracer instance to be passed around.

Example of the deprecated pattern:

// OLD (DEPRECATED) PATTERN:
// const tracer = await HoneyHiveTracer.init({...});
// const getRelevantDocs = tracer.traceFunction()(function getRelevantDocs(...) { ... });
// tracer.enrichSpan({...}); 
// tracer.enrichSession({...});

In this example, enrichSpan is being used to add metrics on particular steps: getRelevantDocs and generateResponse, while enrichSession is used to set metrics that apply to the entire session or pipeline run.

Offline Experiments

You should define your evaluators with the appropriate parameter signature - they can accept two parameters (input, output), where input contains the data passed to your function and output contains the result returned by your function.

interface MarketAnalysisInput {
    product_type: string;
    region: string;
}

interface MarketAnalysisOutput {
    content: string;
    role: string;
}

interface EvaluatorMetrics {
    sample_metric: number;
    sample_metric_2: boolean;
}

export async function functionToEvaluate(input: MarketAnalysisInput): Promise<MarketAnalysisOutput> {
    try {
        const dummyResponse: MarketAnalysisOutput = {
            content: `This is a simulated analysis of ${input.product_type} in ${input.region}. 
            Market trends show significant growth with increasing consumer adoption.
            Regional challenges include supply chain constraints and regulatory considerations.`,
            role: "assistant"
        };
        
        return dummyResponse;
    } catch (error) {
        console.error('Error in function:', error);
        throw error;
    }
}

const dataset: MarketAnalysisInput[] = [
    {
        product_type: "electric vehicles",
        region: "western europe"
    },
    {
        product_type: "gaming consoles",
        region: "north america"
    }
];

function sampleEvaluator(input: MarketAnalysisInput, output: MarketAnalysisOutput): EvaluatorMetrics {
    return {
        sample_metric: 0.5,
        sample_metric_2: true
    };
}

evaluate({
    evaluationFunction: functionToEvaluate,
    hh_api_key: '<HONEYHIVE_API_KEY>',
    hh_project: '<HONEYHIVE_PROJECT>',
    name: 'Sample Experiment',
    dataset: dataset,
    evaluators: [sampleEvaluator],
    server_url: '<HONEYHIVE_SERVER_URL>'
});

This will run the experiment with the datapoints contained in your dataset and run the evaluation on the target function’s output for each of the datapoint.

For a complete explanation of running experiments, refer to the Experiments Quickstart Example.

Next Steps

Client-Side Evaluations

Learn more about adding metrics to traces

Experiment Quickstart

Learn how to run experiments with HoneyHive

Server-Side Evaluators

Learn about evaluators that run on HoneyHive’s infrastructure

Introduction

Guides

Tutorials

Learn more

Client-Side Evaluators

Online Evaluation

Offline Experiments

Multi-step Evaluation in Experiment Runs

Online Evaluation

Offline Experiments

Multi-step Evaluation in Experiment Runs

Online Evaluation

Offline Experiments

Next Steps

Client-Side Evaluations

Experiment Quickstart

Server-Side Evaluators

Introduction

Guides

Tutorials

Learn more

​Online Evaluation

​Offline Experiments

​Multi-step Evaluation in Experiment Runs

​Online Evaluation

​Offline Experiments

​Multi-step Evaluation in Experiment Runs

​Online Evaluation

​Offline Experiments

​Next Steps

Client-Side Evaluations

Experiment Quickstart

Server-Side Evaluators

Online Evaluation

Offline Experiments

Multi-step Evaluation in Experiment Runs

Online Evaluation

Offline Experiments

Multi-step Evaluation in Experiment Runs

Online Evaluation

Offline Experiments

Next Steps