> ## Documentation Index
> Fetch the complete documentation index at: https://docs.honeyhive.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Client-Side Evaluators

> Learn how to use client-side evaluators for both tracing and experiments

Client-side evaluators run within your application environment, providing immediate feedback and integration with your existing infrastructure.

Evaluators can be utilized either:

* **online**: real-time metrics for live applications
* **offline**: controlled experimental environments

For online evaluation, HoneyHive enables you to [log evaluation results directly alongside your traces](/tracing/client-side-evals) at various stages of your pipeline.
For offline evaluation, evaluators are most effective when used with HoneyHive's evaluation harness, which is designed to [run and manage experiments](/evaluation/introduction) seamlessly.

<Tabs>
  <Tab title="Python">
    ## Online Evaluation

    Once [tracing is set up](/introduction/quickstart) for your application, performing client-side online evaluations becomes straightforward. It simply involves enriching your traces and spans with additional context using the metrics field. This field allows you to pass any custom metric, using any of the primary data types. Metrics can be logged for any type of event and at every step of your pipeline.

    For example, consider a Retrieval-Augmented Generation (RAG) scenario where you have evaluator functions implemented to assess:
    a) retrieval quality,
    b) model response generation, and
    c) overall pipeline performance.

    These metrics can be seamlessly passed alongside your traces within each relevant span, as shown below:

    ```python theme={null}
    from honeyhive import HoneyHiveTracer, trace, enrich_span, enrich_session


    HoneyHiveTracer.init(
    api_key="my-api-key",
    project="my-project",
    )

    @trace
    def get_relevant_docs(query):
        medical_docs = [
            "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
            "Studies show morning exercises have better impact on blood sugar levels."
        ]
        enrich_span(metrics={"retrieval_relevance": 0.5})
        return medical_docs

    @trace
    def generate_response(docs, query):
        prompt = f"Question: {query}\nContext: {docs}\nAnswer:"
        response = "This is a test response."
        enrich_span(metrics={"contains_citations": True})
        return response

    @trace
    def rag_pipeline(query):
        docs = get_relevant_docs(query)
        response = generate_response(docs, query)
        
        
        # Add session-level metrics
        enrich_session(metrics={
            "rag_pipeline": {
                "num_retrieved_docs": len(docs),
                "query_length": len(query.split())   
            }
        })
        
        return docs, response

    def main():
        query = "How does exercise affect diabetes?"
        retrieved_docs, generated_response = rag_pipeline(query)

    if __name__ == "__main__":
        main()
    ```

    In this example, `enrich_span` is being used to add metrics on particular steps: `get_relevant_docs` and `generate_response`,
    while `enrich_session` is used to set metrics that apply to the entire session or pipeline run.

    <Note>
      You can learn more about logging external evaluation results [here](/tracing/client-side-evals).
    </Note>

    ## Offline Experiments

    You can also use client-side evaluators as part of your experiments sessions. In an experiments settings, your evaluator will have access to outputs (as generated by the evaluated function), inputs and ground truth (as defined in your dataset).

    You should define your evaluators with the appropriate parameter signature - they can accept one parameter (`outputs`), two parameters (`outputs`, `inputs`), or three parameters (`outputs`, `inputs`, `ground_truths`) depending on what data your evaluation logic requires.

    ```python theme={null}
    def sample_evaluator(outputs, inputs, ground_truths):
        # Code here
        return random.randint(1, 5)
    ```

    By default, evaluation results are stored at the session level. The return values of evaluator functions should represent meaningful evaluation metrics, such as numerical scores, booleans, or other significant measurements.

    You can use your evaluators to evaluate a target function in a controlled setting with curated datasets, like this:

    ```python theme={null}
        from honeyhive import evaluate

        import random

        def sample_evaluator(outputs, inputs, ground_truths):
            # Code here
            return random.randint(1, 5)


        # Create function to be evaluated
        # inputs -> parameter to which datapoint or json value will be passed
        # (optional) ground_truths -> ground truth value for the input
        def function_to_evaluate(inputs, ground_truths):
            complete_prompt = f"You are an expert analyst specializing in {inputs['product_type']} market trends. Could you provide an analysis of the current market performance and consumer reception of {inputs['product_type']} in {inputs['region']}?"
            response = "This is a test response."
            return response

        if __name__ == "__main__":
            # Run experiment
            evaluate(
                function = function_to_evaluate,               # Function to be evaluated
                hh_api_key = '<HONEYHIVE_API_KEY>',
                hh_project = '<HONEYHIVE_PROJECT>',
                name = 'Sample Experiment',
                dataset_id = '<DATASET_ID>',                      # this example assumes the existence of a managed dataset in HoneyHive
                evaluators=[sample_evaluator]                 # to compute client-side metrics on each run
            )
    ```

    This will run the experiment with the datapoints contained in your dataset and run the evaluation on the target function's output for each of the datapoint.

    <Note>
      For a complete explanation of running experiments, refer to the [Experiments Quickstart Example](/evaluation/quickstart).
    </Note>

    ### Multi-step Evaluation in Experiment Runs

    If your experiment involves complex, multi-step pipelines, you can log metrics either at the trace level or on a per-span level to gain more detailed insights.

    In this example, we define two evaluators: `consistency_evaluator` for the main `rag_pipeline` function, and `retrieval_relevance_evaluator` for the document retrieval step. The first is passed directly to `evaluate()`, while the second is enriched within the retrieval step itself.

    ```python theme={null}
    from honeyhive import evaluate, evaluator
    from honeyhive import trace, enrich_span

    def retrieval_relevance_evaluator(query, docs):
        # code here
        avg_relevance = 0.5
        return avg_relevance

    @evaluator()
    def consistency_evaluator(outputs, inputs, ground_truths):
        # code here
        consistency_score = 0.66
        return consistency_score


    @trace
    def get_relevant_docs(query):
        retrieved_docs = [
            "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
            "Studies show morning exercises have better impact on blood sugar levels."
        ]
        retrieval_relevance = retrieval_relevance_evaluator(query, retrieved_docs)
        enrich_span(metrics={"retrieval_relevance": retrieval_relevance})
        return retrieved_docs

    def generate_response(docs, query):
        prompt = f"Question: {query}\nContext: {docs}\nAnswer:"
        response = "This is a test response"
        return response

    def rag_pipeline(inputs, ground_truths):
        query = inputs["query"]
        docs = get_relevant_docs(query)
        response = generate_response(docs, query)
            
        return response

    dataset = [
        {
            "inputs": {
                "query": "How does exercise affect diabetes?",
            },
            "ground_truths": {
                "response": "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
            }
        },
    ]


    if __name__ == "__main__":
        # Run experiment
        evaluate(
            function = rag_pipeline,               # Function to be evaluated
            hh_api_key = '<your-api-key>',
            hh_project = '<your-project-name>',
            name = 'Multi Step Evals',
            dataset = dataset,
            evaluators=[consistency_evaluator],                 # to compute client-side metrics on each run
        )
    ```

    After running this script, you should be able to see both metrics displayed in your Experiments dashboard.
  </Tab>

  <Tab title="TypeScript">
    ## Online Evaluation

    Once [tracing is set up](/introduction/quickstart) for your application, performing client-side online evaluations becomes straightforward. It simply involves enriching your traces and spans with additional context using the metrics field. This field allows you to pass any custom metric, using any of the primary data types. Metrics can be logged for any type of event and at every step of your pipeline.

    For example, consider a Retrieval-Augmented Generation (RAG) scenario where you have evaluator functions implemented to assess:
    a) retrieval quality,
    b) model response generation, and
    c) overall pipeline performance.

    These metrics can be seamlessly passed alongside your traces within each relevant span, as shown below:

    ```typescript theme={null}
    import { HoneyHiveTracer, traceTool, traceModel, traceChain, enrichSpan, enrichSession } from "honeyhive";

    // Keep interfaces used in the functions
    interface MedicalDocument {
        docs: string[];
        response: string;
    }

    interface RagPipelineMetrics {
        num_retrieved_docs: number;
        query_length: number;
    }

    // Initialize tracer 
    // Ensure HH_API_KEY and HH_PROJECT are set in your environment
    const tracer = await HoneyHiveTracer.init({
        sessionName: "online-client-evals",
        // apiKey and project will be picked from environment variables
    });

    // Define the get_relevant_docs function with traceTool
    const getRelevantDocs = traceTool(function getRelevantDocs(
        query: string
    ): string[] {
        const medicalDocs = [
            "Regular exercise reduces diabetes risk by 30%. Daily walking is recommended.",
            "Studies show morning exercises have better impact on blood sugar levels."
        ];

        enrichSpan({
            metrics: { retrieval_relevance: 0.5 }
        });

        return medicalDocs;
    });

    // Define generateResponse with traceModel (or traceTool if not an LLM call)
    const generateResponse = traceModel(function generateResponse(
        docs: string[],
        query: string
    ): string {
        const prompt = `Question: ${query}\nContext: ${docs}\nAnswer:`;
        const response = "This is a test response."; 

        enrichSpan({
            metrics: { contains_citations: true }
        });

        return response;
    });

    // Define ragPipeline with traceChain
    const ragPipeline = traceChain(function ragPipeline(
        query: string
    ): MedicalDocument {
        const docs = getRelevantDocs(query);
        const response = generateResponse(docs, query);

        enrichSession({
            metrics: {
                rag_pipeline: {
                    num_retrieved_docs: docs.length,
                    query_length: query.split(" ").length
                } as RagPipelineMetrics 
            }
        });

        return { docs, response };
    });

    // --- Main Execution Logic ---
    // Wrap the execution in tracer.trace() to establish context
    await tracer.trace(async () => {
        const query = "How does exercise affect diabetes?";
        await ragPipeline(query); // Assuming ragPipeline might become async
    });

    // Don't forget to flush the tracer if your script exits immediately after
    // await tracer.flush(); 
    ```

    <Note title="Legacy Tracing Method (Deprecated)">
      Previously, tracing and enrichment involved calling methods directly on the `tracer` instance (e.g., `tracer.traceFunction()`, `tracer.enrichSpan()`). While this pattern still works, it is now deprecated and will be removed in a future major version.

      Please update your code to use the imported functions (`traceTool`, `traceModel`, `traceChain`, `enrichSpan`, `enrichSession`) along with the `tracer.trace()` wrapper as shown in the example above. This new approach simplifies usage within nested functions by not requiring the `tracer` instance to be passed around.

      Example of the **deprecated** pattern:

      ```typescript theme={null}
      // OLD (DEPRECATED) PATTERN:
      // const tracer = await HoneyHiveTracer.init({...});
      // const getRelevantDocs = tracer.traceFunction()(function getRelevantDocs(...) { ... });
      // tracer.enrichSpan({...}); 
      // tracer.enrichSession({...});
      ```
    </Note>

    In this example, `enrichSpan` is being used to add metrics on particular steps: `getRelevantDocs` and `generateResponse`, while `enrichSession` is used to set metrics that apply to the entire session or pipeline run.

    ## Offline Experiments

    You can also use client-side evaluators as part of your experiments sessions. In an experiments settings, your evaluator will have access to outputs (as generated by the evaluated function), and inputs (as defined in your dataset).

    You should define your evaluators with the appropriate parameter signature - they can accept two parameters (`input`, `output`), where input contains the data passed to your function and output contains the result returned by your function.

    ```typescript theme={null}
    interface MarketAnalysisInput {
        product_type: string;
        region: string;
    }

    interface MarketAnalysisOutput {
        content: string;
        role: string;
    }

    interface EvaluatorMetrics {
        sample_metric: number;
        sample_metric_2: boolean;
    }

    export async function functionToEvaluate(input: MarketAnalysisInput): Promise<MarketAnalysisOutput> {
        try {
            const dummyResponse: MarketAnalysisOutput = {
                content: `This is a simulated analysis of ${input.product_type} in ${input.region}. 
                Market trends show significant growth with increasing consumer adoption.
                Regional challenges include supply chain constraints and regulatory considerations.`,
                role: "assistant"
            };
            
            return dummyResponse;
        } catch (error) {
            console.error('Error in function:', error);
            throw error;
        }
    }

    const dataset: MarketAnalysisInput[] = [
        {
            product_type: "electric vehicles",
            region: "western europe"
        },
        {
            product_type: "gaming consoles",
            region: "north america"
        }
    ];

    function sampleEvaluator(input: MarketAnalysisInput, output: MarketAnalysisOutput): EvaluatorMetrics {
        return {
            sample_metric: 0.5,
            sample_metric_2: true
        };
    }

    evaluate({
        evaluationFunction: functionToEvaluate,
        hh_api_key: '<HONEYHIVE_API_KEY>',
        hh_project: '<HONEYHIVE_PROJECT>',
        name: 'Sample Experiment',
        dataset: dataset,
        evaluators: [sampleEvaluator],
        server_url: '<HONEYHIVE_SERVER_URL>'
    });
    ```

    This will run the experiment with the datapoints contained in your dataset and run the evaluation on the target function's output for each of the datapoint.

    <Note>
      For a complete explanation of running experiments, refer to the [Experiments Quickstart Example](/evaluation/quickstart).
    </Note>
  </Tab>
</Tabs>

## Next Steps

<CardGroup cols={2}>
  <Card title="Client-Side Evaluations" icon="code" href="/tracing/client-side-evals">
    Learn more about adding metrics to traces
  </Card>

  <Card title="Experiment Quickstart" icon="flask" href="/evaluation/quickstart">
    Learn how to run experiments with HoneyHive
  </Card>

  <Card title="Server-Side Evaluators" icon="server" href="/evaluators/python">
    Learn about evaluators that run on HoneyHive's infrastructure
  </Card>
</CardGroup>