TypeScript

​

Example Usage

import { evaluate } from "honeyhive";

// Define evaluation function
const evaluationFunction = async (inputs: Record<string, any>, ground_truths: Record<string, any>) => {
    const response = await llm.generate(inputs.prompt);
    return response;
};

// Define evaluator
const qualityEvaluator = (outputs: any, inputs: Record<string, any>, ground_truths: Record<string, any>) => {
    return {
        responseQuality: calculateQuality(outputs),
        promptRelevance: measureRelevance(inputs.prompt, outputs)
    };
};

// Run evaluation
const result = await evaluate({
    function: evaluationFunction,
    hh_api_key: "your-api-key",
    hh_project: "Project Name",
    name: "LLM Quality Test",
    dataset_id: "test_dataset_001",
    evaluators: [qualityEvaluator]
});

console.log(`Evaluation Run ID: ${result.run_id}`);
console.log(`Session IDs: ${result.session_ids}`);

​

Function Signature and Interfaces

async function evaluate(config: EvaluationConfig): Promise<EvaluationResult>;

interface EvaluationConfig {
    apiKey?: string | undefined;
    project?: string | undefined;
    name?: string | undefined;
    suite?: string | undefined;
    function?: (...args: any[]) => any | undefined;
    evaluators?: ((...args: any[]) => any)[] | undefined;
    dataset?: Dict<any>[] | undefined;
    datasetId?: string | undefined;
    maxWorkers?: number | undefined;
    runConcurrently?: boolean | undefined;
    serverUrl?: string | undefined;
    verbose?: boolean | undefined;
    disableHttpTracing?: boolean | undefined;
    metadata?: Dict<any> | undefined;
    instrumentModules?: Record<string, any> | undefined;
}

interface EvaluationResult {
    runId: string;
    datasetId: string | undefined;
    sessionIds: string[];
    status: Status;
    suite: string;
    stats: Dict<any>;
    data: Dict<any>;
}

​

Parameters

​

Required Parameters

• function (Function): The function to evaluate. The function parameters are positional arguments and must be specified in this order: (1) an inputs object, (2) an optional ground truth object, and return a serializable output.
• apiKey (string): API key for authenticating with HoneyHive services.
• project (string): Project identifier in HoneyHive.
• name (string): Identifier for this evaluation run.
• suite (string, optional): Name of the evaluation suite. If not provided, uses the directory name of the calling script.
• maxWorkers (number, optional): Maximum number of concurrent workers for parallel evaluation. Defaults to 10.
• runConcurrently (boolean, optional): Whether to run evaluations concurrently. Defaults to false.
• serverUrl (string, optional): Custom server URL for HoneyHive API.
• verbose (boolean, optional): Whether to print detailed logs during evaluation. Defaults to false.
• disableHttpTracing (boolean, optional): Whether to disable automatic HTTP request tracing. Defaults to false.
• metadata (Record<string, any>, optional): Additional metadata to attach to the evaluation run.
• instrumentModules (Record<string, any>, optional): Modules to instrument for automatic tracing.

​

Optional Parameters

• datasetId (string, optional): ID of an existing HoneyHive dataset to use for evaluation inputs.
• dataset (Record<string, any>[], optional): List of input objects to evaluate against. Alternative to using a dataset.
• evaluators (Function[], optional): List of evaluator functions. The function parameters are positional arguments and must be specified in this order: (1) outputs, (2) inputs, (3) and ground truths to generate metrics.

​

Return Value

{
    runId: string;           // HoneyHive run identifier
    datasetId: string | undefined;  // Dataset ID (HoneyHive or generated) 
    sessionIds: string[];    // Individual evaluation session IDs
    status: Status;         // Run status (e.g., "COMPLETED")
    suite: string;          // Name of the evaluation suite
    stats: Dict<any>;       // Statistics about the evaluation run
    data: Dict<any>;        // Additional data associated with the run
}

​

Technical Notes

1. Execution Flow
   • Validates configuration requirements and credentials
   • Initializes evaluation state and HoneyHive client
   • Loads dataset (HoneyHive dataset or generates ID for in-code datasets)
   • Creates evaluation run in HoneyHive
   • For each iteration:
     • Retrieves input data from dataset
     • Initializes HoneyHiveTracer for the iteration
     • Executes evaluation function with inputs
     • Runs evaluators on function outputs
     • Enriches trace with metadata and metrics
     • Collects session ID
   • Updates evaluation status to completed
   • Returns evaluation metadata
2. Dataset Processing
   • Supports both HoneyHive datasets and external datasets
   • Generates MD5 hashes for external datasets
   • Handles datapoint fetching and validation
   • Manages dataset linkage in traces
3. Tracing Integration
   • Creates individual trace sessions per evaluation
   • Captures:
     • Input/output pairs
     • Evaluator metrics
     • Runtime metadata
     • Dataset linkage
   • Automatically flushes traces after each run
4. Error Management
   • Validates configuration requirements
   • Handles API communication errors
   • Manages evaluator failures independently
   • Preserves partial results on failure

​

Notes

• Either dataset_id or dataset must be provided
• External datasets are automatically assigned a dataset ID
• Evaluator functions should handle both inputs and outputs
• All evaluation runs are automatically traced using HoneyHiveTracer
• Evaluation status is updated to reflect completion or failure