> ## Documentation Index
> Fetch the complete documentation index at: https://docs.honeyhive.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Quickstart

> Get started with running experiments with HoneyHive

Running experiments is a natural extension of the tracing capabilities of HoneyHive. We recommend you to go through the [tracing quickstart](/introduction/quickstart) before proceeding with this guide.

<Tabs>
  <Tab title="Python">
    ## Full code

    Here's a minimal example to get you started with experiments in HoneyHive:

    <AccordionGroup>
      <Accordion title="Sample eval script">
        ```python theme={null}
        from honeyhive import evaluate, evaluator
        import os
        from openai import OpenAI
        import random

        openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

        # Create function to be evaluated
        # inputs -> parameter to which datapoint or json value will be passed
        # (optional) ground_truths -> ground truth value for the input
        def function_to_evaluate(inputs, ground_truths):
            completion = openai_client.chat.completions.create(
                model="gpt-4o",
                messages=[
                    {"role": "system", "content": f"You are an expert analyst specializing in {inputs['product_type']} market trends."},
                    {"role": "user", "content": f"Could you provide an analysis of the current market performance and consumer reception of {inputs['product_type']} in {inputs['region']}? Please include any notable trends or challenges specific to this region."}
                ]
            )

            # Output -> session output
            return completion.choices[0].message.content

        dataset = [
            {
                "inputs": {
                    "product_type": "electric vehicles",
                    "region": "western europe",
                    "time_period": "first half of 2023",
                    "metric_1": "total revenue",
                    "metric_2": "market share"
                },
                "ground_truths": {
                    "response": "As of 2023, the electric vehicle (EV) market in Western Europe is experiencing significant growth, with the region maintaining its status as a global leader in EV adoption. [continue...]",
                }
            },
            {
                "inputs": {
                    "product_type": "gaming consoles",
                    "region": "north america",
                    "time_period": "holiday season 2022",
                    "metric_1": "units sold",
                    "metric_2": "gross profit margin"
                },
                "ground_truths": {
                    "response": "As of 2023, the gaming console market in North America is characterized by intense competition, steady consumer demand, and evolving trends influenced by technological advancements and changing consumer preferences. [continue...]",
                }
            },
            {
                "inputs": {
                    "product_type": "smart home devices",
                    "region": "australia and new zealand",
                    "time_period": "fiscal year 2022-2023",
                    "metric_1": "customer acquisition cost",
                    "metric_2": "average revenue per user"
                },
                "ground_truths": {
                    "response": "As of 2023, the market for smart home devices in Australia and New Zealand is experiencing robust growth, driven by increasing consumer interest in home automation and the enhanced convenience and security these devices offer. [continue...]",
                }
            },
        ]

        @evaluator()
        def sample_evaluator(outputs, inputs, ground_truths):
            # Code here
            return random.randint(1, 5)

        if __name__ == "__main__":
            # Run experiment
            evaluate(
                function = function_to_evaluate,               # Function to be evaluated
                api_key = '<HONEYHIVE_API_KEY>',
                project = '<HONEYHIVE_PROJECT>',
                name = 'Sample Experiment',
                dataset = dataset,                      # to be passed for json_list
                evaluators=[sample_evaluator],                 # to compute client-side metrics on each run
                server_url='<HONEYHIVE_SERVER_URL>'  # Optional / Required for self-hosted or dedicated deployments
            )

        ```
      </Accordion>
    </AccordionGroup>

    ## Running an experiment

    **Prerequisites**

    * You have already created a project in HoneyHive, as explained [here](/workspace/projects).
    * You have an API key for your project, as explained [here](/sdk-reference/authentication).

    ***Expected Time***: 5 minutes

    **Steps**

    <Steps>
      <Step title="Setup input data">
        Let's create our dataset by inputting data directly into our code using a list of JSON objects:

        ```python theme={null}
        dataset = [
            {
                "inputs": {
                    "product_type": "electric vehicles",
                    "region": "western europe"   
                },
                "ground_truths": {
                    "response": "As of 2023, the electric vehicle (EV) ... ",
                }
            },
            {
                "inputs": {
                    "product_type": "gaming consoles",
                    "region": "north america"
                },
                "ground_truths": {
                    "response": "As of 2023, the gaming console market ... ",
                }
            },
            {
                "inputs": {
                    "product_type": "smart home devices",
                    "region": "australia and new zealand" 
                },
                "ground_truths": {
                    "response": "As of 2023, the market for smart home devices in Australia and New Zealand ... ",
                }
            },
        ]
        ```

        <Note> The `inputs` and `ground_truths` fields will be accessible in both the function we want to evaluate and the evaluator function, as we will see below. </Note>
      </Step>

      <Step title="Define the function you want to evaluate">
        Define the function you want to evaluate. This can be arbitrarily complex, anywhere from a prompt or a simple retrieval pipeline, to an end-to-end multi-agent system:

        ```python theme={null}
        # inputs -> parameter to which datapoint or json value will be passed
        # (optional) ground_truths -> ground truth values for the input
        def function_to_evaluate(inputs, ground_truths):

            # Code here

            return result
        ```

        **Important Note About Parameters**

        The function parameters are positional arguments and must be specified in this order:

        1. `inputs` (first parameter): dictionary of parameters from your dataset
        2. `ground_truths` (second parameter): optional ground truth dictionary

        The value returned by the function would map to the `outputs` field of each trace in the experiment and will be accessible to your evaluator function, as we will see below.
      </Step>

      <Step title="(Optional) Setup Evaluators">
        Define client-side evaluators in your code that run immediately after each experiment iteration. These evaluators have direct access to inputs, outputs, and ground truths, and run synchronously with your experiment.

        ```python theme={null}
        @evaluator()
        def sample_evaluator(outputs, inputs, ground_truths):
            # Code here
            import random
            return random.randint(1, 5)
        ```

        **Important Note About Evaluator Parameters**

        The evaluator parameters are positional arguments and must be specified in this order:

        1. `outputs` (first parameter): the output returned by the evaluated function
        2. `inputs` (second parameter): the original input dictionary
        3. `ground_truths` (third parameter): the ground truth dictionary

        <Note>For more complex multi-step pipelines, you can [compute and log client-side evaluators on specific traces and spans](/tracing/client-side-evals) directly in your experiment harness.</Note>
      </Step>

      <Step title="Run experiment">
        Finally, you can run your experiment with `evaluate`:

        ```python theme={null}
        from honeyhive import evaluate
        from your_module import function_to_evaluate

        if __name__ == "__main__":
            evaluate(
                function = function_to_evaluate,
                api_key = '<HONEYHIVE_API_KEY>',
                project = '<HONEYHIVE_PROJECT>',
                name = 'Sample Experiment',
                # To be passed for datasets managed in code
                dataset = dataset,
                # Add evaluators to your trace at the end of each execution
                evaluators=[sample_evaluator, ...],
                server_url='<HONEYHIVE_SERVER_URL>'  # Optional / Required for self-hosted or dedicated deployments
            )
        ```

        <Note>
          If you are using a [self-hosted](/setup/self-hosted) or [dedicated](/setup/dedicated) deployment, you also need to pass:

          * `server_url`: The private HoneyHive endpoint found in the Settings page in the HoneyHive app.
        </Note>
      </Step>
    </Steps>

    ## Dashboard View

    Remember to review the results in your HoneyHive dashboard to gain insights into your model's performance across different inputs. The dashboard provides a comprehensive view of the experiment results and performance across multiple runs.

    <Frame>
      <img src="https://mintcdn.com/honeyhiveai/oxFnPfpA79Ja1RXX/images/sample_python_evaluation_screenshot.png?fit=max&auto=format&n=oxFnPfpA79Ja1RXX&q=85&s=c647312e7016c617e500428d8464b4f5" width="2630" height="1592" data-path="images/sample_python_evaluation_screenshot.png" />
    </Frame>

    ## Conclusion

    By following these steps, you can set up and run experiments using HoneyHive. This allows you to systematically test your LLM-based systems across various scenarios and collect performance data for analysis.

    ### Next Steps

    If you are interested in a specific workflow, we recommend reading the walkthrough for the relevant product area.

    <CardGroup>
      <Card title="Introduction to Evaluators" icon="user-check" href="/evaluators/introduction">
        Learn how to evaluate and monitor your AI applications with HoneyHive's flexible evaluation framework.
      </Card>

      <Card title="Comparing Experiments" icon="flask-vial" href="/evaluation/comparing_evals">
        Compare experiments side-by-side in HoneyHive to identify improvements, regressions, and optimize your workflows.
      </Card>

      <Card title="Running Experiments with HoneyHive's managed datasets" icon="table" href="/evaluation/managed_datasets">
        Run experiments using HoneyHive's managed datasets, enabling centralized dataset management and version control.
      </Card>

      <Card title="Running Experiments with HoneyHive's server-side evaluators" icon="cloud" href="/evaluation/server_side_evaluators">
        Server-side evaluators are centralized, scalable, and versioned, making them ideal for resource-intensive or asynchronous tasks.
      </Card>
    </CardGroup>
  </Tab>

  <Tab title="TypeScript">
    ## Full code

    Here's a minimal example to get you started with experiments in HoneyHive:

    <AccordionGroup>
      <Accordion title="Sample eval script">
        ```typescript theme={null}
        import { evaluate } from "honeyhive";
        import { OpenAI } from 'openai';
        const openai = new OpenAI({ apiKey: process.env.OPENAI_KEY });

        // Create function to be evaluated
        // input -> parameter to which datapoint or json value will be passed
        // ground_truths -> optional parameter - ground truth value
        export async function functionToEvaluate(input: Record<string, any>, ground_truths: Record<string, any>) {
            
            try {
                const response = await openai.chat.completions.create({
                    model: "gpt-4",
                    messages: [
                        {
                            role: 'system',
                            content: `You are an expert analyst specializing in ${input.product_type} market trends.`
                        },
                        {
                            role: 'user',
                            content: `Could you provide an analysis of the current market performance and consumer reception of ${input.product_type} in ${input.region}? Please include any notable trends or challenges specific to this region.`
                        }
                    ],
                });

                // Output -> session output
                return response.choices[0].message;
            } catch (error) {
                console.error('Error making GPT-4 call:', error);
                throw error;
            }
        }

        const dataset = [
            {
                "inputs": {
                    "product_type": "electric vehicles",
                    "region": "western europe"   
                },
                "ground_truths": {
                    "response": "As of 2023, the electric vehicle (EV) ... ",
                }
            },
            {
                "inputs": {
                    "product_type": "gaming consoles",
                    "region": "north america"
                },
                "ground_truths": {
                    "response": "As of 2023, the gaming console market ... ",
                }
            },
            {
                "inputs": {
                    "product_type": "smart home devices",
                    "region": "australia and new zealand" 
                },
                "ground_truths": {
                    "response": "As of 2023, the market for smart home devices in Australia and New Zealand ... ",
                }
            }
        ]

        // Sample evaluator that returns fixed metrics
        function sampleEvaluator(outputs: any, inputs: Record<string, any>, ground_truths: Record<string, any>) {
            // Code here
            return {
                sample_metric: 0.5,
                sample_metric_2: true
            };
        }

        evaluate({
            function: functionToEvaluate,       // Function to be evaluated
            apiKey: '<HONEYHIVE_API_KEY>',
            project: '<HONEYHIVE_PROJECT>',
            name: 'Sample Experiment',
            dataset: dataset,                        // to be passed for json_list
            evaluators: [sampleEvaluator],                 // to compute client-side metrics on each run
            serverUrl: '<HONEYHIVE_SERVER_URL>'  // Optional / Required for self-hosted or dedicated deployments
        })
        ```
      </Accordion>
    </AccordionGroup>

    ## Running an experiment

    **Prerequisites**

    * You have already created a project in HoneyHive, as explained [here](/workspace/projects).
    * You have an API key for your project, as explained [here](/sdk-reference/authentication).

    ***Expected Time***: 5 minutes

    **Steps**

    <Steps>
      <Step title="Setup input data">
        Let's create our dataset by inputting data directly into our code using a list of JSON objects:

        ```typescript theme={null}
        const dataset = [
            {
                "inputs": {
                    "product_type": "electric vehicles",
                    "region": "western europe"   
                },
                "ground_truths": {
                    "response": "As of 2023, the electric vehicle (EV) ... ",
                }
            },
            {
                "inputs": {
                    "product_type": "gaming consoles",
                    "region": "north america"
                },
                "ground_truths": {
                    "response": "As of 2023, the gaming console market ... ",
                }
            },
            {
                "inputs": {
                    "product_type": "smart home devices",
                    "region": "australia and new zealand" 
                },
                "ground_truths": {
                    "response": "As of 2023, the market for smart home devices in Australia and New Zealand ... ",
                }
            }
        ]
        ```

        <Note>The input fields in the dataset should map to the fields mapped in the `evaluate` function.</Note>
      </Step>

      <Step title="Create the flow you want to evaluate">
        Define the function you want to evaluate in your experiment:

        ```typescript theme={null}
            // Create function to be evaluated
            export async function functionToEvaluate(input: Record<string, any>, ground_truths: Record<string, any>) {
                try {
                    // your code here

                    return result;
                } catch (error) {
                    console.error('Error:', error);
                    throw error;
                }
            }        
        ```

        **Important Note About Parameters**

        The function parameters are positional arguments and must be specified in this order:

        1. `inputs` (first parameter): dictionary of parameters from your dataset
        2. `ground_truths` (second parameter): optional ground truth dictionary

        The value returned by the function would map to the `outputs` field of each run in the experiment.
      </Step>

      <Step title="(Optional) Setup Evaluators">
        Define client-side evaluators in your code that run immediately after each experiment iteration. These evaluators have direct access to inputs, outputs, and ground truths, and run synchronously with your experiment.

        ```typescript theme={null}
        // input -> input defined above
        // output -> output returned by the function
        function sampleEvaluator(outputs: any, inputs: Record<string, any>, ground_truths: Record<string, any>) {
            // Code here
            // Each evaluator can return a dictionary of metrics
            return { sample_metric: 0.5, sample_metric_2: true };
        }
        ```

        **Important Note About Evaluator Parameters**

        The evaluator parameters are positional arguments and must be specified in this order:

        1. `outputs` (first parameter): the output returned by the evaluated function
        2. `inputs` (second parameter): the original input dictionary
        3. `ground_truths` (third parameter): the ground truth dictionary

        <Note>For more complex multi-step pipelines, you can [compute and log client-side evaluators on specific traces and spans](/tracing/client-side-evals) directly in your experiment harness.</Note>
      </Step>

      <Step title="Run experiment">
        ```typescript theme={null}
        import { evaluate } from "honeyhive";
        import { functionToEvaluate } from "./your-module";

        evaluate({
            function: functionToEvaluate,  // Direct reference since signature matches
            apiKey: '<HONEYHIVE_API_KEY>',
            project: '<HONEYHIVE_PROJECT>',
            name: 'Sample Experiment',
            dataset: dataset,            // to be passed for json_list
            evaluators: [sampleEvaluator],     // Add evaluators to run at the end of each run
            serverUrl: '<HONEYHIVE_SERVER_URL>'  // Optional / Required for self-hosted or dedicated deployments
        })
        ```

        <Note>
          If you are using a [self-hosted](/setup/self-hosted) or [dedicated](/setup/dedicated) deployment, you also need to pass:

          * `server_url`: The private HoneyHive endpoint found in the Settings page in the HoneyHive app.
        </Note>
      </Step>
    </Steps>

    ## Dashboard View

    Remember to review the results in your HoneyHive dashboard to gain insights into your model's performance across different inputs. The dashboard provides a comprehensive view of the experiment results and performance across multiple runs.

    <Frame>
      <img src="https://mintcdn.com/honeyhiveai/oxFnPfpA79Ja1RXX/images/sample_python_evaluation_screenshot.png?fit=max&auto=format&n=oxFnPfpA79Ja1RXX&q=85&s=c647312e7016c617e500428d8464b4f5" width="2630" height="1592" data-path="images/sample_python_evaluation_screenshot.png" />
    </Frame>

    ## Conclusion

    By following these steps, you can set up and run experiments using HoneyHive. This allows you to systematically test your LLM-based systems across various scenarios and collect performance data for analysis.

    ### Next Steps

    If you are interested in a specific workflow, we recommend reading the walkthrough for the relevant product area.

    <CardGroup>
      <Card title="Introduction to Evaluators" icon="user-check" href="/evaluators/introduction">
        Learn how to evaluate and monitor your AI applications with HoneyHive's flexible evaluation framework.
      </Card>

      <Card title="Comparing Experiments" icon="flask-vial" href="/evaluation/comparing_evals">
        Compare experiments side-by-side in HoneyHive to identify improvements, regressions, and optimize your workflows.
      </Card>
    </CardGroup>
  </Tab>
</Tabs>