Tracing in Python

​

Introduction

Our Python SDK allows you to trace your custom pipelines with per-event visibility. This allows you to monitor your pipeline’s performance and log user feedback and ground truth labels associated with each step.

For an in-depth overview of how trace data is structured, please see our Logging Overview page.

​

Setup HoneyHive and get your API key

If you haven’t already done so, then the first thing you will need to do is create a HoneyHive project.

After creating the project, you can find your API key in the Settings page under Account.

Once you have created a HoneyHive project and got your API keys, you can now start tracing your custom pipeline.

​

Install the SDK

!pip install -U honeyhive

​

Initializing HoneyHive tracer

First, let’s start by initializing the HoneyHive tracer.

To initialize the tracer, we need to provide 2 necessary parameters:

1. project - the name of the HoneyHive project you want to log to
2. name - the name of the pipeline you are tracing

We also provide 2 optional parameters:

1. source - the source of the pipeline (e.g. “production” or “testing”)
2. user_properties - a dictionary of user properties for whom this pipeline was ran

import os
from honeyhive.utils.tracer import HoneyHiveTracer

tracer = HoneyHiveTracer(
    project="Sandbox - Email Writer",  # necessary field: specify which project within HoneyHive
    name="Paul Graham Q&A",            # optional field: name of the chain/agent you are running
    source="testing",                  # optional field: source (to separate production & testing in monitoring)
    user_properties={                  # optional field: specify user properties for whom this was ran
        "user_id": "1234",
        "user_country": "US"
    },
    api_key="YOUR_HONEYHIVE_API_KEY"
)

​

Tracing a model completion call

Next, let’s trace a model completion call.

We place the code we want to trace within a with block. This automatically calculates latency, metadata, inputs and outputs of the call.

To trace the model completion call via tracer.model, we need to provide 2 necessary parameters:

• event_name - the name of the event you are tracing
• input - the input to the event

On top of these, we also provide 2 optional parameters:

• config - a dictionary of configuration parameters for the model
  • For example, this might include the provider field, which specifies the model provider (e.g. openai)
• description - a description of what the model is doing

with tracer.model(
    event_name="Email Writer",
    description="Generate an email response based on context provided",
    input=YOUR_INPUT_HERE,
    config={
        "provider": "openai",
        "model": "gpt-3.5-turbo",
        "chat_template": YOUR_TEMPLATE_HERE
    }
) as model_call:
    openai_response = openai_client.chat.completions.create(stream=False, ...)

You can also trace an OpenAI streaming model completion call by passing a config with provider set to openai and endpoint set to streaming and retrieve the response like so:

with tracer.model(
    event_name="Email Writer",
    description="Generate an email response based on context provided",
    input=YOUR_INPUT_HERE,
    config={
        "model": "gpt-3.5-turbo",
        "chat_template": YOUR_TEMPLATE_HERE,
        "provider": "openai",
        "endpoint": "streaming"
    }
) as model_call:
    openai.chat.completions.create(stream=True, ...)
openai_response = model_call.get_output()

You can access the event_id for this model completion call via model_call.event_id.

​

View trace

You can now view this trace from within the HoneyHive platform by clicking on Data Store in the sidebar.