HoneyHive OpenAI Tracing Guide
This comprehensive guide explains how to use HoneyHive to trace and monitor OpenAI API calls. We’ll cover the setup process and explore each type of trace with practical examples from our cookbook code.
Getting Started
Installation
First, install the required packages as specified in requirements.txt:
pip install openai honeyhive pydantic
Basic Setup
To start tracing your OpenAI calls, initialize the HoneyHive tracer at the beginning of your application:
from openai import OpenAI
from honeyhive import HoneyHiveTracer, trace
# Initialize HoneyHive tracer
HoneyHiveTracer.init(
api_key='your-honeyhive-api-key',
project='OpenAI-traces',
# Optional parameters
source='dev', # Environment: 'dev', 'staging', 'prod', etc.
session_name='openai-session' # Custom session name for better organization
)
# Initialize OpenAI client
client = OpenAI(api_key='your-openai-api-key')
This initialization, found in all our example files, enables automatic instrumentation for all OpenAI API calls.
Types of OpenAI Traces
HoneyHive provides automatic instrumentation for various OpenAI features. Let’s examine each type in detail:
1. Basic Chat Completions
The most common OpenAI interaction is the chat completion, which HoneyHive traces automatically.
From basic_chat.py:
# Simple function to call OpenAI chat completions API
@trace(name="basic_chat_completion", tags={"type": "chat_completion"})
def basic_chat_completion():
"""Make a simple chat completion call to OpenAI API."""
try:
# This call will be automatically traced by HoneyHive
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
temperature=0.7,
max_tokens=150
)
# Return the response content
return response.choices[0].message.content
except Exception as e:
# Errors will be captured in the trace
print(f"Error: {e}")
raise
What HoneyHive captures:
- Request details (model, messages, parameters)
- Response content
- Token usage (prompt, completion, total)
- Latency metrics
- Any errors or exceptions
Enhancing Chat Completion Traces
For richer context, add custom metadata and tags to your traces, as shown in basic_chat.py:
@trace(name="annotated_chat_completion",
tags={"type": "chat_completion", "purpose": "geography_question"},
metadata={"user_id": "test-user-123"})
def annotated_chat_completion(question):
"""Make a chat completion call with custom annotations and metadata."""
# Implementation...
This additional information makes it easier to filter, search, and analyze your traces in the HoneyHive dashboard.
2. Function Calling
Function calling is a powerful OpenAI feature that HoneyHive captures in detail. The trace includes the initial request, function execution, and final response.
From function_calling.py:
@trace(name="basic_function_calling", tags={"type": "function_calling"})
def basic_function_calling():
# Define the tools (functions) the model can use
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a specified location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and country, e.g., 'San Francisco, CA' or 'Paris, France'"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "The temperature unit to use. Default is celsius."
}
},
"required": ["location"]
}
}
}
]
# Make a request to the OpenAI API
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What's the weather like in Paris today?"}
]
# This API call will be traced by HoneyHive
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
tool_choice="auto"
)
# Process response and function calls...
Additionally, tracing the actual functions being called provides a complete picture:
@trace(name="get_weather_function", tags={"type": "external_function"})
def get_weather(location, unit="celsius"):
"""
Get the current weather in a given location.
This is a mock function that would typically call a weather API.
"""
# Implementation...
return weather_data
What HoneyHive captures for function calling:
- The initial request with tools definition
- Function call arguments from the model
- Function execution details
- Second API call with function results
- Final assistant response
3. Structured Outputs
Structured outputs ensure the model’s response adheres to a specific format, either JSON or a Pydantic model. HoneyHive traces these specialized responses including the schema definition.
From structured_output.py:
# Simple JSON schema response format
@trace(name="json_response_format", tags={"type": "structured_output", "format": "json"})
def get_structured_json():
"""Get a structured JSON response using the response_format parameter."""
try:
response = client.chat.completions.create(
model="gpt-4o-2024-08-06", # Make sure to use a model that supports JSON response format
messages=[
{"role": "system", "content": "You are a helpful assistant that provides weather information."},
{"role": "user", "content": "What's the weather like in New York today?"}
],
response_format={"type": "json_object"}
)
return response.choices[0].message.content
except Exception as e:
print(f"Error: {e}")
raise
More advanced structured outputs using JSON schema:
@trace(name="json_schema_output", tags={"type": "structured_output", "format": "json_schema"})
def get_json_schema_output():
"""Get a structured response using a JSON schema."""
try:
# Define a JSON schema
json_schema = {
"type": "object",
"properties": {
"location": {"type": "string"},
"current_weather": {
"type": "object",
"properties": {
"temperature": {"type": "number"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
"conditions": {"type": "string"},
"precipitation_chance": {"type": "number"}
},
"required": ["temperature", "unit", "conditions", "precipitation_chance"]
},
"forecast": {
"type": "array",
"items": {
"type": "object",
"properties": {
"day": {"type": "string"},
"temperature": {"type": "number"},
"conditions": {"type": "string"}
},
"required": ["day", "temperature", "conditions"]
}
}
},
"required": ["location", "current_weather", "forecast"]
}
response = client.chat.completions.create(
model="gpt-4o-2024-08-06",
messages=[...],
response_format={"type": "json_schema", "schema": json_schema}
)
return response.choices[0].message.content
except Exception as e:
print(f"Error: {e}")
raise
And using Pydantic models:
@trace(name="pydantic_structured_output", tags={"type": "structured_output", "format": "pydantic"})
def get_pydantic_structured_output():
"""Get a structured response using Pydantic models."""
try:
completion = client.beta.chat.completions.parse(
model="gpt-4o-2024-08-06",
messages=[...],
response_format=Person
)
# The parsed attribute contains the structured data
person = completion.choices[0].message.parsed
return person
except Exception as e:
print(f"Error: {e}")
raise
What HoneyHive captures for structured outputs:
- The schema or model definition
- Response parsing process
- Structured data output
- Any parsing errors
4. Reasoning Models
OpenAI’s reasoning models (o1, o3-mini) have unique tracing needs, particularly around reasoning tokens and effort levels.
From reasoning_models.py:
@trace(name="reasoning_model_o1", tags={"type": "reasoning_model", "model": "o1"})
def call_o1_model():
"""
Demonstrate calling the o1 reasoning model and trace the request/response.
"""
try:
# Complex math problem that benefits from reasoning capability
response = client.chat.completions.create(
model="o1",
messages=[
{"role": "system", "content": "You are a helpful math assistant."},
{"role": "user", "content": "Solve this step by step: Integrate x^3 * ln(x) with respect to x."}
],
reasoning_effort="high" # Use high reasoning effort for complex problems
)
# Extract the response and the usage information
content = response.choices[0].message.content
reasoning_tokens = response.usage.completion_tokens_details.reasoning_tokens if hasattr(response.usage, "completion_tokens_details") else None
return {
"content": content,
"usage": {
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"total_tokens": response.usage.total_tokens,
"reasoning_tokens": reasoning_tokens
}
}
except Exception as e:
print(f"Error: {e}")
raise
You can also compare different reasoning effort levels:
@trace(name="reasoning_model_o1_with_effort", tags={"type": "reasoning_model", "model": "o1"})
def call_o1_model_with_effort(problem, effort="medium"):
"""
Demonstrate calling the o1 model with different reasoning efforts.
Args:
problem: Math problem to solve
effort: Reasoning effort ('low', 'medium', or 'high')
"""
# Implementation...
What HoneyHive captures for reasoning models:
- Standard request and response details
- Reasoning token usage
- Reasoning effort level
- Model-specific parameters
5. Multi-turn Conversations
Tracing conversations across multiple turns provides a complete history and context. From multi_turn_conversation.py:
class Conversation:
"""
Class to manage a conversation with the OpenAI API.
Each turn in the conversation is traced by HoneyHive.
"""
def __init__(self, system_message="You are a helpful assistant."):
self.messages = [{"role": "system", "content": system_message}]
self.turn_count = 0
@trace(name="conversation_turn", tags={"type": "conversation"})
def add_user_message(self, content):
"""Add a user message to the conversation and get the assistant's response."""
# Increment turn count
self.turn_count += 1
# Add user message to the conversation
self.messages.append({"role": "user", "content": content})
try:
# Get assistant response
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=self.messages,
temperature=0.7,
max_tokens=150
)
# Extract the assistant's message
assistant_message = response.choices[0].message
# Add assistant message to the conversation
self.messages.append({"role": "assistant", "content": assistant_message.content})
return {
"role": assistant_message.role,
"content": assistant_message.content,
"turn": self.turn_count,
"usage": {
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"total_tokens": response.usage.total_tokens
}
}
except Exception as e:
print(f"Error in turn {self.turn_count}: {e}")
raise
Using this class in a full conversation:
@trace(name="rich_conversation", tags={"type": "conversation", "topic": "varied"})
def run_rich_conversation():
"""Run a multi-turn conversation with the assistant on various topics."""
# Initialize conversation with a broad system message
conversation = Conversation(
system_message="You are a knowledgeable assistant able to discuss a wide range of topics."
)
# First turn - Ask about a historical event
turn1 = conversation.add_user_message("Can you tell me about the Apollo 11 mission?")
# Second turn - Follow up on the same topic
turn2 = conversation.add_user_message("What were the names of the astronauts on that mission?")
# Third turn - Change the topic
turn3 = conversation.add_user_message("Let's switch topics. Can you explain how photosynthesis works?")
# Fourth turn - Ask for a summary of the conversation
turn4 = conversation.add_user_message("Can you summarize what we've discussed so far?")
return conversation.get_conversation_history()
What HoneyHive captures for multi-turn conversations:
- Individual turns as separate traces
- Message history accumulation
- Token usage across turns
- Context of the entire conversation
- Relationships between turns
Conclusion
HoneyHive provides comprehensive observability for your OpenAI applications, giving you insights into performance, costs, and behavior. With automatic instrumentation and custom tracing, you can easily monitor and optimize your AI system.
Get started by initializing HoneyHive in your application and watch as your OpenAI calls are automatically traced!
