Azure OpenAI HoneyHive Tracing Quickstart

This guide provides a comprehensive overview of tracing Azure OpenAI API calls using HoneyHive, with practical examples for different tracing scenarios.

Getting Started

Prerequisites

Before you begin, make sure you have:

  • Python 3.8+
  • An Azure OpenAI resource with API access
  • A HoneyHive API key

Installation

Install the required packages:

pip install openai honeyhive pydantic

Environment Setup

Create a .env file with your API credentials:

HONEYHIVE_API_KEY=your_honeyhive_api_key
AZURE_OPENAI_API_KEY=your_azure_openai_api_key
AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com
AZURE_OPENAI_API_VERSION=2023-07-01-preview
GPT4_DEPLOYMENT_NAME=your-gpt4-deployment-name

Basic Configuration

Here’s how to initialize HoneyHive tracing and the Azure OpenAI client:

import os
from openai import AzureOpenAI
from honeyhive import HoneyHiveTracer, trace

# Initialize HoneyHive tracer
HoneyHiveTracer.init(
    api_key=os.getenv("HONEYHIVE_API_KEY"),
    project="Azure-OpenAI-traces"
)

# Initialize Azure OpenAI client
client = AzureOpenAI(
    api_version=os.getenv("AZURE_OPENAI_API_VERSION", "2023-07-01-preview"),
    azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
)

Tracing Types

1. Basic Chat Completions

The simplest form of tracing captures basic chat completions with the Azure OpenAI API:

@trace
def basic_chat_completion():
    """Make a simple chat completion call to Azure OpenAI API."""
    try:
        # This call will be automatically traced by HoneyHive
        response = client.chat.completions.create(
            model="deployment-name",  # Replace with your actual deployment name
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": "What is the capital of France?"}
            ],
            temperature=0.7,
            max_tokens=150
        )
        
        # Return the response content
        return response.choices[0].message.content
    except Exception as e:
        # Errors will be captured in the trace
        print(f"Error: {e}")
        raise

2. Function Calling Traces

Trace function calling with tools and handling of tool responses:

@trace
def basic_function_calling():
    """
    Demonstrate basic function calling with Azure OpenAI API.
    The model will decide when to call the function based on the user query.
    """
    # Define the tools (functions) the model can use
    tools = [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather in a specified location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and country, e.g., 'San Francisco, CA' or 'Paris, France'"
                        },
                        "unit": {
                            "type": "string",
                            "enum": ["celsius", "fahrenheit"],
                            "description": "The temperature unit to use. Default is celsius."
                        }
                    },
                    "required": ["location"]
                }
            }
        }
    ]
    
    # Make a request to the Azure OpenAI API
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What's the weather like in Paris today?"}
    ]
    
    # This API call will be traced by HoneyHive
    response = client.chat.completions.create(
        model="deployment-name",  # Replace with your actual deployment name
        messages=messages,
        tools=tools,
        tool_choice="auto"
    )
    
    # Continue processing the response...

3. Structured Output Traces

Trace structured outputs using response formats:

@trace
def get_structured_json():
    """Get a structured JSON response using the response_format parameter."""
    try:
        response = client.chat.completions.create(
            model="deployment-name",  # Replace with your actual deployment name
            messages=[
                {"role": "system", "content": "You are a helpful assistant that provides weather information."},
                {"role": "user", "content": "What's the weather like in New York today?"}
            ],
            response_format={"type": "json_object"}
        )
        
        return response.choices[0].message.content
    except Exception as e:
        print(f"Error: {e}")
        raise

You can also trace Pydantic model parsing:

@trace
def get_weather_structured_output(location: str):
    """Get structured weather information for a location using Pydantic."""
    try:
        completion = client.beta.chat.completions.parse(
            model="deployment-name",  # Replace with your actual deployment name
            messages=[
                {"role": "system", "content": "You are a helpful assistant that provides weather information."},
                {"role": "user", "content": f"What's the weather like in {location} today?"}
            ],
            response_format=WeatherInfo
        )
        
        # The parsed attribute contains the structured data
        weather_info = completion.choices[0].message.parsed
        return weather_info
    except Exception as e:
        print(f"Error: {e}")
        raise

4. Multi-Turn Conversation Traces

Track conversations across multiple turns:

class Conversation:
    """
    Class to manage a conversation with the Azure OpenAI API.
    Each turn in the conversation is traced by HoneyHive.
    """
    def __init__(self, system_message="You are a helpful assistant."):
        self.messages = [{"role": "system", "content": system_message}]
        self.turn_count = 0
    
    @trace
    def add_user_message(self, content):
        """Add a user message to the conversation and get the assistant's response."""
        # Increment turn count
        self.turn_count += 1
        
        # Add user message to the conversation
        self.messages.append({"role": "user", "content": content})
        
        try:
            # Get assistant response
            response = client.chat.completions.create(
                model="deployment-name",  # Replace with your actual deployment name
                messages=self.messages,
                temperature=0.7,
                max_tokens=150
            )
            
            # Process response...

Usage example:

@trace
def run_rich_conversation():
    """Run a multi-turn conversation with the assistant on various topics."""
    # Initialize conversation with a broad system message
    conversation = Conversation(
        system_message="You are a knowledgeable assistant able to discuss a wide range of topics."
    )
    
    # First turn
    turn1 = conversation.add_user_message("Can you tell me about the Apollo 11 mission?")
    
    # Second turn
    turn2 = conversation.add_user_message("What were the names of the astronauts on that mission?")
    
    # Third turn
    turn3 = conversation.add_user_message("Let's switch topics. Can you explain how photosynthesis works?")
    
    # And so on...

5. Reasoning Model Traces

Trace model behavior for complex reasoning tasks with temperature control:

@trace
def call_reasoning_model_math():
    """
    Demonstrate calling a reasoning-capable model for math problems and trace the request/response.
    Note: Use your Azure OpenAI deployed model that supports advanced reasoning.
    """
    try:
        # Complex math problem that benefits from reasoning capability
        response = client.chat.completions.create(
            model="gpt-4-deployment",  # Replace with your actual GPT-4 deployment name
            messages=[
                {"role": "system", "content": "You are a helpful math assistant."},
                {"role": "user", "content": "Solve this step by step: Integrate x^3 * ln(x) with respect to x."}
            ],
            temperature=0.1  # Lower temperature for more precise reasoning
        )
        
        # Extract the response and the usage information
        content = response.choices[0].message.content
        
        return {
            "content": content,
            "usage": {
                "prompt_tokens": response.usage.prompt_tokens,
                "completion_tokens": response.usage.completion_tokens,
                "total_tokens": response.usage.total_tokens
            }
        }
    except Exception as e:
        print(f"Error: {e}")
        raise

Conclusion

HoneyHive provides comprehensive observability for your Azure OpenAI applications, allowing you to monitor API usage, performance, and quality. By integrating HoneyHive tracing into your Azure OpenAI applications, you can:

  • Debug issues more effectively
  • Optimize token usage
  • Improve response quality
  • Monitor application performance
  • Track user interactions

For more information, refer to the HoneyHive Documentation and Azure OpenAI Documentation. Happy tracing!