Integrating HoneyHive with your LLM pipeline provides a detailed view of your chain’s performance and behavior, which is crucial for debugging and optimization. In this guide, we’ll delve into the details of how to use the TypeScript SessionTracer object to trace the execution of a ReAct-style pipeline with HoneyHive, exploring how to pass tracers to different pipeline stages, understand the parent-child nesting relationships, and how to utilize this information for debugging, evaluation, and monitoring.

​
Understanding the HoneyHive TypeScript SessionTracer

The SessionTracer class is designed to track various properties of your LLM pipeline execution, including inputs, outputs, error messages, and metadata. These properties are logged and sent to HoneyHive’s platform, where they are visualized and stored for future debugging and optimization.

The Tracer functions by wrapping around different stages of your pipeline and monitoring the execution of each stage’s tasks. It tracks each event in the pipeline’s life cycle, from receiving inputs to generating outputs. This allows you to have a detailed view of how your pipeline is performing, where it’s spending most of its time, and where potential issues or bottlenecks might be.

​
Properties Tracked by the Tracer

The SessionTracer captures a wealth of information including:

  • Event type: The type of event that occurred. This could be a model prediction (model), an operation by a tool (tool), a chain of events (chain), or a generic event (generic).
  • Event name: The name of the function in the chain that the event is associated with.
  • Inputs and Outputs: The inputs to and outputs from the function. For models, this would be the input prompt and the generated text, respectively.
  • Error: Any error that occurred during the execution of the function.
  • Duration: The time it took to execute the function.
  • Metadata and User Properties: Additional data about the event or the user associated with it.

These properties provide deep insights into the pipeline’s behavior and performance, and they can be viewed and analyzed on the HoneyHive platform.

​
Overview of SessionTracer Methods

The SessionTracer class defines several key methods for tracking events in your ReAct Pipeline:

  • constructor(api_key, project, session_name, user_properties = {}, source = 'HoneyHive Typescript SDK'): Initializes a new SessionTracer instance. Accepts parameters for HoneyHive API key, project name, session name, user properties (optional), and source (optional).

  • startSession(inputs?: { [key: string]: any }): Starts a new session. It accepts an optional inputs parameter which contains any inputs for the session.

  • startEvent(event_type, event_name, config, inputs): Starts a new event. Requires parameters for event type, event name, configuration, and inputs. The event type can be a tool (tool), a model (model), a chain of events (chain), or a generic event (generic).

  • endEvent(outputs?: { [key: string]: any }, error?: string): Ends an event. Accepts optional parameters for outputs and error message. If an error occurred during the event, you can pass the error message to this method.

  • endSession(outputs?: { [key:string]: any }, error?: string): Ends a session. Accepts optional parameters for outputs and error message. If an error occurred during the session, you can pass the error message to this method.

The SessionTracer works by maintaining a stack of events. When you call startEvent, it adds a new event to the top of the stack. When you call endEvent, it removes the event from the top of the stack and adds it to the list of children of the next event in the stack (which is now the new top event). If the stack is empty when you call endEvent, it adds the event to the list of children of the parent event. When you call endSession, it ensures that the stack is empty, ending any remaining events, then sends the parent event (which now includes all other events as children) to HoneyHive.

This approach allows the SessionTracer to handle nested events (events that start and end within the duration of another event) and to correctly record the parent-child relationships between events. This becomes extremely useful when you need to understand the sequence of events that occurred during a session, especially when debugging complex pipelines.

​
Visualizing Tracer Data in HoneyHive

Once your pipeline has completed its tasks and the Tracer has logged the necessary information, you can view the traces on the HoneyHive platform. Each trace is visualized as a tree, with each node representing an event in your pipeline’s life cycle. This provides a clear and intuitive way to understand how your pipeline is operating and where potential issues might be.

The properties tracked by the Tracer are presented in a structured way, allowing you to quickly identify important information such as inputs, outputs, and any errors.

You can use these traces to:

  • Debug your application: If an error occurs during execution, you can use the traces to find out where the error happened and what caused it.
  • Evaluate your model: By comparing the inputs and outputs of your model across different runs,you can evaluate its performance and identify areas for improvement.
  • Monitor your application: Regularly checking the traces can help you spot any unusual behavior or performance issues early on.

​
Example Tracer Usage in ReAct Pipeline

Before you can use the SessionTracer class in your project, you need to add the HoneyHive library to your project. You can do this using either npm or yarn:

# With npm
npm add honeyhive

# With yarn
yarn add honeyhive

Here’s an example of using the SessionTracer in a ReAct Pipeline:

import OpenAI from 'openai';
import axios from 'axios';
import { SessionTracer, Config, ToolConfig, ModelConfig } from 'honeyhive';

// Configure your API keys here
const OPENAI_KEY = 'my-openai-key';
const SERP_API_KEY = 'my-serpapi-key';
const openai = new OpenAI({ apiKey: OPENAI_KEY });

// Function to search Google using the SERP API
async function searchGoogle(query: string, tracer: SessionTracer): Promise<string[]> {
    const searchConfig: ToolConfig = { type: 'tool', name: 'GoogleSearch' };
    tracer.startEvent("tool", 'Google Search', searchConfig, { query });

    const url = `https://serpapi.com/search.json?q=${encodeURIComponent(query)}&hl=en&gl=us&api_key=${SERP_API_KEY}`;
    let results = [];
    try {
        const response = await axios.get(url);
        results = response.data.organic_results.map((result: any) => result.snippet);
    } catch (error) {
        console.error('Error during Google search:', error);
    }

    tracer.endEvent({ results });
    return results;
}

// Function to generate a summary of text using OpenAI's LLM
async function generateAnswer(text: string, question: string, tracer: SessionTracer): Promise<string | null> {
    const summarizeConfig: ModelConfig = {
        type: 'model',
        name: 'OpenAI Summarization',
        provider: 'OpenAI',
        model: 'gpt-3.5-turbo',
    };
    const prompt = `Answer the question "${question}" with the help of the following text:\n${text}`;
    tracer.startEvent("model", 'Text Summarization', summarizeConfig, { prompt });

    let summary = null;
    try {
        const chatCompletion = await openai.chat.completions.create({
            messages: [{ role: 'user', content: prompt }],
            model: 'gpt-3.5-turbo',
        });
        summary = chatCompletion.choices[0].message.content;
    } catch (error) {
        console.error('Error during text summarization:', error);
    }

    tracer.endEvent({ summary });
    return summary;
}

// Function to determine if the answer is satisfactory
async function isAnswerSatisfactory(question: string, text: string, tracer: SessionTracer): Promise<boolean> {
    const answerConfig: ModelConfig = {
        type: 'model',
        name: 'OpenAI Summarization',
        provider: 'OpenAI',
        model: 'gpt-3.5-turbo',
    };
    const prompt = `Question: "${question}"\n\nAnswer: "${text}"\n\nIs the provided answer clear and satisfactory? Please respond with "Yes" or "No".`;
    tracer.startEvent("model", 'Answer Verification', answerConfig, { prompt });
    let ans = false;
    let response = null;

    try {
        const chatCompletion = await openai.chat.completions.create({
            messages: [{ role: 'user', content: prompt }],
            model: 'gpt-3.5-turbo',
        });
        response = chatCompletion.choices[0].message.content?.toLowerCase();
        // Assume a simple "Yes" or "No" for a straightforward answer.
        if (response) {
            ans = response.startsWith('yes');
        }
    } catch (error) {
        console.error('Error determining if the answer is satisfactory:', error);
    }

    const outputs = { answer: ans, model_response: response };
    tracer.endEvent({ outputs });
    return ans;
}

// Main pipeline function
async function ReActPipeline(question: string): Promise<void> {
    const tracer = new SessionTracer('my-honeyhive-key', 'My HoneyHive Project', 'My ReAct Pipeline');
    await tracer.startSession({ question: question });
    let attempts = 0;
    const maxAttempts = 5;
    let satisfactory = false;
    let summary: string | null = '';

    while (!satisfactory && attempts < maxAttempts) {
        // Step 1: Turn the user question into a search query
        const searchQuery = `${question}`;
        console.log(`Searching for: ${searchQuery}`);
        const searchResults = await searchGoogle(searchQuery, tracer);

        if (searchResults.length === 0) {
            console.log('No search results found. Trying a different phrasing...');
            attempts++;
            continue;
        }

        // Step 2: Combine the search results into a single text
        const combinedSearchResults = searchResults.join('\n\n');

        // Step 3: Generate a summary of the search results
        summary = await generateAnswer(combinedSearchResults, question, tracer);
        
        // Step 4: Check if the summary is satisfactory
        if (summary) {
            satisfactory = await isAnswerSatisfactory(question, summary, tracer);
        } else {
            console.log('Failed to generate a summary. Trying again...');
        }

        attempts++;
    }

    // Output the final summary if satisfactory, or indicate failure
    if (satisfactory && summary) {
        console.log('Satisfactory Summary Found:', summary);
    } else {
        console.log('Failed to find a satisfactory summary after several attempts.');
    }

    await tracer.endSession({ finalSummary: summary });
}

// Example usage
const userQuestion = 'What is the effect of climate change on the polar bear population?';
ReActPipeline(userQuestion);

In this example, the SessionTracer is initialized at the start of the ReAct Pipeline. At each stage of the pipeline, the SessionTracer starts a new event, logs the necessary information, and then ends the event. Finally, when the pipeline has finished executing, the SessionTracer ends the session and sends the logged information to HoneyHive.

By integrating the SessionTracer into your LLM pipeline like this, you can gain detailed insights into your pipeline’s performance and behavior, which can be invaluable for debugging, optimization, and monitoring.

Tracing in PythonTracing via API