Trace Distributed Systems

Problem: You have a multi-service AI application and need to trace requests as they flow across service boundaries to understand performance, errors, and dependencies. Solution: Use HoneyHive’s distributed tracing with context propagation to create unified traces across multiple services.

How Context Propagation Works

When a request crosses a service boundary, the calling service injects its trace context into outgoing HTTP headers. The receiving service extracts that context and attaches all of its spans to the same trace. The result is a single unified session in HoneyHive, even though the work happened in different processes. HoneyHive provides two helpers that handle the plumbing:

Helper	Where	What it does
`inject_context_into_carrier(headers, tracer)`	Client side	Injects trace ID, session ID, and project into HTTP headers
`with_distributed_trace_context(headers, tracer)`	Server side	Extracts context from headers and attaches it to all spans in the block

What You’ll Build

A distributed AI agent architecture with a client orchestrator calling both a remote and a local agent:

Prerequisites

Python 3.11+
HoneyHive API key from https://app.us.honeyhive.ai
Google Gemini API key from https://aistudio.google.com/apikey

Installation

pip install honeyhive google-adk openinference-instrumentation-google-adk flask[async] requests

Step 1: Set Environment Variables

export HH_API_KEY=your_honeyhive_api_key_here
export GOOGLE_API_KEY=your_google_gemini_api_key_here
export AGENT_SERVER_URL=http://localhost:5003

Step 2: Create the Agent Server (Remote Service)

The agent server runs a Google ADK research agent. The key line is with_distributed_trace_context(), which extracts the incoming trace context from HTTP headers and attaches it to all spans created inside the block. Create agent_server.py:

agent_server.py

"""Google ADK Agent Server - Demonstrates with_distributed_trace_context() helper."""

from flask import Flask, request, jsonify
from honeyhive import HoneyHiveTracer
from honeyhive.tracer.processing.context import with_distributed_trace_context
from openinference.instrumentation.google_adk import GoogleADKInstrumentor
from google.adk.agents import LlmAgent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types
import os

# Initialize HoneyHive tracer
tracer = HoneyHiveTracer.init(
    api_key=os.getenv("HH_API_KEY"),
    source="agent-server"
)

# Initialize Google ADK instrumentor
instrumentor = GoogleADKInstrumentor()
instrumentor.instrument(tracer_provider=tracer.provider)

app = Flask(__name__)
session_service = InMemorySessionService()
app_name = "distributed_agent_demo"

async def run_agent(user_id: str, query: str, agent_name: str = "research_agent") -> str:
    """Run Google ADK agent - automatically part of distributed trace."""
    agent = LlmAgent(
        model="gemini-2.0-flash-exp",
        name=agent_name,
        description="A research agent that gathers information on topics",
        instruction="""You are a research assistant. When given a topic, provide 
        key facts and important information in 2-3 clear sentences. 
        Focus on accuracy and relevance."""
    )
    
    runner = Runner(agent=agent, app_name=app_name, session_service=session_service)
    session_id = f"{app_name}_{user_id}"
    
    try:
        await session_service.create_session(
            app_name=app_name, user_id=user_id, session_id=session_id
        )
    except Exception:
        pass
    
    user_content = types.Content(role='user', parts=[types.Part(text=query)])
    final_response = ""
    
    async for event in runner.run_async(
        user_id=user_id, session_id=session_id, new_message=user_content
    ):
        if event.is_final_response() and event.content and event.content.parts:
            final_response = event.content.parts[0].text
    
    return final_response or ""

@app.route("/agent/invoke", methods=["POST"])
async def invoke_agent():
    """Invoke agent with distributed tracing."""
    
    # One line: extract context, attach to all spans in this block
    with with_distributed_trace_context(dict(request.headers), tracer):
        try:
            data = request.get_json()
            result = await run_agent(
                data.get("user_id", "default_user"),
                data.get("query", ""),
                data.get("agent_name", "research_agent")
            )
            return jsonify({
                "response": result,
                "agent": data.get("agent_name", "research_agent")
            })
        except Exception as e:
            return jsonify({"error": str(e)}), 500

if __name__ == "__main__":
    app.run(port=5003, debug=True, use_reloader=False)

The critical pattern on the server side:

@app.route("/agent/invoke", methods=["POST"])
async def invoke_agent():
    with with_distributed_trace_context(dict(request.headers), tracer):
        # Everything here is part of the client's trace
        result = await run_agent(...)

with_distributed_trace_context() handles extracting the trace ID, session ID, and project from the incoming headers, attaching context so all spans link to the caller’s trace, and cleaning up when the block exits (even on exceptions).

Step 3: Create the Client Application

The client orchestrates both remote and local agent calls. For the remote call, it uses inject_context_into_carrier() to propagate trace context via HTTP headers. Create client_app.py:

client_app.py

"""Client Application - Orchestrates remote and local agent calls."""

import asyncio
import os
import requests

from google.adk.sessions import InMemorySessionService
from google.adk.agents import LlmAgent
from google.adk.runners import Runner
from google.genai import types

from honeyhive import HoneyHiveTracer, trace
from openinference.instrumentation.google_adk import GoogleADKInstrumentor
from honeyhive.tracer.processing.context import (
    enrich_span_context,
    inject_context_into_carrier
)

# Initialize HoneyHive tracer
tracer = HoneyHiveTracer.init(
    api_key=os.getenv("HH_API_KEY"),
    source="client-app"
)

# Initialize Google ADK instrumentor (for local agent calls)
instrumentor = GoogleADKInstrumentor()
instrumentor.instrument(tracer_provider=tracer.provider)

session_service = InMemorySessionService()
app_name = "distributed_agent_demo"

async def main():
    """Main entry point - demonstrates multi-turn conversation."""
    user_id = "demo_user"
    result1 = await user_call(user_id, "Explain the benefits of renewable energy")
    print(result1)
    result2 = await user_call(user_id, "What are the main challenges?")
    print(result2)

@trace(event_type="chain", event_name="user_call")
async def user_call(user_id: str, user_query: str) -> str:
    agent_server_url = os.getenv("AGENT_SERVER_URL", "http://localhost:5003")
    return await call_principal(user_id, user_query, agent_server_url)

@trace(event_type="chain", event_name="call_principal")
async def call_principal(user_id: str, query: str, agent_server_url: str) -> str:
    research_result = await call_remote_agent(user_id, query, agent_server_url)
    analysis_result = await call_local_agent(user_id, research_result)
    return f"Research: {research_result}\n\nAnalysis: {analysis_result}"

async def call_remote_agent(user_id: str, query: str, agent_server_url: str) -> str:
    """REMOTE invocation: Call agent server via HTTP with context propagation."""
    with enrich_span_context(event_name="call_remote_agent", inputs={"query": query}):
        headers = {"Content-Type": "application/json"}
        inject_context_into_carrier(headers, tracer)
        
        response = requests.post(
            f"{agent_server_url}/agent/invoke",
            json={"user_id": user_id, "query": query, "agent_name": "research_agent"},
            headers=headers,
            timeout=60
        )
        response.raise_for_status()
        result = response.json().get("response", "")
        tracer.enrich_span(outputs={"response": result}, metadata={"mode": "remote"})
        return result

async def call_local_agent(user_id: str, research_input: str) -> str:
    """LOCAL invocation: Run analysis agent in same process."""
    with enrich_span_context(event_name="call_local_agent", inputs={"research": research_input}):
        agent = LlmAgent(
            model="gemini-2.0-flash-exp",
            name="analysis_agent",
            description="An analysis agent that provides insights",
            instruction="""You are an analytical assistant. Review the research 
            provided and give key insights and conclusions in 2-3 sentences."""
        )
        
        runner = Runner(agent=agent, app_name=app_name, session_service=session_service)
        session_id = f"{app_name}_{user_id}_local"
        
        try:
            await session_service.create_session(
                app_name=app_name, user_id=user_id, session_id=session_id
            )
        except Exception:
            pass
        
        user_content = types.Content(
            role='user', 
            parts=[types.Part(text=f"Analyze this research: {research_input}")]
        )
        result = ""
        
        async for event in runner.run_async(
            user_id=user_id, session_id=session_id, new_message=user_content
        ):
            if event.is_final_response() and event.content and event.content.parts:
                result = event.content.parts[0].text
        
        tracer.enrich_span(outputs={"response": result}, metadata={"mode": "local"})
        return result or ""

if __name__ == "__main__":
    asyncio.run(main())

The critical pattern on the client side:

async def call_remote_agent(user_id, query, agent_server_url):
    with enrich_span_context(event_name="call_remote_agent", inputs={"query": query}):
        headers = {"Content-Type": "application/json"}
        inject_context_into_carrier(headers, tracer)  # Inject trace context
        
        response = requests.post(url, json=payload, headers=headers)

inject_context_into_carrier() adds the W3C traceparent header plus HoneyHive baggage (session ID, project, source) to your outgoing HTTP headers.

Step 4: Run and Test

Start the Agent Server

python agent_server.py

You should see:

* Running on http://127.0.0.1:5003

Run the Client Application (in a separate terminal)

python client_app.py

You should see research and analysis results for each query.

View in HoneyHive

Go to https://app.us.honeyhive.ai, open the distributed-tracing-tutorial project, and click Traces. You’ll see a unified trace hierarchy:

user_call [ROOT]
└── call_principal
    ├── call_remote_agent (Remote - Process B)
    │   └── agent_run [research_agent] (on server)
    │       └── call_llm (Google ADK instrumentation)
    └── call_local_agent (Local - Process A)
        └── agent_run [analysis_agent] (same process)
            └── call_llm (Google ADK instrumentation)

Spans from agent-server appear as children of client-app spans, even though they ran in different processes.

Troubleshooting

Problem	Solution
Remote spans don’t appear	Ensure `inject_context_into_carrier()` is called before the HTTP request, and the server uses `with_distributed_trace_context()`
Connection refused	Start the agent server first
Missing `GOOGLE_API_KEY`	Export it in your environment
Different projects in dashboard	Both server and client must use the same `project` in `HoneyHiveTracer.init()`

Getting Started

Observability

Evaluation

Prompt Management

Administration

Learn More

How Context Propagation Works

What You’ll Build

Prerequisites

Installation

Step 1: Set Environment Variables

Step 2: Create the Agent Server (Remote Service)

Step 3: Create the Client Application

Step 4: Run and Test

Troubleshooting

Next Steps

Distributed Tracing Reference

Tracing Introduction

​How Context Propagation Works

​What You’ll Build

​Prerequisites

​Installation

​Step 1: Set Environment Variables

​Step 2: Create the Agent Server (Remote Service)

​Step 3: Create the Client Application

​Step 4: Run and Test

​Troubleshooting

​Next Steps

Distributed Tracing Reference

Tracing Introduction

How Context Propagation Works

What You’ll Build

Prerequisites

Installation

Step 1: Set Environment Variables

Step 2: Create the Agent Server (Remote Service)

Step 3: Create the Client Application

Step 4: Run and Test

Troubleshooting

Next Steps