Skip to main content
Problem: You have a multi-service AI application and need to trace requests as they flow across service boundaries to understand performance, errors, and dependencies. Solution: Use HoneyHive’s distributed tracing with context propagation to create unified traces across multiple services.

​
How Context Propagation Works

When a request crosses a service boundary, the calling service injects its trace context into outgoing HTTP headers. The receiving service extracts that context and attaches all of its spans to the same trace. The result is a single unified session in HoneyHive, even though the work happened in different processes. HoneyHive provides two helpers that handle the plumbing:
HelperWhereWhat it does
inject_context_into_carrier(headers, tracer)Client sideInjects trace ID, session ID, and project into HTTP headers
with_distributed_trace_context(headers, tracer)Server sideExtracts context from headers and attaches it to all spans in the block

​
What You’ll Build

A distributed AI agent architecture with a client orchestrator calling both a remote and a local agent:

​
Prerequisites

​
Installation

pip install "honeyhive>=1.0.0rc0" google-adk openinference-instrumentation-google-adk flask[async] requests

​
Step 1: Set Environment Variables

export HH_API_KEY=your_honeyhive_api_key_here
export HH_PROJECT=distributed-tracing-tutorial
export GOOGLE_API_KEY=your_google_gemini_api_key_here
export AGENT_SERVER_URL=http://localhost:5003

​
Step 2: Create the Agent Server (Remote Service)

The agent server runs a Google ADK research agent. The key line is with_distributed_trace_context(), which extracts the incoming trace context from HTTP headers and attaches it to all spans created inside the block. Create agent_server.py: 
"""Google ADK Agent Server - Demonstrates with_distributed_trace_context() helper."""

from flask import Flask, request, jsonify
from honeyhive import HoneyHiveTracer
from honeyhive.tracer.processing.context import with_distributed_trace_context
from openinference.instrumentation.google_adk import GoogleADKInstrumentor
from google.adk.agents import LlmAgent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types
import os

# Initialize HoneyHive tracer
tracer = HoneyHiveTracer.init(
    api_key=os.getenv("HH_API_KEY"),
    project=os.getenv("HH_PROJECT", "distributed-tracing-tutorial"),
    source="agent-server"
)

# Initialize Google ADK instrumentor
instrumentor = GoogleADKInstrumentor()
instrumentor.instrument(tracer_provider=tracer.provider)

app = Flask(__name__)
session_service = InMemorySessionService()
app_name = "distributed_agent_demo"

async def run_agent(user_id: str, query: str, agent_name: str = "research_agent") -> str:
    """Run Google ADK agent - automatically part of distributed trace."""
    agent = LlmAgent(
        model="gemini-2.0-flash-exp",
        name=agent_name,
        description="A research agent that gathers information on topics",
        instruction="""You are a research assistant. When given a topic, provide 
        key facts and important information in 2-3 clear sentences. 
        Focus on accuracy and relevance."""
    )
    
    runner = Runner(agent=agent, app_name=app_name, session_service=session_service)
    session_id = f"{app_name}_{user_id}"
    
    try:
        await session_service.create_session(
            app_name=app_name, user_id=user_id, session_id=session_id
        )
    except Exception:
        pass
    
    user_content = types.Content(role='user', parts=[types.Part(text=query)])
    final_response = ""
    
    async for event in runner.run_async(
        user_id=user_id, session_id=session_id, new_message=user_content
    ):
        if event.is_final_response() and event.content and event.content.parts:
            final_response = event.content.parts[0].text
    
    return final_response or ""

@app.route("/agent/invoke", methods=["POST"])
async def invoke_agent():
    """Invoke agent with distributed tracing."""
    
    # One line: extract context, attach to all spans in this block
    with with_distributed_trace_context(dict(request.headers), tracer):
        try:
            data = request.get_json()
            result = await run_agent(
                data.get("user_id", "default_user"),
                data.get("query", ""),
                data.get("agent_name", "research_agent")
            )
            return jsonify({
                "response": result,
                "agent": data.get("agent_name", "research_agent")
            })
        except Exception as e:
            return jsonify({"error": str(e)}), 500

if __name__ == "__main__":
    app.run(port=5003, debug=True, use_reloader=False)
The critical pattern on the server side: 
@app.route("/agent/invoke", methods=["POST"])
async def invoke_agent():
    with with_distributed_trace_context(dict(request.headers), tracer):
        # Everything here is part of the client's trace
        result = await run_agent(...)
with_distributed_trace_context() handles extracting the trace ID, session ID, and project from the incoming headers, attaching context so all spans link to the caller’s trace, and cleaning up when the block exits (even on exceptions).

​
Step 3: Create the Client Application

The client orchestrates both remote and local agent calls. For the remote call, it uses inject_context_into_carrier() to propagate trace context via HTTP headers. Create client_app.py: 
"""Client Application - Orchestrates remote and local agent calls."""

import asyncio
import os
import requests

from google.adk.sessions import InMemorySessionService
from google.adk.agents import LlmAgent
from google.adk.runners import Runner
from google.genai import types

from honeyhive import HoneyHiveTracer, trace
from openinference.instrumentation.google_adk import GoogleADKInstrumentor
from honeyhive.tracer.processing.context import (
    enrich_span_context,
    inject_context_into_carrier
)

# Initialize HoneyHive tracer
tracer = HoneyHiveTracer.init(
    api_key=os.getenv("HH_API_KEY"),
    project=os.getenv("HH_PROJECT", "distributed-tracing-tutorial"),
    source="client-app"
)

# Initialize Google ADK instrumentor (for local agent calls)
instrumentor = GoogleADKInstrumentor()
instrumentor.instrument(tracer_provider=tracer.provider)

session_service = InMemorySessionService()
app_name = "distributed_agent_demo"

async def main():
    """Main entry point - demonstrates multi-turn conversation."""
    user_id = "demo_user"
    result1 = await user_call(user_id, "Explain the benefits of renewable energy")
    print(result1)
    result2 = await user_call(user_id, "What are the main challenges?")
    print(result2)

@trace(event_type="chain", event_name="user_call")
async def user_call(user_id: str, user_query: str) -> str:
    agent_server_url = os.getenv("AGENT_SERVER_URL", "http://localhost:5003")
    return await call_principal(user_id, user_query, agent_server_url)

@trace(event_type="chain", event_name="call_principal")
async def call_principal(user_id: str, query: str, agent_server_url: str) -> str:
    research_result = await call_remote_agent(user_id, query, agent_server_url)
    analysis_result = await call_local_agent(user_id, research_result)
    return f"Research: {research_result}\n\nAnalysis: {analysis_result}"

async def call_remote_agent(user_id: str, query: str, agent_server_url: str) -> str:
    """REMOTE invocation: Call agent server via HTTP with context propagation."""
    with enrich_span_context(event_name="call_remote_agent", inputs={"query": query}):
        headers = {"Content-Type": "application/json"}
        inject_context_into_carrier(headers, tracer)
        
        response = requests.post(
            f"{agent_server_url}/agent/invoke",
            json={"user_id": user_id, "query": query, "agent_name": "research_agent"},
            headers=headers,
            timeout=60
        )
        response.raise_for_status()
        result = response.json().get("response", "")
        tracer.enrich_span(outputs={"response": result}, metadata={"mode": "remote"})
        return result

async def call_local_agent(user_id: str, research_input: str) -> str:
    """LOCAL invocation: Run analysis agent in same process."""
    with enrich_span_context(event_name="call_local_agent", inputs={"research": research_input}):
        agent = LlmAgent(
            model="gemini-2.0-flash-exp",
            name="analysis_agent",
            description="An analysis agent that provides insights",
            instruction="""You are an analytical assistant. Review the research 
            provided and give key insights and conclusions in 2-3 sentences."""
        )
        
        runner = Runner(agent=agent, app_name=app_name, session_service=session_service)
        session_id = f"{app_name}_{user_id}_local"
        
        try:
            await session_service.create_session(
                app_name=app_name, user_id=user_id, session_id=session_id
            )
        except Exception:
            pass
        
        user_content = types.Content(
            role='user', 
            parts=[types.Part(text=f"Analyze this research: {research_input}")]
        )
        result = ""
        
        async for event in runner.run_async(
            user_id=user_id, session_id=session_id, new_message=user_content
        ):
            if event.is_final_response() and event.content and event.content.parts:
                result = event.content.parts[0].text
        
        tracer.enrich_span(outputs={"response": result}, metadata={"mode": "local"})
        return result or ""

if __name__ == "__main__":
    asyncio.run(main())
The critical pattern on the client side: 
async def call_remote_agent(user_id, query, agent_server_url):
    with enrich_span_context(event_name="call_remote_agent", inputs={"query": query}):
        headers = {"Content-Type": "application/json"}
        inject_context_into_carrier(headers, tracer)  # Inject trace context
        
        response = requests.post(url, json=payload, headers=headers)
inject_context_into_carrier() adds the W3C traceparent header plus HoneyHive baggage (session ID, project, source) to your outgoing HTTP headers.

​
Step 4: Run and Test

1

Start the Agent Server

python agent_server.py
You should see:
* Running on http://127.0.0.1:5003
2

Run the Client Application (in a separate terminal)

python client_app.py
You should see research and analysis results for each query.
3

View in HoneyHive

Go to https://app.honeyhive.ai, open the distributed-tracing-tutorial project, and click Traces. You’ll see a unified trace hierarchy:
user_call [ROOT]
└── call_principal
    ├── call_remote_agent (Remote - Process B)
    │   └── agent_run [research_agent] (on server)
    │       └── call_llm (Google ADK instrumentation)
    └── call_local_agent (Local - Process A)
        └── agent_run [analysis_agent] (same process)
            └── call_llm (Google ADK instrumentation)
Spans from agent-server appear as children of client-app spans, even though they ran in different processes.

​
Troubleshooting

ProblemSolution
Remote spans don’t appearEnsure inject_context_into_carrier() is called before the HTTP request, and the server uses with_distributed_trace_context()
Connection refusedStart the agent server first
Missing GOOGLE_API_KEYExport it in your environment
Different projects in dashboardBoth server and client must use the same project in HoneyHiveTracer.init()

​
Next Steps

Distributed Tracing Reference

Session ID approach and serverless patterns (Lambda)

Tracing Introduction

How sessions, events, and context propagation work