> ## Documentation Index
> Fetch the complete documentation index at: https://docs.honeyhive.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace Distributed Systems

> Trace requests across service boundaries with context propagation

**Problem:** You have a multi-service AI application and need to trace requests as they flow across service boundaries to understand performance, errors, and dependencies.

**Solution:** Use HoneyHive's distributed tracing with context propagation to create unified traces across multiple services.

## How Context Propagation Works

When a request crosses a service boundary, the calling service injects its trace context into outgoing HTTP headers. The receiving service extracts that context and attaches all of its spans to the same trace. The result is a single unified session in HoneyHive, even though the work happened in different processes.

```mermaid theme={null}
sequenceDiagram
    participant Client as Client Request
    participant Gateway as API Gateway
    participant UserSvc as User Service
    participant LLMSvc as LLM Service
    
    Client->>Gateway: HTTP Request<br/>trace-id: abc123
    
    Gateway->>UserSvc: Internal Call<br/>trace-id: abc123
    
    UserSvc->>LLMSvc: LLM Request<br/>trace-id: abc123
    LLMSvc->>LLMSvc: OpenAI Call
    LLMSvc-->>UserSvc: LLM Response
    
    UserSvc-->>Gateway: Result
    Gateway-->>Client: Final Response
    
    Note over Client,LLMSvc: All operations linked by trace-id
```

HoneyHive provides two helpers that handle the plumbing:

| Helper                                            | Where       | What it does                                                            |
| ------------------------------------------------- | ----------- | ----------------------------------------------------------------------- |
| `inject_context_into_carrier(headers, tracer)`    | Client side | Injects trace ID, session ID, and project into HTTP headers             |
| `with_distributed_trace_context(headers, tracer)` | Server side | Extracts context from headers and attaches it to all spans in the block |

## What You'll Build

A distributed AI agent architecture with a client orchestrator calling both a remote and a local agent:

```mermaid theme={null}
graph LR
    Client[Client App] -->|user_call| Principal[Principal Agent]
    Principal -->|HTTP + Context| RemoteAgent[Research Agent<br/>Process B]
    Principal -->|Direct Call| LocalAgent[Analysis Agent<br/>Process A]
```

## Prerequisites

* Python 3.11+
* HoneyHive API key from [https://app.us.honeyhive.ai](https://app.us.honeyhive.ai)
* Google Gemini API key from [https://aistudio.google.com/apikey](https://aistudio.google.com/apikey)

## Installation

```bash theme={null}
pip install honeyhive google-adk openinference-instrumentation-google-adk flask[async] requests
```

## Step 1: Set Environment Variables

```bash theme={null}
export HH_API_KEY=your_honeyhive_api_key_here
export HH_PROJECT=distributed-tracing-tutorial
export GOOGLE_API_KEY=your_google_gemini_api_key_here
export AGENT_SERVER_URL=http://localhost:5003
```

## Step 2: Create the Agent Server (Remote Service)

The agent server runs a Google ADK research agent. The key line is `with_distributed_trace_context()`, which extracts the incoming trace context from HTTP headers and attaches it to all spans created inside the block.

Create `agent_server.py`:

<Accordion title="agent_server.py" icon="server">
  ```python theme={null}
  """Google ADK Agent Server - Demonstrates with_distributed_trace_context() helper."""

  from flask import Flask, request, jsonify
  from honeyhive import HoneyHiveTracer
  from honeyhive.tracer.processing.context import with_distributed_trace_context
  from openinference.instrumentation.google_adk import GoogleADKInstrumentor
  from google.adk.agents import LlmAgent
  from google.adk.runners import Runner
  from google.adk.sessions import InMemorySessionService
  from google.genai import types
  import os

  # Initialize HoneyHive tracer
  tracer = HoneyHiveTracer.init(
      api_key=os.getenv("HH_API_KEY"),
      project=os.getenv("HH_PROJECT", "distributed-tracing-tutorial"),
      source="agent-server"
  )

  # Initialize Google ADK instrumentor
  instrumentor = GoogleADKInstrumentor()
  instrumentor.instrument(tracer_provider=tracer.provider)

  app = Flask(__name__)
  session_service = InMemorySessionService()
  app_name = "distributed_agent_demo"

  async def run_agent(user_id: str, query: str, agent_name: str = "research_agent") -> str:
      """Run Google ADK agent - automatically part of distributed trace."""
      agent = LlmAgent(
          model="gemini-2.0-flash-exp",
          name=agent_name,
          description="A research agent that gathers information on topics",
          instruction="""You are a research assistant. When given a topic, provide 
          key facts and important information in 2-3 clear sentences. 
          Focus on accuracy and relevance."""
      )
      
      runner = Runner(agent=agent, app_name=app_name, session_service=session_service)
      session_id = f"{app_name}_{user_id}"
      
      try:
          await session_service.create_session(
              app_name=app_name, user_id=user_id, session_id=session_id
          )
      except Exception:
          pass
      
      user_content = types.Content(role='user', parts=[types.Part(text=query)])
      final_response = ""
      
      async for event in runner.run_async(
          user_id=user_id, session_id=session_id, new_message=user_content
      ):
          if event.is_final_response() and event.content and event.content.parts:
              final_response = event.content.parts[0].text
      
      return final_response or ""

  @app.route("/agent/invoke", methods=["POST"])
  async def invoke_agent():
      """Invoke agent with distributed tracing."""
      
      # One line: extract context, attach to all spans in this block
      with with_distributed_trace_context(dict(request.headers), tracer):
          try:
              data = request.get_json()
              result = await run_agent(
                  data.get("user_id", "default_user"),
                  data.get("query", ""),
                  data.get("agent_name", "research_agent")
              )
              return jsonify({
                  "response": result,
                  "agent": data.get("agent_name", "research_agent")
              })
          except Exception as e:
              return jsonify({"error": str(e)}), 500

  if __name__ == "__main__":
      app.run(port=5003, debug=True, use_reloader=False)
  ```
</Accordion>

The critical pattern on the server side:

```python highlight={2} theme={null}
@app.route("/agent/invoke", methods=["POST"])
async def invoke_agent():
    with with_distributed_trace_context(dict(request.headers), tracer):
        # Everything here is part of the client's trace
        result = await run_agent(...)
```

`with_distributed_trace_context()` handles extracting the trace ID, session ID, and project from the incoming headers, attaching context so all spans link to the caller's trace, and cleaning up when the block exits (even on exceptions).

## Step 3: Create the Client Application

The client orchestrates both remote and local agent calls. For the remote call, it uses `inject_context_into_carrier()` to propagate trace context via HTTP headers.

Create `client_app.py`:

<Accordion title="client_app.py" icon="desktop">
  ```python theme={null}
  """Client Application - Orchestrates remote and local agent calls."""

  import asyncio
  import os
  import requests

  from google.adk.sessions import InMemorySessionService
  from google.adk.agents import LlmAgent
  from google.adk.runners import Runner
  from google.genai import types

  from honeyhive import HoneyHiveTracer, trace
  from openinference.instrumentation.google_adk import GoogleADKInstrumentor
  from honeyhive.tracer.processing.context import (
      enrich_span_context,
      inject_context_into_carrier
  )

  # Initialize HoneyHive tracer
  tracer = HoneyHiveTracer.init(
      api_key=os.getenv("HH_API_KEY"),
      project=os.getenv("HH_PROJECT", "distributed-tracing-tutorial"),
      source="client-app"
  )

  # Initialize Google ADK instrumentor (for local agent calls)
  instrumentor = GoogleADKInstrumentor()
  instrumentor.instrument(tracer_provider=tracer.provider)

  session_service = InMemorySessionService()
  app_name = "distributed_agent_demo"

  async def main():
      """Main entry point - demonstrates multi-turn conversation."""
      user_id = "demo_user"
      result1 = await user_call(user_id, "Explain the benefits of renewable energy")
      print(result1)
      result2 = await user_call(user_id, "What are the main challenges?")
      print(result2)

  @trace(event_type="chain", event_name="user_call")
  async def user_call(user_id: str, user_query: str) -> str:
      agent_server_url = os.getenv("AGENT_SERVER_URL", "http://localhost:5003")
      return await call_principal(user_id, user_query, agent_server_url)

  @trace(event_type="chain", event_name="call_principal")
  async def call_principal(user_id: str, query: str, agent_server_url: str) -> str:
      research_result = await call_remote_agent(user_id, query, agent_server_url)
      analysis_result = await call_local_agent(user_id, research_result)
      return f"Research: {research_result}\n\nAnalysis: {analysis_result}"

  async def call_remote_agent(user_id: str, query: str, agent_server_url: str) -> str:
      """REMOTE invocation: Call agent server via HTTP with context propagation."""
      with enrich_span_context(event_name="call_remote_agent", inputs={"query": query}):
          headers = {"Content-Type": "application/json"}
          inject_context_into_carrier(headers, tracer)
          
          response = requests.post(
              f"{agent_server_url}/agent/invoke",
              json={"user_id": user_id, "query": query, "agent_name": "research_agent"},
              headers=headers,
              timeout=60
          )
          response.raise_for_status()
          result = response.json().get("response", "")
          tracer.enrich_span(outputs={"response": result}, metadata={"mode": "remote"})
          return result

  async def call_local_agent(user_id: str, research_input: str) -> str:
      """LOCAL invocation: Run analysis agent in same process."""
      with enrich_span_context(event_name="call_local_agent", inputs={"research": research_input}):
          agent = LlmAgent(
              model="gemini-2.0-flash-exp",
              name="analysis_agent",
              description="An analysis agent that provides insights",
              instruction="""You are an analytical assistant. Review the research 
              provided and give key insights and conclusions in 2-3 sentences."""
          )
          
          runner = Runner(agent=agent, app_name=app_name, session_service=session_service)
          session_id = f"{app_name}_{user_id}_local"
          
          try:
              await session_service.create_session(
                  app_name=app_name, user_id=user_id, session_id=session_id
              )
          except Exception:
              pass
          
          user_content = types.Content(
              role='user', 
              parts=[types.Part(text=f"Analyze this research: {research_input}")]
          )
          result = ""
          
          async for event in runner.run_async(
              user_id=user_id, session_id=session_id, new_message=user_content
          ):
              if event.is_final_response() and event.content and event.content.parts:
                  result = event.content.parts[0].text
          
          tracer.enrich_span(outputs={"response": result}, metadata={"mode": "local"})
          return result or ""

  if __name__ == "__main__":
      asyncio.run(main())
  ```
</Accordion>

The critical pattern on the client side:

```python highlight={3,4} theme={null}
async def call_remote_agent(user_id, query, agent_server_url):
    with enrich_span_context(event_name="call_remote_agent", inputs={"query": query}):
        headers = {"Content-Type": "application/json"}
        inject_context_into_carrier(headers, tracer)  # Inject trace context
        
        response = requests.post(url, json=payload, headers=headers)
```

`inject_context_into_carrier()` adds the W3C `traceparent` header plus HoneyHive baggage (session ID, project, source) to your outgoing HTTP headers.

## Step 4: Run and Test

<Steps>
  <Step title="Start the Agent Server">
    ```bash theme={null}
    python agent_server.py
    ```

    You should see:

    ```text theme={null}
    * Running on http://127.0.0.1:5003
    ```
  </Step>

  <Step title="Run the Client Application (in a separate terminal)">
    ```bash theme={null}
    python client_app.py
    ```

    You should see research and analysis results for each query.
  </Step>

  <Step title="View in HoneyHive">
    Go to [https://app.us.honeyhive.ai](https://app.us.honeyhive.ai), open the `distributed-tracing-tutorial` project, and click **Traces**. You'll see a unified trace hierarchy:

    ```text theme={null}
    user_call [ROOT]
    └── call_principal
        ├── call_remote_agent (Remote - Process B)
        │   └── agent_run [research_agent] (on server)
        │       └── call_llm (Google ADK instrumentation)
        └── call_local_agent (Local - Process A)
            └── agent_run [analysis_agent] (same process)
                └── call_llm (Google ADK instrumentation)
    ```

    Spans from `agent-server` appear as children of `client-app` spans, even though they ran in different processes.
  </Step>
</Steps>

## Troubleshooting

| Problem                         | Solution                                                                                                                         |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Remote spans don't appear       | Ensure `inject_context_into_carrier()` is called before the HTTP request, and the server uses `with_distributed_trace_context()` |
| Connection refused              | Start the agent server first                                                                                                     |
| Missing `GOOGLE_API_KEY`        | Export it in your environment                                                                                                    |
| Different projects in dashboard | Both server and client must use the same `project` in `HoneyHiveTracer.init()`                                                   |

## Next Steps

<CardGroup cols={2}>
  <Card title="Distributed Tracing Reference" icon="network-wired" href="/v2/tracing/distributed-tracing">
    Session ID approach and serverless patterns (Lambda)
  </Card>

  <Card title="Tracing Introduction" icon="route" href="/v2/tracing/introduction">
    How sessions, events, and context propagation work
  </Card>
</CardGroup>