HoneyHive Docs

HoneyHive combines logs, metrics, and traces into a unified data model, leveraging the concept of high cardinality to provide a comprehensive view of your AI system’s performance and behavior. By consolidating these traditionally separate observability pillars into a single, flexible event-based structure, we enable developers to gain deeper insights and perform more sophisticated analyses. This approach offers several key benefits:

Unified Context: Each event captures not just raw data, but also the surrounding context, allowing for more meaningful correlations and insights.
Flexible Querying: High cardinality enables precise filtering and aggregation across multiple dimensions, facilitating complex analyses and troubleshooting.
Scalability: The event-based model scales efficiently with the growing complexity of AI systems and the increasing volume of observability data.
Faster Debugging: The ability to trace a request through various components while simultaneously accessing logs and metrics streamlines the debugging process.

Introducing Events

The base unit of data in HoneyHive is called an event, which represents a span in a trace. A root event in a trace is of the type session, while all non-root events in a trace can be of 3 core types - model, tool and chain.

All events have a parent-child relationship, except session event, which being a root event does not have any parents.

session: A root event used to group together multiple model, tool, and chain events into a single trace. This is achieved by having a common session_id across all children.
model events: Used to track the execution of any LLM requests.
tool events: Used to track execution of any deterministic functions like requests to vector DBs, requests to an external API, regex parsing, document reranking, and more.
chain events: Used to group together multiple model and tool events into composable units that can be evaluated and monitored independently. Typical examples of chains include retrieval pipelines, post-processing pipelines, and more.

Here’s a visual representation of the event hierarchy:

Event Properties

All events have the following properties:

event_id: A unique identifier for the event.
event_type: The type of event. Can be model, tool or chain.
event_name: The name of the event. This can be the model name, the tool name, etc.
source: The source of the event. This can be “production”, “dev”, etc.
session_id: A unique identifier for the session. For a session, all events have the same session_id.
project_id: A unique identifier for the project. For a project, all events have the same project_id.
start_time: The UTC timestamp (in milliseconds) when the event started.
end_time: The UTC timestamp (in milliseconds) when the event ended.
duration: The duration of the event in milliseconds. This can be the model latency, the tool latency, etc.
config: The configuration of the event. This can be the model configuration, the tool configuration, etc.
inputs: The inputs to the event. This can be the prompt, the vector query, etc.
outputs: The outputs of the event. This can be a completion, a vector response, an API response, etc.
error: The error message of the event. This can be a rate limit error, failed retrieval error, etc.
metadata: Additional metadata about the event. This can be the product metadata, error metadata, etc.
user_properties: The user properties of the event. This can be the user id, country, tier, etc.
metrics: The metrics associated with the event. This can be model metrics, tool metrics, etc.
feedback: The feedback for the event. This can be the user feedback, the model feedback, etc.

Session Events

Session events are used to track the execution of your application. These can be used to capture

Session configuration like the application version, environment, etc.
Session metrics like session latency, session throughput, etc.
Session properties like user id, country, tier, etc.
Session feedback like overall session feedback, etc.

Schema for Session Events

Root Field	Field	Type	Description	Reserved
`config`	`app_version`	string	The version of the LLM application currently running.	No
`source`	-	string	The environment/deployment context (`production`, `staging`, etc.).	No
`session`	`session_id`	string	Unique identifier for the session/interaction.	No
	`start_time`	Number	Minimum UTC timestamp (ms) of start_time in session hierarchy.	No
	`end_time`	Number	Maximum UTC timestamp (ms) of end_time in session hierarchy.	No
	`duration`	Number	Calculated difference between end_time and start_time (ms).	No
`metadata`	`num_events`	Number	Total number of events captured during the session.	Yes
	`num_model_events`	Number	Number of model-related events (LLM requests) in session.	Yes
	`has_feedback`	Boolean	Indicates if session contains user feedback events.	Yes
	`cost`	Number	Total LLM usage cost based on provider’s pricing model.	Yes
	`total_tokens`	Number	Total tokens processed (input + output).	Yes
	`prompt_tokens`	Number	Tokens in user prompts/input.	Yes
	`completion_tokens`	Number	Tokens in LLM-generated responses.	Yes
`user_properties`	`user_id`	string	Unique identifier for the user.	No
	`user_tier`	string	User subscription tier (`free`/`pro`).	No
	`user_tenant`	string	Tenant/organization for multi-tenant applications.	No

Properties marked as “Reserved” in the schema are automatically calculated and managed internally by HoneyHive’s auto-tracing system.

Example for Session Events

Here’s an example session event:

{
  "source": "evaluation",
  "project_id": "65e0fc2d6a2eb95f55a92cbc",
  "session_id": "d22c2b1d-b2cf-4593-b489-bb9ed2841d13",
  "event_id": "d22c2b1d-b2cf-4593-b489-bb9ed2841d13",
  "parent_id": null,
  "children_ids": [
    "441de3d0-5e73-4351-ad05-5c60886937d1",
    "15e41853-ff4e-4355-a691-a4d366b3635e"
  ],
  "event_type": "session",
  "event_name": "Ramp Docs Assistant",
  "start_time": 1710161932.7,
  "end_time": 1710147613.894,
  "duration": 80509.507,
  "config": {
    "app_version": "1.0.1"
  },
  "inputs": {
    "question": "How do I build an integration using Ramp API?",
    "chat_history": [
      {
        "role": "system",
        "content": "\nAnswer the user's question only using provided context. Don't lie.\n\nContext: Getting started\nWelcome to the Ramp API. Use the Ramp API to access transactions, issue cards, invite users, and so on.\n\nWe recommend getting started by connecting a new app and going through the request authorization documentation.\n\nFor Ramp developer partners\nIf you are a Ramp partner and want to offer your application to other Ramp customers, please contact your Ramp liaison and we will help set up your application.\n\n\nEnvironments\nThe API is available in two environments that can be accessed by sending requests to different hosts.\n\nEnvironment\nHost\nOpenAPI spec\nDescription\nProduction\nhttps://api.ramp.com\nProduction spec\nUse our production environment to go live with your application.\nSandbox\nhttps://demo-api.ramp.com\nSandbox spec ↗\nFill out this form ↗ to request a sandbox. A sandbox is a full-fledged environment in which you can explore different API endpoints and test your application.\n\n\nContact us\nHave feedback, questions, or ideas? Get in touch via email at developer-support@ramp.com ↗.\n\n\n\nRate limiting\nWe rate limit requests to preserve availability responsibly. The current limit (subject to change) is 200 requests, and gets refreshed in a 10 second rolling window.\n\nWhen the limit is reached, API calls will start getting 429 Too Many Requests responses.\n\nAfter a minute, the request limit will be replenished and you'll be able to make requests again. Note that any API calls made during this window will restart the clock, delaying the replenishment.\n\nPlease contact your Ramp liaison if you would like to request a limit increase for your account.\n\n\n\nApp connection\nAdmin user privileges required\nPlease note that only business admin or owner may register and configure the application. It is not recommended to downgrade the admin that created the app to a non-admin role.\n\n\nRegistering your application in the Ramp developer dashboard is the first step of building an integration based on Ramp API.\n\n\nFrom the Ramp developer ↗ settings page, click on Create new app to register a new application. Provide app name and app description, sign the Terms of service ↗, and click Create app.\n\n\nNow you have registered a new application. Click into it and configure the following parameters:\n\nClient ID and client secret: Credentials for your application; store securely.\nApp name and description\nGrant types: A list of grant types that the application may use to get access token. See authorization guide for more information.\nScopes: Defines scopes that may be granted to access token.\nRedirect URIs: A list of URIs telling Ramp where to send back the users in the authorization process.\nRedirect URI format\nNote that redirect URIs must either use https protocol or be in localhost.\n\n✅ https://example.com/callback is valid\n❎ http://example.com/callback is invalid\n✅ http://localhost:8000/callback is valid\n\n\n\n\nOAuth 2.0\nRamp API uses the OAuth 2.0 protocol ↗ to handle authorization and access control.\n\nWhich grant type should you use?\nIf you are a Ramp customer and your application only accesses your own Ramp data, then you can use either client credentials grant or authorization code grant. If your application is used by other Ramp customers, the authorization code grant is required.\n\nClient Credentials Grant\nClient Credentials ↗ grant can be used to get an access token outside of the context of a user. It is typically used by applications to directly access their own resources, not on behalf of a user.\n\nTo obtain a token, make a request to POST /developer/v1/token. You must include an Authorization header containing a base-64 representation of client_id:client_secret.\n\n\nShell\n\nJavaScript\n\nPython\n\ncurl --location --request POST 'https://api.ramp.com/developer/v1/token' \\\n    --header 'Authorization: Basic <base64-encoded client_id:client_secret>' \\\n    --header 'Content-Type: application/x-www-form-urlencoded' \\\n    --data-urlencode 'grant_type=client_credentials' \\\n    --data-urlencode 'scope=business:read transactions:read' \nThe response JSON payload contains a ready-to-use access_token. The Client Credentials Grant does not produce refresh tokens - you manually obtain new access tokens before the existing ones expire.\n\nAuthorization Code Grant\nThere are three parties involved in the Authorization Code flow -- the client (your application), the server (Ramp) and the user (data owner). The overall flow follows these steps:\n\nYour application sends the user to authenticate with Ramp.\nThe user sees the authorization prompt and approves the app's request for data access.\nThe user is redirected back via a redirect_uri with a temporary authorization_code.\nYour application exchanges the authorization_code for an access_token.\nRamp verifies the params and returns an access_token.\nYour application gets a new access_token with the refresh_token."
      },
      {
        "role": "user",
        "content": "How do I build an integration using Ramp API?"
      }
    ]
  },
  "outputs": {
    "content": "To build an integration using Ramp API, you need to:\n\nget started\nwith Ramp API\n\nRecommended steps:\n\n1. register a new application in the Ramp developer dashboard.\n2. configure parameters for the application, such as client ID, client secret, grant types, scopes, and redirect URIs.\n3.Obtain an access token to use the Ramp API on behalf of the user\nReference: OAuth 2.0 Ramp API"
  },
  "metrics": {
    "output_length": 394
  },
  "feedback": {
    "rating": 5
  },
  "metadata": {
    "num_events": 2,
    "num_model_events": 1,
    "has_feedback": true,
    "cost": 0,
    "total_tokens": 305,
    "prompt_tokens": 203,
    "completion_tokens": 102
  },
  "user_properties": {
    "user_id": "user_123",
	"user_tier": "free"
  },
  "error": null,
}

Model Events

Model events represent a request made to an LLM. These can be used to capture

Model configuration like model name, model hyperparameters, prompt template, etc.
Model metrics like completion token count, cost, tokens per second, etc.
API-level metrics like request latency, rate limit errors, etc.

Schema for Model Events

Root Field	Field	Type	Description	Reference	Critical
`config`	`model`	String	The name or identifier of the LLM model being used for the request.		Yes
	`provider`	String	The provider or vendor of the LLM model (e.g., Anthropic, OpenAI, etc.).	Based on LiteLLM’s list of providers	Yes
	`temperature`	Number	The temperature hyperparameter value used for the LLM, which controls the randomness or creativity of the generated output.		Yes
	`max_tokens`	Number	The maximum number of tokens allowed to be generated by the LLM for the current request.		Yes
	`top_p`	Number	The top-p sampling hyperparameter value used for the LLM, which controls the diversity of the generated output.		Yes
	`top_k`	Number	The top-k sampling hyperparameter value used for the LLM, which controls the diversity of the generated output.		Yes
	`template`	Array	The prompt template or format used for structuring the input to the LLM.		Yes
	`type`	String	Type of model request - “chat” or “completion”.		Yes
	`tools`	Array	Array of OpenAI compatible tool list.	OpenAI API - Function Calling	Yes
	`tool_choice`	String	Tool selection choice.		Yes
	`frequency_penalty`	Number	Controls the model’s likelihood to repeat information.		Yes
	`presence_penalty`	Number	Controls the model’s likelihood to introduce new information.		Yes
	`stop_sequences`	Array	Array of strings that will cause the model to stop generating.		Yes
	`is_streaming`	Boolean	Boolean indicating if the response is streamed.		Yes
	`repetition_penalty`	Number	Controls repetition in the model’s output.		Yes
	`user`	String	Person who created the prompt.		No
	`headers`	Object	Object containing request headers.		No
	`decoding_method`	String	String specifying the decoding method.		No
	`random_seed`	Number	Number used for reproducible outputs.		No
	`min_new_tokens`	Number	Minimum number of new tokens to generate.		No
	`{custom}`	Any	Any additional configuration properties to track		No
`inputs`	`chat_history`	Array	The messages or context provided as input to the LLM, typically in a conversational or chat-like format.	OpenAI API - Chat Messages	Yes
	`functions`	Object	OpenAI compatible functions schema.	OpenAI API - Function Calling	No
	`nodes`	Array	Array of strings - text chunks from retrievers.		No
	`chunks`	Array	Array of strings - text chunks from retrievers.		No
	`{custom}`	Any	Any arbitrary input properties to track		No
`outputs`	`choices`	Array	Array of OpenAI compatible choices schema.	OpenAI API - Chat Completion	Yes
	`role`	String	The role or perspective from which the LLM generated the response (e.g., assistant, user, system).		No
	`content`	String	The actual response message generated by the LLM.		No
	`{custom}`	Any	Any additional output properties to track		No
`metadata`	`total_tokens`	Number	The total number of tokens in the LLM’s response, including the prompt and completion.		Yes
	`completion_tokens`	Number	The number of tokens in the generated completion or output from the LLM.		Yes
	`prompt_tokens`	Number	The number of tokens in the prompt or input provided to the LLM.		Yes
	`cost`	Number	The cost or pricing information associated with the LLM request, if available.		Yes
	`system_fingerprint`	String	System fingerprint string.		No
	`response_model`	String	Response model string.		No
	`status_code`	Number	HTTP status code of the request.		No
	`{custom}`	Any	Any additional metadata properties		No
`metrics`	`{custom}`	Any	Any custom metrics or performance indicators		No
`feedback`	`{custom}`	Any	Any end-user provided feedback		No
`duration`	-	Number	The total time taken for the LLM request, measured in milliseconds, which can help identify performance bottlenecks or slow operations.		No
`error`	-	String	Any errors, exceptions, or error messages that occurred during the LLM request, which can aid in debugging and troubleshooting.		No

Properties marked as reserved are required by HoneyHive for core functionality:

Model configuration, inputs, and outputs properties are used for rendering and replaying requests in the HoneyHive playground
Token counts and cost metadata are used for aggregating session-level analytics

All other properties are recommendations based on our auto-tracing system and can be customized based on your needs.

Example for Model Events

Here’s an example model event:

{
  "source": "evaluation",
  "project_id": "65e0fc2d6a2eb95f55a92cbc",
  "event_id": "fead4996-5bec-4710-bc71-c1f97d311782",
  "parent_id": "397c9cbc-297f-42e9-bc1d-b2b0db850df5",
  "session_id": "397c9cbc-297f-42e9-bc1d-b2b0db850df5",
  "children_ids": [],
  "event_name": "Ramp Docs Answerer",
  "event_type": "model",
  "config": {
    "model": "mistralai/mistral-7b-instruct:free",
    "provider": "openrouter",
    "template": [
      {
        "role": "system",
        "content": "\nAnswer the user's question only using provided context. Don't lie.\n\nContext: {{context}}\n    "
      },
      {
        "role": "user",
        "content": "{{question}}"
      }
    ]
  },
  "inputs": {
    "question": "How do I find all the limits that have been set?",
    "context": "Search documentation\nOverview\nGetting started\nRate limiting\nConventions\nAccounting setup\nError Codes\nChangelog\nAuthorization\nApp connection\nRequest authorization\nOAuth scopes\nREST API\nAccounting\nAccounting Connections\nBills\nBusiness\nCard Programs\nCards\nCashbacks\nDepartments\nEntities\nLeads\nLedger Accounts\nLimits\nList limits\nCreate a limit\nFetch deferred task status\nFetch a limit\nUpdate a limit\nTerminate a limit\nSuspend a limit\nUnsuspend a limit\nLocations\nMemos\nMerchants\nReceipt Integrations\nReceipts\nReimbursements\nSpend Programs\nStatements\nToken\nTransactions\nTransfers\nUsers\nVendors\n\nSwitch to Light theme\nLimits\nList limits\nOAuth scopes\nlimits:read\nRequest Schemas\nRequest Body\nThis request has no body.\nRequest query string parameters\nentity_id string<uuid>\noptional\nspend_program_id string<uuid>\noptional\nuser_id string<uuid>\noptional\nstart string<uuid>\noptional\npage_size integer\noptional\nResponse Schemas\nHTTP 200\n\ndata array<object>\nrequired\n\npage object\nrequired\nGET /developer/v1/limits\n\nShell\n\nJavascript\n\nPython\n\ncurl \\\n  -H \"Accept: application/json\" \\\n  -H \"Authorization: Bearer $RAMP_API_TOKEN\" \\\n    'https://api.ramp.com/developer/v1/limits'\nSample response\nHTTP 200\n{\n  \"data\": [\n    {\n      \"balance\": {\n        \"cleared\": 65,\n        \"pending\": 35,\n        \"total\": 100\n      },\n      \"cards\": [\n        {\n          \"card_id\": \"a40a6ce8-70d4-4d06-91e1-0728ad9bbe39\"\n        }\n      ],\n      \"display_name\": \"T&E\",\n      \"entity_id\": \"c18d9d2e-964f-476d-8bb3-9ac078f00e11\",\n      \"has_program_overridden\": false,\n      \"id\": \"d8135cfe-0396-4b2d-b2cf-ad809fb04731\",\n      \"permitted_spend_types\": {\n        \"primary_card_enabled\": true,\n        \"reimbursements_enabled\": false\n      },\n      \"restrictions\": {\n        \"auto_lock_date\": null,\n        \"categories_whitelist\": [\n          35\n        ],\n        \"interval\": \"MONTHLY\",\n        \"limit\": 500,\n        \"next_interval_reset\": \"2022-12-01T00:00:00+00:00\",\n        \"start_of_interval_date\": \"2022-11-01T00:00:00+00:00\",\n        \"temporary_limit\": null,\n        \"transaction_amount_limit\": 200,\n        \"vendor_blacklist\": [\n          61\n        ]\n      },\n      \"spend_program_id\": \"3a5b1f62-988f-4190-bf31-b7ae87c5dfee\",\n      \"state\": \"ACTIVE\",\n      \"suspension\": {\n        \"acting_user_id\": \"e9186c3a-4650-48ef-aee4-56b77f5019bd\",\n        \"inserted_at\": \"2022-11-03T00:00:00+00:00\",\n        \"suspended_by_ramp\": false\n      },\n      \"users\": [\n        {\n          \"user_id\": \"2ba219ba-5867-453f-bec2-b8d0414b7f75\"\n        }\n      ]\n    }\n  ],\n  \"page\": {\n    \"next\": \"https://api.ramp.com/developer/v1/<resources>?<new_params>\"\n  }\n}\nCreate a limit\nLimit may either be created with spend program id (can provide display name and spending restrictions, cannot provide payment restrictions) or without (must provide display name, spending restrictions, and payment restrictions).\n\nOAuth scopes\nlimits:write\nRequest Schemas\nRequest Body\ndisplay_name string\noptional\nCosmetic display name of the limit.\n\n\nfulfillment object\noptional\nFulfillment details of the limit's card.\n\nidempotency_key string\nrequired\nAn idempotency key is a unique value generated by the client which the server uses to recognize subsequent retries of the same request. To avoid collisions, we encourage clients to use random generated UUIDs.\n\n\npermitted_spend_types object\noptional\nSpecifies the permitted spend types.\n\nspend_program_id string<uuid>\noptional\nThe id of the associated spend program.\n    "
      },
      {
        "role": "user",
        "content": "How do I find all the limits that have been set?"
      }
    ]
  },
  "outputs": {
    "content": "To find all the limits, you can use the Ramp API. You can make an HTTP GET request to \"https://api.ramp.com/developer/v1/limits\". This request will return a list of limits in response. The response will contain an array of objects, each representing a limit, and will also include additional metadata, such as the page of results and the total number ofles.\n\nFor more information about the Ramp API, and how to use it, please refer to the Ramp API documentation. The documentation provides details on the request and response formats, as well as information on each of the available operations and their parameters."
  },
  "start_time": 1710147521.798,
  "end_time": 1710147531.367,
  "duration": 9569.497,
  "metrics": {
    "Answer Faithfulness": 4.0,
    "Answer Faithfulness_explanation": "The AI assistant's answer provides a clear and accurate explanation of the two environments available for the Ramp API: Production and Sandbox. It correctly mentions that API calls in the Production environment should be directed to "https://api.ramp.com" and that this environment is intended for releasing the application to the public. It also correctly states that API calls in the Sandbox environment should be directed to "https://demo-api.ramp.com" and that this environment is for exploring different API endpoints and testing applications.",
    "Number of words": 100
  },
  "feedback": {},
  "metadata": {
    "completion_length": 139
  },
  "user_properties": {},
  "error": null,
}

Tool Events

When your LLM application interacts with external APIs, databases, or vector databases like Pinecone, you can instrument these interactions to evaluate performance, debug issues, and gain insights. Tool events are used to track the execution of anything other than the model. These can be used to capture

Tool configuration like vector index name, vector index hyperparameters, any internal tool configuration, etc.
Tool metrics like retrieved chunk similarity, internal tool response validation, etc.
API-level metrics like request latency, index errors, internal tool errors, etc.

Schema for Tool Events

The tool event represents an interaction with an external resource. Send the following fields:

Root Field	Field	Type	Description	Reserved
`config`	`provider`	string	The name of the external service provider offering vector database, API, or other relevant services (e.g., Pinecone, Weaviate, etc.).	No
	`instance`	string	The specific instance or deployment name of the service within the provider’s infrastructure, allowing for differentiation between multiple instances or deployments.	No
	`embedding_model`	string	The name or identifier of the embedding model used for calculating vector similarity, which is particularly relevant for vector databases or services that rely on vector representations of data.	No
	`chunk_size`	integer	The size (in characters or tokens) of the chunks into which data is split before being converted into vectors, if applicable to the service being used. This is important for services that operate on chunked data.	No
	`chunk_overlap`	integer	The amount of overlap (in characters or tokens) between consecutive chunks of data, if applicable to the service being used. This is also relevant for services that operate on chunked data with overlapping segments.	No
	`db_vendor`	string	Vector database provider name.	No
	`{custom}`	Any	Any additional configuration properties to track	No
`inputs`	`top_k`	integer	The number of top-ranked or most similar results to be retrieved from the vector database or service during a similarity search or ranking operation.	No
	`query`	string	The query string, vector representation, or any other input data used for retrieval, search, or processing by the external service.	No
	`url`	string	External API URL.	No
	`{custom}`	Any	Any arbitrary input properties to track	No
`outputs`	`chunks`	array	The data chunks, documents, or any other output retrieved or obtained from the external service as a result of the query or operation performed.	No
	`scores`	array<number>	The similarity scores, relevance scores, or any other scoring metrics associated with the retrieved chunks or documents, if applicable to the service being used.	No
	`nodes`	array<string>	Text chunks from retrievers.	No
	`{custom}`	Any	Any additional output properties to track	No
`metrics`	`read_units`	number	Vector Database Utilization metric.	No
	`write_units`	number	Vector Database Utilization metric.	No
	`{custom}`	Any	Any custom metrics or performance indicators	No
`metadata`	`operationId`	string	Operation identifier.	No
	`{custom}`	Any	Any additional metadata properties	No
`duration`	-	integer	The total time taken for the request or interaction with the external service, measured in milliseconds, which can be useful for identifying performance bottlenecks or slow operations.	No
`error`	-	string	Any errors, exceptions, or error messages that occurred during the retrieval request or interaction with the external service, which can aid in debugging and troubleshooting.	No
`feedback`	`{custom}`	Any	Any end-user provided feedback	No

Example for Tool Events

Here’s an example tool event:

{
  "source": "evaluation",
  "project_id": "65e0fc2d6a2eb95f55a92cbc",
  "session_id": "d22c2b1d-b2cf-4593-b489-bb9ed2841d13",
  "event_id": "441de3d0-5e73-4351-ad05-5c60886937d1",
  "parent_id": "d22c2b1d-b2cf-4593-b489-bb9ed2841d13",
  "children_ids": [],
  "event_name": "Ramp Docs Retriever",
  "event_type": "tool",
  "config": {
    "provider": "pinecone"
  },
  "inputs": {
    "question": "How do I build an integration using Ramp API?"
  },
  "outputs": {
    "content": "Getting started\nWelcome to the Ramp API. Use the Ramp API to access transactions, issue cards, invite users, and so on.\n\nWe recommend getting started by connecting a new app and going through the request authorization documentation.\n\nFor Ramp developer partners\nIf you are a Ramp partner and want to offer your application to other Ramp customers, please contact your Ramp liaison and we will help set up your application.\n\n\nEnvironments\nThe API is available in two environments that can be accessed by sending requests to different hosts.\n\nEnvironment\nHost\nOpenAPI spec\nDescription\nProduction\nhttps://api.ramp.com\nProduction spec\nUse our production environment to go live with your application.\nSandbox\nhttps://demo-api.ramp.com\nSandbox spec ↗\nFill out this form ↗ to request a sandbox. A sandbox is a full-fledged environment in which you can explore different API endpoints and test your application.\n\n\nContact us\nHave feedback, questions, or ideas? Get in touch via email at developer-support@ramp.com ↗.\n\n\n\nRate limiting\nWe rate limit requests to preserve availability responsibly. The current limit (subject to change) is 200 requests, and gets refreshed in a 10 second rolling window.\n\nWhen the limit is reached, API calls will start getting 429 Too Many Requests responses.\n\nAfter a minute, the request limit will be replenished and you'll be able to make requests again. Note that any API calls made during this window will restart the clock, delaying the replenishment.\n\nPlease contact your Ramp liaison if you would like to request a limit increase for your account.\n\n\n\nApp connection\nAdmin user privileges required\nPlease note that only business admin or owner may register and configure the application. It is not recommended to downgrade the admin that created the app to a non-admin role.\n\n\nRegistering your application in the Ramp developer dashboard is the first step of building an integration based on Ramp API.\n\n\nFrom the Ramp developer ↗ settings page, click on Create new app to register a new application. Provide app name and app description, sign the Terms of service ↗, and click Create app.\n\n\nNow you have registered a new application. Click into it and configure the following parameters:\n\nClient ID and client secret: Credentials for your application; store securely.\nApp name and description\nGrant types: A list of grant types that the application may use to get access token. See authorization guide for more information.\nScopes: Defines scopes that may be granted to access token.\nRedirect URIs: A list of URIs telling Ramp where to send back the users in the authorization process.\nRedirect URI format\nNote that redirect URIs must either use https protocol or be in localhost.\n\n✅ https://example.com/callback is valid\n❎ http://example.com/callback is invalid\n✅ http://localhost:8000/callback is valid\n\n\n\n\nOAuth 2.0\nRamp API uses the OAuth 2.0 protocol ↗ to handle authorization and access control.\n\nWhich grant type should you use?\nIf you are a Ramp customer and your application only accesses your own Ramp data, then you can use either client credentials grant or authorization code grant. If your application is used by other Ramp customers, the authorization code grant is required.\n\nClient Credentials Grant\nClient Credentials ↗ grant can be used to get an access token outside of the context of a user. It is typically used by applications to directly access their own resources, not on behalf of a user.\n\nTo obtain a token, make a request to POST /developer/v1/token. You must include an Authorization header containing a base-64 representation of client_id:client_secret.\n\n\nShell\n\nJavaScript\n\nPython\n\ncurl --location --request POST 'https://api.ramp.com/developer/v1/token' \\\n    --header 'Authorization: Basic <base64-encoded client_id:client_secret>' \\\n    --header 'Content-Type: application/x-www-form-urlencoded' \\\n    --data-urlencode 'grant_type=client_credentials' \\\n    --data-urlencode 'scope=business:read transactions:read' \nThe response JSON payload contains a ready-to-use access_token. The Client Credentials Grant does not produce refresh tokens - you manually obtain new access tokens before the existing ones expire.\n\nAuthorization Code Grant\nThere are three parties involved in the Authorization Code flow -- the client (your application), the server (Ramp) and the user (data owner). The overall flow follows these steps:\n\nYour application sends the user to authenticate with Ramp.\nThe user sees the authorization prompt and approves the app's request for data access.\nThe user is redirected back via a redirect_uri with a temporary authorization_code.\nYour application exchanges the authorization_code for an access_token.\nRamp verifies the params and returns an access_token.\nYour application gets a new access_token with the refresh_token."
  },
  "start_time": 1710147532.796,
  "end_time": 1710147533.133,
  "duration": 337.009,
  "metrics": {
    "Context Relevance": 5,
    "Context Relevance_explanation": "The fetched context from the retriever performs well relative to the user's query. It directly addresses the user's question by providing information on how to build an integration using Ramp API. \n\n"
  },
  "feedback": {},
  "metadata": {},
  "user_properties": {},
  "error": null,
}

Chain Events

Chain events help with categorizing the events into different stages of the pipeline. These can be synchronous or asynchronous stages. How Chain Events Work Any event that has its “parent” set to a chain event becomes a step within that chain. This simple mechanism allows you to consolidate various events into a single unit, making it easier to monitor the progress of your pipeline. Nesting for Hierarchy You can also nest chains within each other. This hierarchical approach lets you track the execution of your pipeline in a structured and organized manner. This nesting feature can be particularly useful for complex workflows.

Session Event as a Special Case: As a special case, the “session event” for a pipeline is essentially a chain event with all other events as its children. This means you can encapsulate the entire pipeline within a single session event, making it easy to manage and analyze.

By separating events into chains, you can track properties like:

Chain configuration like chain name, chain settings, etc.
Chain metrics like chain latency, chain throughput, etc.

Here’s an example chain event:

{
  "source": "development",
  "project_id": "64d69442f9fa4485aa1cc582",
  "event_id": "52f22f37-289c-4718-bc40-0231cc5c7a99",
  "session_id": "fa78fb31-5bf9-4717-bca1-88fee7fb026b",
  "parent_id": "fa78fb31-5bf9-4717-bca1-88fee7fb026b",
  "children_ids": [
    "a809865a-8663-4201-b70b-7f4fc355175b",
    "8af7a04a-e91e-4f42-b345-29eeb614e3e1"
  ],
  "event_type": "chain",
  "event_name": "query",
  "config": {
    "name": "query_rewriter_v1",
    "description": "Rewrite the query to improve retriever performance"
  },
  "inputs": {
    "query_str": "What did the author do growing up?"
  },
  "outputs": {
    "rewritten_query": "What did Paul Graham do growing up?"
  },
  "start_time": 1710244017.942,
  "end_time": 1710244019.976,
  "duration": 2033.809,
  "metrics": {},
  "feedback": {},
  "metadata": {
	"total_tokens": 10,
	"num_events": 2,
  },
  "user_properties": {},
  "error": null,
}

Next Steps

Refer to our tracing introduction guide to get started with tracing in HoneyHive.

Tracing Introduction

Getting started with tracing in HoneyHive.

​Introducing Events

​Session Events

​Schema for Session Events

​Example for Session Events

​Model Events

​Schema for Model Events

​Example for Model Events

​Tool Events

​Schema for Tool Events

​Example for Tool Events

​Chain Events

​Next Steps

Tracing Introduction

Introducing Events

Session Events

Schema for Session Events

Example for Session Events

Model Events

Schema for Model Events

Example for Model Events

Tool Events

Schema for Tool Events

Example for Tool Events

Chain Events

Next Steps