Skip to main content
Portkey is an AI gateway that provides a unified OpenAI-compatible API across 1,600+ models from providers including OpenAI, Anthropic, AWS Bedrock, Google, Mistral, and more. Because HoneyHive’s LLM evaluators use an OpenAI-compatible interface, you can point them at Portkey’s gateway to access any provider Portkey supports. This is useful when you need to:
  • Run evaluators on models not natively supported in HoneyHive (AWS Bedrock, Mistral, Cohere, etc.)
  • Apply Portkey routing rules (fallbacks, load balancing, budget limits) to evaluation calls
  • Manage provider credentials centrally in Portkey rather than per-tool

​
How it works

HoneyHive’s LLM evaluators call a provider using two fields from the evaluator definition:
  • model_provider - determines which credential set to use
  • model_name - passed verbatim as the model identifier
Setting model_provider to openai and pointing the OpenAI api_base at Portkey’s gateway redirects all evaluator calls through Portkey. The model_name field uses Portkey’s @provider-slug/model-id format to tell Portkey which backend provider to target.

​
Step 1: Configure provider keys

In HoneyHive, go to Settings > Workspace > AI Providers and click OpenAI. Enter:
FieldValue
api_keyYour Portkey API key
api_basehttps://api.portkey.ai/v1
Save. The OpenAI provider will now show as connected.
Provider secrets are configured at the workspace level. Changing the OpenAI api_base to Portkey’s gateway affects all LLM evaluators in the workspace that use the openai provider.To keep some evaluators on a direct OpenAI connection, use a separate workspace.

​
Step 2: Set up a provider in Portkey’s Model Catalog

In the Portkey dashboard, add your provider credentials under Model Catalog:
  1. Add credentials for the provider you want to use (e.g. Anthropic API key, AWS credentials)
  2. Give it a slug, for example anthropic-prod or bedrock-prod
  3. That slug becomes the @provider-slug prefix in your model names
See Portkey’s Model Catalog docs for supported providers and credential formats.

​
Step 3: Update the evaluator’s model name

The HoneyHive UI shows a fixed list of model names, so set the Portkey model name via API. Use a PUT request with just the fields to change - all other evaluator settings are preserved: 
curl -X PUT 'https://api.dp1.us.honeyhive.ai/v1/metrics/<metric_id>' \
  -H 'Authorization: Bearer <your-honeyhive-api-key>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model_provider": "openai",
    "model_name": "@<provider-slug>/<model-id>"
  }'
Replace:
  • <metric_id> - the ID of your LLM evaluator (visible in the evaluator URL in the HoneyHive console)
  • <your-honeyhive-api-key> - your HoneyHive API key
  • <provider-slug> - the slug you assigned in Portkey’s Model Catalog
  • <model-id> - the model ID in the format your target provider expects
After setting the model via API, the evaluator editor will show “Model unavailable” for the Model field - this is expected. If you open the evaluator in the UI and save it, the dropdown will overwrite model_name with a listed value, reverting your Portkey configuration. Make all future changes to model_provider and model_name via the API.

​
Example model names

Target providerExample model_name
OpenAI (via Portkey)@openai-prod/gpt-4o
Anthropic (via Portkey)@anthropic-prod/claude-sonnet-4-6-20250929
Mistral (via Portkey)@mistral-prod/mistral-large-latest
AWS Bedrock - Anthropic@bedrock-prod/us.anthropic.claude-haiku-4-5-20251001-v1:0
AWS Bedrock - Meta@bedrock-prod/us.meta.llama3-3-70b-instruct-v1:0
The model ID after the slug must match the format the target provider expects. Check the provider’s own documentation for the correct model identifier string.

​
Verify the configuration

Trigger an evaluator run by sending a trace that matches the evaluator’s filters, or by running an experiment. If it succeeds, the evaluator result appears as normal. If it fails, check:
  1. Provider secret values - confirm api_key and api_base are set correctly under Settings > AI Providers > OpenAI
  2. Portkey provider slug - ensure the slug in model_name matches exactly what you configured in Portkey’s Model Catalog
  3. Model ID format - verify the model ID after the slug is valid for the target provider
Portkey logs all gateway requests in its dashboard. If HoneyHive’s evaluator call reaches Portkey but the underlying provider returns an error, the Portkey logs will show the provider’s error response.

LLM Evaluators

Create and configure LLM evaluators

Provider Keys

Manage AI provider credentials in HoneyHive

Portkey Integration

Add HoneyHive tracing to Portkey-routed applications