Config as Code - HoneyHive AI Docs

Keep your HoneyHive evaluators, datasets, charts, and alerts as YAML files in your repo. Review changes in pull requests, then apply them with the HoneyHive CLI. Start with a guide:

Evaluators and Datasets

Define evaluators, datasets, and datapoints as files and apply them to one project per run.

Charts and Alerts

Define charts and alerts as files and apply them to one or more projects.

Why config as code

Reviewable: changes to an evaluator’s Python code or an LLM prompt go through the same PR review as the rest of your app.
Reproducible: the resource definitions live next to the code they evaluate, pinned to a commit.
Portable: the same files apply to staging and production projects by swapping the plane URLs and API keys.
Agent-friendly: every command exposes its JSON Schema, so coding agents like Cursor and Claude Code can author and validate files without guessing.

One project or many

You can apply the same files to one project or to many. What changes is the key you use:

Resources	Key	To cover several projects
Evaluators, datasets, datapoints	Project API key, bound to one project	Run the sync once per project, with that project’s key
Charts, alerts	Data plane and control plane keys rooted at the workspace	One run covers every project in `projects.yaml`

Directory layout

A common convention is to keep one file per resource under .honeyhive/:

.honeyhive/
├── projects.yaml
├── evaluators/
│   ├── keyword-check.yaml
│   └── relevance-llm.yaml
├── datasets/
│   └── qa-eval-set.yaml
├── datapoints/
│   ├── q1.yaml
│   └── q2.yaml
├── charts/
│   └── model-latency.yaml
└── alerts/
    └── high-model-latency.yaml

The directory names map one-to-one to CLI namespaces:

Folder	CLI namespace	API resource
`evaluators/`	`honeyhive metrics`	Metrics
`datasets/`	`honeyhive datasets`	Datasets
`datapoints/`	`honeyhive datapoints`	Datapoints
`charts/`	`honeyhive charts`	Charts
`alerts/`	`honeyhive alerts`	Alerts

“Evaluator” and “metric” are the same resource. The product UI and docs call them evaluators; the API and CLI call them metrics.

Discover the schema

Every CLI command that takes arguments supports two read-only flags:

--show-file-schema, which prints the JSON Schema for the full request object (the exact shape --filename accepts).
--show-argument-schema <flag-name>, which prints the JSON Schema for a single argument. Pass the kebab-case flag name without the leading --.

Both write pure JSON to stdout and never call the API.

# Schema for a new evaluator file
honeyhive metrics create --show-file-schema

# Schema for just the `criteria` field
honeyhive metrics create --show-argument-schema criteria

Save it to scaffold or validate files:

honeyhive metrics create --show-file-schema > .honeyhive/schemas/metric.schema.json

File schemas use the API field names (snake_case or camelCase, matching the public OpenAPI spec) rather than the CLI’s --kebab-case flag names. The CLI’s --filename flag passes the parsed file straight through to the API with no field translation.

For HoneyHive Skills that wrap this workflow for coding agents, see AI Coding Agents. For every command and flag, see the CLI Command Reference.

Evaluators and Datasets

Charts and Alerts

​Why config as code

​One project or many

​Directory layout

​Discover the schema

Why config as code

One project or many

Directory layout

Discover the schema