Charts and Alerts as Code

Write your charts and alerts once as YAML files. A short script then creates them in every project you list, and keeps the charts up to date when you change the files. You need:

The HoneyHive CLI, jq, and yq v4.
Permission to create API keys in your workspace.

Set up

Create two API keys

Charts and alerts use different keys. Go to Settings > Workspace > API Keys and create one key on each tab:

Data Plane tab, for charts. Grant project.chart.list, .post, and .put. Save it as HH_DATA_PLANE_API_KEY.
Control Plane tab, for alerts. Grant project.alert.list and .post. Save it as HH_CONTROL_PLANE_API_KEY.

Keys created at the workspace reach every project in it, including new ones. See API Keys for other options.

List your projects

Create .honeyhive/projects.yaml with one or more project IDs. You find each ID in the project’s settings, on the Data Plane tab. The name is only a label for you.

.honeyhive/projects.yaml

projects:
  - id: <support-agent-project-id>
    name: support-agent
  - id: <claims-agent-project-id>
    name: claims-agent

Define a chart

Each file under .honeyhive/charts/ is one chart. This one shows the median model latency per day:

.honeyhive/charts/model-latency.yaml

name: Model latency p50
description: Median model latency per day.
metric: duration
func: median
bucketing: day
dateRange:
  relative: 7d
query:
  - field: event_type
    operator: is
    value: model
    type: string

To see every field, run honeyhive charts create --show-file-schema.

Define an alert

Each file under .honeyhive/alerts/ is one alert. This one emails project members when average model latency goes above 5 seconds:

.honeyhive/alerts/high-model-latency.yaml

name: High model latency
description: Model latency above 5 seconds.
frequency: HOURLY
alert_type: AGGREGATE
aggregation: AVERAGE
projections:
  - duration
filters:
  - field: event_type
    operator: is
    value: model
    type: string
thresholds:
  critical:
    operator: greater_than
    value: 5000
  resolved:
    operator: less_than
    value: 3000
notification_details:
  critical:
    channel: EMAIL
    scope: ALL_PROJECT_MEMBERS
    metadata: {}
  resolved:
    channel: EMAIL
    scope: ALL_PROJECT_MEMBERS
    metadata: {}

To see every field, run honeyhive alerts create --show-file-schema.

Add the sync script

Save this script as sync-dashboards.sh at the root of your repo. It needs no changes.

sync-dashboards.sh

#!/usr/bin/env bash
# Applies .honeyhive/charts and .honeyhive/alerts to every project
# in .honeyhive/projects.yaml. Run from the repo root.
set -euo pipefail
shopt -s nullglob  # Skip an empty charts/ or alerts/ folder.

with_project() {
  # Copy the file and add project_id. Keep .yaml so the CLI parses it.
  local tmp
  tmp="$(mktemp -d)/definition.yaml"
  yq ".project_id = \"$1\"" "$2" > "$tmp"
  echo "$tmp"
}

failed=0
for project_id in $(yq '.projects[].id' .honeyhive/projects.yaml); do
  echo "Project $project_id"

  charts=$(honeyhive charts list --project-id "$project_id")
  for file in .honeyhive/charts/*.yaml; do
    name=$(yq '.name' "$file")
    chart_id=$(jq -r --arg n "$name" '[.data[] | select(.name == $n)][0].id // ""' <<< "$charts")
    tmp=$(with_project "$project_id" "$file")
    if [ -n "$chart_id" ]; then
      yq -i ".chart_id = \"$chart_id\"" "$tmp"
      action=update
    else
      action=create
    fi
    if honeyhive charts "$action" --filename "$tmp" > /dev/null; then
      echo "  ${action}d chart: $name"
    else
      echo "  FAILED chart: $name" >&2
      failed=1
    fi
    rm -rf "$(dirname "$tmp")"
  done

  alerts=$(honeyhive alerts list --project-id "$project_id" --limit 100)
  for file in .honeyhive/alerts/*.yaml; do
    name=$(yq '.name' "$file")
    if jq -e --arg n "$name" 'any(.data[]; .name == $n)' <<< "$alerts" > /dev/null; then
      echo "  alert exists, skipped: $name"
      continue
    fi
    tmp=$(with_project "$project_id" "$file")
    if honeyhive alerts create --filename "$tmp" > /dev/null; then
      echo "  created alert: $name"
    else
      echo "  FAILED alert: $name" >&2
      failed=1
    fi
    rm -rf "$(dirname "$tmp")"
  done
done

exit "$failed"

Run it

export HH_DATA_PLANE_API_KEY=...
export HH_CONTROL_PLANE_API_KEY=...
bash sync-dashboards.sh

The script prints one line per chart and alert, for each project. Run it again whenever you change a file.

How updates work

The script matches charts and alerts by name, so keep names unique within a project.

Charts: if a chart with the same name exists, the script updates it. Otherwise it creates it.
Alerts: the script only creates alerts that don’t exist yet. It can’t change an existing alert, because the CLI has no alert update. To change one, edit it in the UI, or delete it there and run the script again.

If a chart or alert fails, the script keeps going and exits with status 1 at the end.

Run from CI

To apply changes on every merge, add a workflow with both keys saved as repository secrets:

.github/workflows/honeyhive-dashboards.yml

name: Sync HoneyHive charts and alerts

on:
  push:
    branches: [main]
    paths: [".honeyhive/**"]

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install HoneyHive CLI
        run: curl -fsSL https://github.com/honeyhiveai/honeyhive-cli/releases/download/v1.8.0/install.sh | sh
      - name: Apply charts and alerts
        env:
          HH_DATA_PLANE_API_KEY: ${{ secrets.HH_DATA_PLANE_API_KEY }}
          HH_CONTROL_PLANE_API_KEY: ${{ secrets.HH_CONTROL_PLANE_API_KEY }}
        run: bash sync-dashboards.sh

Set it up with a coding agent

Paste this prompt into Claude Code, Cursor, or another coding agent in your repository:

Set up HoneyHive charts and alerts as code in this repo.
Follow https://docs.honeyhive.ai/v2/cli-reference/config-as-code-charts-alerts
Use the project IDs I give you.
Add one example chart and alert, then run the sync script once.

Limits

Chart names can only use letters, numbers, spaces, and _ - ' &. Other characters, such as parentheses, return a 400.
New projects: creating an alert returns 400: No schema data found until the project has logged matching events. Run the script again after events arrive.
Many alerts: the script reads the 100 newest alerts per project. With more, add --page to the alerts list call, or it can create duplicates.
Self-hosted and dedicated deployments: also set HH_DATA_PLANE_URL and HH_CONTROL_PLANE_URL, in your shell and in the workflow’s env.

Get Started

Config as Code

Reference

Set up

How updates work

Run from CI

Set it up with a coding agent

Limits

​Set up

​How updates work

​Run from CI

​Set it up with a coding agent

​Limits

Set up

How updates work

Run from CI

Set it up with a coding agent

Limits