HoneyHive Alerts

​
Alert Page Components

  • The chart displays a real-time preview of your alert. Adjusting the date does not impact the alert configuration
  • The right panel is where you configure the alert or view the saved configuration
  • Action buttons at the top allow you to pause, mute, resolve, or delete the alert
  • Recent activity at the bottom shows a list of triggered events for this alert

​
Quick Start: Building Your First Alert

1

Name and describe your alert

Give your alert a clear name and description so your team knows what it monitors.
2

Choose monitoring frequency

Select how often to evaluate your alert:
  • Hourly: Checked every hour for immediate detection
  • Daily: Checked every hour but evaluated over daily periods
  • Weekly: Checked daily but evaluated over weekly periods
  • Monthly: Checked daily but evaluated over monthly periods
Note: More frequent checking means faster detection, even for longer evaluation periods.
3

Select what to monitor

Event Type: Choose the data source for your alert
  • Models: Individual LLM API calls and completions
  • Sessions: Complete user conversations or workflows
  • Tools: Function calls and external integrations
  • Chains: Multi-step workflows and complex pipelines
Event Filters: Narrow down to specific segments (optional)
  • Filter by event name, tenant, or any custom metadata
  • Example: Only monitor production traffic or specific model versions
4

Define the metric and aggregation

Metric: What to measure (latency, error rate, custom evaluator scores, etc.)Aggregation: How to summarize the data
  • COUNT: Total number of events
  • AVERAGE: Mean value across events
  • P90/P95/P99: Percentile values for performance monitoring
  • SUM: Total of all values
  • MIN/MAX: Extreme values
  • MEDIAN: Middle value
5

Set alert thresholds

For Aggregate Alerts: Set the threshold value and comparison operator
  • Example: “Average latency > 2 seconds”
For Drift Alerts: Set the percentage change threshold
  • Example: “Current period is 25% worse than previous period”
Resolution Threshold: Optionally set when the alert should auto-resolve (defaults to the inverse of your trigger threshold)
6

Configure notifications

  • Channel: Email (Slack, Teams, PagerDuty coming soon)
  • Recipients: Choose specific team members or notify all project members
  • Get rich notifications with actual values and direct links to investigate

​
Managing Your Alerts

HoneyHive Alerts
Once your alerts are running, you can manage them from the main alerts page. Each alert shows its current status and recent activity.

​
Feature Requests or Unresolved Issues?

Reach out to us at support@honeyhive.ai