Human annotation allows for manual review and evaluation of AI outputs by human reviewers.

​
Creating a Human Evaluator

  1. Navigate to the Evaluators tab in the HoneyHive console.
  2. Click Add Evaluator and select Human Evaluator.

​
Evaluation Criteria

Define clear evaluation criteria for annotators: 
1. Relevance: Is the response directly related to the prompt without unnecessary details?
2. Clarity: Is the message clear and easily understandable?
3. Word Economy: Are unnecessary words, phrases, or sentences eliminated?
4. Precision: Does the response use precise language without being vague?
5. Elimination of Filler: Are redundant or filler words removed?
6. Logical Flow: Does the response follow a logical sequence without unnecessary jumps?
7. Brevity vs. Completeness: Is the response concise while still covering all necessary points?
8. Consistency: Does the response maintain consistent conciseness throughout?
9. Engagement: Does the response keep the reader's interest despite its brevity?
10. Overall Impact: Does the response effectively convey the message concisely?
metrichuman

​
Configuration

​
Return Type

Options:
  1. Numeric: For ratings on a scale
  2. Binary: For yes/no evaluations
  3. Notes: For free-form text feedback

​
Rating Scale

For numeric return types, specify the scale (e.g., 1-5).

​
Passing Range

Define the range of scores considered acceptable.

​
Annotation in UI

You can invite domain experts to annotate traces in any experiment. Once in, experts can annotate each trace and quickly navigate across events using keyboard shortcuts (⬆️ and ⬇️).
humaneval