Get API key

After signing up on the app, you can find your API key in the Settings page under Account.

Install the SDK

We currently support a native Python SDK. For other languages, we encourage using HTTP request libraries to send requests.

!pip install honeyhive -q

Log user feedback and ground truth labels

If you have already logged a LLM request in HoneyHive, you can now log user feedback and any ground truth labels associated with that completion.

Using the generation_id that is returned, you can send arbitrary feedback to HoneyHive using the feedback endpoint.

We recommend three general categories of feedback:-

  1. Rating: Helps you log explicit ratings from your users on model performance. Examples include ”👍” / ”👎” or on a numeric scale (eg: “1-5”).
  2. Action: Helps you log implicit actions from your users. Examples include “Edited”, “Regenerated”, “Saved”, “Exited”, etc.
  3. Issues: Helps you log explicit issues pointed out by your users. Examples include “Inappropriate Content”, “Incorrect Answer”, etc.

Below is an example implementation from a writing-assistant app where we log ratings (thumbs_up, rating), user actions (corrected, saved) and a ground truth label (here, we use the end-user’s edited response as the ground truth label).

import os
from openai import OpenAI
import honeyhive
import time

from honeyhive.sdk.feedback import generation_feedback

HONEYHIVE_API_KEY = ""
OPENAI_API_KEY = ""

os.environ["HONEYHIVE_API_KEY"] = HONEYHIVE_API_KEY
client = OpenAI(api_key=OPENAI_API_KEY)

conversation = [
    {"role": "system", "content": "You are a movie recommendation agent who has in-depth knowledge of how to recommend movies to users. You are talking to a user who is looking for a movie to watch. "},
    {"role": "user", "content": "Recommend a movie on the language python"}
]

start = time.time()
openai_response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=conversation,
)
end = time.time()

generation_response = honeyhive.generations.log(
    project="New Project",
    model="gpt-3.5-turbo",
    hyperparameters={
        "temperature": 1
    },
    prompt_template=conversation[0]["content"],
    inputs={
        "message": "Recommend a movie on the language python"
    },
    generation=openai_response.choices[0].message.content
)

generation_feedback(
    project="New Project",
    generation_id=generation_response.generation_id,
    feedback_json={
        "accepted": True
    }
)