> ## Documentation Index
> Fetch the complete documentation index at: https://docs.honeyhive.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluators

> Manage project-scoped evaluator definitions, immutable versions, publication and archival. Requires a fine-grained data plane API key.

## Schema introspection

Every command below with arguments supports two read-only flags for tooling and AI agents:

* `--show-file-schema`: print the JSON Schema for the full request object (the format `--filename` accepts).
* `--show-argument-schema <flag-name>`: print the JSON Schema for one argument's value. Pass the kebab flag name **without** the leading `--` (e.g. `project-id`, not `--project-id`).

Both write pure JSON to stdout and never call the API. They cannot be combined with any other command-specific flag.

## `list`

List evaluators

List active evaluators using their latest definitions. Archived evaluators are excluded. Requires project.evaluator.list.

### Usage

```sh theme={null}
honeyhive evaluators list [options]
```

**Accepted keys:** [Data plane fine-grained API key](https://docs.honeyhive.ai/v2/workspace/api-keys.md#key-types)

### Options

| Flag | Type | Required | Description |
| - | - | - | - |
| `--project-id` | string | yes | Project that owns the evaluator; must be within the API key root scope. |
| `--limit` | number | no | Maximum number of evaluators or versions per page. |
| `--page` | number | no | One-based page number. (page - 1) × limit must not exceed 2147483647. |

Also supports `--show-file-schema`, `--show-argument-schema <flag-name>`, and `--filename`. See [Schema introspection](#schema-introspection) for details.

## `create`

Create an evaluator

Create a project-scoped evaluator and immutable version 1. Slugs are unique within a project and cannot subsequently be changed. Requires project.evaluator.post; publish=true additionally requires project.evaluator.publish.

### Usage

```sh theme={null}
honeyhive evaluators create [options]
```

**Accepted keys:** [Data plane fine-grained API key](https://docs.honeyhive.ai/v2/workspace/api-keys.md#key-types)

### Options

| Flag | Type | Required | Description |
| - | - | - | - |
| `--definition` | json | yes | Evaluator definition with type python, llm, or human and the matching configuration object under that key. |
| `--project-id` | string | yes | Project that owns the evaluator; must be within the API key root scope. |
| `--slug` | string | yes | Immutable, case-sensitive project-unique slug containing 1–64 ASCII letters, digits, underscores or hyphens. Must not be a valid ULID, regardless of case. Cannot change after creation. |
| `--message` | string | no | Message for the initial version; defaults to an empty string. |
| `--publish` / `--no-publish` | boolean | no | Publish the saved version atomically. Requires project.evaluator.publish in addition to the operation permission. |

Also supports `--show-file-schema`, `--show-argument-schema <flag-name>`, and `--filename`. See [Schema introspection](#schema-introspection) for details.

## `get`

Get the latest evaluator version

Return the latest definition with current lifecycle state, including archived evaluators. Requires project.evaluator.get.

### Usage

```sh theme={null}
honeyhive evaluators get [options]
```

**Accepted keys:** [Data plane fine-grained API key](https://docs.honeyhive.ai/v2/workspace/api-keys.md#key-types)

### Options

| Flag | Type | Required | Description |
| - | - | - | - |
| `--evaluator-id-or-slug` | string | yes | Evaluator ULID or its immutable, case-sensitive project-unique slug. |
| `--project-id` | string | yes | Project that owns the evaluator; must be within the API key root scope. |

Also supports `--show-file-schema`, `--show-argument-schema <flag-name>`, and `--filename`. See [Schema introspection](#schema-introspection) for details.

## `replace`

Replace an evaluator definition

Replace the complete definition of an active evaluator. A changed definition creates the next server-assigned version; an identical definition retains its version and message. Requires project.evaluator.update; publish=true additionally requires project.evaluator.publish. Archived evaluators return 409.

### Usage

```sh theme={null}
honeyhive evaluators replace [options]
```

**Accepted keys:** [Data plane fine-grained API key](https://docs.honeyhive.ai/v2/workspace/api-keys.md#key-types)

### Options

| Flag | Type | Required | Description |
| - | - | - | - |
| `--definition` | json | yes | Evaluator definition with type python, llm, or human and the matching configuration object under that key. |
| `--evaluator-id-or-slug` | string | yes | Evaluator ULID or its immutable, case-sensitive project-unique slug. |
| `--project-id` | string | yes | Project that owns the evaluator; must be within the API key root scope. |
| `--message` | string | no | Message for a new version; defaults to an empty string. An unchanged definition retains its existing message. |
| `--publish` / `--no-publish` | boolean | no | Publish the saved version atomically. Requires project.evaluator.publish in addition to the operation permission. |

Also supports `--show-file-schema`, `--show-argument-schema <flag-name>`, and `--filename`. See [Schema introspection](#schema-introspection) for details.

## `archive`

Archive an evaluator

Archive and unpublish the evaluator, preserving its slug and immutable history. Repeated archival succeeds without changing its timestamps. Requires project.evaluator.delete.

### Usage

```sh theme={null}
honeyhive evaluators archive [options]
```

**Accepted keys:** [Data plane fine-grained API key](https://docs.honeyhive.ai/v2/workspace/api-keys.md#key-types)

### Options

| Flag | Type | Required | Description |
| - | - | - | - |
| `--evaluator-id-or-slug` | string | yes | Evaluator ULID or its immutable, case-sensitive project-unique slug. |
| `--project-id` | string | yes | Project that owns the evaluator; must be within the API key root scope. |

Also supports `--show-file-schema`, `--show-argument-schema <flag-name>`, and `--filename`. See [Schema introspection](#schema-introspection) for details.

## `list-versions`

List evaluator versions

List immutable definitions in descending version order, including archived evaluators. Each item includes current evaluator lifecycle state. Requires project.evaluator.list.

### Usage

```sh theme={null}
honeyhive evaluators list-versions [options]
```

**Accepted keys:** [Data plane fine-grained API key](https://docs.honeyhive.ai/v2/workspace/api-keys.md#key-types)

### Options

| Flag | Type | Required | Description |
| - | - | - | - |
| `--evaluator-id-or-slug` | string | yes | Evaluator ULID or its immutable, case-sensitive project-unique slug. |
| `--project-id` | string | yes | Project that owns the evaluator; must be within the API key root scope. |
| `--limit` | number | no | Maximum number of evaluators or versions per page. |
| `--page` | number | no | One-based page number. (page - 1) × limit must not exceed 2147483647. |

Also supports `--show-file-schema`, `--show-argument-schema <flag-name>`, and `--filename`. See [Schema introspection](#schema-introspection) for details.

## `get-version`

Get a specific evaluator version

Return the requested immutable definition with current evaluator lifecycle state. The selected version may differ from latest\_version and published\_version. Requires project.evaluator.get.

### Usage

```sh theme={null}
honeyhive evaluators get-version [options]
```

**Accepted keys:** [Data plane fine-grained API key](https://docs.honeyhive.ai/v2/workspace/api-keys.md#key-types)

### Options

| Flag | Type | Required | Description |
| - | - | - | - |
| `--evaluator-id-or-slug` | string | yes | Evaluator ULID or its immutable, case-sensitive project-unique slug. |
| `--project-id` | string | yes | Project that owns the evaluator; must be within the API key root scope. |
| `--resource-version` | string | yes | Server-assigned evaluator version. |

Also supports `--show-file-schema`, `--show-argument-schema <flag-name>`, and `--filename`. See [Schema introspection](#schema-introspection) for details.

## `publish`

Publish an evaluator version

Select an existing version for automatic evaluation. Repeated publication of the same version is a no-op. Archived evaluators return 409. Requires project.evaluator.publish.

### Usage

```sh theme={null}
honeyhive evaluators publish [options]
```

**Accepted keys:** [Data plane fine-grained API key](https://docs.honeyhive.ai/v2/workspace/api-keys.md#key-types)

### Options

| Flag | Type | Required | Description |
| - | - | - | - |
| `--evaluator-id-or-slug` | string | yes | Evaluator ULID or its immutable, case-sensitive project-unique slug. |
| `--project-id` | string | yes | Project that owns the evaluator; must be within the API key root scope. |
| `--resource-version` | string | yes | Server-assigned evaluator version. |

Also supports `--show-file-schema`, `--show-argument-schema <flag-name>`, and `--filename`. See [Schema introspection](#schema-introspection) for details.

## `unpublish`

Unpublish an evaluator

Clear the publication pointer without changing definitions or history. Already-unpublished and archived evaluators succeed. Requires project.evaluator.publish.

### Usage

```sh theme={null}
honeyhive evaluators unpublish [options]
```

**Accepted keys:** [Data plane fine-grained API key](https://docs.honeyhive.ai/v2/workspace/api-keys.md#key-types)

### Options

| Flag | Type | Required | Description |
| - | - | - | - |
| `--evaluator-id-or-slug` | string | yes | Evaluator ULID or its immutable, case-sensitive project-unique slug. |
| `--project-id` | string | yes | Project that owns the evaluator; must be within the API key root scope. |

Also supports `--show-file-schema`, `--show-argument-schema <flag-name>`, and `--filename`. See [Schema introspection](#schema-introspection) for details.


## Related topics

- [LLM Evaluators](https://docs.honeyhive.ai/v2/evaluators/llm.md)
- [Human Evaluators](https://docs.honeyhive.ai/v2/evaluators/human.md)
- [Python Evaluators](https://docs.honeyhive.ai/v2/evaluators/python.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.