Self-Hosting

​

chi-installation v1.10.1

• Fixed the restore job failing to list a backup’s tables when backup.image.tag is set to a clickhouse-backup 2.8 release. The job now works with both the default 2.6.33 and 2.8 releases.
• Fixed the restore job’s log including the clickhouse-backup sidecar’s API username and password when listing a backup’s tables failed. These credentials are now redacted.

​

chi-installation v1.10.0

• New restore.mode selects how the restore job treats existing tables. fill (default) never drops a table: it creates missing tables with the chart’s bootstrap scripts and attaches the backup’s data. Backups taken before and after the v1.8.1 session_aggregates change both restore into a current cluster, including a fresh cluster on ClickHouse 26.7 or later.
• restore.mode: replace drops the backup’s tables on every replica, after checking that all replicas are reachable, then restores as fill does. Any other value fails at render time.
• New restore.allowNonEmpty (default: false). In fill mode the job stops before changing anything if a table already holds rows. Set it to true only when the tables do not already hold rows from the backup; otherwise project_event_details, session_aggregates, and event_schemas are double-counted.
• Before changing anything, the restore job checks that every ClickHouse pod is reachable and that no clickhouse-backup sidecar runs in --watch mode (backup.schedule.watch, default: true). A watching sidecar blocks every restore action, so the job now fails immediately instead of waiting forever.
• The restore job restores only the default database, which holds the chart’s tables. It resumes a restore action already running on the sidecar instead of starting a second one, and logs per-table row counts while it runs.
• Fixed scheduled backups failing once new rows arrived after the v1.9.0 column conversion or after restoring an older backup. The conversion is metadata-only, so existing session_aggregates parts kept the old insertion_time type, and clickhouse-backup rejects a table whose parts disagree. The init script and the restore job now rewrite those parts with a replicated background mutation, and the restore job waits for it to finish on every replica.
• Fixed the ClickHouseInstallation failing to apply when neither server.service.labels nor server.service.annotations is set, because its CRD rejected the empty service metadata. If you added a placeholder service label or annotation to work around this, you can remove it.
• The restore job no longer drops and recreates existing tables by default. To keep the previous behavior, set restore.mode: replace.
• To run a restore: set backup.schedule.watch: false, upgrade, and wait for the ClickHouse pods to restart. Then set restore.enabled: true and, if needed, restore.backupName (empty restores the latest backup), restore.mode, or restore.allowNonEmpty, and upgrade. When the job completes, set restore.enabled: false and backup.schedule.watch: true, and upgrade.
• Action Required (only if your session_aggregates table predates v1.8.1 and a ClickHouse pod restarted on v1.9.0, or you restored an older backup on v1.9.0): scheduled backups keep failing until the old-type parts are rewritten. helm upgrade does not restart ClickHouse pods, so after the upgrade restart one ClickHouse pod per shard to start the rewrite.
• Otherwise, if you are not restoring, no action is required. Any remaining old-type session_aggregates parts are rewritten the next time a ClickHouse pod in each shard restarts.

​

hive-control-plane v1.12.0

• This release ships HoneyHive platform v1.18.0, up from v1.17.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.18.0. See the Product changelog for the platform changes.
• The cp-writer-service Deployment now sets OTEL_LOGS_EXPORTER from common.observability.otel.logsExporter (default: "none"), as cp-backend-service, cp-notification-service, and the control plane jobs already do. It renders only when common.observability.otel.enabled is true (default: true). With the default, writer logs are not exported.
• If you already set common.observability.otel.logsExporter: "otlp", the writer service also starts sending its logs to common.observability.otel.collectorEndpoint after the upgrade.

​

hive-data-plane v1.13.1

• This release ships HoneyHive platform v1.18.0, up from v1.17.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.18.0. See the Product changelog for the platform changes.
• No configuration or template migration is required. A standard helm upgrade rolls the pods that use the chart defaults.

​

chi-installation v1.9.0

• New optional PrometheusRule named clickhouse-alerts with three ClickHouse Keeper alerts. Set prometheus.prometheusRule.enabled: true to render it (default: false). It requires the Prometheus Operator CRDs. The resource carries the label release: <prometheus.prometheusRule.release> (default: monitoring), which must match the ruleSelector on your Prometheus.
• ClickHouseReadonlyReplica fires when a replica is read-only, usually because its Keeper session did not re-establish (defaults: 5m, critical).
• ClickHouseZooKeeperSessionLost fires when a pod that held a Keeper session in the last hour has none (defaults: 5m, critical).
• ClickHouseZooKeeperHardwareExceptions fires when Keeper connection errors rise by more than 5 in 10 minutes (defaults: 0m, warning).
• Configure each alert under prometheus.prometheusRule.rules.<name> with enabled, for, and severity. The names are readonlyReplica, zooKeeperSessionLost, and zooKeeperHardwareExceptions. The alerts overlap with Altinity’s upstream alert manifests, so enable only one set.
• Fixed a replica that starts on an empty volume crash-looping with Code: 122 when the cluster’s session_aggregates table predates v1.8.1. This covers a replica rebuilt after volume loss and a replica added by raising server.replica_count. The init script now creates the table with the insertion_time type that ClickHouse Keeper already records.
• The init script converts a pre-v1.8.1 insertion_time column to SimpleAggregateFunction(max, DateTime64(3)). The conversion runs once, is metadata-only, and replicates to the other replicas in the shard. If two replicas convert at the same time, one logs Code: 517 and continues to start.
• This corrects the v1.8.1 note, which said no action was required. Under v1.8.1, a replica that starts on an empty volume crash-loops in the case above. Upgrade to v1.9.0 before you rebuild or add replicas.
• Action Required (only if a replica already crash-loops with Code: 122): upgrade to v1.9.0, then delete that replica’s pod.
• helm upgrade does not restart ClickHouse pods. Existing tables convert the next time a ClickHouse pod in each shard restarts. Until then, a rebuilt or added replica still starts, because v1.9.0 creates the table with the column type ClickHouse Keeper records. To convert immediately, restart one ClickHouse pod per shard.

​

hive-control-plane v1.11.3

• This release ships HoneyHive platform v1.17.0, up from v1.16.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.17.0. See the Product changelog for the platform changes.
• The default nginx sidecar image tag common.nginxSidecar.image.tag changes from 1.27-alpine to 1.30-alpine (repository nginxinc/nginx-unprivileged). This affects only services with the sidecar enabled through common.nginxSidecar.enabled (default: false) or a per-service nginxSidecar.enabled. Their pods restart onto the new image during the upgrade.
• No configuration or template migration is required. A standard helm upgrade rolls the pods that use the chart defaults.

​

hive-data-plane v1.13.0

• This release ships HoneyHive platform v1.17.0, up from v1.16.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.17.0. See the Product changelog for the platform changes.
• The default nginx sidecar image tag common.nginxSidecar.image.tag changes from 1.27-alpine to 1.30-alpine (repository nginxinc/nginx-unprivileged). This affects only services with the sidecar enabled through common.nginxSidecar.enabled (default: false) or a per-service nginxSidecar.enabled. Their pods restart onto the new image during the upgrade.
• New opt-in jobs.sessionDeletion CronJob (session-deletion-dp) processes session deletion requests. Each run deletes the queued sessions’ blob store objects and Redis cache entries, has the control plane hide the sessions, and purges them from ClickHouse once a day. Runs exit quickly when the queue is empty. It renders only when both jobs.enabled and jobs.sessionDeletion.enabled are true, and both default to false. It runs on the shared jobs.image with concurrencyPolicy: Forbid and uses jobs.serviceAccount.
• jobs.sessionDeletion.schedule defaults to "*/15 * * * *". Run it at least hourly: the ClickHouse purge runs during jobs.sessionDeletion.purgeHourUtc (UTC hour 0..23, job default 8), and a slower schedule delays the purge until the job’s overdue catch-up runs it outside that hour.
• jobs.sessionDeletion.jobTimeoutMs defaults to "900000". It must stay below jobs.sessionDeletion.staleClaimMs (job default 1800000), or the job refuses to start. Keep it above 600000 (10 minutes) to leave room for the job’s worst-case startup time.
• Other settings and defaults: jobs.sessionDeletion.activeDeadlineSeconds (1200), backoffLimit (0), startingDeadlineSeconds (600), batchSize (job default 100, range 1..1000), objectConcurrency (job default 8), and env ([], extra env vars merged after the built-in env).
• When the job is enabled, rendering fails if any of jobs.image.repository, common.region, common.blobStore.bucket, common.controlPlane.apiPublicUrl, or common.encryption.keyId is unset. Rendering also fails unless activeDeadlineSeconds × 1000 exceeds (backoffLimit + 1) × jobTimeoutMs + 60000.
• The dp-backend-service Deployment now sets SESSION_DELETION_ENABLED: "true" only when both jobs.enabled and jobs.sessionDeletion.enabled are true, and "false" otherwise. With the defaults, the data plane API rejects new session deletion requests with a 501 error (Session deletion is not enabled on this data plane).
• To serve session deletion requests: in the IAM role set in jobs.serviceAccount.roleArn (set one if the value is empty), grant s3:DeleteObject on the blob store bucket, and allow the role to use the common.encryption.keyId key if you use KMS encryption. Make sure the required values above are set, then set both jobs.enabled: true and jobs.sessionDeletion.enabled: true and run helm upgrade.

​

hive-control-plane v1.11.2

• This release ships HoneyHive platform v1.16.0, up from v1.15.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.16.0. See the Product changelog for the platform changes.
• No configuration or template migration is required. A standard helm upgrade rolls the pods that use the chart defaults.

​

hive-data-plane v1.12.0

• This release ships HoneyHive platform v1.16.0, up from v1.15.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.16.0. See the Product changelog for the platform changes, including Python evaluators now running on Python 3.14.
• New optional dpLlmproxyService.bedrockApplicationInferenceProfiles map (default: {}) routes Bedrock models through AWS Bedrock application inference profiles. Use it when your AWS account allows Bedrock calls only through application inference profiles. Keys are Bedrock catalog model IDs (for example, anthropic.claude-sonnet-4-5-20250929-v1:0), and values are application inference profile ARNs (arn:aws:bedrock:<region>:<account-id>:application-inference-profile/<profile-id>).
• The map takes effect only when dp-llmproxy-service runs with DISABLE_SECRETS=true, set through dpLlmproxyService.env. That setting applies to every provider: the service stops using provider keys saved in HoneyHive and calls Bedrock with the deployment’s own AWS credentials. Don’t turn it on just for this map if your workspaces rely on saved provider keys.
• Once the map has at least one valid entry, requests for Bedrock models missing from it fail with a 400 error (application_inference_profile_not_configured). Entries whose value is not an application inference profile ARN are ignored, and the service logs a warning.
• Requests that use a profile go to the AWS region in the profile ARN, which overrides common.region and dpLlmproxyService.assumedRole.region.
• With the default empty map, the rendered Deployment and the service’s behavior are unchanged. No configuration or template migration is required.

​

hive-control-plane v1.11.1

• This release ships HoneyHive platform v1.15.0, up from v1.14.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.15.0. See the Product changelog for the platform changes.
• The top-level license block accepts a new optional license.license_policy.controlplane_admin_sso_group. Set it to an identity provider group name to grant control plane admin access to that group’s members. It is matched against the claim named by license.license_policy.sso_group_field and replaces the built-in admin entitlement groups. It requires license.license_policy.sso_group_scheme: entitlement-roles-2026-8 and cannot be combined with license.license_policy.controlplane_admin_emails. A license that breaks either rule stops cp-backend-service from starting. Licenses that set neither field are unaffected.
• No template migration is required.
• Action Required (only if your license sets license.license_policy.sso_group_scheme: entitlement-roles-2026-8 and license.license_policy.controlplane_admin_emails): review controlplane_admin_emails before you upgrade and remove stale entries. After the upgrade, every listed address and wildcard pattern keeps control plane admin access after sign-in.

​

hive-data-plane v1.11.1

• This release ships HoneyHive platform v1.15.0, up from v1.14.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.15.0. See the Product changelog for the platform changes.
• No configuration or template migration is required. A standard helm upgrade rolls the pods that use the chart defaults.

​

hive-control-plane v1.11.0

• This release ships HoneyHive platform v1.14.0, up from v1.13.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.14.0. See the Product changelog for the platform changes.
• New opt-in jobs.postgresCleanup CronJob deletes expired auth sessions from the control-plane Postgres database. It renders only when both jobs.enabled (default: true) and jobs.postgresCleanup.enabled (default: false) are true, so nothing new is deployed unless you turn it on. It runs on the shared jobs.image with concurrencyPolicy: Forbid and deletes in batches.
• The schedule, retention, batching, and dryRun settings, with their defaults, are under jobs.postgresCleanup in hive-control-plane/values.yaml.
• When the job metrics exporter is enabled (jobs.statsdExporter.enabled), it also exposes the cleanup job’s hh_postgres_cleanup_task_duration_seconds histogram.
• Graceful shutdown for every control-plane Deployment (cp-backend-service, cp-controller-service, cp-frontend-service, cp-notification-service, cp-writer-service): each application container now has a preStop hook that waits 10 seconds before the container receives SIGTERM, so the pod is removed from Service endpoints and ingress before it stops accepting connections. When the nginx sidecar is enabled (common.nginxSidecar.enabled or <service>.nginxSidecar.enabled, default: false), the sidecar gets the same 10-second wait.
• The pods also set terminationGracePeriodSeconds explicitly: 40 for cp-writer-service and 30 (the Kubernetes default) for the others. These values are fixed in the templates and are not configurable. Pods take about 10 seconds longer to terminate during rollouts, scale-downs, and node drains.
• Action Required (only if your cluster runs Kubernetes older than 1.30): upgrade the cluster to Kubernetes 1.30 or later before you upgrade to this chart version. The preStop hooks use the native Kubernetes sleep lifecycle action, which older API servers reject, so the Deployments fail to apply.

​

hive-data-plane v1.11.0

• This release ships HoneyHive platform v1.14.0, up from v1.13.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.14.0. See the Product changelog for the platform changes.
• Graceful shutdown for every data-plane Deployment (dp-backend-service, dp-controller-service, dp-evaluation-service, dp-ingestion-service, dp-llmproxy-service, dp-pythonmetric-service): each application container now has a preStop hook that waits 10 seconds before the container receives SIGTERM, so the pod is removed from Service endpoints and ingress before it stops accepting connections. When the nginx sidecar is enabled (common.nginxSidecar.enabled or <service>.nginxSidecar.enabled, default: false), the sidecar gets the same 10-second wait.
• The pods also set terminationGracePeriodSeconds explicitly: 190 for dp-evaluation-service and dp-llmproxy-service, 70 for dp-ingestion-service, and 30 (the Kubernetes default) for the others. For dp-llmproxy-service, 190 seconds covers the 10-second wait plus the 180-second LLM request timeout, so rollouts no longer cut off in-flight LLM requests. These values are fixed in the templates and are not configurable. Pods take at least 10 seconds longer to terminate during rollouts, scale-downs, and node drains, and up to the full grace period while they finish in-flight requests.
• When the nginx sidecar is enabled for dp-llmproxy-service, its proxy_read_timeout and proxy_send_timeout are now fixed at 180s to match the LLM request timeout. common.nginxSidecar.proxyReadTimeout and common.nginxSidecar.proxySendTimeout (default: 300s) no longer apply to dp-llmproxy-service, but still apply to every other data-plane sidecar.
• Action Required (only if your cluster runs Kubernetes older than 1.30): upgrade the cluster to Kubernetes 1.30 or later before you upgrade to this chart version. The preStop hooks use the native Kubernetes sleep lifecycle action, which older API servers reject, so the Deployments fail to apply.

​

hive-control-plane v1.10.3

• This release ships HoneyHive platform v1.13.0, up from v1.12.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.13.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• No configuration or template migration is required. A standard helm upgrade rolls the pods that use the chart defaults.

​

hive-data-plane v1.10.0

• This release ships HoneyHive platform v1.13.0, up from v1.12.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.13.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• No configuration or template migration is required. A standard helm upgrade rolls the pods that use the chart defaults.

​

hive-control-plane v1.10.2

• This release ships HoneyHive platform v1.12.0, up from v1.11.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.12.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• No configuration or template migration is required. A standard helm upgrade rolls the pods that use the chart defaults.

​

hive-data-plane v1.9.3

• This release ships HoneyHive platform v1.12.0, up from v1.11.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.12.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• No configuration or template migration is required. A standard helm upgrade rolls the pods that use the chart defaults.

​

hive-control-plane v1.10.1

• This release ships HoneyHive platform v1.11.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.11.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• cp-backend-service now always reads its license from a file. The license ConfigMap (hive-control-plane-license-config, or <nameOverride>-license-config if you set nameOverride) is rendered on every install from the top-level license block, mounted read-only at /var/hh, and the container always sets LICENSE_PATH=/var/hh/license.yaml. Previously the ConfigMap, its volume, and the mount were rendered only when license was non-empty, and cp-backend-service otherwise received a LICENSE environment variable taken from common.license.
• license in values.yaml now ships a complete, renderable example instead of an entirely commented-out block: title, version, description, license_id, license_type: selfhost, and license_policy: {} (with license_policy.controlplane_admin_emails shown as a commented example). The shipped values are placeholders so a default render succeeds; replace the whole block, in particular license_id, with the license HoneyHive issued for your deployment before you install or upgrade.
• common.license is no longer read by any template, so setting it has no effect. The key is still accepted so existing values files keep rendering, and it will be removed in a future release.
• Every PodDisruptionBudget the chart renders (cp-backend, cp-controller, cp-frontend, cp-notification, and cp-writer, each behind <service>.pdb.enabled, default: false) now sets unhealthyPodEvictionPolicy: AlwaysAllow. Pods that are running but not Ready can then be evicted even while the budget is already at its limit, so a node drain or cluster upgrade is no longer blocked by a pod that is itself unhealthy. Healthy pods stay protected by pdb.minAvailable / pdb.maxUnavailable exactly as before. Kubernetes 1.27 and later honor the field; earlier API servers ignore it. No values change is required.
• Job and CronJob containers (both alert-transition CronJobs, the post-deploy alert-transition Job, and the legacy federated migration Job) now set HOST_IP from the pod’s status.hostIP whenever common.observability.otel.enabled is true (the default), declared ahead of OTEL_EXPORTER_OTLP_ENDPOINT. This makes it possible to point common.observability.otel.collectorEndpoint (default: http://otel-collector.opentelemetry-system.svc.cluster.local:4317) at a node-local collector with http://$(HOST_IP):4317. Kubernetes substitutes the reference only when HOST_IP is defined earlier in the same container, so such an endpoint previously reached these jobs as a literal string and their telemetry was dropped. Endpoints that do not use $(HOST_IP) are unaffected. Service Deployments are unchanged and do not define HOST_IP.
• Action Required (unless you already populate the top-level license block): before upgrading, put the license HoneyHive issued for your deployment in the top-level license block (title, version, description, license_id, license_type, and any license_policy entries such as controlplane_admin_emails), and delete common.license from your values, which is now ignored. The license block ships placeholder content rather than being empty, so an upgrade that leaves it unset starts cp-backend-service against the placeholder instead of failing: the control plane comes up, but under a placeholder license_id and with no controlplane_admin_emails, so no user is granted controlplane admin permissions and nothing reports an error. Check this explicitly rather than waiting for a failure.

​

hive-data-plane v1.9.2

• This release ships HoneyHive platform v1.11.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.11.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• Every PodDisruptionBudget the chart renders (dp-backend, dp-controller, dp-evaluation, dp-ingestion, dp-llmproxy, and dp-pythonmetric, each behind <service>.pdb.enabled, default: false, except dpControllerService.pdb.enabled, which is true) now sets unhealthyPodEvictionPolicy: AlwaysAllow. Pods that are running but not Ready can then be evicted even while the budget is already at its limit, so a node drain or cluster upgrade is no longer blocked by a pod that is itself unhealthy. Healthy pods stay protected by pdb.minAvailable / pdb.maxUnavailable exactly as before. Kubernetes 1.27 and later honor the field; earlier API servers ignore it. No values change is required.
• Job containers (the S3 time-window backfill Job, jobs.s3TimeWindowBackfill.enabled, and the legacy federated migration Job, jobs.legacyFederatedMigration.enabled) now set HOST_IP from the pod’s status.hostIP whenever common.observability.otel.enabled is true (the default), declared ahead of OTEL_EXPORTER_OTLP_ENDPOINT. This makes it possible to point common.observability.otel.collectorEndpoint (default: otel-collector.opentelemetry-system.svc.cluster.local:4317) at a node-local collector with $(HOST_IP):4317. Kubernetes substitutes the reference only when HOST_IP is defined earlier in the same container, so such an endpoint previously reached these jobs as a literal string and their telemetry was dropped. Endpoints that do not use $(HOST_IP) are unaffected. Service Deployments are unchanged and do not define HOST_IP.

​

hive-control-plane v1.10.0

• This release ships HoneyHive platform v1.10.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.10.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• New cpFrontendService.providersConfigMap serves your own AI provider catalog in place of the one bundled in the frontend image. Use it to restrict which providers and models appear in the UI, or to add a provider the bundled catalog omits.
• New cpFrontendService.providersConfigMap.enabled (default: false). When false, nothing is rendered and the bundled catalog is served, so existing installs are unchanged.
• New cpFrontendService.providersConfigMap.providers (default: []) holds provider entries in the same schema as the bundled catalog. An empty list falls back to the bundled catalog.
• When enabled, the chart renders a ConfigMap named <cpFrontendService.name>-providers-config (default: cp-frontend-service-providers-config) and mounts it read-only into the frontend container as providers-config.yaml under the app’s public directory. The frontend prefers that file over the bundled catalog.
• The frontend Deployment gains a checksum/providers-config pod annotation derived from cpFrontendService.providersConfigMap, so editing the catalog rolls the frontend pods. This is required: the mount is a subPath, which the kubelet never refreshes in place, and the frontend reads its public file list once at startup.
• Setting providers while leaving enabled: false now fails rendering with an explicit message instead of silently discarding your catalog.
• Two things to know when authoring entries. A provider whose credentials come from the pod’s IAM role rather than from stored secrets must mark those secret keys required: false, or the UI treats the provider as unconfigured and hides its models. The catalog is served from the frontend’s public directory, so name credential keys only, never credential values, which operators enter in the UI and which are stored server-side.

​

hive-data-plane v1.9.1

• This release ships HoneyHive platform v1.10.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.10.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• This release makes it possible to run two data planes in one Kubernetes cluster. Values that previously hard-coded a single plane’s identity are now configurable, and the objects the chart renders now all land in global.namespace. Single-plane installs that leave global.namespace at its default see no change beyond the image pin.
• New serviceMonitor.namePrefix (default: "") is prepended to the name and app label of every ServiceMonitor the chart renders. ServiceMonitors are created in the shared serviceMonitor.namespace (default: monitoring) rather than in the plane’s own namespace, so their names are the only thing separating two planes. Leave it empty for a single plane; give each additional plane its own prefix.
• New ingress.alb.groupName (default: data-plane-alb, the previously hard-coded value) sets the alb.ingress.kubernetes.io/group.name annotation on every data-plane ALB Ingress. The AWS Load Balancer Controller treats this name as cluster-wide, so two planes sharing it merge into one ALB. A distinct value provisions a separate ALB with its own DNS name, which then needs its own DNS record and its own ACM certificate; neither is created by this chart. The per-Ingress group.order values are unchanged; they order rules within a group, and separate groups do not contend.
• Every common.serviceUrls.* entry (dpBackendService, ingestionService, evaluationService, pythonMetricService, llmProxyService, dpController) now defaults to "" instead of a literal http://<service>.data-plane.svc.cluster.local:<port> URL. When an entry is empty, the chart derives the URL from that service’s own name, port, and global.namespace, so inter-service traffic follows the namespace you install into. A non-empty entry still wins verbatim, which is what mesh gateways and egress proxies rely on. If you set these values explicitly, note that they are namespace-specific: reusing them for a second plane points that plane’s traffic at the first one without any error.
• Every object the chart renders now sets metadata.namespace from global.namespace (default: data-plane) rather than inheriting the release namespace. This affects the dp-backend, dp-llmproxy, and dp-pythonmetric ServiceAccounts and the JetStream evaluation-stream Stream and evaluation-consumer Consumer; other objects already set it. global.namespace now decides on its own where an install lands.
• dpControllerService.pdb now renders maxUnavailable: 1 instead of minAvailable: 1. At the default dpControllerService.replicas: 1, minAvailable: 1 permitted zero voluntary disruptions, so the API rejected evictions of the controller pod and node drains (cluster upgrades, node consolidation, node-termination handling) never completed. maxUnavailable: 1 accepts a brief controller outage while the pod reschedules, and becomes a real budget once the service runs multiple replicas. minAvailable is commented out in values.yaml; it takes precedence over maxUnavailable when set, so leave it unset while replicas is 1.
• common.dataplane.id (default: dp-prod), common.blobStore.bucket, common.encryption.keyId, and ingress.tls.hosts are per-plane values, not per-environment ones. A second plane that inherits them registers with the control plane as the first plane, writes to the first plane’s bucket, encrypts under the first plane’s key, or requests the first plane’s certificate, none of which collide at the object level or report unhealthy. Only the documentation changed here; the defaults are the same.
• Fixed ServiceAccounts rendering the eks.amazonaws.com/role-arn annotation with a null value when the corresponding <service>.serviceAccount.roleArn is empty. The annotation is now omitted entirely, so leaving roleArn unset, for example on a plane that does not use IAM Roles for Service Accounts, produces a clean ServiceAccount.
• Action Required (only if your release namespace differs from global.namespace): before upgrading, reconcile the two, either by installing into the namespace named by global.namespace or by setting global.namespace to your current release namespace. The JetStream evaluation-stream Stream and evaluation-consumer Consumer previously inherited the release namespace and now render into global.namespace. Helm identifies a resource by kind, namespace, and name, so it deletes the old objects and creates new ones; the NATS controller’s finalizer removes the underlying JetStream stream and its durable consumer as part of that deletion, dropping in-flight evaluation messages. Nothing fails loudly if you skip this, so check it explicitly. Installs where the two already match, the normal case, are unaffected.

​

chi-installation v1.8.1

• Fixed fresh installs with empty ClickHouse volumes crash-looping on ClickHouse 26.7 and later. Creating the session_aggregates table failed with Code: 36 (BAD_ARGUMENTS), so the ClickHouse pods never started. New installations now get the corrected table schema automatically.
• Only affects deployments that override the chart’s default ClickHouse image to 26.7 or later. Clusters on the chart defaults (server.image.tag and restore.image.tag, both 25.7.3.13) were never affected.
• No migration required. Existing deployments keep the previous table definition after helm upgrade with no impact, and the table continues to read and write normally if you later upgrade the ClickHouse image to 26.7 or later.

​

hive-control-plane v1.9.0

• This release ships HoneyHive platform v1.9.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.9.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• New common.containerSecurityContext applies a hardened container securityContext to every HoneyHive service, init, and job container. Defaults: allowPrivilegeEscalation: false, runAsNonRoot: true, runAsUser: 65532, runAsGroup: 65532, readOnlyRootFilesystem: true, capabilities.drop: [ALL], and seccompProfile.type: RuntimeDefault. The HoneyHive images run as UID 65532 and write nothing to their own filesystem, so no scratch volume is needed (cp-frontend-service’s Next.js cache already has its own emptyDir).
• New per-workload overrides via <service>.containerSecurityContext, for example cpBackendService.containerSecurityContext, cpFrontendService.containerSecurityContext, jobs.containerSecurityContext, and jobs.statsdExporter.containerSecurityContext. Keys merge one at a time and the per-service value wins, so you can relax a single field (for example readOnlyRootFilesystem: false) without restating the whole map. Setting common.containerSecurityContext: {} with no per-service override renders no container securityContext at all.
• Every ServiceAccount the chart creates (cp-backend, cp-controller, cp-frontend, cp-notification, cp-writer, and the shared jobs ServiceAccount) now sets automountServiceAccountToken: false, and the statsd exporter Deployment sets the same field on its pod spec. Pods no longer receive the default Kubernetes API credential at /var/run/secrets/kubernetes.io/serviceaccount. No HoneyHive workload calls the Kubernetes API, and IAM Roles for Service Accounts is unaffected because the EKS webhook injects its own projected token volume.
• Every workload now sets both AWS_REGION and AWS_DEFAULT_REGION from common.region (default: us-west-2), instead of only some containers setting AWS_REGION. This stops AWS SDKs, in particular Python’s botocore/boto3, which ignores AWS_REGION, from falling back to the instance metadata service to discover a region. If common.region is overridden to an empty string, neither variable is rendered at all (previously an empty AWS_REGION was rendered on some workloads and treated by the SDKs as a real, invalid region), which leaves the SDKs’ normal region resolution intact.
• The nginx sidecar security context (common.nginxSidecar.securityContext) now also sets seccompProfile.type: RuntimeDefault.
• Fixed alert notification emails omitting their dashboard, alert, and alert-settings links. cp-notification-service now receives FRONTEND_PUBLIC_URL from common.auth.frontendPublicUrl, which is already required for login (default: ""). If it is unset, emails still send but their links remain omitted.
• Action Required (only if you have added a sidecar or a mutating webhook to these pods that relies on the automounted Kubernetes API token): mount it explicitly with a serviceAccountToken projected volume.

​

hive-data-plane v1.9.0

• This release ships HoneyHive platform v1.9.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.9.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• New common.containerSecurityContext applies a hardened container securityContext to every HoneyHive service, init, and job container. Defaults: allowPrivilegeEscalation: false, runAsNonRoot: true, runAsUser: 65532, runAsGroup: 65532, readOnlyRootFilesystem: true, capabilities.drop: [ALL], and seccompProfile.type: RuntimeDefault. The HoneyHive images run as UID 65532 and write nothing to their own filesystem, with one exception handled by the chart: dp-pythonmetric-service now mounts an emptyDir at /tmp for gunicorn’s worker heartbeat files and the Prometheus multiprocess registry.
• New per-workload overrides via <service>.containerSecurityContext, for example dpBackendService.containerSecurityContext, dpPythonmetricService.containerSecurityContext, and jobs.containerSecurityContext (covers the S3 time-window backfill Job and the legacy federated migration Job). The prepare-certs init container inherits the security context of the service it runs in, so override it under dpLlmproxyService.containerSecurityContext or dpPythonmetricService.containerSecurityContext. Per-service keys merge into the global map one key at a time and win on conflict, so you can relax a single field (for example readOnlyRootFilesystem: false) without restating the whole map. Setting common.containerSecurityContext: {} with no per-service override renders no container securityContext at all.
• Every ServiceAccount the chart creates (dp-backend, dp-controller, dp-evaluation, dp-ingestion, dp-llmproxy, dp-pythonmetric, and the shared jobs ServiceAccount) now sets automountServiceAccountToken: false. Pods no longer receive the default Kubernetes API credential at /var/run/secrets/kubernetes.io/serviceaccount. No HoneyHive workload calls the Kubernetes API, and IAM Roles for Service Accounts is unaffected because the EKS webhook injects its own projected token volume.
• Every workload, including all jobs, now sets both AWS_REGION and AWS_DEFAULT_REGION from common.region (default: us-west-2), instead of only some containers setting AWS_REGION. This stops AWS SDKs, in particular Python’s botocore/boto3, which ignores AWS_REGION, from falling back to the instance metadata service to discover a region. If common.region is overridden to an empty string, neither variable is rendered at all (previously an empty AWS_REGION was rendered on some workloads and treated by the SDKs as a real, invalid region), which leaves the SDKs’ normal region resolution intact.
• dp-pythonmetric-service no longer receives the Postgres connection URL, the REDIS_MASTER_HOST / REDIS_REPLICAS_HOST / REDIS_MASTER_PORT / REDIS_REPLICAS_PORT / REDIS_PASSWORD variables, or the HH_ENCRYPTION_KEY_ID / HH_ENCRYPTION_SECRET variables. The service reads none of them, so it no longer depends on the Secrets named by common.externalSecrets.postgres, common.externalSecrets.redis, and common.externalSecrets.encryption; its pods now start even when those Secrets are absent. Other data-plane services still consume all three, so keep common.externalSecrets.*, redis.*, and common.encryption.* configured as before. No values change is required.
• The nginx sidecar security context (common.nginxSidecar.securityContext) now also sets seccompProfile.type: RuntimeDefault.
• Fixed the prepare-certs init container that builds the merged CA bundle for dp-llmproxy-service and dp-pythonmetric-service when common.tls.caCerts is set. It no longer requires a shell or cat in the service image, so it works on shell-less (distroless) images where it previously failed immediately and blocked the pod from starting. It builds the same /certs/ca-bundle.pem (the image’s system trust store followed by each *.crt from your common.tls.caCerts ConfigMap, consumed via SSL_CERT_FILE and REQUESTS_CA_BUNDLE) and now fails with a clear error if the image’s system CA bundle is missing rather than emitting a bundle without it. The init container also declares resources (requests: 10m CPU / 32Mi memory; limits: 50m CPU / 64Mi memory). Kubernetes schedules on the larger of the init container’s request and the app containers’ combined request, so at the chart’s default service requests your pods’ scheduling footprint does not change.
• Action Required (only if you have added a sidecar or a mutating webhook to these pods that relies on the automounted Kubernetes API token): mount it explicitly with a serviceAccountToken projected volume.

​

hive-control-plane v1.8.0

• This release ships HoneyHive platform v1.8.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.8.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• New common.auth.scopes (unset by default) for additional space-separated OAuth scopes requested at login, rendered as the AUTH_SCOPES env var on the frontend Deployment. openid email profile is always requested, so list only extra scopes. Leaving the key unset renders no AUTH_SCOPES variable and preserves existing behavior.
• Identity providers that scope access tokens per resource (for example Microsoft Entra ID) must also request the scope of the API exposed for this deployment; otherwise the access token is issued for the provider’s own API and fails verification. For Entra ID, set common.auth.scopes: "api://<client-id>/access_as_user".
• When ingress.tls.enabled: true, the HTTP and gRPC ALB Ingress resources now carry the alb.ingress.kubernetes.io/ssl-policy annotation taken from ingress.tls.sslPolicy (default: ELBSecurityPolicy-TLS13-1-2-2021-06). These two listeners previously fell back to the AWS Load Balancer Controller’s default policy, so the negotiated TLS versions and ciphers may change on upgrade.
• Every service and job container now sets HH_SERVICE_NAME to its service name, so structured logs keep their service name even when common.observability.otel.enabled: false. When OpenTelemetry is enabled, OTEL_SERVICE_NAME still takes precedence. No values change is required.
• Fixed the wait-for-cp-backend init container of the cron-alert-transitions-long-range-on-deploy Job. On job images without a shell, the init container failed immediately and the post-deploy job never ran. It now polls http://<cpBackendService.name>:<cpBackendService.port>/healthcheck for up to 20 attempts with a 5s request timeout and 10s between attempts (about 290s).
• Action Required (only if you override ingress.tls.sslPolicy to an empty string): set it to a non-empty AWS policy name. With TLS enabled, an empty value now fails rendering with ingress.tls.sslPolicy is required when ingress.tls.enabled is true.
• Action Required (only if you raise jobs.alertTransitionsLongRange.jobTimeoutMs, default: 600000): raise jobs.alertTransitionsLongRange.activeDeadlineSeconds (default: 900) by at least as much. The deadline runs from the Job’s start time and covers the backend wait; exceeding it fails the Job with no retry.

​

hive-data-plane v1.8.0

• This release ships HoneyHive platform v1.8.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.8.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• When ingress.tls.enabled: true, every ALB Ingress rendered by the chart now carries the alb.ingress.kubernetes.io/ssl-policy annotation taken from ingress.tls.sslPolicy (default: ELBSecurityPolicy-TLS13-1-2-2021-06). These listeners previously fell back to the AWS Load Balancer Controller’s default policy, so the negotiated TLS versions and ciphers may change on upgrade.
• Every service and job container now sets HH_SERVICE_NAME to its service name, so structured logs keep their service name even when common.observability.otel.enabled: false. When OpenTelemetry is enabled, OTEL_SERVICE_NAME still takes precedence. No values change is required.
• Removed the Sentry Secret and the secrets.sentry block (secrets.sentry.enabled, secrets.sentry.dsns.*) from hive-data-plane/values.yaml. The Secret was disabled by default and its contents were never consumed by any data-plane workload, so no data-plane service loses configuration. If you still set secrets.sentry.* in your own values, the keys are now ignored and can be deleted.
• hive-data-plane/values.yaml now documents supported keys that previously existed only in the templates, as commented-out entries with their defaults. Rendering behavior is unchanged; these keys already worked. Newly documented: nameOverride, fullnameOverride, global.labels, global.org, global.env, global.sregion, global.deployment, common.image.pullSecret, <service>.pdb.maxUnavailable, dpEvaluationService.resources, dpLlmproxyService.assumedRole (roleArn, roleSessionName, externalId, region), config, and secrets.postgres.useExternalSecret / host / port / database / username.
• Action Required (only if you override ingress.tls.sslPolicy to an empty string): set it to a non-empty AWS policy name. With TLS enabled, an empty value now fails rendering with ingress.tls.sslPolicy is required when ingress.tls.enabled is true.

​

chi-installation v1.8.0

• New opt-in coordination-aware readiness probe for the ClickHouse server container, configured under server.readinessProbe in chi-installation/values.yaml. When server.readinessProbe.useReplicaCheck is true, the chart sets an exec readiness probe that queries system.replicas and marks the pod unready while any replicated table in an application database is read-only or has lost its ClickHouse Keeper session. This catches degraded replicas that still answer /ping with HTTP 200, so traffic is withheld until coordination recovers. ClickHouse’s own system, information_schema, and INFORMATION_SCHEMA databases are excluded from the check.
• New probe timing keys, applied only when server.readinessProbe.useReplicaCheck is enabled: server.readinessProbe.initialDelaySeconds (default: 30), server.readinessProbe.periodSeconds (default: 10), server.readinessProbe.timeoutSeconds (default: 5), server.readinessProbe.failureThreshold (default: 3), and server.readinessProbe.successThreshold (default: 1).
• No default behavior change: server.readinessProbe.useReplicaCheck defaults to false, in which case the chart sets no readinessProbe on the ClickHouse container and the ClickHouse Operator continues to inject its own default /ping-based probe. Disabling the flag reverts to that operator default rather than turning readiness checking off, so a standard helm upgrade requires no action.
• Enabling the probe changes the ClickHouse container’s pod spec, so the ClickHouse pods roll on the next helm upgrade. Schedule the upgrade accordingly. A pod that restores a backup fails the replica check until the restore completes, so it stays unready during the restore. Take this into account on clusters that perform in-place restores.

​

hive-control-plane v1.7.2

• This release ships HoneyHive platform v1.7.0. The chart’s appVersion and default HoneyHive workload image tags in hive-control-plane/values.yaml are now v1.7.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• A standard helm upgrade rolls the pods that use the chart defaults. No configuration or template migration is required.

​

hive-data-plane v1.7.2

• This release ships HoneyHive platform v1.7.0. The chart’s appVersion and default HoneyHive workload image tags in hive-data-plane/values.yaml are now v1.7.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• A standard helm upgrade rolls the pods that use the chart defaults. No configuration or template migration is required.

​

v1.6.3

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts (values.yaml, templates/, and Chart.yaml) are unchanged, so a standard helm upgrade requires no action.

​

v1.6.2

• New custom annotations on service Deployment metadata in hive-control-plane and hive-data-plane. Set common.deploymentAnnotations (default: {}) to apply a map of annotations to every service Deployment’s metadata.annotations, or <service>.deploymentAnnotations (default: {}) for a single service, which merges over the common map with per-service keys winning on conflict. Useful for GitOps controllers, config-reloaders, cost-allocation, and similar Deployment-level integrations. Both default empty and render identically to before when unset, so a standard helm upgrade requires no action.
• New per-service resource overrides in hive-control-plane/values.yaml and hive-data-plane/values.yaml. Each service can now set its own resources block (CPU/memory requests and limits) instead of sharing common.resources. Newly configurable: dpBackendService.resources, cpBackendService.resources, cpFrontendService.resources, cpWriterService.resources, and cpNotificationService.resources (each default: {}, falling back to common.resources). Set any in your values overlay to override only that service, e.g. dpBackendService.resources.limits.memory: 4Gi. No action required for a standard upgrade.
• New file-based license configuration in hive-control-plane. Set the structured license document issued by HoneyHive under the top-level license key (default: empty, feature off when unset). When set, the chart renders it into a new hive-control-plane-license-config ConfigMap, mounts it into the cp-backend pod at /var/hh/license.yaml, and points the backend at it via the LICENSE_PATH env var. See the annotated example block in hive-control-plane/values.yaml for the full structure (title, version, license_id, license_type, and license_policy holding controlplane_admin_emails and allowed identity_providers).
• Deprecated common.license (the inline key injected as the LICENSE env var) in favor of license. It still works, but only as a fallback when license is unset, and will be removed in a future release. Recommended (not required): move your license into the new license block and remove common.license. While license is set it takes precedence and common.license is ignored; if license is left empty the chart falls back to common.license, so existing deployments keep working until you migrate.

​

v1.6.0

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts (values.yaml, templates/, and Chart.yaml) are unchanged, so a standard helm upgrade requires no action.

​

v1.5.0

• Documented dozens of previously-undeclared configuration knobs in hive-control-plane/values.yaml (name overrides, global.*, common.*, cp-controller tuning, per-service PDB alternates, frontend TLS policy, and a legacy config block). These are surfaced as commented, render-neutral documentation; merged values are unchanged, so a standard helm upgrade requires no action.
• Fixed the cp-controller PodDisruptionBudget so setting cpControllerService.pdb.minAvailable or cpControllerService.pdb.maxUnavailable to null now omits the field instead of rendering an invalid empty value. maxUnavailable: 0 still works, and default values render identically to v1.4.1.
• Removed a dead beekeeper Sentry DSN block from the control-plane secrets template (the beekeeper service was retired). Template-only change, applied automatically on helm upgrade.

​

v1.4.1

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts are unchanged, so a standard helm upgrade requires no action.

​

v1.4.0

• New optional generic ALB backend Service overrides in hive-control-plane/values.yaml and hive-data-plane/values.yaml, letting you route individual ALB backends to a mesh gateway or customer-managed reverse proxy Service instead of the HoneyHive app Services. All disabled by default, so no action is required for a standard upgrade.
• Control-plane override keys: ingress.backendServices.http.* (HTTP backend), ingress.backendServices.grpc.* (gRPC backend), and frontendIngress.backendService.* (frontend backend), each with enabled, name, port, protocol, and healthcheck.* fields.
• Data-plane override keys: ingress.backendServices.ingestion.*, ingress.backendServices.backend.*, and ingress.backendServices.controller.*. When any data-plane override is enabled, the chart splits the data-plane ALB rules into separate Ingress resources so each target class gets its own backend protocol and health check annotations.
• Each configured Service must be in the same namespace as the rendered ALB Ingress. With the default healthcheck.port: status-port, the gateway Service must expose a port named status-port; otherwise override healthcheck.port to match the gateway Service. Example: set ingress.backendServices.http.enabled: true with name: cp-http-gateway and port: 443.
• New optional gzip-capable nginx sidecars for hive-control-plane and hive-data-plane service pods, controlled by common.nginxSidecar.enabled (default: false). When enabled, HTTP Service traffic targets the nginx sidecar, which proxies to the app container over localhost and gzip-compresses responses; gRPC and metrics ports are unchanged.
• nginx sidecar tuning keys and defaults: common.nginxSidecar.port (default: 18080), common.nginxSidecar.image.repository (default: nginxinc/nginx-unprivileged), common.nginxSidecar.image.tag (default: 1.27-alpine), common.nginxSidecar.proxyReadTimeout and proxySendTimeout (default: 300s), common.nginxSidecar.clientMaxBodySize (default: 64m), common.nginxSidecar.workerProcesses (default: 1), common.nginxSidecar.upstreamKeepalive (default: 16), common.nginxSidecar.gzip.compLevel (default: 5), common.nginxSidecar.gzip.minLength (default: 1024), plus gzip.types, securityContext, and resources (requests 100m CPU / 128Mi memory, limits 500m CPU / 512Mi memory). The sidecar runs non-root with a read-only root filesystem and an emptyDir at /tmp. To enable, set common.nginxSidecar.enabled: true in each chart where compression is needed.

​

v1.3.0

• New configurable ClickHouse connection pool size for control-plane services in hive-control-plane/values.yaml: cpBackendService.clickhouseMaxOpenConnections, cpWriterService.clickhouseMaxOpenConnections, cpControllerService.clickhouseMaxOpenConnections, and cpNotificationService.clickhouseMaxOpenConnections (all default: 50). Each sets the service’s CLICKHOUSE_MAX_OPEN_CONNECTIONS env var; previously the pool size was not configurable from Helm values. No action required for a standard upgrade; override any of these keys in your values overlay if a service needs a different pool size.
• New data-plane ALB route for /v1/traces in hive-data-plane, which now forwards to ingestion alongside the existing ingestion routes.
• Fixed method-aware data-plane ALB routing for overlapping /v1 event and session paths. POST /v1/events, POST /v1/events/batch, POST /v1/sessions, POST /v1/sessions/{session_id}/events/batch, and PUT /v1/events/{event_id} route to ingestion; GET /v1/events/{event_id} and GET /v1/sessions/{session_id} now fall through to the data-plane backend instead of being captured by broader ingestion prefix routes; PUT and OPTIONS to /v1/events/{event_id}/enrich route to the data-plane backend from the primary ingress (the separate wildcard ingress was removed). Template-only change, applied automatically on helm upgrade.
• Action Required (data-plane ALB ingress only): before upgrading environments with ingress.enabled: true for the data-plane chart, verify the AWS Load Balancer Controller is v2.4.3 or newer. The new ALB ingress relies on the controller preserving manifest order for ImplementationSpecific paths; on older controllers the overlapping /v1/events and /v1/sessions routes may be prioritized incorrectly after helm upgrade. Upgrade the controller first if needed. Standard values overlays need no changes.

​

v1.2.1

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts are unchanged, so a standard helm upgrade requires no action.

​

v1.2.0

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts are unchanged, so a standard helm upgrade requires no action.

​

v1.1.0

• New optional prom/statsd-exporter deployment for cp-jobs that bridges StatsD metrics emitted by job pods to a Prometheus /metrics scrape endpoint. Disabled by default, so no action is required unless you want to enable job metrics.
• New jobs.statsdExporter.enabled (default: false) in hive-control-plane/values.yaml. Set to true to deploy the exporter. All new resources are gated on this key.
• New jobs.statsdExporter.name (default: "cp-jobs-statsd-exporter"), jobs.statsdExporter.image.repository (default: prom/statsd-exporter), and jobs.statsdExporter.image.tag (default: "v0.28.0").
• New jobs.statsdExporter.statsdPort (default: 9125, UDP port job pods send metrics to) and jobs.statsdExporter.metricsPort (default: 9102, TCP port Prometheus or the Datadog Agent scrapes).
• New jobs.statsdExporter.podAnnotations (default: {}) for Datadog Agent autodiscovery checks, and jobs.statsdExporter.resources (requests: 10m CPU / 32Mi memory; limits: 100m CPU / 64Mi memory).
• To enable job metrics scraping, set jobs.statsdExporter.enabled: true. For Prometheus Operator, also set serviceMonitor.enabled: true to auto-create a ServiceMonitor for the metrics port (9102). For Datadog, add an ad.datadoghq.com/statsd-exporter.checks openmetrics annotation to jobs.statsdExporter.podAnnotations pointing at http://%%host%%:9102/metrics.
• Fixed a manifest rendering bug that produced invalid YAML when jobs.statsdExporter.enabled: true. Template-only fix, applied automatically on helm upgrade with no values changes required.

​

v1.0.0

1. If you override dpIngestionService.ports.metrics, rename the key to dpIngestionService.metricsPort in your data-plane values overlay. The old nested ports.metrics key no longer exists; if you don’t rename, the dp-ingestion metrics port silently reverts to the default (9091).
2. If you use Prometheus Operator and relied on the default serviceMonitor.enabled: true, add serviceMonitor.enabled: true to both your control-plane and data-plane values overlays. The default was changed to false to avoid CRD-missing errors for customers who do not run Prometheus Operator. Without it, ServiceMonitor resources are not created and Prometheus stops scraping HoneyHive pods.
3. (Optional) To opt into ACM cert auto-discovery on the control-plane frontend ALB, leave frontendIngress.tls.certificateArn empty (with frontendIngress.tls.enabled: true) and set frontendIngress.tls.hosts to the hostname(s) the ALB must serve.
4. (Optional) To opt into ACM cert auto-discovery on the data-plane ALB (primary + wildcard), leave ingress.tls.certificateArn empty (with ingress.tls.enabled: true) and set ingress.tls.hosts to your data-plane hostname(s).

• Action Required: serviceMonitor.enabled default changed from true to false in both hive-control-plane/values.yaml and hive-data-plane/values.yaml. The default was true in v0.104.0, which caused Helm to render ServiceMonitor resources even when the Prometheus Operator CRDs were not installed, failing the deploy. If you use Prometheus Operator and relied on the default, add serviceMonitor.enabled: true to both your control-plane and data-plane values overlays. Without it, ServiceMonitor resources are not created and Prometheus stops scraping HoneyHive pods.
• Action Required: dpIngestionService.ports.metrics renamed to dpIngestionService.metricsPort in hive-data-plane/values.yaml. The nested ports.metrics key was the only service using that structure; all other services use a flat metricsPort key. The old key no longer exists; if you don’t rename, the dp-ingestion metrics port silently reverts to the default (9091).
• New dpPythonmetricService.metricsPort (default: 9091) and dpLlmproxyService.metricsPort (default: 9091) in hive-data-plane/values.yaml for dedicated Prometheus metrics ports on dp-pythonmetric and dp-llmproxy. When serviceMonitor.enabled: true, ServiceMonitors are now also created for these two services.
• New frontendIngress.tls.hosts (default: []) in hive-control-plane/values.yaml and ingress.tls.hosts (default: []) in hive-data-plane/values.yaml for ACM certificate auto-discovery. frontendIngress.tls.hosts controls the control-plane frontend ALB; ingress.tls.hosts controls the data-plane API ALB (applied to both the primary and wildcard Ingresses). Rendered as spec.tls[*].hosts so the AWS Load Balancer Controller can match an ACM cert by SNI/SAN when certificateArn is empty. Leave certificateArn empty (with tls.enabled: true) and set tls.hosts to your hostname(s).
• Fixed ACM certificate auto-discovery on the data-plane wildcard Ingress and control-plane frontend Ingress. Previously, leaving certificateArn blank emitted an empty alb.ingress.kubernetes.io/certificate-arn annotation that ArgoCD rejected with a nil-annotation error. The annotation is now omitted entirely when certificateArn is empty. The control-plane frontend Ingress now contributes a spec.tls[*].hosts cert-discovery hint when frontendIngress.tls.hosts is set, so HTTPS listener creation no longer fails on overlays that rely on auto-discovery. Overlays that set an explicit certificateArn continue to render identically.

​

v0.104.0

• Action Required: common.dataplane.dpPublicUrl default changed to "" in data-plane/services/values.yaml. Set this to your environment’s public data plane URL (e.g., "https://api.my-dp.example.com"). Without it, the Admin Center Data Planes view and /settings/project/keys page show no URL.
• Action Required: dpIngestionService HPA defaults changed in data-plane/services/values.yaml. targetCPUUtilizationPercentage is now 60 (was 80), and a new targetMemoryUtilizationPercentage (default: 60) was added. To keep prior behavior, set CPU to 80 and memory to 0.
• Action Required: ingress.maxBodySize removed from both control-plane/services/values.yaml and data-plane/services/values.yaml. Remove any overrides for this key.
• Action Required: ingress.authHost removed from control-plane/services/values.yaml. Remove any overrides for this key.
• Action Required: jobs.serviceAccount.create now defaults to true in both planes. Set to false if you manage the jobs ServiceAccount externally.
• Action Required (ArgoCD users): Deployments omit spec.replicas when autoscaling is enabled. Add ignoreDifferences with jsonPointers: ["/spec/replicas"] for Deployment kind in your ArgoCD Application spec.
• New <service>.affinity (default: {}) for all CP/DP services and jobs.affinity for CronJobs/batch jobs. Supports podAffinity, podAntiAffinity, and nodeAffinity.
• New <service>.autoscaling.enabled (default: true) for all 11 CP/DP services. Set to false to disable HPA and pin replicas via <service>.replicas.
• New frontendIngress.tls.redirect (default: false) in control-plane/services/values.yaml. When true, the ALB listens on HTTP:80 and redirects to HTTPS.
• New dpIngestionService.resources (default: {}) in data-plane/services/values.yaml for per-service resource overrides on ingestion.
• New dpPythonmetricService.gunicornWorkers (default: 4) and dpPythonmetricService.pythonExecutionTimeout (default: 0.1) in data-plane/services/values.yaml for tuning custom metric concurrency and per-metric runtime.
• New cpNotificationService.ses.domain, .ses.rps (default: 14), .ses.senderRoleArn, .ses.senderRoleDurationSeconds (default: 3600) in control-plane/services/values.yaml for SES email sender configuration.
• New serviceMonitor.enabled (default: true), serviceMonitor.namespace (default: "monitoring"), serviceMonitor.labels (default: {release: monitoring}), serviceMonitor.interval (default: "30s"), serviceMonitor.scrapeTimeout (default: "10s") in both planes for Prometheus Operator ServiceMonitor support.
• New <service>.metricsPort (default: 9091), a dedicated Prometheus metrics port for all CP/DP services.
• ingress.host and ingress.grpcHost now accept a string or a list of strings for multi-host ingress support.
• Deployments no longer reset replica count on helm upgrade when autoscaling is enabled.
• Fixed NATS anti-affinity selector to correctly distinguish nats-box from nats server pods.
• Custom CA certs from common.tls.caCerts now apply to the DP S3 backfill Job.
• Added app.kubernetes.io/version label and OTel service.version attribute to all CP/DP services.
• Distributed tracing is now correctly emitted from dp-backend, dp-llmproxy, and dp-controller (the OTEL_ENABLED=true env var was previously missing on these deployments).
• Ingress routing improvements: /v1/events/*/annotate routes to dp-backend, /v1/* ingestion paths route to ingestion-service, API and frontend traffic use separate ALBs.
• Fixed PDB template to correctly handle minAvailable: 0 and maxUnavailable: 0.

​

v0.103.0

• New jobs configuration block in control-plane/services/values.yaml for Kubernetes CronJobs that process alert transitions
• New jobs configuration block in data-plane/services/values.yaml for operator-triggered S3 time-window backfill batch workloads
• New per-service keys: <service>.service.annotations (default: {}), <service>.service.labels (default: {}), <service>.service.type (default: "ClusterIP") now exposed on cpWriterService, cpControllerService, cpNotificationService, dpControllerService, dpIngestionService, dpEvaluationService, dpLlmproxyService, dpPythonmetricService — previously hardcoded
• Service port protocol hints (name: http, protocol: TCP, appProtocol: http/grpc) added to all service templates for correct L7 traffic classification by service meshes (Istio, Linkerd)
• appProtocol added to ClickHouse instance Service ports (http, interserver, metrics: appProtocol: http; native: appProtocol: tcp)
• ClickHouse instances chi-installation chart version bumped from 1.1.6 to 1.1.7
• No breaking changes — fully backward-compatible with v0.102.0 configurations

​

v0.102.0

• New podAnnotations (default: {}) in control-plane/infrastructure/clickhouse/clickhouse_instances/values.yaml for arbitrary annotations on ClickHouse pods (useful for Datadog autodiscovery, Prometheus scraping)
• Fixed missing DP_DATABASE_URL env var in dp-pythonmetric-service deployment template, now reads from common.externalSecrets.postgres.secretName / uriKey like all other data-plane services
• No breaking changes — fully backward-compatible with v0.101.0 configurations

​

v0.101.0

• Disabled ClickHouse replica check before attaching backup parts (CLICKHOUSE_CHECK_REPLICAS_BEFORE_ATTACH set to "false" in backup container) - prevents backup restore failures in environments where replica availability cannot be confirmed
• ClickHouse instances chi-installation chart version bumped from 1.1.5 to 1.1.6
• No breaking changes — fully backward-compatible with v0.100.0 configurations

​

v0.100.0

• New ingress.albClassName (default: "alb") and frontendIngress.albClassName (default: "alb") in control-plane and data-plane services for configurable ALB ingress class
  • Useful for shared CP+DP cluster scenarios where each plane needs its own ALB IngressClass (e.g., "cp-alb", "dp-alb")
  • Existing deployments using the default "alb" class require no changes
• New scheduling config for CP prometheus-nats-exporter in control-plane/infrastructure/nats/values.yaml: prometheusExporter.tolerations, prometheusExporter.nodeSelector, prometheusExporter.affinity, prometheusExporter.additionalLabels
• common.extraLabels now propagated to all Service, ServiceAccount, HPA, and Ingress resources across both planes (previously only applied to Deployments)

​

v0.99.3

• Action Required: Add common.controlPlane.id in data-plane/services/values.yaml - set it to match common.controlPlane.id from your control-plane values. Without this, dp-controller-service cannot identify its parent control plane, causing data-plane-to-control-plane communication failures
• New global.labels (default: {}) in control-plane/infrastructure/nats/values.yaml and data-plane/infrastructure/nats/values.yaml - applied to all NATS-generated Kubernetes resources (StatefulSet, Service, PVC, ConfigMap, PDB, etc.)
• New nack.additionalLabels, nack.tolerations, nack.nodeSelector, nack.affinity in control-plane NATS values for JetStream Controller scheduling
• prometheusExporter.additionalLabels now propagated to Prometheus NATS Exporter Deployment, Service, and ServiceMonitor labels in both planes
• Fixed topologySpreadConstraints typo in control-plane NATS values (topolicySpreadConstraints → topologySpreadConstraints)

​

v0.99.2

• ClickHouse instances chi-installation chart version bumped from 1.1.4 to 1.1.5 in control-plane/infrastructure/clickhouse/clickhouse_instances/Chart.yaml

​

v0.99.1

• Fixed cp-notification-service healthcheck port - renamed env var from EXPRESS_PORT to PORT in deployment template
• Control plane hive-control-plane chart version bumped from 0.2.0 to 0.2.1

​

v0.99.0

• Added readiness and liveness probes (GET /healthcheck) to all 9 service deployments across control-plane and data-plane
• cp-backend-service and dp-backend-service include a startup probe (allows up to 310s for Prisma migrations before liveness checks begin)
• Eliminates intermittent 502 errors during rolling updates caused by traffic routing to pods not yet ready to serve
• No values.yaml changes required - probes use hardcoded values in deployment templates

​

v0.98.9

• Unified encryption configuration for cp-controller, dp-controller, and dp-llmproxy-service
• Supports at-rest encryption for identity management and provider secrets via KMS or environment variable mode
• New common.encryption.keyId value in both control-plane and data-plane services
• New ExternalSecret templates for encryption in both control-plane and data-plane secret-store charts
• Replaced dpLlmproxyService.kmsKeyId with unified HH_ENCRYPTION_KEY_ID and HH_ENCRYPTION_SECRET env vars in dp-llmproxy-service
• Fixed perpetual ArgoCD OutOfSync caused by Redis PDB enabled: true in control-plane Redis
• Action Required: Remove dpLlmproxyService.kmsKeyId from data-plane values and add common.encryption.keyId in both control-plane and data-plane values

​

v0.98.1

• Added common.extraLabels for custom governance/compliance labels on all Kubernetes resources (control-plane, data-plane, shared dependencies)
• Added common.observability.otel.exporterProtocol (default: "grpc") for OTLP exporter protocol configuration in data-plane
• Added dpLlmproxyService.kmsKeyId (default: "alias/hh-provider-secrets") for AWS KMS encryption of LLM provider secrets
• Removed duplicate common.observability block in data-plane services values
• Fixed Next.js cache permission errors in cp-frontend-service with nextjs-cache emptyDir volume
• Fixed OTEL service name in cp-frontend-service (was hardcoded to cp-controller-service)
• Added custom CA certificate support (SSL_CERT_FILE, REQUESTS_CA_BUNDLE) for dp-llmproxy-service and dp-pythonmetric-service
• Added DP_DATABASE_URL env var and KMS config to dp-llmproxy-service
• Fixed ClickHouse logging configuration (moved to config.d/99-logger.xml with replace="1")
• Action Required: Set common.extraLabels if your organization requires specific labels on all resources

​

v0.90.17

• Added kube-prometheus-stack monitoring for both control-plane and data-plane (Prometheus, Grafana, Alertmanager with 30-day retention)
• Added Tempo for distributed tracing in both control-plane and data-plane
• Added Loki and Promtail for centralized log aggregation in control-plane
• Added legacy nats-old chart for backward compatibility
• Added Datadog integration support for OTEL collectors (disabled by default, set datadog.enabled: true)
• Added common.tls.caCerts dictionary for custom root CA certificates in data-plane
• Added common.controlPlane.apiPublicUrl for data-plane to call control-plane API
• Added resource limits for dp-llmproxy-service and dp-pythonmetric-service (500m/512Mi requests, 1000m/1Gi limits)
• Added persistent storage for ClickHouse Keeper (storage.enabled: true, storage.size: "10Gi")
• Updated ClickHouse Keeper image to altinity/clickhouse-keeper:25.3.6.10034.altinitystable-alpine
• Simplified ClickHouse Operator values from 903 lines to 29 lines
• Updated ExternalSecret API version from v1beta1 to v1 (requires External Secrets Operator 0.9.0+)
• Moved OTEL collector nodeSelector/affinity/tolerations under opentelemetry-collector key
• Action Required: Set common.controlPlane.apiPublicUrl to your control-plane API endpoint
• Action Required: Ensure ClickHouse Keeper persistent storage is enabled for production
• Action Required: Update External Secrets Operator to 0.9.0+ if not already

• Removed OpenUnison authentication infrastructure (all charts, operators, CRDs)
• Removed Nginx ingress infrastructure
• Added NATS infrastructure for data-plane with independent cluster deployment (3 replicas, JetStream, PDB)
• Disabled S3 DLQ and disk spool in writer service (cpWriterService.dlq.enabled: false)
• Added frontendIngress.alb.annotations for custom ALB annotations
• Removed PVC functionality from cp-writer-service
• Changed cp-frontend-service NEXTJS_PORT env var to PORT
• Added auth config env vars (AUTH_ISSUER_DOMAIN, AUTH_CLIENT_ID, AUTH_CLIENT_SECRET) to cp-frontend-service
• Added NATS connection settings for data-plane services (dp-evaluation-service, dp-ingestion-service)
• Enabled Redis authentication in control-plane (auth: true, existingSecret: redis-secrets)
• Removed gRPC ingress from data-plane services
• Switched from NLB to ALB for both control-plane and data-plane ingress
• Added NATS HA streams configuration with configurable replicas
• Added common.dataPlane.dpPublicUrl and common.controlPlane.frontendPublicUrl for cross-plane communication
• Added Prometheus monitoring for NATS (exporter on port 7777) and ClickHouse (built-in on port 9363)
• Added Redis authentication for data-plane (auth: true, existingSecret: redis-secrets)
• Fixed Redis PDB in data-plane (removed invalid enabled field)
• Action Required: Remove any NLB-related values overrides and switch to ALB configuration
• Action Required: Remove any OpenUnison or Beekeeper-related overrides from values files
• Action Required: Configure auth secrets in AWS Secrets Manager with client-secret and cp-jwt-private-key