Self-Hosting

​

hive-control-plane v1.9.0

• This release ships HoneyHive platform v1.9.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.9.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• New common.containerSecurityContext applies a hardened container securityContext to every HoneyHive service, init, and job container. Defaults: allowPrivilegeEscalation: false, runAsNonRoot: true, runAsUser: 65532, runAsGroup: 65532, readOnlyRootFilesystem: true, capabilities.drop: [ALL], and seccompProfile.type: RuntimeDefault. The HoneyHive images run as UID 65532 and write nothing to their own filesystem, so no scratch volume is needed (cp-frontend-service’s Next.js cache already has its own emptyDir).
• New per-workload overrides via <service>.containerSecurityContext, for example cpBackendService.containerSecurityContext, cpFrontendService.containerSecurityContext, jobs.containerSecurityContext, and jobs.statsdExporter.containerSecurityContext. Keys merge one at a time and the per-service value wins, so you can relax a single field (for example readOnlyRootFilesystem: false) without restating the whole map. Setting common.containerSecurityContext: {} with no per-service override renders no container securityContext at all.
• Every ServiceAccount the chart creates (cp-backend, cp-controller, cp-frontend, cp-notification, cp-writer, and the shared jobs ServiceAccount) now sets automountServiceAccountToken: false, and the statsd exporter Deployment sets the same field on its pod spec. Pods no longer receive the default Kubernetes API credential at /var/run/secrets/kubernetes.io/serviceaccount. No HoneyHive workload calls the Kubernetes API, and IAM Roles for Service Accounts is unaffected because the EKS webhook injects its own projected token volume.
• Every workload now sets both AWS_REGION and AWS_DEFAULT_REGION from common.region (default: us-west-2), instead of only some containers setting AWS_REGION. This stops AWS SDKs, in particular Python’s botocore/boto3, which ignores AWS_REGION, from falling back to the instance metadata service to discover a region. If common.region is overridden to an empty string, neither variable is rendered at all (previously an empty AWS_REGION was rendered on some workloads and treated by the SDKs as a real, invalid region), which leaves the SDKs’ normal region resolution intact.
• The nginx sidecar security context (common.nginxSidecar.securityContext) now also sets seccompProfile.type: RuntimeDefault.
• Fixed alert notification emails omitting their dashboard, alert, and alert-settings links. cp-notification-service now receives FRONTEND_PUBLIC_URL from common.auth.frontendPublicUrl, which is already required for login (default: ""). If it is unset, emails still send but their links remain omitted.
• Action Required (only if you pin HoneyHive image tags): set your tag overrides to v1.9.0 before upgrading. Overlays that use the chart defaults need no changes.
• Action Required (only if you have added a sidecar or a mutating webhook to these pods that relies on the automounted Kubernetes API token): mount it explicitly with a serviceAccountToken projected volume.

​

hive-data-plane v1.9.0

• This release ships HoneyHive platform v1.9.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.9.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• New common.containerSecurityContext applies a hardened container securityContext to every HoneyHive service, init, and job container. Defaults: allowPrivilegeEscalation: false, runAsNonRoot: true, runAsUser: 65532, runAsGroup: 65532, readOnlyRootFilesystem: true, capabilities.drop: [ALL], and seccompProfile.type: RuntimeDefault. The HoneyHive images run as UID 65532 and write nothing to their own filesystem, with one exception handled by the chart: dp-pythonmetric-service now mounts an emptyDir at /tmp for gunicorn’s worker heartbeat files and the Prometheus multiprocess registry.
• New per-workload overrides via <service>.containerSecurityContext, for example dpBackendService.containerSecurityContext, dpPythonmetricService.containerSecurityContext, and jobs.containerSecurityContext (covers the S3 time-window backfill Job and the legacy federated migration Job). The prepare-certs init container inherits the security context of the service it runs in, so override it under dpLlmproxyService.containerSecurityContext or dpPythonmetricService.containerSecurityContext. Per-service keys merge into the global map one key at a time and win on conflict, so you can relax a single field (for example readOnlyRootFilesystem: false) without restating the whole map. Setting common.containerSecurityContext: {} with no per-service override renders no container securityContext at all.
• Every ServiceAccount the chart creates (dp-backend, dp-controller, dp-evaluation, dp-ingestion, dp-llmproxy, dp-pythonmetric, and the shared jobs ServiceAccount) now sets automountServiceAccountToken: false. Pods no longer receive the default Kubernetes API credential at /var/run/secrets/kubernetes.io/serviceaccount. No HoneyHive workload calls the Kubernetes API, and IAM Roles for Service Accounts is unaffected because the EKS webhook injects its own projected token volume.
• Every workload, including all jobs, now sets both AWS_REGION and AWS_DEFAULT_REGION from common.region (default: us-west-2), instead of only some containers setting AWS_REGION. This stops AWS SDKs, in particular Python’s botocore/boto3, which ignores AWS_REGION, from falling back to the instance metadata service to discover a region. If common.region is overridden to an empty string, neither variable is rendered at all (previously an empty AWS_REGION was rendered on some workloads and treated by the SDKs as a real, invalid region), which leaves the SDKs’ normal region resolution intact.
• dp-pythonmetric-service no longer receives the Postgres connection URL, the REDIS_MASTER_HOST / REDIS_REPLICAS_HOST / REDIS_MASTER_PORT / REDIS_REPLICAS_PORT / REDIS_PASSWORD variables, or the HH_ENCRYPTION_KEY_ID / HH_ENCRYPTION_SECRET variables. The service reads none of them, so it no longer depends on the Secrets named by common.externalSecrets.postgres, common.externalSecrets.redis, and common.externalSecrets.encryption; its pods now start even when those Secrets are absent. Other data-plane services still consume all three, so keep common.externalSecrets.*, redis.*, and common.encryption.* configured as before. No values change is required.
• The nginx sidecar security context (common.nginxSidecar.securityContext) now also sets seccompProfile.type: RuntimeDefault.
• Fixed the prepare-certs init container that builds the merged CA bundle for dp-llmproxy-service and dp-pythonmetric-service when common.tls.caCerts is set. It no longer requires a shell or cat in the service image, so it works on shell-less (distroless) images where it previously failed immediately and blocked the pod from starting. It builds the same /certs/ca-bundle.pem (the image’s system trust store followed by each *.crt from your common.tls.caCerts ConfigMap, consumed via SSL_CERT_FILE and REQUESTS_CA_BUNDLE) and now fails with a clear error if the image’s system CA bundle is missing rather than emitting a bundle without it. The init container also declares resources (requests: 10m CPU / 32Mi memory; limits: 50m CPU / 64Mi memory). Kubernetes schedules on the larger of the init container’s request and the app containers’ combined request, so at the chart’s default service requests your pods’ scheduling footprint does not change.
• Action Required (only if you pin HoneyHive image tags): set your tag overrides to v1.9.0 before upgrading. Overlays that use the chart defaults need no changes.
• Action Required (only if you have added a sidecar or a mutating webhook to these pods that relies on the automounted Kubernetes API token): mount it explicitly with a serviceAccountToken projected volume.

​

hive-control-plane v1.8.0

• This release ships HoneyHive platform v1.8.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-control-plane/values.yaml are now v1.8.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• New common.auth.scopes (unset by default) for additional space-separated OAuth scopes requested at login, rendered as the AUTH_SCOPES env var on the frontend Deployment. openid email profile is always requested, so list only extra scopes. Leaving the key unset renders no AUTH_SCOPES variable and preserves existing behavior.
• Identity providers that scope access tokens per resource (for example Microsoft Entra ID) must also request the scope of the API exposed for this deployment; otherwise the access token is issued for the provider’s own API and fails verification. For Entra ID, set common.auth.scopes: "api://<client-id>/access_as_user".
• When ingress.tls.enabled: true, the HTTP and gRPC ALB Ingress resources now carry the alb.ingress.kubernetes.io/ssl-policy annotation taken from ingress.tls.sslPolicy (default: ELBSecurityPolicy-TLS13-1-2-2021-06). These two listeners previously fell back to the AWS Load Balancer Controller’s default policy, so the negotiated TLS versions and ciphers may change on upgrade.
• Every service and job container now sets HH_SERVICE_NAME to its service name, so structured logs keep their service name even when common.observability.otel.enabled: false. When OpenTelemetry is enabled, OTEL_SERVICE_NAME still takes precedence. No values change is required.
• Fixed the wait-for-cp-backend init container of the cron-alert-transitions-long-range-on-deploy Job. On job images without a shell, the init container failed immediately and the post-deploy job never ran. It now polls http://<cpBackendService.name>:<cpBackendService.port>/healthcheck for up to 20 attempts with a 5s request timeout and 10s between attempts (about 290s).
• Action Required (only if you pin HoneyHive image tags): set your tag overrides to v1.8.0 before upgrading. Overlays that use the chart defaults need no changes.
• Action Required (only if you override ingress.tls.sslPolicy to an empty string): set it to a non-empty AWS policy name. With TLS enabled, an empty value now fails rendering with ingress.tls.sslPolicy is required when ingress.tls.enabled is true.
• Action Required (only if you raise jobs.alertTransitionsLongRange.jobTimeoutMs, default: 600000): raise jobs.alertTransitionsLongRange.activeDeadlineSeconds (default: 900) by at least as much. The deadline runs from the Job’s start time and covers the backend wait; exceeding it fails the Job with no retry.

​

hive-data-plane v1.8.0

• This release ships HoneyHive platform v1.8.0. The chart’s appVersion and the default HoneyHive workload and job image tags in hive-data-plane/values.yaml are now v1.8.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• When ingress.tls.enabled: true, every ALB Ingress rendered by the chart now carries the alb.ingress.kubernetes.io/ssl-policy annotation taken from ingress.tls.sslPolicy (default: ELBSecurityPolicy-TLS13-1-2-2021-06). These listeners previously fell back to the AWS Load Balancer Controller’s default policy, so the negotiated TLS versions and ciphers may change on upgrade.
• Every service and job container now sets HH_SERVICE_NAME to its service name, so structured logs keep their service name even when common.observability.otel.enabled: false. When OpenTelemetry is enabled, OTEL_SERVICE_NAME still takes precedence. No values change is required.
• Removed the Sentry Secret and the secrets.sentry block (secrets.sentry.enabled, secrets.sentry.dsns.*) from hive-data-plane/values.yaml. The Secret was disabled by default and its contents were never consumed by any data-plane workload, so no data-plane service loses configuration. If you still set secrets.sentry.* in your own values, the keys are now ignored and can be deleted.
• hive-data-plane/values.yaml now documents supported keys that previously existed only in the templates, as commented-out entries with their defaults. Rendering behavior is unchanged; these keys already worked. Newly documented: nameOverride, fullnameOverride, global.labels, global.org, global.env, global.sregion, global.deployment, common.image.pullSecret, <service>.pdb.maxUnavailable, dpEvaluationService.resources, dpLlmproxyService.assumedRole (roleArn, roleSessionName, externalId, region), config, and secrets.postgres.useExternalSecret / host / port / database / username.
• Action Required (only if you pin HoneyHive image tags): set your tag overrides to v1.8.0 before upgrading. Overlays that use the chart defaults need no changes.
• Action Required (only if you override ingress.tls.sslPolicy to an empty string): set it to a non-empty AWS policy name. With TLS enabled, an empty value now fails rendering with ingress.tls.sslPolicy is required when ingress.tls.enabled is true.

​

chi-installation v1.8.0

• New opt-in coordination-aware readiness probe for the ClickHouse server container, configured under server.readinessProbe in chi-installation/values.yaml. When server.readinessProbe.useReplicaCheck is true, the chart sets an exec readiness probe that queries system.replicas and marks the pod unready while any replicated table in an application database is read-only or has lost its ClickHouse Keeper session. This catches degraded replicas that still answer /ping with HTTP 200, so traffic is withheld until coordination recovers. ClickHouse’s own system, information_schema, and INFORMATION_SCHEMA databases are excluded from the check.
• New probe timing keys, applied only when server.readinessProbe.useReplicaCheck is enabled: server.readinessProbe.initialDelaySeconds (default: 30), server.readinessProbe.periodSeconds (default: 10), server.readinessProbe.timeoutSeconds (default: 5), server.readinessProbe.failureThreshold (default: 3), and server.readinessProbe.successThreshold (default: 1).
• No default behavior change: server.readinessProbe.useReplicaCheck defaults to false, in which case the chart sets no readinessProbe on the ClickHouse container and the ClickHouse Operator continues to inject its own default /ping-based probe. Disabling the flag reverts to that operator default rather than turning readiness checking off, so a standard helm upgrade requires no action.
• Enabling the probe changes the ClickHouse container’s pod spec, so the ClickHouse pods roll on the next helm upgrade. Schedule the upgrade accordingly. A pod that restores a backup fails the replica check until the restore completes, so it stays unready during the restore. Take this into account on clusters that perform in-place restores.

​

hive-control-plane v1.7.2

• This release ships HoneyHive platform v1.7.0. The chart’s appVersion and default HoneyHive workload image tags in hive-control-plane/values.yaml are now v1.7.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• A standard helm upgrade rolls the pods that use the chart defaults. No configuration or template migration is required.
• Action Required (only if you pin HoneyHive image tags): set your tag overrides to v1.7.0 before upgrading. Overlays that use the chart defaults need no changes.

​

hive-data-plane v1.7.2

• This release ships HoneyHive platform v1.7.0. The chart’s appVersion and default HoneyHive workload image tags in hive-data-plane/values.yaml are now v1.7.0. Third-party image tags are unchanged. See the Product changelog for the platform changes.
• A standard helm upgrade rolls the pods that use the chart defaults. No configuration or template migration is required.
• Action Required (only if you pin HoneyHive image tags): set your tag overrides to v1.7.0 before upgrading. Overlays that use the chart defaults need no changes.

​

v1.6.3

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts (values.yaml, templates/, and Chart.yaml) are unchanged, so a standard helm upgrade requires no action.

​

v1.6.2

• New custom annotations on service Deployment metadata in hive-control-plane and hive-data-plane. Set common.deploymentAnnotations (default: {}) to apply a map of annotations to every service Deployment’s metadata.annotations, or <service>.deploymentAnnotations (default: {}) for a single service, which merges over the common map with per-service keys winning on conflict. Useful for GitOps controllers, config-reloaders, cost-allocation, and similar Deployment-level integrations. Both default empty and render identically to before when unset, so a standard helm upgrade requires no action.
• New per-service resource overrides in hive-control-plane/values.yaml and hive-data-plane/values.yaml. Each service can now set its own resources block (CPU/memory requests and limits) instead of sharing common.resources. Newly configurable: dpBackendService.resources, cpBackendService.resources, cpFrontendService.resources, cpWriterService.resources, and cpNotificationService.resources (each default: {}, falling back to common.resources). Set any in your values overlay to override only that service, e.g. dpBackendService.resources.limits.memory: 4Gi. No action required for a standard upgrade.
• New file-based license configuration in hive-control-plane. Set the structured license document issued by HoneyHive under the top-level license key (default: empty, feature off when unset). When set, the chart renders it into a new hive-control-plane-license-config ConfigMap, mounts it into the cp-backend pod at /var/hh/license.yaml, and points the backend at it via the LICENSE_PATH env var. See the annotated example block in hive-control-plane/values.yaml for the full structure (title, version, license_id, license_type, and license_policy holding controlplane_admin_emails and allowed identity_providers).
• Deprecated common.license (the inline key injected as the LICENSE env var) in favor of license. It still works, but only as a fallback when license is unset, and will be removed in a future release. Recommended (not required): move your license into the new license block and remove common.license. While license is set it takes precedence and common.license is ignored; if license is left empty the chart falls back to common.license, so existing deployments keep working until you migrate.

​

v1.6.0

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts (values.yaml, templates/, and Chart.yaml) are unchanged, so a standard helm upgrade requires no action.

​

v1.5.0

• Documented dozens of previously-undeclared configuration knobs in hive-control-plane/values.yaml (name overrides, global.*, common.*, cp-controller tuning, per-service PDB alternates, frontend TLS policy, and a legacy config block). These are surfaced as commented, render-neutral documentation; merged values are unchanged, so a standard helm upgrade requires no action.
• Fixed the cp-controller PodDisruptionBudget so setting cpControllerService.pdb.minAvailable or cpControllerService.pdb.maxUnavailable to null now omits the field instead of rendering an invalid empty value. maxUnavailable: 0 still works, and default values render identically to v1.4.1.
• Removed a dead beekeeper Sentry DSN block from the control-plane secrets template (the beekeeper service was retired). Template-only change, applied automatically on helm upgrade.

​

v1.4.1

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts are unchanged, so a standard helm upgrade requires no action.

​

v1.4.0

• New optional generic ALB backend Service overrides in hive-control-plane/values.yaml and hive-data-plane/values.yaml, letting you route individual ALB backends to a mesh gateway or customer-managed reverse proxy Service instead of the HoneyHive app Services. All disabled by default, so no action is required for a standard upgrade.
• Control-plane override keys: ingress.backendServices.http.* (HTTP backend), ingress.backendServices.grpc.* (gRPC backend), and frontendIngress.backendService.* (frontend backend), each with enabled, name, port, protocol, and healthcheck.* fields.
• Data-plane override keys: ingress.backendServices.ingestion.*, ingress.backendServices.backend.*, and ingress.backendServices.controller.*. When any data-plane override is enabled, the chart splits the data-plane ALB rules into separate Ingress resources so each target class gets its own backend protocol and health check annotations.
• Each configured Service must be in the same namespace as the rendered ALB Ingress. With the default healthcheck.port: status-port, the gateway Service must expose a port named status-port; otherwise override healthcheck.port to match the gateway Service. Example: set ingress.backendServices.http.enabled: true with name: cp-http-gateway and port: 443.
• New optional gzip-capable nginx sidecars for hive-control-plane and hive-data-plane service pods, controlled by common.nginxSidecar.enabled (default: false). When enabled, HTTP Service traffic targets the nginx sidecar, which proxies to the app container over localhost and gzip-compresses responses; gRPC and metrics ports are unchanged.
• nginx sidecar tuning keys and defaults: common.nginxSidecar.port (default: 18080), common.nginxSidecar.image.repository (default: nginxinc/nginx-unprivileged), common.nginxSidecar.image.tag (default: 1.27-alpine), common.nginxSidecar.proxyReadTimeout and proxySendTimeout (default: 300s), common.nginxSidecar.clientMaxBodySize (default: 64m), common.nginxSidecar.workerProcesses (default: 1), common.nginxSidecar.upstreamKeepalive (default: 16), common.nginxSidecar.gzip.compLevel (default: 5), common.nginxSidecar.gzip.minLength (default: 1024), plus gzip.types, securityContext, and resources (requests 100m CPU / 128Mi memory, limits 500m CPU / 512Mi memory). The sidecar runs non-root with a read-only root filesystem and an emptyDir at /tmp. To enable, set common.nginxSidecar.enabled: true in each chart where compression is needed.

​

v1.3.0

• New configurable ClickHouse connection pool size for control-plane services in hive-control-plane/values.yaml: cpBackendService.clickhouseMaxOpenConnections, cpWriterService.clickhouseMaxOpenConnections, cpControllerService.clickhouseMaxOpenConnections, and cpNotificationService.clickhouseMaxOpenConnections (all default: 50). Each sets the service’s CLICKHOUSE_MAX_OPEN_CONNECTIONS env var; previously the pool size was not configurable from Helm values. No action required for a standard upgrade; override any of these keys in your values overlay if a service needs a different pool size.
• New data-plane ALB route for /v1/traces in hive-data-plane, which now forwards to ingestion alongside the existing ingestion routes.
• Fixed method-aware data-plane ALB routing for overlapping /v1 event and session paths. POST /v1/events, POST /v1/events/batch, POST /v1/sessions, POST /v1/sessions/{session_id}/events/batch, and PUT /v1/events/{event_id} route to ingestion; GET /v1/events/{event_id} and GET /v1/sessions/{session_id} now fall through to the data-plane backend instead of being captured by broader ingestion prefix routes; PUT and OPTIONS to /v1/events/{event_id}/enrich route to the data-plane backend from the primary ingress (the separate wildcard ingress was removed). Template-only change, applied automatically on helm upgrade.
• Action Required (data-plane ALB ingress only): before upgrading environments with ingress.enabled: true for the data-plane chart, verify the AWS Load Balancer Controller is v2.4.3 or newer. The new ALB ingress relies on the controller preserving manifest order for ImplementationSpecific paths; on older controllers the overlapping /v1/events and /v1/sessions routes may be prioritized incorrectly after helm upgrade. Upgrade the controller first if needed. Standard values overlays need no changes.

​

v1.2.1

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts are unchanged, so a standard helm upgrade requires no action.

​

v1.2.0

• No customer-facing chart changes. The packaged hive-control-plane and hive-data-plane service charts are unchanged, so a standard helm upgrade requires no action.

​

v1.1.0

• New optional prom/statsd-exporter deployment for cp-jobs that bridges StatsD metrics emitted by job pods to a Prometheus /metrics scrape endpoint. Disabled by default, so no action is required unless you want to enable job metrics.
• New jobs.statsdExporter.enabled (default: false) in hive-control-plane/values.yaml. Set to true to deploy the exporter. All new resources are gated on this key.
• New jobs.statsdExporter.name (default: "cp-jobs-statsd-exporter"), jobs.statsdExporter.image.repository (default: prom/statsd-exporter), and jobs.statsdExporter.image.tag (default: "v0.28.0").
• New jobs.statsdExporter.statsdPort (default: 9125, UDP port job pods send metrics to) and jobs.statsdExporter.metricsPort (default: 9102, TCP port Prometheus or the Datadog Agent scrapes).
• New jobs.statsdExporter.podAnnotations (default: {}) for Datadog Agent autodiscovery checks, and jobs.statsdExporter.resources (requests: 10m CPU / 32Mi memory; limits: 100m CPU / 64Mi memory).
• To enable job metrics scraping, set jobs.statsdExporter.enabled: true. For Prometheus Operator, also set serviceMonitor.enabled: true to auto-create a ServiceMonitor for the metrics port (9102). For Datadog, add an ad.datadoghq.com/statsd-exporter.checks openmetrics annotation to jobs.statsdExporter.podAnnotations pointing at http://%%host%%:9102/metrics.
• Fixed a manifest rendering bug that produced invalid YAML when jobs.statsdExporter.enabled: true. Template-only fix, applied automatically on helm upgrade with no values changes required.

​

v1.0.0

1. If you override dpIngestionService.ports.metrics, rename the key to dpIngestionService.metricsPort in your data-plane values overlay. The old nested ports.metrics key no longer exists; if you don’t rename, the dp-ingestion metrics port silently reverts to the default (9091).
2. If you use Prometheus Operator and relied on the default serviceMonitor.enabled: true, add serviceMonitor.enabled: true to both your control-plane and data-plane values overlays. The default was changed to false to avoid CRD-missing errors for customers who do not run Prometheus Operator. Without it, ServiceMonitor resources are not created and Prometheus stops scraping HoneyHive pods.
3. (Optional) To opt into ACM cert auto-discovery on the control-plane frontend ALB, leave frontendIngress.tls.certificateArn empty (with frontendIngress.tls.enabled: true) and set frontendIngress.tls.hosts to the hostname(s) the ALB must serve.
4. (Optional) To opt into ACM cert auto-discovery on the data-plane ALB (primary + wildcard), leave ingress.tls.certificateArn empty (with ingress.tls.enabled: true) and set ingress.tls.hosts to your data-plane hostname(s).

• Action Required: serviceMonitor.enabled default changed from true to false in both hive-control-plane/values.yaml and hive-data-plane/values.yaml. The default was true in v0.104.0, which caused Helm to render ServiceMonitor resources even when the Prometheus Operator CRDs were not installed, failing the deploy. If you use Prometheus Operator and relied on the default, add serviceMonitor.enabled: true to both your control-plane and data-plane values overlays. Without it, ServiceMonitor resources are not created and Prometheus stops scraping HoneyHive pods.
• Action Required: dpIngestionService.ports.metrics renamed to dpIngestionService.metricsPort in hive-data-plane/values.yaml. The nested ports.metrics key was the only service using that structure; all other services use a flat metricsPort key. The old key no longer exists; if you don’t rename, the dp-ingestion metrics port silently reverts to the default (9091).
• New dpPythonmetricService.metricsPort (default: 9091) and dpLlmproxyService.metricsPort (default: 9091) in hive-data-plane/values.yaml for dedicated Prometheus metrics ports on dp-pythonmetric and dp-llmproxy. When serviceMonitor.enabled: true, ServiceMonitors are now also created for these two services.
• New frontendIngress.tls.hosts (default: []) in hive-control-plane/values.yaml and ingress.tls.hosts (default: []) in hive-data-plane/values.yaml for ACM certificate auto-discovery. frontendIngress.tls.hosts controls the control-plane frontend ALB; ingress.tls.hosts controls the data-plane API ALB (applied to both the primary and wildcard Ingresses). Rendered as spec.tls[*].hosts so the AWS Load Balancer Controller can match an ACM cert by SNI/SAN when certificateArn is empty. Leave certificateArn empty (with tls.enabled: true) and set tls.hosts to your hostname(s).
• Fixed ACM certificate auto-discovery on the data-plane wildcard Ingress and control-plane frontend Ingress. Previously, leaving certificateArn blank emitted an empty alb.ingress.kubernetes.io/certificate-arn annotation that ArgoCD rejected with a nil-annotation error. The annotation is now omitted entirely when certificateArn is empty. The control-plane frontend Ingress now contributes a spec.tls[*].hosts cert-discovery hint when frontendIngress.tls.hosts is set, so HTTPS listener creation no longer fails on overlays that rely on auto-discovery. Overlays that set an explicit certificateArn continue to render identically.

​

v0.104.0

• Action Required: common.dataplane.dpPublicUrl default changed to "" in data-plane/services/values.yaml. Set this to your environment’s public data plane URL (e.g., "https://api.my-dp.example.com"). Without it, the Admin Center Data Planes view and /settings/project/keys page show no URL.
• Action Required: dpIngestionService HPA defaults changed in data-plane/services/values.yaml. targetCPUUtilizationPercentage is now 60 (was 80), and a new targetMemoryUtilizationPercentage (default: 60) was added. To keep prior behavior, set CPU to 80 and memory to 0.
• Action Required: ingress.maxBodySize removed from both control-plane/services/values.yaml and data-plane/services/values.yaml. Remove any overrides for this key.
• Action Required: ingress.authHost removed from control-plane/services/values.yaml. Remove any overrides for this key.
• Action Required: jobs.serviceAccount.create now defaults to true in both planes. Set to false if you manage the jobs ServiceAccount externally.
• Action Required (ArgoCD users): Deployments omit spec.replicas when autoscaling is enabled. Add ignoreDifferences with jsonPointers: ["/spec/replicas"] for Deployment kind in your ArgoCD Application spec.
• New <service>.affinity (default: {}) for all CP/DP services and jobs.affinity for CronJobs/batch jobs. Supports podAffinity, podAntiAffinity, and nodeAffinity.
• New <service>.autoscaling.enabled (default: true) for all 11 CP/DP services. Set to false to disable HPA and pin replicas via <service>.replicas.
• New frontendIngress.tls.redirect (default: false) in control-plane/services/values.yaml. When true, the ALB listens on HTTP:80 and redirects to HTTPS.
• New dpIngestionService.resources (default: {}) in data-plane/services/values.yaml for per-service resource overrides on ingestion.
• New dpPythonmetricService.gunicornWorkers (default: 4) and dpPythonmetricService.pythonExecutionTimeout (default: 0.1) in data-plane/services/values.yaml for tuning custom metric concurrency and per-metric runtime.
• New cpNotificationService.ses.domain, .ses.rps (default: 14), .ses.senderRoleArn, .ses.senderRoleDurationSeconds (default: 3600) in control-plane/services/values.yaml for SES email sender configuration.
• New serviceMonitor.enabled (default: true), serviceMonitor.namespace (default: "monitoring"), serviceMonitor.labels (default: {release: monitoring}), serviceMonitor.interval (default: "30s"), serviceMonitor.scrapeTimeout (default: "10s") in both planes for Prometheus Operator ServiceMonitor support.
• New <service>.metricsPort (default: 9091), a dedicated Prometheus metrics port for all CP/DP services.
• ingress.host and ingress.grpcHost now accept a string or a list of strings for multi-host ingress support.
• Deployments no longer reset replica count on helm upgrade when autoscaling is enabled.
• Fixed NATS anti-affinity selector to correctly distinguish nats-box from nats server pods.
• Custom CA certs from common.tls.caCerts now apply to the DP S3 backfill Job.
• Added app.kubernetes.io/version label and OTel service.version attribute to all CP/DP services.
• Distributed tracing is now correctly emitted from dp-backend, dp-llmproxy, and dp-controller (the OTEL_ENABLED=true env var was previously missing on these deployments).
• Ingress routing improvements: /v1/events/*/annotate routes to dp-backend, /v1/* ingestion paths route to ingestion-service, API and frontend traffic use separate ALBs.
• Fixed PDB template to correctly handle minAvailable: 0 and maxUnavailable: 0.

​

v0.103.0

• New jobs configuration block in control-plane/services/values.yaml for Kubernetes CronJobs that process alert transitions
• New jobs configuration block in data-plane/services/values.yaml for operator-triggered S3 time-window backfill batch workloads
• New per-service keys: <service>.service.annotations (default: {}), <service>.service.labels (default: {}), <service>.service.type (default: "ClusterIP") now exposed on cpWriterService, cpControllerService, cpNotificationService, dpControllerService, dpIngestionService, dpEvaluationService, dpLlmproxyService, dpPythonmetricService — previously hardcoded
• Service port protocol hints (name: http, protocol: TCP, appProtocol: http/grpc) added to all service templates for correct L7 traffic classification by service meshes (Istio, Linkerd)
• appProtocol added to ClickHouse instance Service ports (http, interserver, metrics: appProtocol: http; native: appProtocol: tcp)
• ClickHouse instances chi-installation chart version bumped from 1.1.6 to 1.1.7
• No breaking changes — fully backward-compatible with v0.102.0 configurations

​

v0.102.0

• New podAnnotations (default: {}) in control-plane/infrastructure/clickhouse/clickhouse_instances/values.yaml for arbitrary annotations on ClickHouse pods (useful for Datadog autodiscovery, Prometheus scraping)
• Fixed missing DP_DATABASE_URL env var in dp-pythonmetric-service deployment template, now reads from common.externalSecrets.postgres.secretName / uriKey like all other data-plane services
• No breaking changes — fully backward-compatible with v0.101.0 configurations

​

v0.101.0

• Disabled ClickHouse replica check before attaching backup parts (CLICKHOUSE_CHECK_REPLICAS_BEFORE_ATTACH set to "false" in backup container) - prevents backup restore failures in environments where replica availability cannot be confirmed
• ClickHouse instances chi-installation chart version bumped from 1.1.5 to 1.1.6
• No breaking changes — fully backward-compatible with v0.100.0 configurations

​

v0.100.0

• New ingress.albClassName (default: "alb") and frontendIngress.albClassName (default: "alb") in control-plane and data-plane services for configurable ALB ingress class
  • Useful for shared CP+DP cluster scenarios where each plane needs its own ALB IngressClass (e.g., "cp-alb", "dp-alb")
  • Existing deployments using the default "alb" class require no changes
• New scheduling config for CP prometheus-nats-exporter in control-plane/infrastructure/nats/values.yaml: prometheusExporter.tolerations, prometheusExporter.nodeSelector, prometheusExporter.affinity, prometheusExporter.additionalLabels
• common.extraLabels now propagated to all Service, ServiceAccount, HPA, and Ingress resources across both planes (previously only applied to Deployments)

​

v0.99.3

• Action Required: Add common.controlPlane.id in data-plane/services/values.yaml - set it to match common.controlPlane.id from your control-plane values. Without this, dp-controller-service cannot identify its parent control plane, causing data-plane-to-control-plane communication failures
• New global.labels (default: {}) in control-plane/infrastructure/nats/values.yaml and data-plane/infrastructure/nats/values.yaml - applied to all NATS-generated Kubernetes resources (StatefulSet, Service, PVC, ConfigMap, PDB, etc.)
• New nack.additionalLabels, nack.tolerations, nack.nodeSelector, nack.affinity in control-plane NATS values for JetStream Controller scheduling
• prometheusExporter.additionalLabels now propagated to Prometheus NATS Exporter Deployment, Service, and ServiceMonitor labels in both planes
• Fixed topologySpreadConstraints typo in control-plane NATS values (topolicySpreadConstraints → topologySpreadConstraints)

​

v0.99.2

• ClickHouse instances chi-installation chart version bumped from 1.1.4 to 1.1.5 in control-plane/infrastructure/clickhouse/clickhouse_instances/Chart.yaml

​

v0.99.1

• Fixed cp-notification-service healthcheck port - renamed env var from EXPRESS_PORT to PORT in deployment template
• Control plane hive-control-plane chart version bumped from 0.2.0 to 0.2.1

​

v0.99.0

• Added readiness and liveness probes (GET /healthcheck) to all 9 service deployments across control-plane and data-plane
• cp-backend-service and dp-backend-service include a startup probe (allows up to 310s for Prisma migrations before liveness checks begin)
• Eliminates intermittent 502 errors during rolling updates caused by traffic routing to pods not yet ready to serve
• No values.yaml changes required - probes use hardcoded values in deployment templates

​

v0.98.9

• Unified encryption configuration for cp-controller, dp-controller, and dp-llmproxy-service
• Supports at-rest encryption for identity management and provider secrets via KMS or environment variable mode
• New common.encryption.keyId value in both control-plane and data-plane services
• New ExternalSecret templates for encryption in both control-plane and data-plane secret-store charts
• Replaced dpLlmproxyService.kmsKeyId with unified HH_ENCRYPTION_KEY_ID and HH_ENCRYPTION_SECRET env vars in dp-llmproxy-service
• Fixed perpetual ArgoCD OutOfSync caused by Redis PDB enabled: true in control-plane Redis
• Action Required: Remove dpLlmproxyService.kmsKeyId from data-plane values and add common.encryption.keyId in both control-plane and data-plane values

​

v0.98.1

• Added common.extraLabels for custom governance/compliance labels on all Kubernetes resources (control-plane, data-plane, shared dependencies)
• Added common.observability.otel.exporterProtocol (default: "grpc") for OTLP exporter protocol configuration in data-plane
• Added dpLlmproxyService.kmsKeyId (default: "alias/hh-provider-secrets") for AWS KMS encryption of LLM provider secrets
• Removed duplicate common.observability block in data-plane services values
• Fixed Next.js cache permission errors in cp-frontend-service with nextjs-cache emptyDir volume
• Fixed OTEL service name in cp-frontend-service (was hardcoded to cp-controller-service)
• Added custom CA certificate support (SSL_CERT_FILE, REQUESTS_CA_BUNDLE) for dp-llmproxy-service and dp-pythonmetric-service
• Added DP_DATABASE_URL env var and KMS config to dp-llmproxy-service
• Fixed ClickHouse logging configuration (moved to config.d/99-logger.xml with replace="1")
• Action Required: Set common.extraLabels if your organization requires specific labels on all resources

​

v0.90.17

• Added kube-prometheus-stack monitoring for both control-plane and data-plane (Prometheus, Grafana, Alertmanager with 30-day retention)
• Added Tempo for distributed tracing in both control-plane and data-plane
• Added Loki and Promtail for centralized log aggregation in control-plane
• Added legacy nats-old chart for backward compatibility
• Added Datadog integration support for OTEL collectors (disabled by default, set datadog.enabled: true)
• Added common.tls.caCerts dictionary for custom root CA certificates in data-plane
• Added common.controlPlane.apiPublicUrl for data-plane to call control-plane API
• Added resource limits for dp-llmproxy-service and dp-pythonmetric-service (500m/512Mi requests, 1000m/1Gi limits)
• Added persistent storage for ClickHouse Keeper (storage.enabled: true, storage.size: "10Gi")
• Updated ClickHouse Keeper image to altinity/clickhouse-keeper:25.3.6.10034.altinitystable-alpine
• Simplified ClickHouse Operator values from 903 lines to 29 lines
• Updated ExternalSecret API version from v1beta1 to v1 (requires External Secrets Operator 0.9.0+)
• Moved OTEL collector nodeSelector/affinity/tolerations under opentelemetry-collector key
• Action Required: Set common.controlPlane.apiPublicUrl to your control-plane API endpoint
• Action Required: Ensure ClickHouse Keeper persistent storage is enabled for production
• Action Required: Update External Secrets Operator to 0.9.0+ if not already

• Removed OpenUnison authentication infrastructure (all charts, operators, CRDs)
• Removed Nginx ingress infrastructure
• Added NATS infrastructure for data-plane with independent cluster deployment (3 replicas, JetStream, PDB)
• Disabled S3 DLQ and disk spool in writer service (cpWriterService.dlq.enabled: false)
• Added frontendIngress.alb.annotations for custom ALB annotations
• Removed PVC functionality from cp-writer-service
• Changed cp-frontend-service NEXTJS_PORT env var to PORT
• Added auth config env vars (AUTH_ISSUER_DOMAIN, AUTH_CLIENT_ID, AUTH_CLIENT_SECRET) to cp-frontend-service
• Added NATS connection settings for data-plane services (dp-evaluation-service, dp-ingestion-service)
• Enabled Redis authentication in control-plane (auth: true, existingSecret: redis-secrets)
• Removed gRPC ingress from data-plane services
• Switched from NLB to ALB for both control-plane and data-plane ingress
• Added NATS HA streams configuration with configurable replicas
• Added common.dataPlane.dpPublicUrl and common.controlPlane.frontendPublicUrl for cross-plane communication
• Added Prometheus monitoring for NATS (exporter on port 7777) and ClickHouse (built-in on port 9363)
• Added Redis authentication for data-plane (auth: true, existingSecret: redis-secrets)
• Fixed Redis PDB in data-plane (removed invalid enabled field)
• Action Required: Remove any NLB-related values overrides and switch to ALB configuration
• Action Required: Remove any OpenUnison or Beekeeper-related overrides from values files
• Action Required: Configure auth secrets in AWS Secrets Manager with client-secret and cp-jwt-private-key