feat(helm): add Prometheus operator ServiceMonitor support (#3847)

* feat(helm): add Prometheus operator ServiceMonitor support

The api (port 8888) and worker (port 8889) containers expose Prometheus
format metrics at /metrics (verified against app source); the chart had
no wiring for them — the worker's scrape annotations are gated behind
the unrelated podAnnotations value and the api service had nothing.

Add a gated metrics.serviceMonitor block that emits per-component
ServiceMonitor resources (api always when enabled, worker only when
worker.enabled). Selection labels are configurable for the Prometheus
operator's serviceMonitorSelector (e.g. release: kube-prometheus-stack).
Also scrape the dedicated worker in the dev LGTM compose stack, which
previously only scraped the api on :8888.

Verified end-to-end on k3d + kube-prometheus-stack: all three targets
(api + 2 worker pods via headless endpoints) discovered and up=1.

* fix(helm): address ServiceMonitor review

helm/hindsight/README.md

@@ -82,6 +82,8 @@ helm install hindsight ./helm/hindsight -n hindsight --create-namespace -f value
 | `postgresql.external.username` | Database username | `hindsight` |
 | `ingress.enabled` | Enable ingress | `false` |
 | `autoscaling.enabled` | Enable HPA | `false` |
+| `metrics.serviceMonitor.enabled` | Create ServiceMonitors for api/worker (needs Prometheus operator) | `false` |
+| `metrics.serviceMonitor.labels` | Labels for Prometheus operator selection, e.g. `release: kube-prometheus-stack` | `{}` |
 
 ### Environment Variables
 
@@ -163,6 +165,25 @@ ingress:
         - hindsight.example.com
 ```
 
+### Prometheus metrics
+
+The api (port 8888) and worker (port 8889) containers expose Prometheus
+format metrics at `/metrics`. The control plane does not expose metrics.
+On clusters running the Prometheus operator (e.g. kube-prometheus-stack),
+enable ServiceMonitor discovery:
+
+```yaml
+metrics:
+  serviceMonitor:
+    enabled: true
+    labels:
+      release: kube-prometheus-stack  # must match the stack's serviceMonitorSelector
+```
+
+The worker monitor requires `worker.enabled: true`. Without the operator,
+scrape `svc/<release>-api:8888/metrics` and `svc/<release>-worker:8889/metrics`
+directly with annotation-based discovery or static targets.
+
 ## Upgrading
 
 ```bash

helm/hindsight/templates/servicemonitor.yaml

@@ -0,0 +1,43 @@
+{{- if .Values.metrics.serviceMonitor.enabled }}
+{{- if .Values.api.enabled }}
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: {{ include "hindsight.fullname" . }}-api
+  labels:
+    {{- include "hindsight.api.labels" . | nindent 4 }}
+    {{- with .Values.metrics.serviceMonitor.labels }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
+spec:
+  selector:
+    matchLabels:
+      {{- include "hindsight.api.selectorLabels" . | nindent 6 }}
+  endpoints:
+    - port: http
+      path: {{ .Values.metrics.serviceMonitor.path }}
+      interval: {{ .Values.metrics.serviceMonitor.interval }}
+      scrapeTimeout: {{ .Values.metrics.serviceMonitor.scrapeTimeout }}
+{{- end }}
+---
+{{- if .Values.worker.enabled }}
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: {{ include "hindsight.fullname" . }}-worker
+  labels:
+    {{- include "hindsight.worker.labels" . | nindent 4 }}
+    {{- with .Values.metrics.serviceMonitor.labels }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
+spec:
+  selector:
+    matchLabels:
+      {{- include "hindsight.worker.selectorLabels" . | nindent 6 }}
+  endpoints:
+    - port: http
+      path: {{ .Values.metrics.serviceMonitor.path }}
+      interval: {{ .Values.metrics.serviceMonitor.interval }}
+      scrapeTimeout: {{ .Values.metrics.serviceMonitor.scrapeTimeout }}
+{{- end }}
+{{- end }}

helm/hindsight/values.yaml

@@ -501,3 +501,20 @@ autoscaling:
   maxReplicas: 10
   targetCPUUtilizationPercentage: 80
   targetMemoryUtilizationPercentage: 80
+
+# Metrics scraping via Prometheus Operator ServiceMonitors.
+# Requires the prometheus-operator CRDs (monitoring.coreos.com/v1) and a
+# Prometheus instance whose serviceMonitorSelector matches the labels below —
+# e.g. kube-prometheus-stack selects release: <stack-release-name> by default.
+# The api and worker containers serve Prometheus format at /metrics on their
+# service ports; the control plane (Next.js) does not expose metrics.
+metrics:
+  serviceMonitor:
+    enabled: false
+    # Labels added to the ServiceMonitor for Prometheus operator selection,
+    # e.g. { release: kube-prometheus-stack }
+    labels: {}
+    # Endpoints shared by the api and worker monitors
+    path: /metrics
+    interval: 30s
+    scrapeTimeout: 10s

scripts/dev/monitoring/prometheus.yml

@@ -28,3 +28,9 @@ scrape_configs:
       - targets: ['host.docker.internal:8888']
     metrics_path: '/metrics'
     scrape_interval: 5s
+  - job_name: 'hindsight-worker'
+    static_configs:
+      # Dedicated workers (helm: worker.enabled, port 8889). Harmless 404-less
+      # no-op when no worker runs on the host.
+      - targets: ['host.docker.internal:8889']
+    metrics_path: '/metrics'