Files
vectorize-io__hindsight/hindsight-api-slim/tests
Nicolò Boschi 2cd0561f14 feat(metrics): /metrics covers every worker (labelled api_worker), and event-loop lag as a histogram (#4319)
* feat(metrics): a metrics port per worker, and event-loop lag as a histogram

With --workers N every worker is its own process with its own metrics, but they
share one port, so a scrape of /metrics reaches one worker at random. Counters
jump between processes from one scrape to the next (a rate over them reads each
switch as a reset), and process_cpu_seconds_total describes a random worker, so
a worker whose event loop is saturated is invisible while the pod total still
looks like headroom.

- HINDSIGHT_API_METRICS_WORKER_BASE_PORT (default 0, off): each worker also
  serves its own registry on BASE + slot. The slot (0..N-1) is claimed with an
  exclusive flock on a per-slot lock file; the kernel drops it when the process
  exits, so a respawned worker takes over its predecessor's port. /metrics on
  the API port is unchanged.
- HINDSIGHT_API_LOOP_LAG_METRIC (default false): the existing loop-lag probe
  records every sample in a hindsight.event_loop.lag histogram (seconds, with
  sub-second buckets), independent of its log reports.

* feat(metrics): one /metrics covering every worker, labelled api_worker=<slot>

Replaces the per-worker ports from the previous commit. Labelling alone would
not fix the scrape: each scrape still reaches one worker, the others' series go
missing from about half the scrapes, and Prometheus marks them stale. So with
HINDSIGHT_API_METRICS_WORKER_LABEL on, each worker claims a slot (flock on a
per-slot lock file; the kernel frees it when the process exits) and publishes a
snapshot of its registry every 5 s to a directory the server's workers share.
/metrics, whichever worker answers, returns every live worker's series with
api_worker="<slot>": its own read live, the others from their latest snapshot,
never summed. A snapshot older than 15 s is a gone worker and is skipped; a
corrupt one is skipped without breaking the scrape.

One port and one scrape target, so no change to charts or scrape configs. The
event-loop lag histogram from the previous commit is unchanged.
2026-09-14 09:52:57 +02:00
..