feat(metrics): /metrics covers every worker (labelled api_worker), and event-loop lag as a histogram (#4319)

* feat(metrics): a metrics port per worker, and event-loop lag as a histogram

With --workers N every worker is its own process with its own metrics, but they
share one port, so a scrape of /metrics reaches one worker at random. Counters
jump between processes from one scrape to the next (a rate over them reads each
switch as a reset), and process_cpu_seconds_total describes a random worker, so
a worker whose event loop is saturated is invisible while the pod total still
looks like headroom.

- HINDSIGHT_API_METRICS_WORKER_BASE_PORT (default 0, off): each worker also
  serves its own registry on BASE + slot. The slot (0..N-1) is claimed with an
  exclusive flock on a per-slot lock file; the kernel drops it when the process
  exits, so a respawned worker takes over its predecessor's port. /metrics on
  the API port is unchanged.
- HINDSIGHT_API_LOOP_LAG_METRIC (default false): the existing loop-lag probe
  records every sample in a hindsight.event_loop.lag histogram (seconds, with
  sub-second buckets), independent of its log reports.

* feat(metrics): one /metrics covering every worker, labelled api_worker=<slot>

Replaces the per-worker ports from the previous commit. Labelling alone would
not fix the scrape: each scrape still reaches one worker, the others' series go
missing from about half the scrapes, and Prometheus marks them stale. So with
HINDSIGHT_API_METRICS_WORKER_LABEL on, each worker claims a slot (flock on a
per-slot lock file; the kernel frees it when the process exits) and publishes a
snapshot of its registry every 5 s to a directory the server's workers share.
/metrics, whichever worker answers, returns every live worker's series with
api_worker="<slot>": its own read live, the others from their latest snapshot,
never summed. A snapshot older than 15 s is a gone worker and is skipped; a
corrupt one is skipped without breaking the scrape.

One port and one scrape target, so no change to charts or scrape configs. The
event-loop lag histogram from the previous commit is unchanged.

.env.example

@@ -196,6 +196,10 @@ HINDSIGHT_API_LOG_LEVEL=info
 # HINDSIGHT_API_GZIP_MIN_SIZE=1024
 # Diagnostic: log event-loop lag percentiles every N seconds. 0 disables.
 # HINDSIGHT_API_LOOP_LAG_REPORT_SECONDS=0
+# Record every event-loop lag sample as a histogram (hindsight_event_loop_lag_seconds).
+# HINDSIGHT_API_LOOP_LAG_METRIC=false
+# With --workers N, make /metrics cover every worker, each series labelled api_worker=<slot>.
+# HINDSIGHT_API_METRICS_WORKER_LABEL=false
 # Record the diagnostic (subset) recall phase metrics.
 # HINDSIGHT_API_RECALL_DIAGNOSTIC_PHASES=true
 # Record 1 in N recall-phase metric observations (1 = all). Cuts metrics CPU at high recall rates.

hindsight-api-slim/hindsight_api/api/http.py

@@ -4729,7 +4729,7 @@ def create_app(
 
         # Started here rather than at import time because it needs a running loop, and it must run
         # on the loop that actually serves requests — that is the only one whose lag says anything.
-        _install_loop_lag(config.loop_lag_report_seconds)
+        _install_loop_lag(config.loop_lag_report_seconds, metric=config.loop_lag_metric)
 
         poller = None
         poller_task = None
@@ -4744,6 +4744,12 @@ def create_app(
             prometheus_reader = initialize_metrics(service_name="hindsight-api", service_version="1.0.0")
             create_metrics_collector()
             app.state.prometheus_reader = prometheus_reader
+            if config.metrics_worker_label:
+                # With --workers N a scrape of /metrics reaches one random worker; make every
+                # worker's series part of every scrape (see hindsight_api.metrics_multiworker).
+                from hindsight_api.metrics_multiworker import start_worker_metrics
+
+                app.state.worker_metrics = start_worker_metrics(max(1, config.workers))
             logging.info("Metrics initialized - available at /metrics endpoint")
         except Exception as e:
             logging.warning(f"Failed to initialize metrics: {e}. Metrics will be disabled (using no-op collector).")
@@ -5275,7 +5281,8 @@ def _register_routes(app: FastAPI):
         from fastapi.responses import Response
         from prometheus_client import CONTENT_TYPE_LATEST, generate_latest
 
-        metrics_data = generate_latest()
+        worker_metrics = getattr(app.state, "worker_metrics", None)
+        metrics_data = worker_metrics.render() if worker_metrics is not None else generate_latest()
         return Response(content=metrics_data, media_type=CONTENT_TYPE_LATEST)
 
     @app.get(

hindsight-api-slim/hindsight_api/config.py

@@ -663,6 +663,8 @@ ENV_RECALL_DIAGNOSTIC_PHASES = "HINDSIGHT_API_RECALL_DIAGNOSTIC_PHASES"
 ENV_RECALL_PHASE_SAMPLE_EVERY = "HINDSIGHT_API_RECALL_PHASE_SAMPLE_EVERY"
 ENV_GZIP_MIN_SIZE = "HINDSIGHT_API_GZIP_MIN_SIZE"
 ENV_LOOP_LAG_REPORT_SECONDS = "HINDSIGHT_API_LOOP_LAG_REPORT_SECONDS"
+ENV_LOOP_LAG_METRIC = "HINDSIGHT_API_LOOP_LAG_METRIC"
+ENV_METRICS_WORKER_LABEL = "HINDSIGHT_API_METRICS_WORKER_LABEL"
 ENV_LINK_EXPANSION_PER_ENTITY_LIMIT = "HINDSIGHT_API_LINK_EXPANSION_PER_ENTITY_LIMIT"
 ENV_LINK_EXPANSION_TIMEOUT = "HINDSIGHT_API_LINK_EXPANSION_TIMEOUT"
 ENV_RETAIN_BATCH_DOCUMENT_WRITES = "HINDSIGHT_API_RETAIN_BATCH_DOCUMENT_WRITES"
@@ -1472,6 +1474,8 @@ DEFAULT_RECALL_DIAGNOSTIC_PHASES = True  # Record the subset (diagnostic=true) r
 DEFAULT_RECALL_PHASE_SAMPLE_EVERY = 1  # Record 1 in N recall-phase observations; 1 records every one
 DEFAULT_GZIP_MIN_SIZE = 1024  # Min response bytes to gzip; negative disables compression
 DEFAULT_LOOP_LAG_REPORT_SECONDS = 0.0  # Event-loop lag probe report interval; 0 disables it
+DEFAULT_LOOP_LAG_METRIC = False  # Record every event-loop lag sample as a histogram
+DEFAULT_METRICS_WORKER_LABEL = False  # /metrics covers every worker, labelled api_worker=<slot>
 DEFAULT_LINK_EXPANSION_PER_ENTITY_LIMIT = 200  # Max target units per entity in graph expansion
 DEFAULT_LINK_EXPANSION_TIMEOUT = 10.0  # Timeout (seconds) for entity expansion query
 # The bank's own row (name/disposition/mission) and its config, cached per process so a
@@ -3046,6 +3050,8 @@ class HindsightConfig:
     recall_phase_sample_every: int
     gzip_min_size: int
     loop_lag_report_seconds: float
+    loop_lag_metric: bool
+    metrics_worker_label: bool
     link_expansion_per_entity_limit: int
     link_expansion_timeout: float
     retain_batch_document_writes: bool
@@ -4469,6 +4475,8 @@ class HindsightConfig:
             ),
             gzip_min_size=int(os.getenv(ENV_GZIP_MIN_SIZE, str(DEFAULT_GZIP_MIN_SIZE))),
             loop_lag_report_seconds=float(os.getenv(ENV_LOOP_LAG_REPORT_SECONDS, str(DEFAULT_LOOP_LAG_REPORT_SECONDS))),
+            loop_lag_metric=_parse_boolean_env(ENV_LOOP_LAG_METRIC, DEFAULT_LOOP_LAG_METRIC),
+            metrics_worker_label=_parse_boolean_env(ENV_METRICS_WORKER_LABEL, DEFAULT_METRICS_WORKER_LABEL),
             link_expansion_per_entity_limit=int(
                 os.getenv(ENV_LINK_EXPANSION_PER_ENTITY_LIMIT, str(DEFAULT_LINK_EXPANSION_PER_ENTITY_LIMIT))
             ),

hindsight-api-slim/hindsight_api/loop_lag.py

@@ -12,8 +12,9 @@ other callbacks (or blocked in a synchronous call) while this one was ready. If
 requests are slow, the time is in a real await and the phases are missing one; if lag tracks
 request latency, the loop is oversubscribed and no amount of I/O tuning helps.
 
-Enabled by HINDSIGHT_API_LOOP_LAG_REPORT_SECONDS (seconds between reports); 0 means the task never
-starts.
+HINDSIGHT_API_LOOP_LAG_REPORT_SECONDS sets the seconds between log reports (0: no reports), and
+HINDSIGHT_API_LOOP_LAG_METRIC records every sample in the ``hindsight.event_loop.lag`` histogram.
+With neither set the task never starts.
 """
 
 from __future__ import annotations
@@ -41,15 +42,26 @@ def _percentile(sorted_lags: list[float], p: float) -> float:
     return sorted_lags[min(len(sorted_lags) - 1, int(len(sorted_lags) * p / 100))]
 
 
-async def _run(report_every: float) -> None:
+async def _run(report_every: float, *, record_metric: bool) -> None:
+    from hindsight_api.metrics import get_metrics_collector
+
     pid = os.getpid()
+    # Without reports the window only bounds how long `lags` grows before it is dropped.
+    window = report_every if report_every > 0 else 10.0
     while True:
         lags: list[float] = []
-        deadline = time.monotonic() + report_every
+        deadline = time.monotonic() + window
         while time.monotonic() < deadline:
             t0 = time.monotonic()
             await asyncio.sleep(_TICK_S)
-            lags.append((time.monotonic() - t0 - _TICK_S) * 1000.0)
+            lag_s = max(0.0, time.monotonic() - t0 - _TICK_S)
+            lags.append(lag_s * 1000.0)
+            if record_metric:
+                # Looked up per sample: the API lifespan installs the real collector after the probe
+                # starts, and a test may swap it.
+                get_metrics_collector().record_loop_lag(lag_s)
+        if report_every <= 0:
+            continue
         lags.sort()
         logger.info(
             "[loop-lag] pid=%d n=%d p50=%.1fms p90=%.1fms p99=%.1fms max=%.1fms",
@@ -62,13 +74,18 @@ async def _run(report_every: float) -> None:
         )
 
 
-def install(report_every: float) -> asyncio.Task[None] | None:
-    """Start the probe on the running loop. No-op when `report_every` is 0 (the default)."""
-    if report_every <= 0:
+def install(report_every: float, *, metric: bool = False) -> asyncio.Task[None] | None:
+    """Start the probe on the running loop.
+
+    No-op when neither log reports (`report_every` > 0) nor the histogram (`metric`) is enabled,
+    which is the default.
+    """
+    if report_every <= 0 and not metric:
         return None
-    report_every = max(_MIN_REPORT_S, report_every)
-    task = asyncio.get_running_loop().create_task(_run(report_every))
+    if report_every > 0:
+        report_every = max(_MIN_REPORT_S, report_every)
+    task = asyncio.get_running_loop().create_task(_run(report_every, record_metric=metric))
     _tasks.add(task)
     task.add_done_callback(_tasks.discard)
-    logger.info("[loop-lag] armed: reporting every %.0fs", report_every)
+    logger.info("[loop-lag] armed: reports every %.0fs, histogram %s", report_every, "on" if metric else "off")
     return task

hindsight-api-slim/hindsight_api/metrics.py

@@ -165,6 +165,10 @@ logger = logging.getLogger(__name__)
 _meter = None
 
 
+#: Event-loop lag in seconds. Healthy is well under 10 ms; a saturated loop runs into seconds.
+LOOP_LAG_BUCKETS = (0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0)
+
+
 def initialize_metrics(service_name: str = "hindsight-api", service_version: str = "1.0.0"):
     """
     Initialize OpenTelemetry metrics with Prometheus exporter.
@@ -213,7 +217,15 @@ def initialize_metrics(service_name: str = "hindsight-api", service_version: str
     provider = MeterProvider(
         resource=resource,
         metric_readers=[prometheus_reader],
-        views=[duration_view, llm_duration_view, http_duration_view],
+        views=[
+            duration_view,
+            llm_duration_view,
+            http_duration_view,
+            View(
+                instrument_name="hindsight.event_loop.lag",
+                aggregation=ExplicitBucketHistogramAggregation(boundaries=LOOP_LAG_BUCKETS),
+            ),
+        ],
     )
 
     # Set the global meter provider
@@ -321,6 +333,13 @@ class MetricsCollectorBase:
         """Record a detected event-loop stall (blocked longer than the watchdog threshold)."""
         raise NotImplementedError
 
+    def record_loop_lag(self, lag_seconds: float):
+        """Record one event-loop lag sample (see ``hindsight_api.loop_lag``).
+
+        A no-op here rather than abstract: the probe calls it on every tick, and a collector that
+        predates it must not kill the probe.
+        """
+
     def record_consolidation_batch_failure(self, failure_class: str, error_type: str):
         """Record one consolidation LLM batch call that failed.
 
@@ -407,6 +426,10 @@ class NoOpMetricsCollector(MetricsCollectorBase):
         """No-op loop-stall recording."""
         pass
 
+    def record_loop_lag(self, lag_seconds: float):
+        """No-op loop-lag recording."""
+        pass
+
     def record_consolidation_batch_failure(self, failure_class: str, error_type: str):
         """No-op consolidation batch-failure recording."""
         pass
@@ -603,6 +626,14 @@ class MetricsCollector(MetricsCollectorBase):
             description="Duration of detected event-loop stalls in seconds",
             unit="s",
         )
+        # How long a ready coroutine waited for the loop (see hindsight_api.loop_lag). Unlike a
+        # stall, which only counts blocks past a threshold, this is the whole distribution, so a
+        # loop that is busy but never blocked still shows up.
+        self.event_loop_lag = self.meter.create_histogram(
+            name="hindsight.event_loop.lag",
+            description="Event-loop lag: extra time a ready coroutine waited before it ran",
+            unit="s",
+        )
 
         # Process metrics (observable gauges - collected on scrape)
         self._setup_process_metrics()
@@ -884,6 +915,10 @@ class MetricsCollector(MetricsCollectorBase):
         self.event_loop_stalls.add(1)
         self.event_loop_stall_duration.record(stall_seconds)
 
+    def record_loop_lag(self, lag_seconds: float):
+        """Record one event-loop lag sample. Called by the probe on every tick."""
+        self.event_loop_lag.record(lag_seconds)
+
     def record_consolidation_batch_failure(self, failure_class: str, error_type: str):
         """Record one failed consolidation LLM batch call.
 

hindsight-api-slim/hindsight_api/metrics_multiworker.py

@@ -0,0 +1,204 @@
+"""One ``/metrics`` that covers every worker, each series labelled with the worker it came from.
+
+With ``--workers N`` every uvicorn worker is its own process with its own metrics registry, but
+they share one listening socket, so a scrape of ``/metrics`` reaches ONE worker, picked by the
+kernel. Two consequences:
+
+* counters and histograms jump between processes from one scrape to the next, and a monotonic
+  counter that goes backwards reads to Prometheus as a reset -- rates over them are wrong;
+* per-process series such as ``process_cpu_seconds_total`` describe a random worker, so a worker
+  whose event loop is saturated is invisible while the pod total still looks like headroom.
+
+Labelling each worker's series is not enough on its own: a scrape would still return one worker's
+series, the others would be missing from about half the scrapes, and Prometheus marks a missing
+series stale. So when ``HINDSIGHT_API_METRICS_WORKER_LABEL`` is on, every worker:
+
+* claims a slot (0..N-1) with an exclusive ``flock`` on a per-slot lock file -- the kernel drops it
+  when the process exits, so a worker the supervisor respawns reuses its predecessor's slot and
+  the label stays bounded to N values;
+* writes its registry's exposition to a directory shared by the server's workers every
+  ``SNAPSHOT_INTERVAL_S``;
+* answers ``/metrics`` with every live worker's series, each carrying ``api_worker="<slot>"``: its
+  own read live, the others from their latest snapshot. A snapshot older than ``STALE_AFTER_S`` is
+  a worker that is gone, and is skipped.
+
+One port, one scrape target, and every worker in every scrape.
+"""
+
+from __future__ import annotations
+
+import fcntl
+import glob
+import logging
+import os
+import tempfile
+import threading
+import time
+from collections.abc import Iterable
+from typing import IO
+
+from prometheus_client import REGISTRY, CollectorRegistry, generate_latest
+from prometheus_client.metrics_core import Metric
+from prometheus_client.parser import text_string_to_metric_families
+
+logger = logging.getLogger(__name__)
+
+#: The label every series gets. Not ``worker``: the task poller already uses that key.
+LABEL = "api_worker"
+
+#: How often a worker publishes its snapshot. Well under a typical 15-30 s scrape interval, so the
+#: other workers' series are at most this old.
+SNAPSHOT_INTERVAL_S = 5.0
+
+#: A snapshot older than this belongs to a worker that is gone (crashed, or replaced and not yet
+#: republished under the same slot).
+STALE_AFTER_S = 3 * SNAPSHOT_INTERVAL_S
+
+
+def shared_dir() -> str:
+    """The directory this server's workers share. Keyed by the supervisor (the workers' parent),
+    so two servers on one host, or a restarted server, never read each other's snapshots."""
+    return os.path.join(tempfile.gettempdir(), f"hindsight-metrics-{os.getppid()}")
+
+
+def claim_slot(directory: str, slots: int) -> tuple[int, IO[str]] | None:
+    """Claim the lowest free slot in ``0..slots-1``; return ``(slot, lock_file)`` or None.
+
+    The caller must keep ``lock_file`` open for as long as it publishes under the slot.
+    """
+    os.makedirs(directory, exist_ok=True)
+    for slot in range(slots):
+        handle = open(os.path.join(directory, f"slot-{slot}.lock"), "w")
+        try:
+            fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except OSError:
+            handle.close()
+            continue
+        handle.write(str(os.getpid()))
+        handle.flush()
+        return slot, handle
+    return None
+
+
+class _Families:
+    """A collector that yields already-built metric families."""
+
+    def __init__(self, families: Iterable[Metric]) -> None:
+        self._families = list(families)
+
+    def collect(self) -> Iterable[Metric]:
+        return iter(self._families)
+
+
+def merge_expositions(parts: Iterable[tuple[str, str]]) -> bytes:
+    """Merge ``(slot, exposition text)`` pairs into one exposition, labelling every sample.
+
+    Series are kept apart per worker, never summed: summing is a query-time decision, and a sum
+    would hide exactly the per-worker skew this exists to show.
+    """
+    families: dict[str, Metric] = {}
+    for slot, text in parts:
+        for family in text_string_to_metric_families(text):
+            merged = families.get(family.name)
+            if merged is None:
+                merged = Metric(family.name, family.documentation, family.type, family.unit)
+                families[family.name] = merged
+            for sample in family.samples:
+                merged.add_sample(
+                    sample.name, {**sample.labels, LABEL: slot}, sample.value, sample.timestamp, sample.exemplar
+                )
+    registry = CollectorRegistry(auto_describe=False)
+    registry.register(_Families(families.values()))
+    return generate_latest(registry)
+
+
+class WorkerMetrics:
+    """This worker's share of the multi-worker ``/metrics``: publishes its snapshot, renders all."""
+
+    def __init__(
+        self,
+        directory: str,
+        slot: int,
+        lock_file: IO[str] | None = None,
+        registry: CollectorRegistry = REGISTRY,
+    ) -> None:
+        self.directory = directory
+        self.slot = slot
+        self._lock_file = lock_file  # held open: closing it would release the slot
+        self._registry = registry
+        self._stop = threading.Event()
+        self._thread: threading.Thread | None = None
+        self._warned: set[str] = set()
+
+    def _snapshot_path(self, slot: int) -> str:
+        return os.path.join(self.directory, f"worker-{slot}.prom")
+
+    def write_snapshot(self) -> None:
+        """Publish this worker's exposition, atomically (a reader never sees half a file)."""
+        path = self._snapshot_path(self.slot)
+        tmp = f"{path}.{os.getpid()}.tmp"
+        with open(tmp, "wb") as f:
+            f.write(generate_latest(self._registry))
+        os.replace(tmp, path)
+
+    def start(self, interval_s: float = SNAPSHOT_INTERVAL_S) -> None:
+        self.write_snapshot()
+
+        def _loop() -> None:
+            while not self._stop.wait(interval_s):
+                try:
+                    self.write_snapshot()
+                except Exception as e:  # never let the publisher die silently or kill the worker
+                    self._warn_once("snapshot", f"[metrics] could not publish worker snapshot: {e}")
+
+        self._thread = threading.Thread(target=_loop, name="hindsight-metrics-snapshot", daemon=True)
+        self._thread.start()
+
+    def stop(self) -> None:
+        self._stop.set()
+
+    def render(self, *, stale_after_s: float = STALE_AFTER_S) -> bytes:
+        """Every live worker's series, each labelled with its slot. This worker's are read live."""
+        parts: list[tuple[str, str]] = [(str(self.slot), generate_latest(self._registry).decode())]
+        now = time.time()
+        for path in sorted(glob.glob(os.path.join(self.directory, "worker-*.prom"))):
+            slot = os.path.basename(path)[len("worker-") : -len(".prom")]
+            if slot == str(self.slot):
+                continue
+            try:
+                if now - os.stat(path).st_mtime > stale_after_s:
+                    continue
+                with open(path, encoding="utf-8") as f:
+                    text = f.read()
+                list(text_string_to_metric_families(text))  # reject a corrupt snapshot up front
+            except Exception as e:
+                self._warn_once(f"read:{slot}", f"[metrics] skipping worker {slot}'s snapshot: {e}")
+                continue
+            parts.append((slot, text))
+        return merge_expositions(parts)
+
+    def _warn_once(self, key: str, message: str) -> None:
+        if key not in self._warned:
+            self._warned.add(key)
+            logger.warning(message)
+
+
+def start_worker_metrics(slots: int, *, directory: str | None = None) -> WorkerMetrics | None:
+    """Claim a slot and start publishing. Returns None -- plain ``/metrics`` -- if no slot is free.
+
+    Never raises: metrics must not stop the API from starting.
+    """
+    directory = directory or shared_dir()
+    try:
+        claimed = claim_slot(directory, max(1, slots))
+        if claimed is None:
+            logger.warning("[metrics] no free worker slot in %s; this worker serves only its own metrics", directory)
+            return None
+        slot, lock_file = claimed
+        worker = WorkerMetrics(directory, slot, lock_file)
+        worker.start()
+        logger.info("[metrics] worker-labelled metrics: slot %d (pid %d)", slot, os.getpid())
+        return worker
+    except Exception as e:
+        logger.warning("[metrics] worker-labelled metrics disabled: %s", e)
+        return None

hindsight-api-slim/tests/test_loop_lag.py

@@ -34,3 +34,35 @@ async def test_reports_lag_percentiles(monkeypatch: pytest.MonkeyPatch, caplog:
             await task
     await asyncio.sleep(0)  # done callbacks run on the next loop iteration
     assert task not in loop_lag._tasks
+
+
+async def test_records_samples_as_a_metric_without_log_reports(
+    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
+) -> None:
+    """With reports off and the metric on, every tick is recorded and nothing is logged."""
+    import hindsight_api.metrics as metrics
+
+    monkeypatch.setattr(loop_lag, "_TICK_S", 0.001)
+    samples: list[float] = []
+
+    class Recorder(metrics.NoOpMetricsCollector):
+        def record_loop_lag(self, lag_seconds: float) -> None:
+            samples.append(lag_seconds)
+
+    monkeypatch.setattr(metrics, "get_metrics_collector", lambda: Recorder())
+    caplog.set_level(logging.INFO, logger=loop_lag.__name__)
+
+    task = loop_lag.install(0, metric=True)
+    assert task is not None
+    try:
+        for _ in range(100):
+            if len(samples) >= 5:
+                break
+            await asyncio.sleep(0.01)
+        assert len(samples) >= 5
+        assert all(s >= 0.0 for s in samples)
+        assert not any("p99=" in r.getMessage() for r in caplog.records)
+    finally:
+        task.cancel()
+        with pytest.raises(asyncio.CancelledError):
+            await task

hindsight-api-slim/tests/test_metrics_multiworker.py

@@ -0,0 +1,114 @@
+"""One /metrics covering every worker, each series labelled with its worker.
+
+What matters: every live worker's series appear in a single scrape, they are kept apart (never
+summed), the answering worker's own values are current, a gone worker drops out, and a bad
+snapshot never breaks the scrape.
+"""
+
+import os
+import time
+
+from prometheus_client import CollectorRegistry, Counter, Histogram
+from prometheus_client.parser import text_string_to_metric_families
+
+from hindsight_api import metrics_multiworker as mw
+
+
+def _registry(requests: float, latency: float) -> CollectorRegistry:
+    r = CollectorRegistry()
+    Counter("demo_requests", "requests", ["route"], registry=r).labels("/recall").inc(requests)
+    Histogram("demo_seconds", "latency", registry=r, buckets=(0.1, 1.0)).observe(latency)
+    return r
+
+
+def _samples(exposition: bytes) -> dict[tuple[str, tuple], float]:
+    out = {}
+    for family in text_string_to_metric_families(exposition.decode()):
+        for s in family.samples:
+            out[(s.name, tuple(sorted(s.labels.items())))] = s.value
+    return out
+
+
+def _worker(tmp_path, slot: int, registry: CollectorRegistry) -> mw.WorkerMetrics:
+    w = mw.WorkerMetrics(str(tmp_path), slot, registry=registry)
+    w.write_snapshot()
+    return w
+
+
+def test_every_worker_appears_in_one_scrape_labelled_and_not_summed(tmp_path):
+    w0 = _worker(tmp_path, 0, _registry(requests=3, latency=0.05))
+    _worker(tmp_path, 1, _registry(requests=5, latency=0.5))
+
+    got = _samples(w0.render())
+
+    assert got[("demo_requests_total", (("api_worker", "0"), ("route", "/recall")))] == 3
+    assert got[("demo_requests_total", (("api_worker", "1"), ("route", "/recall")))] == 5
+    assert got[("demo_seconds_bucket", (("api_worker", "0"), ("le", "0.1")))] == 1
+    assert got[("demo_seconds_bucket", (("api_worker", "1"), ("le", "0.1")))] == 0
+    # Nothing without the label: no series from an unknown worker.
+    assert all(dict(labels).get("api_worker") in ("0", "1") for _, labels in got)
+
+
+def test_the_answering_workers_own_series_are_live_not_its_snapshot(tmp_path):
+    registry = CollectorRegistry()
+    counter = Counter("demo_requests", "requests", registry=registry)
+    w0 = _worker(tmp_path, 0, registry)  # snapshot taken at 0
+    counter.inc(7)
+
+    got = _samples(w0.render())
+
+    assert got[("demo_requests_total", (("api_worker", "0"),))] == 7
+
+
+def test_a_gone_workers_stale_snapshot_is_skipped(tmp_path):
+    w0 = _worker(tmp_path, 0, _registry(requests=1, latency=0.05))
+    _worker(tmp_path, 1, _registry(requests=9, latency=0.05))
+    old = time.time() - mw.STALE_AFTER_S - 5
+    os.utime(tmp_path / "worker-1.prom", (old, old))
+
+    got = _samples(w0.render())
+
+    assert not any(dict(labels).get("api_worker") == "1" for _, labels in got)
+    assert got[("demo_requests_total", (("api_worker", "0"), ("route", "/recall")))] == 1
+
+
+def test_a_corrupt_snapshot_is_skipped_without_breaking_the_scrape(tmp_path):
+    w0 = _worker(tmp_path, 0, _registry(requests=2, latency=0.05))
+    (tmp_path / "worker-1.prom").write_text("this is {not an exposition\n")
+
+    got = _samples(w0.render())
+
+    assert got[("demo_requests_total", (("api_worker", "0"), ("route", "/recall")))] == 2
+
+
+def test_the_merged_output_round_trips_through_the_parser(tmp_path):
+    w0 = _worker(tmp_path, 0, _registry(requests=1, latency=0.2))
+    _worker(tmp_path, 1, _registry(requests=2, latency=2.0))
+
+    families = {f.name: f.type for f in text_string_to_metric_families(w0.render().decode())}
+
+    assert families["demo_requests"] == "counter"
+    assert families["demo_seconds"] == "histogram"
+
+
+def test_slots_are_distinct_and_a_released_slot_is_reused(tmp_path):
+    first = mw.claim_slot(str(tmp_path), 2)
+    second = mw.claim_slot(str(tmp_path), 2)
+    assert (first[0], second[0]) == (0, 1)
+    assert mw.claim_slot(str(tmp_path), 2) is None  # both held
+    first[1].close()  # the worker holding slot 0 exits
+    again = mw.claim_slot(str(tmp_path), 2)
+    assert again is not None and again[0] == 0
+    again[1].close()
+    second[1].close()
+
+
+def test_start_publishes_immediately_and_never_raises(tmp_path):
+    worker = mw.start_worker_metrics(2, directory=str(tmp_path))
+    try:
+        assert worker is not None
+        assert (tmp_path / f"worker-{worker.slot}.prom").exists()
+    finally:
+        worker.stop()
+    # No writable directory: disabled, not an exception.
+    assert mw.start_worker_metrics(2, directory="/proc/definitely-not-writable/x") is None

hindsight-docs/docs/developer/configuration.md

@@ -1405,6 +1405,8 @@ For advanced authentication (JWT, OAuth, multi-tenant schemas), implement a cust
 | `HINDSIGHT_API_MCP_ENABLED` | Enable MCP server at `/mcp/{bank_id}/` | `true` |
 | `HINDSIGHT_API_GZIP_MIN_SIZE` | Minimum response size (bytes) to gzip. Compressing a recall response costs ~5% of its CPU, so a CPU-bound (rather than bandwidth-bound) deployment can raise this past its typical response size. Negative disables compression entirely. | `1024` |
 | `HINDSIGHT_API_LOOP_LAG_REPORT_SECONDS` | Diagnostic: log event-loop lag percentiles (`[loop-lag]`) every N seconds (minimum 1), to tell a slow await from an oversubscribed loop. `0` disables the probe. | `0` |
+| `HINDSIGHT_API_LOOP_LAG_METRIC` | Record every event-loop lag sample in the `hindsight_event_loop_lag_seconds` histogram (sampled every 50 ms), independent of the log reports above. A loop that is busy but never blocked shows up here and nowhere else: the recall phase timers stay fast while requests wait for the loop. | `false` |
+| `HINDSIGHT_API_METRICS_WORKER_LABEL` | With `--workers N` every worker is its own process with its own metrics, but they share one port, so a scrape of `/metrics` reaches one worker at random: counters jump between processes (a rate over them reads every switch as a reset) and a saturated worker is invisible. When on, each worker publishes a snapshot of its metrics every 5 s and `/metrics` returns every live worker's series, each labelled `api_worker="<slot>"` (slot `0..N-1`). One port and one scrape target; the extra label is the only visible change. | `false` |
 | `HINDSIGHT_API_TOKENIZER_ENCODING` | Vocabulary used for every token count and chunk boundary (recall budgets, chunk sizes, prompt fitting, embedding truncation). `o200k_base` matches current OpenAI models and counts non-Latin text far closer to what they actually charge; `cl100k_base` reproduces the counts Hindsight produced before this default changed. Server-level: token budgets are only comparable between banks if they are all counted the same way. Other bundled vocabulary: `o200k_harmony`. | `o200k_base` |
 | `HINDSIGHT_API_MODEL_INIT_TIMEOUT` | Wall-clock cap (seconds) on startup model/connection initialization. If embeddings, the cross-encoder, or LLM verification block (e.g. an offline model download or an unreachable provider), the server fails fast with a clear error instead of hanging forever. Increase if a legitimate first-time model download needs more time. | `300` |
 | `HINDSIGHT_API_STARTUP_WAIT_SECONDS` | **Docker image only.** How long the container waits for the API to answer `/health` before it stops and restarts. Raising `HINDSIGHT_API_MODEL_INIT_TIMEOUT` above the default raises this wait too, so a slow first-time model download is not cut short; set this to override the wait on its own. | `300`, or `HINDSIGHT_API_MODEL_INIT_TIMEOUT` + 30s when that is longer |

skills/hindsight-docs/references/developer/configuration.md

@@ -1405,6 +1405,8 @@ For advanced authentication (JWT, OAuth, multi-tenant schemas), implement a cust
 | `HINDSIGHT_API_MCP_ENABLED` | Enable MCP server at `/mcp/{bank_id}/` | `true` |
 | `HINDSIGHT_API_GZIP_MIN_SIZE` | Minimum response size (bytes) to gzip. Compressing a recall response costs ~5% of its CPU, so a CPU-bound (rather than bandwidth-bound) deployment can raise this past its typical response size. Negative disables compression entirely. | `1024` |
 | `HINDSIGHT_API_LOOP_LAG_REPORT_SECONDS` | Diagnostic: log event-loop lag percentiles (`[loop-lag]`) every N seconds (minimum 1), to tell a slow await from an oversubscribed loop. `0` disables the probe. | `0` |
+| `HINDSIGHT_API_LOOP_LAG_METRIC` | Record every event-loop lag sample in the `hindsight_event_loop_lag_seconds` histogram (sampled every 50 ms), independent of the log reports above. A loop that is busy but never blocked shows up here and nowhere else: the recall phase timers stay fast while requests wait for the loop. | `false` |
+| `HINDSIGHT_API_METRICS_WORKER_LABEL` | With `--workers N` every worker is its own process with its own metrics, but they share one port, so a scrape of `/metrics` reaches one worker at random: counters jump between processes (a rate over them reads every switch as a reset) and a saturated worker is invisible. When on, each worker publishes a snapshot of its metrics every 5 s and `/metrics` returns every live worker's series, each labelled `api_worker="<slot>"` (slot `0..N-1`). One port and one scrape target; the extra label is the only visible change. | `false` |
 | `HINDSIGHT_API_TOKENIZER_ENCODING` | Vocabulary used for every token count and chunk boundary (recall budgets, chunk sizes, prompt fitting, embedding truncation). `o200k_base` matches current OpenAI models and counts non-Latin text far closer to what they actually charge; `cl100k_base` reproduces the counts Hindsight produced before this default changed. Server-level: token budgets are only comparable between banks if they are all counted the same way. Other bundled vocabulary: `o200k_harmony`. | `o200k_base` |
 | `HINDSIGHT_API_MODEL_INIT_TIMEOUT` | Wall-clock cap (seconds) on startup model/connection initialization. If embeddings, the cross-encoder, or LLM verification block (e.g. an offline model download or an unreachable provider), the server fails fast with a clear error instead of hanging forever. Increase if a legitimate first-time model download needs more time. | `300` |
 | `HINDSIGHT_API_STARTUP_WAIT_SECONDS` | **Docker image only.** How long the container waits for the API to answer `/health` before it stops and restarts. Raising `HINDSIGHT_API_MODEL_INIT_TIMEOUT` above the default raises this wait too, so a slow first-time model download is not cut short; set this to override the wait on its own. | `300`, or `HINDSIGHT_API_MODEL_INIT_TIMEOUT` + 30s when that is longer |