mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
08a119eeb6
Two independent gaps meant a deployment that looked fully instrumented was instrumented on one half, and its traces never linked up with the caller's. Worker emits no spans (#3614). initialize_tracing() was called from exactly one place — the FastAPI lifespan — and the standalone `hindsight-worker` entrypoint never goes through it. Since both tracing chokepoints degrade to deliberate no-ops, consolidation, batch retain and mental-model refresh — most of the long-running work and token spend — produced nothing, with no error or warning to say so. The bootstrap moves into a shared tracing.initialize_tracing_from_config() that both entrypoints call. Workers default to the service name "hindsight-worker", matching the name they already report for metrics, while an explicit HINDSIGHT_API_OTEL_SERVICE_NAME still wins. No trace-context propagation (#3604). Every operation opened a new root span, so a caller's request and the Hindsight work it triggered were two unrelated traces. The ASGI instrumentation — already a declared dependency, previously unused — now extracts W3C traceparent and opens a SERVER span, which the engine's existing spans nest under through the ambient context, with no changes at those call sites. Requests without a traceparent still start their own root trace. Health and metrics URLs are excluded so probe traffic doesn't drown out real work, and per-ASGI-message spans are excluded both to cut span volume and because that leaves the raw receive callable untouched for ClientDisconnectCancellationMiddleware (#2122). Also adds a shutdown flush, so spans still queued in the BatchSpanProcessor survive SIGTERM — a consolidation span can be minutes long — and reports the real package version in service.version instead of a hardcoded 0.4.8.
73 lines
2.7 KiB
Python
73 lines
2.7 KiB
Python
"""Tests for hindsight_api.worker.main entry-point helpers."""
|
|
|
|
import asyncio
|
|
import signal
|
|
from unittest.mock import MagicMock
|
|
|
|
from hindsight_api.worker.main import _install_shutdown_signal_handlers
|
|
|
|
|
|
def test_install_shutdown_signal_handlers_unix_path():
|
|
"""On platforms where asyncio supports signal handlers (Unix), both
|
|
SIGINT and SIGTERM are registered and the helper reports success."""
|
|
loop = MagicMock(spec=asyncio.AbstractEventLoop)
|
|
handler = MagicMock()
|
|
|
|
installed = _install_shutdown_signal_handlers(loop, handler)
|
|
|
|
assert installed is True
|
|
loop.add_signal_handler.assert_any_call(signal.SIGINT, handler)
|
|
loop.add_signal_handler.assert_any_call(signal.SIGTERM, handler)
|
|
assert loop.add_signal_handler.call_count == 2
|
|
|
|
|
|
def test_install_shutdown_signal_handlers_windows_path():
|
|
"""On Windows, asyncio's ProactorEventLoop raises NotImplementedError
|
|
from add_signal_handler. The helper must swallow it and report failure
|
|
so the worker keeps running with default Python signal behavior
|
|
(regression test for issue #1411)."""
|
|
loop = MagicMock(spec=asyncio.AbstractEventLoop)
|
|
loop.add_signal_handler.side_effect = NotImplementedError
|
|
handler = MagicMock()
|
|
|
|
installed = _install_shutdown_signal_handlers(loop, handler)
|
|
|
|
assert installed is False
|
|
|
|
|
|
def test_main_bootstraps_tracing_for_the_worker_process(monkeypatch):
|
|
"""The standalone worker must initialize tracing itself.
|
|
|
|
initialize_tracing() used to be called only from the FastAPI lifespan, so a
|
|
`hindsight-worker` process emitted no spans at all — silently, since both
|
|
tracing chokepoints degrade to no-ops (issue #3614). It also identifies
|
|
itself as "hindsight-worker" by default, matching the name it already
|
|
reports for metrics.
|
|
"""
|
|
import dataclasses
|
|
import sys
|
|
|
|
from hindsight_api import tracing
|
|
from hindsight_api.config import _get_raw_config
|
|
from hindsight_api.worker import main as worker_main
|
|
|
|
config = dataclasses.replace(_get_raw_config(), worker_id="test-worker")
|
|
monkeypatch.setattr(config, "configure_logging", lambda: None)
|
|
monkeypatch.setattr(worker_main, "get_config", lambda: config)
|
|
monkeypatch.setattr(worker_main, "load_dotenv_for_entrypoint", lambda: None)
|
|
monkeypatch.setattr(sys, "argv", ["hindsight-worker"])
|
|
|
|
bootstrap_calls = []
|
|
|
|
def _record(cfg, **kwargs):
|
|
bootstrap_calls.append(kwargs)
|
|
return False
|
|
|
|
monkeypatch.setattr(tracing, "initialize_tracing_from_config", _record)
|
|
# Stop before the worker actually runs; we only care about the bootstrap.
|
|
monkeypatch.setattr(worker_main.asyncio, "run", lambda coro: coro.close())
|
|
|
|
worker_main.main()
|
|
|
|
assert bootstrap_calls == [{"default_service_name": "hindsight-worker"}]
|