mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
b1de1b9418
`DELETE /v1/default/banks/{bank}/operations/{id}` only accepted `pending`
operations, so an operation stranded in `processing` — orphaned when a worker
was killed before it could write a terminal status — could only be cleared by
hand-editing `async_operations` and restarting the container.
Cancel now accepts `processing` too. It stays cooperative and is never
immediate: the row is flipped to `cancelled` and the worker running it stops at
its next `_check_op_alive` checkpoint (between retain sub-batches/documents,
between consolidation LLM batches). For the orphaned case nothing is running, so
the flip is the whole fix. No heartbeat and no per-batch bookkeeping is added.
Making the flip stick required guarding the worker writes that had none, and
would otherwise overwrite it:
- `_schedule_retry` — the one that actually resurrected cancelled work: a task
failing after cancellation went back to `pending` and was re-claimed.
- `_mark_failed` (poller and engine), `_defer_operation`.
`_mark_completed` already guarded on `status='processing'`; tests now pin it.
The sibling rollup counted only `completed`/`failed` as done, so a cancelled
child stranded its `batch_retain` parent in `processing` forever — the same
wedge one level up. Both rollup copies now treat `cancelled` as done and settle
the parent on `cancelled` (a real failure still outranks it), cancel performs
the rollup itself so cancelling the last outstanding child terminalizes the
parent, and a cancelled parent is never flipped back by a child finishing later.
Control plane: the Cancel button was gated on `pending`, hiding the fix from the
UI an operator would reach for. It now shows for `processing` rows too.