Edward Cheng-I Wu e90a98d321 ci(eval-harness): render PR comment as verdict + table, fold raw JSON (#479) ...

* ci(eval-harness): render PR comment as verdict + table, fold raw JSON

The eval-harness workflow pasted the entire eval_report.json into every PR
comment as one raw fenced block. Replace that with a unit-tested display
module (scripts/render_eval_comment.py): a one-line verdict (measured
passed/total + pending count), a per-task markdown table (metric, value,
threshold, result), and the full JSON folded into <details>.

Display layer only — run_evals, scripts._eval_threshold_gate, the gate step,
and the [eval-regression-acknowledged] ack contract are untouched. The row
verdict mirrors the gate's failure signal (aggregate AND per-class, #328) so
the table never shows a clean pass on a run the gate blocks.

Registered in scripts/_ci_pytest_manifest.toml; workflow test extended to
pin the renderer so the comment can't silently regress to a raw dump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jcH7gEkVTQ1baMbVedSZp

* fix(eval-comment): escape pipes/newlines in table cells (codex P2)

task_name / metric / comparison come from eval manifests; a `|` or newline
in them would break the markdown table or spoof extra rows/results.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jcH7gEkVTQ1baMbVedSZp

* refactor(eval-comment): /simplify pass — \r line boundaries, fd hygiene, gate-agreement pin

- _cell now normalizes ALL line boundaries via splitlines() (codex re-review:
  \r could still split a table row), not just \n.
- main() reads the report via a context manager (ResourceWarning hygiene).
- New test pins that _task_failures agrees with _eval_threshold_gate.failed_tasks
  on which tasks fail, so the deliberate display-side mirror of the gate's
  failure rule drifts loudly (CI fail) instead of silently rendering green on
  a blocked run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jcH7gEkVTQ1baMbVedSZp

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

2026-07-02 09:33:41 +08:00

6.3 KiB

Raw Permalink Blame History

View Raw