mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
`disable-model-invocation: true` drops a skill from the CLI's `<available_skills>` menu, so the experiment's skilled arm -- which loads exactly one skill -- cannot reach it either. The gate already reasoned about this, but `report_uncovered()` skipped any skill that had an eval, so it only reported the *better* case (no eval, visibly zero evidence) and stayed silent on the worse one (an eval scoring baseline against baseline and labelling the result a pass or a fail). Two evals landed in that blind spot after the reasoning was written down: tests/dotnet-test/platform-detection (#974) and tests/dotnet-test/filter-syntax (#976). Both READMEs still described them as deliberately un-evaluated. - report reference skills that carry a direct eval, naming the spec - self-test both sides of the trigger via a new silent_case helper - correct eng/eval-quality/README.md and plugins/dotnet-test/README.md Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d666b74-a00f-4ed0-a3ce-2d3427cf0d23
This commit is contained in:
Amaury Levé
+23
-13
@@ -368,7 +368,7 @@ way to raise an eval's trial count, because the fixture already exists —
|
||||
`migrate-nullable-references` sits at 3 scenarios with three unreferenced
|
||||
fixtures beside it.
|
||||
|
||||
### Skills with no eval
|
||||
### Skill eval coverage
|
||||
|
||||
A skill that ships with `SKILL.md` but has no `tests/<plugin>/<skill>/eval.yaml`
|
||||
carries zero evidence of impact.
|
||||
@@ -385,22 +385,32 @@ and adding such an eval would make the number worse, not better.
|
||||
|
||||
The honest coverage for these is **dependency-level**: they are exercised
|
||||
through the evals of the skills that load them (for example `run-tests` and
|
||||
`mtp-hot-reload` for `platform-detection`, the polyglot analysis skills for
|
||||
`test-analysis-extensions`, and `code-testing-agent` for
|
||||
`mtp-hot-reload` load `platform-detection` and `filter-syntax`, the polyglot
|
||||
analysis skills load `test-analysis-extensions`, and `code-testing-agent` loads
|
||||
`code-testing-extensions`), and in the plugin arm, where the whole plugin is
|
||||
loaded. Closing this properly needs harness support for declaring a dependency
|
||||
in the skilled variant, not a per-skill eval file.
|
||||
|
||||
> **`filter-syntax` is the exception, added in #976.** It carries a direct
|
||||
> `tests/dotnet-test/filter-syntax/eval.yaml` whose stimuli are ordinary
|
||||
> user requests ("one command that runs only the integration tests but leaves
|
||||
> out the slow ones"), so the skilled arm is graded on whether the answer
|
||||
> carries correct filter syntax rather than on whether the skill self-activated.
|
||||
> Whether that produces a *measurable* gap over baseline for a skill the model
|
||||
> cannot invoke is still unconfirmed: the evaluation on that PR landed during the
|
||||
> PAT-pool outage and reported "no results", so no verdict exists for it yet.
|
||||
> Worth reading its first real result before copying the pattern to the other
|
||||
> three.
|
||||
**A reference skill that already has a direct eval is reported too, and more
|
||||
loudly.** The same argument cuts both ways: if the skilled arm cannot reach the
|
||||
skill, an eval sitting beside it does not measure the skill — it measures the
|
||||
judge comparing baseline to baseline and then labels the result a pass or a
|
||||
fail. That is worse than no eval, because no eval is visibly zero evidence
|
||||
whereas a fabricated verdict is counted in the plugin's pass rate. The gate
|
||||
originally skipped any skill that had an eval, which made the worse case the
|
||||
quieter one; it now names them.
|
||||
|
||||
> **Two `dotnet-test` reference skills currently carry a direct eval:**
|
||||
> `filter-syntax` (added in #976) and `platform-detection` (added in #974).
|
||||
> Their stimuli are ordinary user requests ("one command that runs only the
|
||||
> integration tests but leaves out the slow ones"), so the intent was to grade
|
||||
> the answer on whether it carries the correct syntax rather than on whether the
|
||||
> skill self-activated. Whether that can produce a *measurable* gap over baseline
|
||||
> for a skill the model cannot invoke is still unconfirmed — the evaluation on
|
||||
> #976 landed during the PAT-pool outage and reported "no results", and no
|
||||
> cross-family run has covered either eval since. Read a real result before
|
||||
> copying the pattern to `code-testing-extensions` or `test-analysis-extensions`;
|
||||
> if the gap is zero, retire both evals rather than keep scoring noise.
|
||||
|
||||
### Dormancy guard without an anti-hijack rubric item
|
||||
|
||||
|
||||
@@ -590,13 +590,24 @@ def _is_reference_skill(skill_dir: str) -> bool:
|
||||
def report_uncovered() -> None:
|
||||
missing = []
|
||||
reference = []
|
||||
degenerate = []
|
||||
for plugin_dir in sorted(glob.glob("plugins/*")):
|
||||
plugin = os.path.basename(plugin_dir)
|
||||
evals = {os.path.basename(os.path.dirname(f))
|
||||
for f in glob.glob(f"tests/{plugin}/*/eval.yaml")}
|
||||
for skill_dir in sorted(glob.glob(f"{plugin_dir}/skills/*")):
|
||||
skill = os.path.basename(skill_dir)
|
||||
if not os.path.isdir(skill_dir) or skill in evals:
|
||||
if not os.path.isdir(skill_dir):
|
||||
continue
|
||||
if skill in evals:
|
||||
# A reference skill that *has* a direct eval is the worse half of
|
||||
# this problem, not the solved half: the same argument that says
|
||||
# such an eval would compare two identical arms says the verdict
|
||||
# it produces is judge noise wearing a pass/fail label. Silence
|
||||
# here is how two of these landed after the reasoning was
|
||||
# written down. No eval is honest; a fabricated verdict is not.
|
||||
if _is_reference_skill(skill_dir):
|
||||
degenerate.append(f" {plugin}/{skill} — tests/{plugin}/{skill}/eval.yaml")
|
||||
continue
|
||||
if _is_reference_skill(skill_dir):
|
||||
reference.append(f" {plugin}/{skill}")
|
||||
@@ -612,6 +623,13 @@ def report_uncovered() -> None:
|
||||
f"compare two identical arms. Cover them through the consumers that "
|
||||
f"load them:")
|
||||
warnings.extend(reference)
|
||||
if degenerate:
|
||||
warnings.append(
|
||||
f"{len(degenerate)} reference skill(s) carry a direct-activation eval — they set "
|
||||
f"`disable-model-invocation: true`, so the model cannot reach the skill in the "
|
||||
f"skilled arm either: the eval scores baseline against baseline and its verdict is "
|
||||
f"judge noise. Retire the eval or cover the skill through a consumer:")
|
||||
warnings.extend(degenerate)
|
||||
|
||||
|
||||
def check_floor_agreement() -> None:
|
||||
|
||||
@@ -102,6 +102,28 @@ def output_case(label, mutate, expect_substring):
|
||||
EV = lambda d: os.path.join(d, "tests", "demo", "widget", "eval.yaml")
|
||||
|
||||
|
||||
def silent_case(label, mutate, forbidden_substring):
|
||||
"""Assert the gate stays quiet — the other half of every warning's contract.
|
||||
|
||||
A warning that fires on well-formed input is worse than no warning: it
|
||||
trains the team to skim past the whole report. Pairing each `output_case`
|
||||
with this keeps the trigger condition pinned from both sides.
|
||||
"""
|
||||
d = scratch()
|
||||
try:
|
||||
mutate(d)
|
||||
subprocess.run(["git", "add", "-A"], cwd=d, capture_output=True, check=True)
|
||||
code, out = run_gate(d)
|
||||
ok = code == 0 and forbidden_substring not in out
|
||||
print(f" [{'OK ' if ok else 'BAD'}] {label:<52} forbidden={forbidden_substring!r}")
|
||||
if not ok:
|
||||
print(f" exit={code}")
|
||||
print(" " + out.strip().replace("\n", "\n ")[:900])
|
||||
return ok
|
||||
finally:
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
|
||||
|
||||
def clean(d):
|
||||
pass
|
||||
|
||||
@@ -246,6 +268,29 @@ def guard_ok(d):
|
||||
)
|
||||
|
||||
|
||||
# --- reference skills -------------------------------------------------------
|
||||
# `disable-model-invocation: true` hides a skill from the model-facing menu, so
|
||||
# the skilled arm cannot reach it either and the eval scores baseline against
|
||||
# baseline. The gate used to skip any skill that had an eval, which made the
|
||||
# worse case (a fabricated verdict) quieter than the better one (no verdict).
|
||||
|
||||
def _write_skill_md(d, *, hidden):
|
||||
path = os.path.join(d, "plugins", "demo", "skills", "widget", "SKILL.md")
|
||||
with open(path, "w") as f:
|
||||
f.write("---\nname: widget\ndescription: Does the thing\n")
|
||||
if hidden:
|
||||
f.write("disable-model-invocation: true\n")
|
||||
f.write("---\n\n# Widget\n")
|
||||
|
||||
|
||||
def reference_skill_with_a_direct_eval(d):
|
||||
_write_skill_md(d, hidden=True)
|
||||
|
||||
|
||||
def invocable_skill_with_a_direct_eval(d):
|
||||
_write_skill_md(d, hidden=False)
|
||||
|
||||
|
||||
# --- statistical power ------------------------------------------------------
|
||||
# Trials = scenarios x runs. Below the floor the pass gate cannot reach a
|
||||
# credible verdict at any effect size, so a new eval must not land there.
|
||||
@@ -364,6 +409,12 @@ results = [
|
||||
case("spec declares both config: and defaults:", config_and_defaults_together, expect_fail=True),
|
||||
case("dormancy guard also sets reject_skills", guard_with_reject_skills, expect_fail=True),
|
||||
case("well-formed dormancy guard", guard_ok, expect_fail=False),
|
||||
output_case("reference skill carrying a direct-activation eval",
|
||||
reference_skill_with_a_direct_eval,
|
||||
"1 reference skill(s) carry a direct-activation eval"),
|
||||
silent_case("model-invocable skill with a direct eval",
|
||||
invocable_skill_with_a_direct_eval,
|
||||
"carry a direct-activation eval"),
|
||||
case("eval below the trial floor", underpowered, expect_fail=True),
|
||||
case("below the floor but grandfathered", underpowered_but_allowlisted, expect_fail=False),
|
||||
output_case("grandfathered warning reports scenarios x runs",
|
||||
|
||||
@@ -73,20 +73,22 @@ For non-.NET languages, use the native coverage tool: `coverage.py`/`pytest-cov`
|
||||
| **filter-syntax** *(.NET)* | Test filter syntax reference for VSTest and MTP across all frameworks |
|
||||
|
||||
These four set `disable-model-invocation: true`, so the CLI keeps them out of the
|
||||
model-facing skill menu and a consumer loads them by name. Three of them
|
||||
(`code-testing-extensions`, `test-analysis-extensions`, `platform-detection`)
|
||||
deliberately have no `tests/dotnet-test/<skill>/eval.yaml`: the experiment's
|
||||
skilled arm loads a single skill, which the model could never invoke here, so
|
||||
such an eval would compare two identical arms and score judge noise. They are
|
||||
measured through the evals of the skills that load them — `run-tests` and
|
||||
`mtp-hot-reload` for `platform-detection`, the polyglot analysis skills and
|
||||
model-facing skill menu and a consumer loads them by name. Two of them
|
||||
(`code-testing-extensions`, `test-analysis-extensions`) deliberately have no
|
||||
`tests/dotnet-test/<skill>/eval.yaml`: the experiment's skilled arm loads a
|
||||
single skill, which the model could never invoke here, so such an eval would
|
||||
compare two identical arms and score judge noise. They are measured through the
|
||||
evals of the skills that load them — the polyglot analysis skills and
|
||||
`grade-tests` for `test-analysis-extensions`, and `code-testing-agent` for
|
||||
`code-testing-extensions`.
|
||||
|
||||
`filter-syntax` is the exception: #976 gave it a direct eval built from ordinary
|
||||
user requests, so the answer is graded on carrying the right filter syntax rather
|
||||
than on the skill self-activating. That approach has not produced a verdict yet
|
||||
(its evaluation landed during the PAT-pool outage). See
|
||||
`platform-detection` (#974) and `filter-syntax` (#976) are the exceptions: both
|
||||
were given a direct eval built from ordinary user requests, so the answer is
|
||||
graded on carrying the right detection or filter syntax rather than on the skill
|
||||
self-activating. Neither has produced a verdict yet — `filter-syntax` landed
|
||||
during the PAT-pool outage, and no cross-family run has covered either since —
|
||||
so whether that grading survives an arm the model cannot reach is still an open
|
||||
question. The eval-quality gate reports both until it is answered. See
|
||||
`eng/eval-quality/README.md`.
|
||||
|
||||
## Agents
|
||||
|
||||
Reference in New Issue
Block a user