From 26caf0ecab71aa8176671d2fe7b87ee309d4b79f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Thu, 27 Aug 2026 16:04:37 +0200 Subject: [PATCH] Scale coverage analysis to request scope (#1078) * Scale coverage analysis to request scope Gate CRAP and full reports behind explicit intent, reconcile coverage arithmetic, and strengthen only evidence-backed eval outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address coverage eval review feedback Broaden no-command guards for Markdown formatting and clarify plateau recommendations across both uncovered members. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dc01c723-1c85-446e-873a-c4bd26fdb0e8 * Improve coverage analysis recovery Use the latest cross-model evidence to recover from failed artifact reads, require source-safe branch interpretation, make target combinations concrete, and enforce CRAP-first refactoring risk ranking. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dc01c723-1c85-446e-873a-c4bd26fdb0e8 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dc01c723-1c85-446e-873a-c4bd26fdb0e8 --- .../skills/coverage-analysis/SKILL.md | 299 ++++++++---------- .../references/guidelines.md | 99 +++--- .../references/output-format.md | 10 +- .../references/report-generation.md | 11 +- .../references/setup-discovery.md | 12 +- .../references/test-execution.md | 6 +- tests/dotnet-test/coverage-analysis/eval.yaml | 52 +-- 7 files changed, 241 insertions(+), 248 deletions(-) diff --git a/plugins/dotnet-test/skills/coverage-analysis/SKILL.md b/plugins/dotnet-test/skills/coverage-analysis/SKILL.md index a555a4f5..2da5c7dc 100644 --- a/plugins/dotnet-test/skills/coverage-analysis/SKILL.md +++ b/plugins/dotnet-test/skills/coverage-analysis/SKILL.md @@ -1,18 +1,19 @@ --- name: coverage-analysis description: > - Explains and analyzes .NET Cobertura line, branch, and condition evidence plus - project-wide CRAP risk. MUST USE for "why is branch coverage lower than line - coverage?", condition-coverage="50% (1/2)", a supplied coverage excerpt, - partially covered conditions, untested outcomes, plateaus, members blocking a - project target, project-wide CRAP/refactoring safety, or coverage-backed test - priorities. For project-wide work, uses real coverage, counts every - below-threshold member, and ranks the top N hotspots. DO NOT USE FOR: test - trait/category distributions or coverage shape by test type (test-tagging); - static source-to-test pairing (find-untested-sources); behavioral/pseudo-mutation - gaps (test-gap-analysis); named-target CRAP (crap-score); test-code audits + Interprets .NET Cobertura line, branch, and condition evidence and, when + explicitly requested, computes project-wide CRAP/refactoring-risk hotspots. + MUST USE for "why is branch coverage lower than line coverage?", + condition-coverage="50% (1/2)", a supplied coverage excerpt, partially covered + conditions, coverage plateaus, members blocking a target, project-wide CRAP, + refactoring safety, or coverage-backed risk priorities. A supplied report is + analyzed directly without rerunning tests, installing tools, generating a + report, or calculating CRAP unless the request asks for risk/CRAP/refactoring + safety. DO NOT USE FOR: test trait distributions (test-tagging); static + source-to-test pairing (find-untested-sources); behavioral/pseudo-mutation gaps + (test-gap-analysis); named-target CRAP (crap-score); test-code audits (test-anti-patterns); raw collection/percentage-only requests or just running - tests (run-tests); non-.NET coverage (native tooling); or writing tests. + tests (run-tests); non-.NET coverage; or writing tests. license: MIT --- @@ -20,21 +21,16 @@ license: MIT ## Purpose -Raw coverage percentages answer "what code was executed?" — they don't answer what you actually need to know: - -- **What tests should I write next?** — ranked by risk and impact -- **Which uncovered code is risky vs. trivial?** — CRAP scores separate the two -- **Why has coverage plateaued?** — identify the files blocking further gains -- **Is this code safe to refactor?** — complex + uncovered = dangerous to change - -This skill bridges that gap: from a bare .NET solution to a prioritized risk hotspot list, with no manual tool configuration required. +Explain what .NET coverage evidence proves, reconcile target arithmetic, and +identify the code blocking progress. Add complexity/CRAP ranking only when the +user explicitly asks for risk hotspots, CRAP, priorities by risk, or refactoring +safety. ## When to Use -Use this skill for interpreting a supplied .NET line/branch/condition excerpt or -for project-wide test coverage, coverage gaps, code risk, CRAP scores, test -priorities, plateau diagnosis, or refactoring safety — even if the user does not -explicitly say "coverage analysis". +Use this skill for interpreting supplied .NET line/branch/condition evidence, +coverage gaps and plateaus, target arithmetic, or explicit project-wide +coverage-backed risk analysis. ## When Not to Use @@ -53,173 +49,146 @@ explicitly say "coverage analysis". | Project/solution path | No | Current directory | Path to the .NET solution or project | | Line coverage threshold | No | 80% | Minimum acceptable line coverage | | Branch coverage threshold | No | 70% | Minimum acceptable branch coverage | -| CRAP threshold | No | 30 | Maximum acceptable CRAP score before flagging | -| Top N hotspots | No | 10 | Number of risk hotspots to surface | +| Existing Cobertura path | No | Discover only if not supplied | Preferred input; never rerun tests when usable | +| CRAP threshold | No | 30 | Used only for explicit risk/CRAP requests | +| Hotspot count | No | 3 | Explicit risk requests only; cap at 10 unless the user asks for more | -### Prerequisites +Discover optional inputs from the workspace. Do not ask for a project path when +the current directory or a supplied report is sufficient. -- .NET SDK installed (`dotnet` on PATH) -- At least one test project referencing the production code (xUnit, NUnit, or MSTest) — only required for the from-scratch path; not needed when the user supplies an existing Cobertura XML -- **Optional, only for the SDK-style from-scratch path:** internet/NuGet access for `dotnet add package coverlet.collector` (or `Microsoft.Testing.Extensions.CodeCoverage`) when a test project has no coverage provider yet. Skip for classic projects and when the user supplies an existing Cobertura XML. -- **Optional, only for Phase 5:** internet access for `dotnet tool install` (ReportGenerator). Core CRAP/coverage analysis works from Cobertura XML alone — ReportGenerator only adds HTML/CSV reports as an optional post-summary extra. +## Choose the smallest matching path -The skill auto-detects coverage provider state per test project and selects the least-invasive execution strategy: +| User intent | Required work | Do not do | +|-------------|---------------|-----------| +| Explain a supplied excerpt, condition, or summary | Answer directly from the supplied evidence | Tools, CRAP, discovery, report files | +| Interpret a supplied Cobertura path or diagnose a plateau | Read that report, reconcile totals, name all material gaps, answer directly | Rerun tests, install tools, compute CRAP, or generate files unless explicitly requested | +| Rank risk hotspots, compute project-wide CRAP, or assess refactoring safety | Use the supplied/existing report, read `references/guidelines.md`, and compute CRAP before ranking | Coverage-only ranking or a full report template unless requested | +| Analyze coverage when no report exists | Read `references/setup-discovery.md`; collect once using `references/test-execution.md` if safe | CRAP unless risk was requested | +| Produce a full markdown/HTML/CSV report | First deliver the direct answer; then read `references/output-format.md` or `references/report-generation.md` | Report generation before the answer | -- unified Microsoft CodeCoverage when all projects use it, -- unified Coverlet when no project uses Microsoft CodeCoverage, -- per-project provider execution when the solution is truly mixed. +Words such as **analyze coverage**, **what is blocking coverage**, or **why is +coverage stuck** do not by themselves request CRAP. Explicit signals include +**risk hotspot**, **CRAP**, **complexity-weighted priority**, **safe to refactor**, +or an equivalent request to combine complexity with coverage. -No pre-existing runsettings files or manually installed tools required. +## Existing-data fast path -The automatic from-scratch path applies to SDK-style projects. Classic non-SDK -projects (`ToolsVersion`, explicit ``, `packages.config`) remain -fully supported when the user supplies Cobertura XML. Without an existing report, -use a repository-provided coverage command if one exists; otherwise stop and ask -for Cobertura output. Never convert the project, run `dotnet add package`, or add -`PackageReference` as an incidental coverage setup step. +When the user supplies a coverage excerpt, summary, or valid Cobertura path: -> **CLASSIC-ONLY HARD STOP:** If every discovered test project is classic and no -> existing Cobertura report or checked-in coverage command exists, stop after -> discovery. Do not create a temporary SDK-style wrapper/project, copy the source -> into one, install a collector elsewhere, or generate substitute coverage from -> a different assembly. That data does not describe the requested project. -> Report complexity alone if useful and request real Cobertura from the -> repository's supported toolchain. +- Treat it as authoritative input and start there. +- Do not discover the solution or test projects unless source mapping is necessary. +- Do not run `dotnet test`, install ReportGenerator, add a coverage package, or + read `references/setup-discovery.md`, `references/test-execution.md`, or + `references/report-generation.md`. +- Do not write `coverage-analysis.md` or create a report directory unless the user + requested a saved/full report. +- For interpretation and plateau questions, parse only the evidence needed to + answer. For explicit project-wide risk requests, use the bundled scripts as + described in `references/guidelines.md`. -## Workflow +A failed read/view operation is not proof that a named path does not exist. After +one fails, make one allowed targeted existence probe, such as a workspace-relative +glob, and retry the same artifact with a normalized path or alternate reader. +Report the exact missing-path problem only when that independent check also fails. +Do not broaden the search to unrelated coverage files or present a substitute +artifact. -> **MANDATORY for project-wide analysis: deliver the final assistant response with the CRAP/risk-hotspot summary BEFORE any optional work.** As soon as `Compute-CrapScores.ps1` and `Extract-MethodCoverage.ps1` return data, your **next** assistant response must contain the user-facing analysis (CRAP table, blocking methods, recommendations). Do not run ReportGenerator (Phase 5), do not install global tools, and do not start any heavy parallel work before that response is delivered. The user is judged on the final assistant message, not on side-effect files. -> -> If a phase fails, times out, or budget is running low, skip remaining optional work and immediately return a partial summary containing: (1) what was found in the Cobertura XML, (2) any CRAP/risk-hotspot data already extracted, (3) which methods are blocking coverage, and (4) failures encountered. +## Collection path -If the user supplies only a coverage excerpt or asks what a line/condition entry -means, answer directly from that evidence and stop. Do not run tools or fabricate -a project-wide CRAP dashboard. A line hit proves execution, not both decision -outcomes. `50% (1/2)` proves one reported condition outcome ran but does not say -which outcome; recommend forcing the opposite outcome. For compound predicates, -state that operand and short-circuit combinations may need independent tests, but -do not infer their number without source or fuller XML. +Use this path only when no usable coverage evidence exists and the user asked for +analysis that requires it. -If the user provides a path to existing Cobertura XML (or coverage data is already present in `TestResults/`), **skip Phase 2 entirely** (no test execution) **and skip Phase 5 by default** (no ReportGenerator install or HTML report) — go directly from Phase 3 (analysis scripts) to Phase 4 (user-facing summary). Only run Phase 5 if the user explicitly asks for HTML/CSV reports. The Risk Hotspots table and CRAP scores are mandatory in every project-wide output — they are the skill's core value-add over raw coverage numbers. +1. Read `references/setup-discovery.md`. +2. Prefer existing Cobertura discovered under the requested root. +3. If none exists, read `references/test-execution.md` and run the selected + coverage command once per entry point. +4. Analyze the resulting Cobertura. Compute CRAP only if risk analysis was + explicitly requested. -The project-wide workflow runs in five phases. Phases 1–4 are required; Phase 5 (ReportGenerator HTML/CSV reports) is strictly optional and runs **after** the user-facing summary has been delivered. Do not parallelize Phase 5 with earlier phases — the heavy `dotnet tool install` for ReportGenerator can crash the session before Phase 4 completes. +Do not modify production code. The only permitted incidental project change is +adding one missing coverage provider to an SDK-style test project as described in +`references/test-execution.md`; never add a second provider, and report the +change plus its revert command. -### Phase 1 — Setup (sequential) +The automatic collection path is for SDK-style projects. For classic non-SDK or +`packages.config` projects, use only a repository-owned coverage command. If none +exists, stop and request Cobertura XML. Never migrate the project, inject an +SDK-style provider, create a wrapper project, or report substitute coverage from +another assembly. -Read `references/setup-discovery.md` and run the probes it contains, in order: +## Interpretation and arithmetic invariants -| Step | Emits | Why it matters | -|------|-------|----------------| -| 1. Locate the solution or project | `ENTRY_TYPE`, `ENTRY`, `TEST_PROJECTS`, `CLASSIC_TEST_PROJECTS`, `SDK_TEST_PROJECTS`, `TEST_OUTPUT_ROOT` | Entry point, safe project partitions, and output location | -| 2. Create the output directory | `COVERAGE_DIR` | Skill-owned `TestResults/coverage-analysis/`; never deletes user-supplied reports | -| 2b. Discover or accept existing Cobertura XML | `EXISTING_COBERTURA_COUNT`, `EXISTING_COBERTURA` | A user-supplied path always wins; otherwise probe `TestResults/` | -| 2c. Recommend ignoring `TestResults/` | `GITIGNORE_RECOMMENDATION` | One-line recommendation, reported in the summary | +- A line hit proves execution, not both decision outcomes. +- `condition-coverage="50% (1/2)"` proves one reported outcome ran, but not which + one. Recommend forcing the opposite outcome. Without source, never invent likely + predicates or claim whether true or false is missing. State that compound + predicates need independently exercised operands and short-circuit combinations + when applicable; do not infer the exact combinations without source or fuller XML. +- Derive overall totals from Cobertura's covered/valid line counts. For target + `T`, required covered lines are `ceiling(valid lines × T)`. +- Projected coverage is `(current covered lines + newly covered distinct lines) / + valid lines`. State assumptions such as fully covering a method. +- When asked whether one member can reach a target, show its maximum projected + total and at least one concrete sufficient combination of supplied members or + line gains. If no supplied combination is sufficient, say so. +- Reconcile member gaps against project totals. Method line ranges can overlap or + omit class-level lines, so do not sum method counts as project truth. +- Never call one member the **sole**, **entire**, or **all** remaining gap unless + its distinct uncovered lines exactly reconcile with the project total and no + other below-threshold member remains. +- Name every supplied or extracted below-threshold member, but keep detail + proportional: lead with the blockers, summarize the remainder in one sentence + or a compact table. -Branching after Phase 1: +## Response contract -- `EXISTING_COBERTURA_COUNT` > 0 → **skip Phase 2 entirely**; go to Phase 3 with those paths. Do not read `references/test-execution.md`. -- `EXISTING_COBERTURA_COUNT` == 0 and only SDK-style test projects were found → run Phase 2 normally. -- `EXISTING_COBERTURA_COUNT` == 0 and only classic/packages.config projects were found → use a checked-in coverage command; if none exists, stop and request Cobertura XML. -- Mixed classic/SDK projects → collect the SDK subset per project, never through the solution entry; label results partial until classic-project Cobertura is supplied. -- `ENTRY_TYPE:NotFound` with test projects → use the test projects directly as entry points. -- No test projects and no Cobertura XML → stop: `No test projects found (expected projects with 'Test' or 'Spec' in the name), and no existing Cobertura XML was provided. Add a test project or provide a Cobertura file path.` +Answer the user's question in the first 2–4 sentences. -### Phase 2 — Test execution (skip when Cobertura XML already exists) +- **Excerpt or arithmetic question:** one explanation plus the next test or member + priority. No dashboard. +- **Existing-report interpretation or plateau:** overall line/branch coverage, + blocking members, reconciled target impact, and 1–3 recommendations. Use at + most one compact table. +- **Explicit risk/CRAP request:** top 3 actual hotspots by default, supporting + complexity/coverage/CRAP values, remaining flagged count, and 1–3 priorities. + Exclude fully covered low-risk methods from the hotspot table. Never exceed 10 + rows unless the user requests a larger count. For refactoring safety, rank risky + methods by CRAP rather than raw coverage alone, then name comparatively safe + well-covered methods separately. +- **Explicit full report request:** read `references/output-format.md`. Save the + report only then. -Run only when Phase 1 found no Cobertura XML. If the user already has coverage data, skip directly to Phase 3 — do not read this section's reference file, and do not re-run the suite. +Report only artifacts that exist. Do not announce inaccessible output paths or +failed optional file writes when no file was requested. -Read `references/test-execution.md` for the SDK-style subset. It covers safe -partitioning, provider detection, package addition, `dotnet test`, exit codes, -and report discovery. For classic projects, run only a repository-owned coverage -command; otherwise request Cobertura XML and mark mixed-solution results partial. +## Optional reports -Exit codes: **0** all passed; **1** some tests failed (coverage is still collected — proceed with a warning); anything else is a build failure — stop and report it. +HTML/CSV/markdown files are not part of normal analysis. Generate them only when +the user explicitly requests report files or a CI artifact. -### Phase 3 — Analysis (sequential) - -Run the two bundled PowerShell scripts. Both are cheap and complete in seconds. **Do not** install or invoke ReportGenerator here — that belongs in optional Phase 5, after the user-facing summary has been delivered. - -#### Step 4: Calculate CRAP scores using the bundled script - -Run `scripts/Compute-CrapScores.ps1` (co-located with this SKILL.md). It reads all Cobertura XML files, applies `CRAP(m) = comp² × (1 − cov)³ + comp` per method, and returns the top-N hotspots as JSON. - -To locate the script: find the directory containing this skill's `SKILL.md` file (the skill loader provides this context), then resolve `scripts/Compute-CrapScores.ps1` relative to it. If the script path cannot be determined, calculate CRAP scores inline using the formula below. - -```powershell -& "/scripts/Compute-CrapScores.ps1" ` - -CoberturaPath @() ` - -CrapThreshold ` - -TopN -``` - -Script outputs: `OVERALL_LINE_COVERAGE:`, `OVERALL_BRANCH_COVERAGE:` (aggregated project-wide rates across all provided Cobertura files), `TOTAL_METHODS:`, `FLAGGED_METHODS:`, `HOTSPOTS:` (top-N sorted by CrapScore descending). The OVERALL_* values are exactly what the Phase 4 summary needs for the "Line Coverage" / "Branch Coverage" rows — no separate XML parsing tool call is required. - -#### Step 5: Extract per-method coverage gaps - -Run `scripts/Extract-MethodCoverage.ps1` to get per-method coverage data for the Coverage Gaps table: - -```powershell -& "/scripts/Extract-MethodCoverage.ps1" ` - -CoberturaPath @() ` - -CoverageThreshold ` - -BranchThreshold ` - -Filter below-threshold -``` - -Script outputs: JSON array of methods below the coverage threshold, sorted by coverage ascending. Use this data to populate the Coverage Gaps by File table in the report. - -### Phase 4 — Project-wide user-facing summary (MANDATORY — your next assistant response) - -As soon as Phase 3 completes, **your immediately next assistant response must contain the user-facing analysis** — do not interleave any other tool calls before it. This is the response the user (and any judge) sees. Skipping or deferring this in favor of Phase 5 (ReportGenerator) is a hard failure. - -The response must include, at minimum: - -0. **A direct answer to the question that was actually asked, in the first 2–4 sentences.** For "why is my coverage stuck?" / "what's blocking me?", name the blocking members and the lines involved before any table. The standard sections below still follow. -1. Overall line and branch coverage — read directly from the `OVERALL_LINE_COVERAGE:` / `OVERALL_BRANCH_COVERAGE:` lines emitted by `Compute-CrapScores.ps1` (no extra Cobertura parsing required) -2. The Risk Hotspots table built from `Compute-CrapScores.ps1` `HOTSPOTS:` output (CRAP scores, complexity, coverage) -3. Identification of the highest-risk method(s) and what is blocking coverage -4. 1–3 prioritized, specific recommendations (which method to test, expected CRAP/coverage impact) - -**Every number must come from the script output, and the arithmetic must reconcile.** Uncovered lines attributed to individual members must not exceed the project's total uncovered lines, and the coverage you project after a recommendation must follow from those counts. - -**List every member below threshold, not just the worst one.** `Extract-MethodCoverage.ps1` returns the full below-threshold set: name the others even if briefly. Only say "the rest is fine / leave it alone" when that set is otherwise empty — claiming one method is the entire gap when the extractor found more is a factual error. - -Use `references/output-format.md` verbatim for fixed headings, table structures, symbols, and emoji. Use `references/guidelines.md` for prioritization rules and style. - -If Phase 5 has not yet run when you compose this summary, mark the `## 📁 Reports` section's HTML/Text/CSV/GitHub-markdown rows as `Not generated (optional — request HTML reports to enable)`. Only the `coverage-analysis.md` and raw Cobertura paths are guaranteed to exist. - -Attempt to save the same content to `TestResults/coverage-analysis/coverage-analysis.md` before delivering the response (use the editor's create/edit tool — do not shell out). If the file write fails, still deliver the summary and note the file-write failure explicitly. - -### Phase 5 — Optional: ReportGenerator HTML/CSV reports (post-summary) - -Phase 5 is **strictly optional** and runs **only after** Phase 4 has been delivered. Skip Phase 5 entirely when: - -- The user supplied existing Cobertura XML and only asked for analysis (the default for the existing-data path). -- The user is diagnosing a coverage plateau or asking "what's blocking me?" — they want the answer, not a static-site report. -- ReportGenerator is not already installed and you have no clear signal the user wants HTML reports. - -Run Phase 5 only when the user explicitly asks for HTML/CSV reports, or when the project flow requires them (e.g., a CI artifact upload step). - -Read `references/report-generation.md` for the ReportGenerator install and invocation. It is the only heavy step in this skill: a `dotnet tool install` that can exhaust the session budget, which is why it never runs before the Phase 4 summary has been delivered. - -If the install fails (no internet), leave the existing Phase 4 summary as the final output and note that HTML reports were skipped. Do not retry or block on it. +1. Deliver the direct coverage/risk answer first. +2. For a full markdown report, read `references/output-format.md` and save it. +3. For HTML/CSV, then read `references/report-generation.md`. Do not install + ReportGenerator before the direct answer, and do not retry a failed install. ## Validation -- Verify that at least one `coverage.cobertura.xml` file was generated by the selected SDK-style or repository-owned command (or already exists when the user supplied one) -- Confirm the assistant response contained the CRAP/risk-hotspot table — saving the markdown file is secondary -- Confirm `TestResults/coverage-analysis/coverage-analysis.md` was written and contains data -- Spot-check one method's CRAP score: `comp² × (1 − cov)³ + comp` — a method with 100% coverage should have CRAP = complexity -- If Phase 5 ran, verify `TestResults/coverage-analysis/reports/index.html` exists; otherwise the report file should mark HTML/Text/CSV rows as `Not generated` +- Confirm a supplied report was used without test execution or tool installation. +- Reconcile covered, valid, and uncovered line totals before projecting impact. +- Confirm every stated blocker comes from supplied or extracted evidence. +- If CRAP was requested, spot-check one score using `references/guidelines.md`. +- If files were requested, verify they exist before reporting their paths. ## Common Pitfalls -- **No Cobertura XML generated in an SDK-style project** — the test project may lack a coverage provider. The skill auto-adds one, but if `dotnet add package` fails (offline/proxy), coverage collection silently produces nothing. Check for `.coverage` binary files as a fallback indicator. -- **Classic non-SDK project without a report** — do not inject an SDK-style provider. Use the repository's existing coverage workflow or ask the user for Cobertura XML. -- **Test failures (exit code 1)** — coverage is still collected from passing tests. Do not abort; proceed with partial data and note the failures in the summary. -- **Premature end before user-facing summary** — never start Phase 5 (ReportGenerator install/run) before the Phase 4 assistant response is delivered. The heavy `dotnet tool install` can crash the session or exhaust budget, leaving the user with no analysis even though the CRAP scores were already computed. -- **ReportGenerator install failure** — if `dotnet tool install` fails (no internet) during Phase 5, leave the existing Phase 4 summary as the final output and note that HTML reports were skipped. Do not retry or block on the install. -- **Method name mismatches in Cobertura** — async methods, lambdas, and local functions may have compiler-generated names. The scripts use the Cobertura method name/signature directly; verify against source if results look unexpected. -- **Mixed coverage providers** — when a solution contains both Coverlet and Microsoft CodeCoverage projects, the skill runs per-project to avoid dual-provider conflicts. This is slower but correct. -- **Numbers that don't reconcile** — per-member uncovered lines that exceed the project total, or a projected coverage figure that doesn't follow from the counts, make the whole analysis untrustworthy. Re-read the script output rather than estimating. -- **Declaring one method "the entire gap"** — check the full below-threshold list from `Extract-MethodCoverage.ps1` first; naming a single blocker while other uncovered members exist misdirects the user's next test. +- **Existing report triggers setup work** — stop. Analyze the named artifact first. +- **Generic analysis triggers CRAP** — stop. CRAP requires explicit risk intent. +- **One method is called the entire gap** — reconcile distinct uncovered lines and + the full below-threshold set first. +- **A small request gets a dashboard** — scale down to a direct answer or one table. +- **No Cobertura from collection** — report the collection failure; do not invent + substitute coverage. +- **Compiler-generated method names** — verify suspicious async, lambda, or local + function names against source before presenting them as user-authored members. +- **Test exit code 1** — coverage may still exist; proceed with a warning. Other + nonzero build failures stop the collection path. diff --git a/plugins/dotnet-test/skills/coverage-analysis/references/guidelines.md b/plugins/dotnet-test/skills/coverage-analysis/references/guidelines.md index d3b2893a..2c8d5c48 100644 --- a/plugins/dotnet-test/skills/coverage-analysis/references/guidelines.md +++ b/plugins/dotnet-test/skills/coverage-analysis/references/guidelines.md @@ -1,69 +1,68 @@ -# Guidelines +# Explicit Risk/CRAP Analysis -**Don't modify source or production code.** For SDK-style test projects only, -the permitted project-file modifications are adding a missing coverage provider: -`coverlet.collector` or `Microsoft.Testing.Extensions.CodeCoverage`. Never modify -a classic non-SDK or `packages.config` project; require its repository-owned -coverage command or existing Cobertura XML. Do not add a second provider. Always -log package additions and document revert commands in the report. Write all -other output to `TestResults/coverage-analysis/`. +Read this file only when the user explicitly asks for project-wide risk +hotspots, CRAP scores, complexity-weighted priorities, or refactoring safety. +Do not read it for supplied-excerpt interpretation, plateau diagnosis, or target +arithmetic alone. -**Always show and open the generated markdown report — but only after the assistant response with the CRAP/risk-hotspot summary has been delivered.** Saving and opening `TestResults/coverage-analysis/coverage-analysis.md` is a follow-up action; it must never delay the user-facing summary. +## Compute the risk data -**Don't generate new tests during the initial analysis run.** This skill surfaces where tests are needed. Test generation is a separate follow-up step outside the scope of this skill. +Resolve the scripts relative to this skill's `SKILL.md`: -**For SDK-style projects, use inline `dotnet test` arguments, not runsettings -files.** For classic projects, preserve the repository's coverage workflow. +```powershell +& "/scripts/Compute-CrapScores.ps1" ` + -CoberturaPath @() ` + -CrapThreshold ` + -TopN -**Show the risk hotspots table even when all thresholds pass.** A project at 90% line coverage can still have a method with cyclomatic complexity 20 and 0% branch coverage. The thresholds measure averages; the hotspot table finds outliers. Don't hide it just because the summary looks green. +& "/scripts/Extract-MethodCoverage.ps1" ` + -CoberturaPath @() ` + -CoverageThreshold ` + -BranchThreshold ` + -Filter below-threshold +``` -**Always compute and surface CRAP scores.** The Risk Hotspots table is mandatory in every analysis output, whether analyzing pre-existing data, freshly collected data, or diagnosing a plateau. Never skip CRAP score computation — it is the primary differentiator between this skill and raw `dotnet test` coverage output. +`Compute-CrapScores.ps1` emits aggregate line/branch coverage, method counts, +flagged counts, and sorted hotspots. `Extract-MethodCoverage.ps1` emits every +below-threshold method. Use both for explicit project-wide risk work. -**Continue past test failures (exit code 1).** If some tests fail, coverage is still collected from the passing tests — partial data is better than no data. Note the failures in the summary and proceed. Aborting would leave the developer with nothing actionable. +CRAP is: -**Run the selected coverage command only once per entry point during normal -flow.** For SDK-style projects this is `dotnet test`; for classic projects it is -the repository-owned command. A single recovery rerun is allowed only if the -first run produced no Cobertura XML and only `.coverage` binary output. +`CRAP(m) = complexity² × (1 − lineCoverage)³ + complexity` -**CRAP threshold of 30 is the default for a reason.** Scores above 30 are widely cited (by the original researchers) as "needs immediate attention." Scores between 15 and 30 are moderate — flag them in the table but don't make them sound catastrophic. Scores ≤ 5 are generally fine. +A method at 100% coverage therefore has CRAP equal to its complexity. Use 30 as +the default flagged threshold; treat 15–30 as moderate rather than catastrophic. -**Priority assignment for coverage gaps:** +## Scale the output -- **HIGH** — file has both a CRAP score above threshold AND coverage below threshold (the double failure is what makes it urgent) -- **MED** — coverage below threshold OR CRAP score above threshold, but not both -- **LOW** — coverage below threshold with all methods having complexity ≤ 2 (trivial code — missing coverage here is unlikely to hide real bugs) +- Show the top 3 actual risk hotspots by default. +- Exclude fully covered low-risk methods from the hotspot table. +- State how many additional methods exceeded the threshold instead of listing + them all. +- Honor a user-supplied count up to 10. Exceed 10 only when explicitly requested. +- For five or fewer below-threshold members, name all of them. For larger sets, + show the top hotspots and summarize the remaining count and range. +- Give 1–3 recommendations ordered by expected risk reduction. ---- +## Prioritize -## Coverage Intelligence — Going Beyond the Numbers +- **HIGH** — both CRAP and coverage exceed their risk thresholds. +- **MED** — either CRAP or coverage exceeds its threshold. +- **LOW** — below coverage threshold but complexity is at most 2. -**Prioritize uncovered code that is** complex (cyclomatic complexity > 5), on critical paths (auth, payment, data access, error handling), or changed frequently. **Deprioritize** trivial getters (complexity 1–2), generated files (EF migrations, `*.Designer.cs`, `*.g.cs`), and DI/configuration glue code. +Prefer complex uncovered critical paths (authentication, payment, data access, +error handling). Deprioritize trivial getters, generated code, migrations, and +configuration glue. -**Coverage plateau diagnosis** — if coverage has stopped increasing, check for: `[Exclude]` attributes hiding large code sections, tests that execute code but assert nothing (inflated coverage without verification), or integration code that needs external dependencies (databases, file system). +Do not project a CRAP reduction from an arbitrary target without showing the +assumption. Recalculate with the stated projected method coverage. -**AI-generated test quality** — coverage delta alone is insufficient. Flag methods where CRAP score is still above threshold after coverage increased (tests may be happy-path only), and methods covered by a single test with no branch variation. - ---- +Do not generate tests during analysis. Recommend focused test cases; +implementation is a separate follow-up. ## Style -- **Keep risk hotspots prominent and immediately after the summary section** — developers should find the highest-risk methods quickly -- **Quantify recommendations** — "adding 3 tests for `ProcessOrder` would cut the CRAP score from 48 to ~6" -- **Be direct** — skip preamble, get to the table -- **Emoji for visual scanning in generated output** (defined in `references/output-format.md`): - - | Symbol | Meaning | - |--------|---------| - | 🔥 | hotspots | - | 📋 | gaps | - | 💡 | recommendations | - | 📁 | reports | - | ✅ | passing | - | ❌ | failing | - | ⚠️ | warning | - | 🔴 | HIGH priority | - | 🟡 | MED priority | - | 🟢 | LOW priority | - -- **Always use Unicode emoji in generated output** — never shortcodes like `:x:` or `:fire:` +- Lead with the risk verdict, not setup narration. +- Quantify recommendations only from actual line/coverage evidence. +- Use one compact hotspot table. Do not append the full report template unless + the user requested a report. diff --git a/plugins/dotnet-test/skills/coverage-analysis/references/output-format.md b/plugins/dotnet-test/skills/coverage-analysis/references/output-format.md index 7e3c5b6a..e3647f74 100644 --- a/plugins/dotnet-test/skills/coverage-analysis/references/output-format.md +++ b/plugins/dotnet-test/skills/coverage-analysis/references/output-format.md @@ -1,6 +1,10 @@ # Output Format -Copy the template below **verbatim** for all fixed elements (headings, table headers, emoji, symbols). Only replace `` values with actual data. Do not substitute emoji with text equivalents, do not change `·` to `-`, do not change `×` to `x`, and do not drop section emoji prefixes. +Read and use this template only when the user explicitly requests a full +markdown coverage report. Normal interpretation, plateau, arithmetic, and +hotspot requests use the scaled response contract in `SKILL.md` instead. + +For an explicit full report, copy the template below **verbatim** for all fixed elements (headings, table headers, emoji, symbols). Only replace `` values with actual data. Do not substitute emoji with text equivalents, do not change `·` to `-`, do not change `×` to `x`, and do not drop section emoji prefixes. ```markdown # Coverage Analysis - @@ -25,7 +29,7 @@ Copy the template below **verbatim** for all fixed elements (headings, table hea > Coverage collected from ** of test project(s)**. > Outputs saved to: `/` (markdown summary + raw Cobertura XML). -> *If Phase 5 ran:* HTML/CSV reports also at `/reports/`. +> *If HTML/CSV generation ran:* additional reports are at `/reports/`. If any coverage provider package was added to test projects, include this note after the summary: @@ -84,4 +88,4 @@ Files below the line or branch coverage threshold, ordered by uncovered lines de | CSV data | `/reports/Summary.csv` *or* `Not generated` | ``` -If ReportGenerator (Phase 5) has not run, mark the HTML/Text/GitHub-markdown/CSV rows as `Not generated (optional — request HTML reports to enable)`. Do not invent paths for files that have not been produced. For **Raw Cobertura XML**, list the actual XML file path(s) used in analysis (for from-scratch runs this is typically under `/raw/`; for existing-data runs this may be under `TestResults/` or another user-supplied location). +If ReportGenerator has not run, mark the HTML/Text/GitHub-markdown/CSV rows as `Not generated (optional — request HTML reports to enable)`. Do not invent paths for files that have not been produced. For **Raw Cobertura XML**, list the actual XML file path(s) used in analysis (for from-scratch runs this is typically under `/raw/`; for existing-data runs this may be under `TestResults/` or another user-supplied location). diff --git a/plugins/dotnet-test/skills/coverage-analysis/references/report-generation.md b/plugins/dotnet-test/skills/coverage-analysis/references/report-generation.md index f4bd49cf..f98b473f 100644 --- a/plugins/dotnet-test/skills/coverage-analysis/references/report-generation.md +++ b/plugins/dotnet-test/skills/coverage-analysis/references/report-generation.md @@ -1,8 +1,9 @@ -# Coverage Analysis — ReportGenerator HTML/CSV reports (Phase 5) +# Coverage Analysis — ReportGenerator HTML/CSV reports -Read this file **only** when the user explicitly asked for HTML/CSV reports and the Phase 4 summary has already been delivered. +Read this file **only** when the user explicitly asked for HTML/CSV reports and +the direct coverage/risk answer has already been delivered. -## Step 6: Verify or install ReportGenerator (only if running Phase 5) +## Verify or install ReportGenerator ```powershell $rgAvailable = $false @@ -51,4 +52,6 @@ if ($rgAvailable) { } ``` -After Phase 5 completes successfully, you may follow up with a short message pointing the user to the generated HTML report (one paragraph, no need to repeat the summary). +After report generation completes successfully, you may follow up with a short +message pointing the user to the generated HTML report (one paragraph, no need +to repeat the summary). diff --git a/plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md b/plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md index 31d68520..0db1b92c 100644 --- a/plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md +++ b/plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md @@ -1,6 +1,8 @@ # Coverage Analysis — setup and discovery -Read this file at the start of Phase 1. It carries the discovery, output-directory and existing-report probes referenced by `SKILL.md`. +Read this file only when the user did not supply usable coverage evidence and +the request requires workspace discovery or fresh collection. Do not read or run +these probes for a supplied excerpt or valid Cobertura path. ## Step 1: Locate the solution or project @@ -175,15 +177,15 @@ Write-Host "EXISTING_COBERTURA_COUNT:$($coberturaFiles.Count)" $coberturaFiles | ForEach-Object { Write-Host "EXISTING_COBERTURA:$($_.FullName)" } ``` -- If `EXISTING_COBERTURA_COUNT` > 0 → **skip Phase 2 entirely** and pass these paths to the Phase 3 scripts. +- If `EXISTING_COBERTURA_COUNT` > 0 → skip fresh collection and analyze these paths. - If `EXISTING_COBERTURA_COUNT` == 0 and all test projects are SDK-style → run - Phase 2 to generate fresh coverage. + the collection workflow in `test-execution.md`. - If `EXISTING_COBERTURA_COUNT` == 0 and only classic/packages.config projects exist → use a repository-owned coverage command that emits Cobertura; otherwise stop with the message above. - If `EXISTING_COBERTURA_COUNT` == 0 and both classic and SDK-style projects - exist → continue with Phase 2 for `SDK_TEST_PROJECTS` only and mark the result - partial until classic-project Cobertura is available. + exist → collect only for `SDK_TEST_PROJECTS` and mark the result partial until + classic-project Cobertura is available. ## Step 2c: Recommend ignoring `TestResults/` diff --git a/plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md b/plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md index c767210c..9f724de6 100644 --- a/plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md +++ b/plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md @@ -1,6 +1,8 @@ -# Coverage Analysis — test execution (Phase 2) +# Coverage Analysis — fresh test execution -Read this file **only** when Phase 1 found no existing Cobertura XML (`EXISTING_COBERTURA_COUNT:0`) and fresh coverage must be produced. When a report already exists, skip straight to Phase 3. +Read this file **only** when discovery found no existing Cobertura XML +(`EXISTING_COBERTURA_COUNT:0`) and fresh coverage must be produced. When a report +already exists, return to the existing-data path in `SKILL.md`. This automatic provider workflow is for `SDK_TEST_PROJECTS` only. Exclude every `CLASSIC_TEST_PROJECT` before provider detection, package addition, restore, or diff --git a/tests/dotnet-test/coverage-analysis/eval.yaml b/tests/dotnet-test/coverage-analysis/eval.yaml index 8ed05aba..3bf479b0 100644 --- a/tests/dotnet-test/coverage-analysis/eval.yaml +++ b/tests/dotnet-test/coverage-analysis/eval.yaml @@ -100,23 +100,22 @@ stimuli: - src: fixtures/partial-coverage/coverage.cobertura.xml dest: TestResults/coverage.cobertura.xml graders: - - type: output-matches - config: - pattern: CRAP|crap - type: output-matches config: pattern: CalculateGpa|Calculate_?Gpa - type: output-matches config: pattern: hotspot|risk|coverage + - type: output-not-matches + config: + pattern: (?im)^\s*(?:(?:[-*+$>]|`)\s*)*dotnet\s+(?:test|tool\s+install)\b - type: prompt rubric: - - Discovers the solution and identifies the test project - - Parses the existing Cobertura XML to extract per-method coverage data - - Calculates CRAP scores and produces a sorted risk hotspot table - - Identifies CalculateGpa as the highest-risk method (high complexity, low coverage) - - Provides actionable recommendations for the riskiest methods - - Generates or displays a coverage analysis report + - Uses the existing Cobertura XML rather than rerunning tests or installing report tooling + - Reports the overall line and branch coverage and extracts the below-threshold members + - Identifies CalculateGpa as the dominant coverage gap and names the other material gaps + - Provides concise, actionable recommendations derived from the coverage evidence + - Does not compute CRAP or generate a full report unless the user asks for risk/CRAP or report files - name: Run coverage from scratch without existing data prompt: Analyze code coverage and risk hotspots for this project. I haven't run any coverage yet. environment: @@ -173,13 +172,18 @@ stimuli: - type: output-matches config: pattern: 0%|uncovered|untested|no coverage + - type: output-matches + config: + pattern: (?i)Enroll + - type: output-matches + config: + pattern: (?i)(46\s*(?:of|/)\s*47|9[78](?:\.\d+)?%) - type: prompt rubric: - Identifies CalculateGpa as the method blocking coverage improvement (0% coverage, highest complexity) - - Explains that 3 of 4 methods are well-covered but CalculateGpa drags down the average - - Quantifies the impact — covering CalculateGpa would raise overall coverage significantly - - Uses CRAP scores to distinguish the risky gap (CalculateGpa) from trivial ones - - Provides specific test recommendations for the uncovered method + - Also identifies the remaining uncovered Enroll line instead of calling CalculateGpa the sole or entire gap + - Quantifies the impact correctly — fully covering CalculateGpa raises coverage from 22/47 to 46/47, about 98% + - Provides specific test recommendations for both CalculateGpa and the remaining uncovered Enroll line - name: Reconcile a coverage target spread across several members prompt: | A Cobertura summary reports 25 of 42 lines covered (about 60%). These are @@ -193,12 +197,16 @@ stimuli: Which members hold us back, and can testing only the top hotspot reach 80%? Analyze the arithmetic and prioritize the work. Do not run tools. graders: + - type: output-matches + config: + pattern: (?i)(34\s*(?:of|/)\s*42|35\s*(?:of|/)\s*42|81(?:\.\d+)?%|83(?:\.\d+)?%) - type: prompt rubric: - Listed all four below-threshold members rather than naming only the worst one - Calculated that 80% of 42 lines requires at least 34 covered lines - Explained that fully covering the 5-line Apply hotspot reaches only 30 of 42 lines, about 71%, so it is insufficient alone - - Prioritized Apply first because it ties for most uncovered lines and has the highest complexity, while naming at least one additional member needed to close the remaining gap + - Prioritized Apply first because it ties for most uncovered lines and has the highest complexity + - Gave at least one concrete sufficient combination, such as Apply plus Validate reaching 34/42 or Apply plus Calculate reaching 35/42 constraints: reject_tools: - bash @@ -230,16 +238,22 @@ stimuli: pattern: CRAP|crap - type: output-not-matches config: - pattern: (?im)^\s*(?:[$>]\s*)?dotnet\s+(?:test|tool\s+install)\b + pattern: (?im)^\s*(?:(?:[-*+$>]|`)\s*)*dotnet\s+(?:test|tool\s+install)\b + - type: run-command + config: + command: >- + sh -c 'test ! -e TestResults/coverage-analysis/coverage-analysis.md + && test ! -d TestResults/coverage-analysis/reports' + expected_exit_code: 0 - type: prompt rubric: - Used the Cobertura report at the path the user gave, instead of scanning TestResults/ or reporting that no coverage data was found - Did not run dotnet test — the user supplied coverage data and explicitly asked not to re-run the suite - Did not install ReportGenerator or any other global tool; HTML/CSV reports were not requested - - Delivered the CRAP/risk hotspot table as the response rather than deferring it behind optional report - generation + - Delivered a compact CRAP/risk hotspot answer without creating or attempting to create report files - Identified CalculateGpa as the top hotspot with the supporting complexity and coverage figures + - Limited the table to actual hotspots and omitted fully covered low-risk methods - name: Refactoring safety assessment from coverage data prompt: I need to refactor StudentService before adding a feature. Which methods are safe to change and which ones would I be refactoring blind? Coverage data is in TestResults/coverage.cobertura.xml. @@ -294,8 +308,8 @@ stimuli: - Identified Enroll as only partially covered because each condition was taken one way - Identified CalculateGpa as wholly untested - Recommended tests that force the opposite Enroll condition outcomes, not merely more line execution - - Did not infer which Enroll outcome is missing from `1/2`; explained that compound predicates may require - independent operand and short-circuit combinations when the source warrants them + - Did not infer which Enroll outcome is missing or invent source-like predicates from `1/2`; explained that + compound predicates may require independent operand and short-circuit combinations when the source warrants them constraints: reject_tools: - bash