Scale coverage analysis to request scope (#1078)

* Scale coverage analysis to request scope

Gate CRAP and full reports behind explicit intent, reconcile coverage arithmetic, and strengthen only evidence-backed eval outcomes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Address coverage eval review feedback

Broaden no-command guards for Markdown formatting and clarify plateau recommendations across both uncovered members.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: dc01c723-1c85-446e-873a-c4bd26fdb0e8

* Improve coverage analysis recovery

Use the latest cross-model evidence to recover from failed artifact reads, require source-safe branch interpretation, make target combinations concrete, and enforce CRAP-first refactoring risk ranking.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: dc01c723-1c85-446e-873a-c4bd26fdb0e8

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dc01c723-1c85-446e-873a-c4bd26fdb0e8
This commit is contained in:
Amaury Levé
2026-08-27 16:04:37 +02:00
committed by GitHub
parent 94ca0ca748
commit 26caf0ecab
7 changed files with 241 additions and 248 deletions
@@ -1,18 +1,19 @@
---
name: coverage-analysis
description: >
Explains and analyzes .NET Cobertura line, branch, and condition evidence plus
project-wide CRAP risk. MUST USE for "why is branch coverage lower than line
coverage?", condition-coverage="50% (1/2)", a supplied coverage excerpt,
partially covered conditions, untested outcomes, plateaus, members blocking a
project target, project-wide CRAP/refactoring safety, or coverage-backed test
priorities. For project-wide work, uses real coverage, counts every
below-threshold member, and ranks the top N hotspots. DO NOT USE FOR: test
trait/category distributions or coverage shape by test type (test-tagging);
static source-to-test pairing (find-untested-sources); behavioral/pseudo-mutation
gaps (test-gap-analysis); named-target CRAP (crap-score); test-code audits
Interprets .NET Cobertura line, branch, and condition evidence and, when
explicitly requested, computes project-wide CRAP/refactoring-risk hotspots.
MUST USE for "why is branch coverage lower than line coverage?",
condition-coverage="50% (1/2)", a supplied coverage excerpt, partially covered
conditions, coverage plateaus, members blocking a target, project-wide CRAP,
refactoring safety, or coverage-backed risk priorities. A supplied report is
analyzed directly without rerunning tests, installing tools, generating a
report, or calculating CRAP unless the request asks for risk/CRAP/refactoring
safety. DO NOT USE FOR: test trait distributions (test-tagging); static
source-to-test pairing (find-untested-sources); behavioral/pseudo-mutation gaps
(test-gap-analysis); named-target CRAP (crap-score); test-code audits
(test-anti-patterns); raw collection/percentage-only requests or just running
tests (run-tests); non-.NET coverage (native tooling); or writing tests.
tests (run-tests); non-.NET coverage; or writing tests.
license: MIT
---
@@ -20,21 +21,16 @@ license: MIT
## Purpose
Raw coverage percentages answer "what code was executed?" — they don't answer what you actually need to know:
- **What tests should I write next?** — ranked by risk and impact
- **Which uncovered code is risky vs. trivial?** — CRAP scores separate the two
- **Why has coverage plateaued?** — identify the files blocking further gains
- **Is this code safe to refactor?** — complex + uncovered = dangerous to change
This skill bridges that gap: from a bare .NET solution to a prioritized risk hotspot list, with no manual tool configuration required.
Explain what .NET coverage evidence proves, reconcile target arithmetic, and
identify the code blocking progress. Add complexity/CRAP ranking only when the
user explicitly asks for risk hotspots, CRAP, priorities by risk, or refactoring
safety.
## When to Use
Use this skill for interpreting a supplied .NET line/branch/condition excerpt or
for project-wide test coverage, coverage gaps, code risk, CRAP scores, test
priorities, plateau diagnosis, or refactoring safety — even if the user does not
explicitly say "coverage analysis".
Use this skill for interpreting supplied .NET line/branch/condition evidence,
coverage gaps and plateaus, target arithmetic, or explicit project-wide
coverage-backed risk analysis.
## When Not to Use
@@ -53,173 +49,146 @@ explicitly say "coverage analysis".
| Project/solution path | No | Current directory | Path to the .NET solution or project |
| Line coverage threshold | No | 80% | Minimum acceptable line coverage |
| Branch coverage threshold | No | 70% | Minimum acceptable branch coverage |
| CRAP threshold | No | 30 | Maximum acceptable CRAP score before flagging |
| Top N hotspots | No | 10 | Number of risk hotspots to surface |
| Existing Cobertura path | No | Discover only if not supplied | Preferred input; never rerun tests when usable |
| CRAP threshold | No | 30 | Used only for explicit risk/CRAP requests |
| Hotspot count | No | 3 | Explicit risk requests only; cap at 10 unless the user asks for more |
### Prerequisites
Discover optional inputs from the workspace. Do not ask for a project path when
the current directory or a supplied report is sufficient.
- .NET SDK installed (`dotnet` on PATH)
- At least one test project referencing the production code (xUnit, NUnit, or MSTest) — only required for the from-scratch path; not needed when the user supplies an existing Cobertura XML
- **Optional, only for the SDK-style from-scratch path:** internet/NuGet access for `dotnet add package coverlet.collector` (or `Microsoft.Testing.Extensions.CodeCoverage`) when a test project has no coverage provider yet. Skip for classic projects and when the user supplies an existing Cobertura XML.
- **Optional, only for Phase 5:** internet access for `dotnet tool install` (ReportGenerator). Core CRAP/coverage analysis works from Cobertura XML alone — ReportGenerator only adds HTML/CSV reports as an optional post-summary extra.
## Choose the smallest matching path
The skill auto-detects coverage provider state per test project and selects the least-invasive execution strategy:
| User intent | Required work | Do not do |
|-------------|---------------|-----------|
| Explain a supplied excerpt, condition, or summary | Answer directly from the supplied evidence | Tools, CRAP, discovery, report files |
| Interpret a supplied Cobertura path or diagnose a plateau | Read that report, reconcile totals, name all material gaps, answer directly | Rerun tests, install tools, compute CRAP, or generate files unless explicitly requested |
| Rank risk hotspots, compute project-wide CRAP, or assess refactoring safety | Use the supplied/existing report, read `references/guidelines.md`, and compute CRAP before ranking | Coverage-only ranking or a full report template unless requested |
| Analyze coverage when no report exists | Read `references/setup-discovery.md`; collect once using `references/test-execution.md` if safe | CRAP unless risk was requested |
| Produce a full markdown/HTML/CSV report | First deliver the direct answer; then read `references/output-format.md` or `references/report-generation.md` | Report generation before the answer |
- unified Microsoft CodeCoverage when all projects use it,
- unified Coverlet when no project uses Microsoft CodeCoverage,
- per-project provider execution when the solution is truly mixed.
Words such as **analyze coverage**, **what is blocking coverage**, or **why is
coverage stuck** do not by themselves request CRAP. Explicit signals include
**risk hotspot**, **CRAP**, **complexity-weighted priority**, **safe to refactor**,
or an equivalent request to combine complexity with coverage.
No pre-existing runsettings files or manually installed tools required.
## Existing-data fast path
The automatic from-scratch path applies to SDK-style projects. Classic non-SDK
projects (`ToolsVersion`, explicit `<Compile Include>`, `packages.config`) remain
fully supported when the user supplies Cobertura XML. Without an existing report,
use a repository-provided coverage command if one exists; otherwise stop and ask
for Cobertura output. Never convert the project, run `dotnet add package`, or add
`PackageReference` as an incidental coverage setup step.
When the user supplies a coverage excerpt, summary, or valid Cobertura path:
> **CLASSIC-ONLY HARD STOP:** If every discovered test project is classic and no
> existing Cobertura report or checked-in coverage command exists, stop after
> discovery. Do not create a temporary SDK-style wrapper/project, copy the source
> into one, install a collector elsewhere, or generate substitute coverage from
> a different assembly. That data does not describe the requested project.
> Report complexity alone if useful and request real Cobertura from the
> repository's supported toolchain.
- Treat it as authoritative input and start there.
- Do not discover the solution or test projects unless source mapping is necessary.
- Do not run `dotnet test`, install ReportGenerator, add a coverage package, or
read `references/setup-discovery.md`, `references/test-execution.md`, or
`references/report-generation.md`.
- Do not write `coverage-analysis.md` or create a report directory unless the user
requested a saved/full report.
- For interpretation and plateau questions, parse only the evidence needed to
answer. For explicit project-wide risk requests, use the bundled scripts as
described in `references/guidelines.md`.
## Workflow
A failed read/view operation is not proof that a named path does not exist. After
one fails, make one allowed targeted existence probe, such as a workspace-relative
glob, and retry the same artifact with a normalized path or alternate reader.
Report the exact missing-path problem only when that independent check also fails.
Do not broaden the search to unrelated coverage files or present a substitute
artifact.
> **MANDATORY for project-wide analysis: deliver the final assistant response with the CRAP/risk-hotspot summary BEFORE any optional work.** As soon as `Compute-CrapScores.ps1` and `Extract-MethodCoverage.ps1` return data, your **next** assistant response must contain the user-facing analysis (CRAP table, blocking methods, recommendations). Do not run ReportGenerator (Phase 5), do not install global tools, and do not start any heavy parallel work before that response is delivered. The user is judged on the final assistant message, not on side-effect files.
>
> If a phase fails, times out, or budget is running low, skip remaining optional work and immediately return a partial summary containing: (1) what was found in the Cobertura XML, (2) any CRAP/risk-hotspot data already extracted, (3) which methods are blocking coverage, and (4) failures encountered.
## Collection path
If the user supplies only a coverage excerpt or asks what a line/condition entry
means, answer directly from that evidence and stop. Do not run tools or fabricate
a project-wide CRAP dashboard. A line hit proves execution, not both decision
outcomes. `50% (1/2)` proves one reported condition outcome ran but does not say
which outcome; recommend forcing the opposite outcome. For compound predicates,
state that operand and short-circuit combinations may need independent tests, but
do not infer their number without source or fuller XML.
Use this path only when no usable coverage evidence exists and the user asked for
analysis that requires it.
If the user provides a path to existing Cobertura XML (or coverage data is already present in `TestResults/`), **skip Phase 2 entirely** (no test execution) **and skip Phase 5 by default** (no ReportGenerator install or HTML report) — go directly from Phase 3 (analysis scripts) to Phase 4 (user-facing summary). Only run Phase 5 if the user explicitly asks for HTML/CSV reports. The Risk Hotspots table and CRAP scores are mandatory in every project-wide output — they are the skill's core value-add over raw coverage numbers.
1. Read `references/setup-discovery.md`.
2. Prefer existing Cobertura discovered under the requested root.
3. If none exists, read `references/test-execution.md` and run the selected
coverage command once per entry point.
4. Analyze the resulting Cobertura. Compute CRAP only if risk analysis was
explicitly requested.
The project-wide workflow runs in five phases. Phases 14 are required; Phase 5 (ReportGenerator HTML/CSV reports) is strictly optional and runs **after** the user-facing summary has been delivered. Do not parallelize Phase 5 with earlier phases — the heavy `dotnet tool install` for ReportGenerator can crash the session before Phase 4 completes.
Do not modify production code. The only permitted incidental project change is
adding one missing coverage provider to an SDK-style test project as described in
`references/test-execution.md`; never add a second provider, and report the
change plus its revert command.
### Phase 1 — Setup (sequential)
The automatic collection path is for SDK-style projects. For classic non-SDK or
`packages.config` projects, use only a repository-owned coverage command. If none
exists, stop and request Cobertura XML. Never migrate the project, inject an
SDK-style provider, create a wrapper project, or report substitute coverage from
another assembly.
Read `references/setup-discovery.md` and run the probes it contains, in order:
## Interpretation and arithmetic invariants
| Step | Emits | Why it matters |
|------|-------|----------------|
| 1. Locate the solution or project | `ENTRY_TYPE`, `ENTRY`, `TEST_PROJECTS`, `CLASSIC_TEST_PROJECTS`, `SDK_TEST_PROJECTS`, `TEST_OUTPUT_ROOT` | Entry point, safe project partitions, and output location |
| 2. Create the output directory | `COVERAGE_DIR` | Skill-owned `TestResults/coverage-analysis/`; never deletes user-supplied reports |
| 2b. Discover or accept existing Cobertura XML | `EXISTING_COBERTURA_COUNT`, `EXISTING_COBERTURA` | A user-supplied path always wins; otherwise probe `TestResults/` |
| 2c. Recommend ignoring `TestResults/` | `GITIGNORE_RECOMMENDATION` | One-line recommendation, reported in the summary |
- A line hit proves execution, not both decision outcomes.
- `condition-coverage="50% (1/2)"` proves one reported outcome ran, but not which
one. Recommend forcing the opposite outcome. Without source, never invent likely
predicates or claim whether true or false is missing. State that compound
predicates need independently exercised operands and short-circuit combinations
when applicable; do not infer the exact combinations without source or fuller XML.
- Derive overall totals from Cobertura's covered/valid line counts. For target
`T`, required covered lines are `ceiling(valid lines × T)`.
- Projected coverage is `(current covered lines + newly covered distinct lines) /
valid lines`. State assumptions such as fully covering a method.
- When asked whether one member can reach a target, show its maximum projected
total and at least one concrete sufficient combination of supplied members or
line gains. If no supplied combination is sufficient, say so.
- Reconcile member gaps against project totals. Method line ranges can overlap or
omit class-level lines, so do not sum method counts as project truth.
- Never call one member the **sole**, **entire**, or **all** remaining gap unless
its distinct uncovered lines exactly reconcile with the project total and no
other below-threshold member remains.
- Name every supplied or extracted below-threshold member, but keep detail
proportional: lead with the blockers, summarize the remainder in one sentence
or a compact table.
Branching after Phase 1:
## Response contract
- `EXISTING_COBERTURA_COUNT` > 0 → **skip Phase 2 entirely**; go to Phase 3 with those paths. Do not read `references/test-execution.md`.
- `EXISTING_COBERTURA_COUNT` == 0 and only SDK-style test projects were found → run Phase 2 normally.
- `EXISTING_COBERTURA_COUNT` == 0 and only classic/packages.config projects were found → use a checked-in coverage command; if none exists, stop and request Cobertura XML.
- Mixed classic/SDK projects → collect the SDK subset per project, never through the solution entry; label results partial until classic-project Cobertura is supplied.
- `ENTRY_TYPE:NotFound` with test projects → use the test projects directly as entry points.
- No test projects and no Cobertura XML → stop: `No test projects found (expected projects with 'Test' or 'Spec' in the name), and no existing Cobertura XML was provided. Add a test project or provide a Cobertura file path.`
Answer the user's question in the first 24 sentences.
### Phase 2 — Test execution (skip when Cobertura XML already exists)
- **Excerpt or arithmetic question:** one explanation plus the next test or member
priority. No dashboard.
- **Existing-report interpretation or plateau:** overall line/branch coverage,
blocking members, reconciled target impact, and 13 recommendations. Use at
most one compact table.
- **Explicit risk/CRAP request:** top 3 actual hotspots by default, supporting
complexity/coverage/CRAP values, remaining flagged count, and 13 priorities.
Exclude fully covered low-risk methods from the hotspot table. Never exceed 10
rows unless the user requests a larger count. For refactoring safety, rank risky
methods by CRAP rather than raw coverage alone, then name comparatively safe
well-covered methods separately.
- **Explicit full report request:** read `references/output-format.md`. Save the
report only then.
Run only when Phase 1 found no Cobertura XML. If the user already has coverage data, skip directly to Phase 3 — do not read this section's reference file, and do not re-run the suite.
Report only artifacts that exist. Do not announce inaccessible output paths or
failed optional file writes when no file was requested.
Read `references/test-execution.md` for the SDK-style subset. It covers safe
partitioning, provider detection, package addition, `dotnet test`, exit codes,
and report discovery. For classic projects, run only a repository-owned coverage
command; otherwise request Cobertura XML and mark mixed-solution results partial.
## Optional reports
Exit codes: **0** all passed; **1** some tests failed (coverage is still collected — proceed with a warning); anything else is a build failure — stop and report it.
HTML/CSV/markdown files are not part of normal analysis. Generate them only when
the user explicitly requests report files or a CI artifact.
### Phase 3 — Analysis (sequential)
Run the two bundled PowerShell scripts. Both are cheap and complete in seconds. **Do not** install or invoke ReportGenerator here — that belongs in optional Phase 5, after the user-facing summary has been delivered.
#### Step 4: Calculate CRAP scores using the bundled script
Run `scripts/Compute-CrapScores.ps1` (co-located with this SKILL.md). It reads all Cobertura XML files, applies `CRAP(m) = comp² × (1 cov)³ + comp` per method, and returns the top-N hotspots as JSON.
To locate the script: find the directory containing this skill's `SKILL.md` file (the skill loader provides this context), then resolve `scripts/Compute-CrapScores.ps1` relative to it. If the script path cannot be determined, calculate CRAP scores inline using the formula below.
```powershell
& "<skill-directory>/scripts/Compute-CrapScores.ps1" `
-CoberturaPath @(<all COBERTURA file paths as array>) `
-CrapThreshold <crap_threshold> `
-TopN <top_n>
```
Script outputs: `OVERALL_LINE_COVERAGE:<n>`, `OVERALL_BRANCH_COVERAGE:<n>` (aggregated project-wide rates across all provided Cobertura files), `TOTAL_METHODS:<n>`, `FLAGGED_METHODS:<n>`, `HOTSPOTS:<json>` (top-N sorted by CrapScore descending). The OVERALL_* values are exactly what the Phase 4 summary needs for the "Line Coverage" / "Branch Coverage" rows — no separate XML parsing tool call is required.
#### Step 5: Extract per-method coverage gaps
Run `scripts/Extract-MethodCoverage.ps1` to get per-method coverage data for the Coverage Gaps table:
```powershell
& "<skill-directory>/scripts/Extract-MethodCoverage.ps1" `
-CoberturaPath @(<all COBERTURA file paths as array>) `
-CoverageThreshold <line_threshold> `
-BranchThreshold <branch_threshold> `
-Filter below-threshold
```
Script outputs: JSON array of methods below the coverage threshold, sorted by coverage ascending. Use this data to populate the Coverage Gaps by File table in the report.
### Phase 4 — Project-wide user-facing summary (MANDATORY — your next assistant response)
As soon as Phase 3 completes, **your immediately next assistant response must contain the user-facing analysis** — do not interleave any other tool calls before it. This is the response the user (and any judge) sees. Skipping or deferring this in favor of Phase 5 (ReportGenerator) is a hard failure.
The response must include, at minimum:
0. **A direct answer to the question that was actually asked, in the first 24 sentences.** For "why is my coverage stuck?" / "what's blocking me?", name the blocking members and the lines involved before any table. The standard sections below still follow.
1. Overall line and branch coverage — read directly from the `OVERALL_LINE_COVERAGE:` / `OVERALL_BRANCH_COVERAGE:` lines emitted by `Compute-CrapScores.ps1` (no extra Cobertura parsing required)
2. The Risk Hotspots table built from `Compute-CrapScores.ps1` `HOTSPOTS:` output (CRAP scores, complexity, coverage)
3. Identification of the highest-risk method(s) and what is blocking coverage
4. 13 prioritized, specific recommendations (which method to test, expected CRAP/coverage impact)
**Every number must come from the script output, and the arithmetic must reconcile.** Uncovered lines attributed to individual members must not exceed the project's total uncovered lines, and the coverage you project after a recommendation must follow from those counts.
**List every member below threshold, not just the worst one.** `Extract-MethodCoverage.ps1` returns the full below-threshold set: name the others even if briefly. Only say "the rest is fine / leave it alone" when that set is otherwise empty — claiming one method is the entire gap when the extractor found more is a factual error.
Use `references/output-format.md` verbatim for fixed headings, table structures, symbols, and emoji. Use `references/guidelines.md` for prioritization rules and style.
If Phase 5 has not yet run when you compose this summary, mark the `## 📁 Reports` section's HTML/Text/CSV/GitHub-markdown rows as `Not generated (optional — request HTML reports to enable)`. Only the `coverage-analysis.md` and raw Cobertura paths are guaranteed to exist.
Attempt to save the same content to `TestResults/coverage-analysis/coverage-analysis.md` before delivering the response (use the editor's create/edit tool — do not shell out). If the file write fails, still deliver the summary and note the file-write failure explicitly.
### Phase 5 — Optional: ReportGenerator HTML/CSV reports (post-summary)
Phase 5 is **strictly optional** and runs **only after** Phase 4 has been delivered. Skip Phase 5 entirely when:
- The user supplied existing Cobertura XML and only asked for analysis (the default for the existing-data path).
- The user is diagnosing a coverage plateau or asking "what's blocking me?" — they want the answer, not a static-site report.
- ReportGenerator is not already installed and you have no clear signal the user wants HTML reports.
Run Phase 5 only when the user explicitly asks for HTML/CSV reports, or when the project flow requires them (e.g., a CI artifact upload step).
Read `references/report-generation.md` for the ReportGenerator install and invocation. It is the only heavy step in this skill: a `dotnet tool install` that can exhaust the session budget, which is why it never runs before the Phase 4 summary has been delivered.
If the install fails (no internet), leave the existing Phase 4 summary as the final output and note that HTML reports were skipped. Do not retry or block on it.
1. Deliver the direct coverage/risk answer first.
2. For a full markdown report, read `references/output-format.md` and save it.
3. For HTML/CSV, then read `references/report-generation.md`. Do not install
ReportGenerator before the direct answer, and do not retry a failed install.
## Validation
- Verify that at least one `coverage.cobertura.xml` file was generated by the selected SDK-style or repository-owned command (or already exists when the user supplied one)
- Confirm the assistant response contained the CRAP/risk-hotspot table — saving the markdown file is secondary
- Confirm `TestResults/coverage-analysis/coverage-analysis.md` was written and contains data
- Spot-check one method's CRAP score: `comp² × (1 cov)³ + comp` — a method with 100% coverage should have CRAP = complexity
- If Phase 5 ran, verify `TestResults/coverage-analysis/reports/index.html` exists; otherwise the report file should mark HTML/Text/CSV rows as `Not generated`
- Confirm a supplied report was used without test execution or tool installation.
- Reconcile covered, valid, and uncovered line totals before projecting impact.
- Confirm every stated blocker comes from supplied or extracted evidence.
- If CRAP was requested, spot-check one score using `references/guidelines.md`.
- If files were requested, verify they exist before reporting their paths.
## Common Pitfalls
- **No Cobertura XML generated in an SDK-style project** — the test project may lack a coverage provider. The skill auto-adds one, but if `dotnet add package` fails (offline/proxy), coverage collection silently produces nothing. Check for `.coverage` binary files as a fallback indicator.
- **Classic non-SDK project without a report** — do not inject an SDK-style provider. Use the repository's existing coverage workflow or ask the user for Cobertura XML.
- **Test failures (exit code 1)** — coverage is still collected from passing tests. Do not abort; proceed with partial data and note the failures in the summary.
- **Premature end before user-facing summary** — never start Phase 5 (ReportGenerator install/run) before the Phase 4 assistant response is delivered. The heavy `dotnet tool install` can crash the session or exhaust budget, leaving the user with no analysis even though the CRAP scores were already computed.
- **ReportGenerator install failure** — if `dotnet tool install` fails (no internet) during Phase 5, leave the existing Phase 4 summary as the final output and note that HTML reports were skipped. Do not retry or block on the install.
- **Method name mismatches in Cobertura** — async methods, lambdas, and local functions may have compiler-generated names. The scripts use the Cobertura method name/signature directly; verify against source if results look unexpected.
- **Mixed coverage providers** — when a solution contains both Coverlet and Microsoft CodeCoverage projects, the skill runs per-project to avoid dual-provider conflicts. This is slower but correct.
- **Numbers that don't reconcile** — per-member uncovered lines that exceed the project total, or a projected coverage figure that doesn't follow from the counts, make the whole analysis untrustworthy. Re-read the script output rather than estimating.
- **Declaring one method "the entire gap"** — check the full below-threshold list from `Extract-MethodCoverage.ps1` first; naming a single blocker while other uncovered members exist misdirects the user's next test.
- **Existing report triggers setup work** — stop. Analyze the named artifact first.
- **Generic analysis triggers CRAP** — stop. CRAP requires explicit risk intent.
- **One method is called the entire gap** — reconcile distinct uncovered lines and
the full below-threshold set first.
- **A small request gets a dashboard** — scale down to a direct answer or one table.
- **No Cobertura from collection** — report the collection failure; do not invent
substitute coverage.
- **Compiler-generated method names** — verify suspicious async, lambda, or local
function names against source before presenting them as user-authored members.
- **Test exit code 1** — coverage may still exist; proceed with a warning. Other
nonzero build failures stop the collection path.
@@ -1,69 +1,68 @@
# Guidelines
# Explicit Risk/CRAP Analysis
**Don't modify source or production code.** For SDK-style test projects only,
the permitted project-file modifications are adding a missing coverage provider:
`coverlet.collector` or `Microsoft.Testing.Extensions.CodeCoverage`. Never modify
a classic non-SDK or `packages.config` project; require its repository-owned
coverage command or existing Cobertura XML. Do not add a second provider. Always
log package additions and document revert commands in the report. Write all
other output to `TestResults/coverage-analysis/`.
Read this file only when the user explicitly asks for project-wide risk
hotspots, CRAP scores, complexity-weighted priorities, or refactoring safety.
Do not read it for supplied-excerpt interpretation, plateau diagnosis, or target
arithmetic alone.
**Always show and open the generated markdown report — but only after the assistant response with the CRAP/risk-hotspot summary has been delivered.** Saving and opening `TestResults/coverage-analysis/coverage-analysis.md` is a follow-up action; it must never delay the user-facing summary.
## Compute the risk data
**Don't generate new tests during the initial analysis run.** This skill surfaces where tests are needed. Test generation is a separate follow-up step outside the scope of this skill.
Resolve the scripts relative to this skill's `SKILL.md`:
**For SDK-style projects, use inline `dotnet test` arguments, not runsettings
files.** For classic projects, preserve the repository's coverage workflow.
```powershell
& "<skill-directory>/scripts/Compute-CrapScores.ps1" `
-CoberturaPath @(<all Cobertura paths>) `
-CrapThreshold <crap_threshold> `
-TopN <top_n>
**Show the risk hotspots table even when all thresholds pass.** A project at 90% line coverage can still have a method with cyclomatic complexity 20 and 0% branch coverage. The thresholds measure averages; the hotspot table finds outliers. Don't hide it just because the summary looks green.
& "<skill-directory>/scripts/Extract-MethodCoverage.ps1" `
-CoberturaPath @(<all Cobertura paths>) `
-CoverageThreshold <line_threshold> `
-BranchThreshold <branch_threshold> `
-Filter below-threshold
```
**Always compute and surface CRAP scores.** The Risk Hotspots table is mandatory in every analysis output, whether analyzing pre-existing data, freshly collected data, or diagnosing a plateau. Never skip CRAP score computation — it is the primary differentiator between this skill and raw `dotnet test` coverage output.
`Compute-CrapScores.ps1` emits aggregate line/branch coverage, method counts,
flagged counts, and sorted hotspots. `Extract-MethodCoverage.ps1` emits every
below-threshold method. Use both for explicit project-wide risk work.
**Continue past test failures (exit code 1).** If some tests fail, coverage is still collected from the passing tests — partial data is better than no data. Note the failures in the summary and proceed. Aborting would leave the developer with nothing actionable.
CRAP is:
**Run the selected coverage command only once per entry point during normal
flow.** For SDK-style projects this is `dotnet test`; for classic projects it is
the repository-owned command. A single recovery rerun is allowed only if the
first run produced no Cobertura XML and only `.coverage` binary output.
`CRAP(m) = complexity² × (1 lineCoverage)³ + complexity`
**CRAP threshold of 30 is the default for a reason.** Scores above 30 are widely cited (by the original researchers) as "needs immediate attention." Scores between 15 and 30 are moderate — flag them in the table but don't make them sound catastrophic. Scores ≤ 5 are generally fine.
A method at 100% coverage therefore has CRAP equal to its complexity. Use 30 as
the default flagged threshold; treat 1530 as moderate rather than catastrophic.
**Priority assignment for coverage gaps:**
## Scale the output
- **HIGH** — file has both a CRAP score above threshold AND coverage below threshold (the double failure is what makes it urgent)
- **MED** — coverage below threshold OR CRAP score above threshold, but not both
- **LOW** — coverage below threshold with all methods having complexity ≤ 2 (trivial code — missing coverage here is unlikely to hide real bugs)
- Show the top 3 actual risk hotspots by default.
- Exclude fully covered low-risk methods from the hotspot table.
- State how many additional methods exceeded the threshold instead of listing
them all.
- Honor a user-supplied count up to 10. Exceed 10 only when explicitly requested.
- For five or fewer below-threshold members, name all of them. For larger sets,
show the top hotspots and summarize the remaining count and range.
- Give 13 recommendations ordered by expected risk reduction.
---
## Prioritize
## Coverage Intelligence — Going Beyond the Numbers
- **HIGH** — both CRAP and coverage exceed their risk thresholds.
- **MED** — either CRAP or coverage exceeds its threshold.
- **LOW** — below coverage threshold but complexity is at most 2.
**Prioritize uncovered code that is** complex (cyclomatic complexity > 5), on critical paths (auth, payment, data access, error handling), or changed frequently. **Deprioritize** trivial getters (complexity 12), generated files (EF migrations, `*.Designer.cs`, `*.g.cs`), and DI/configuration glue code.
Prefer complex uncovered critical paths (authentication, payment, data access,
error handling). Deprioritize trivial getters, generated code, migrations, and
configuration glue.
**Coverage plateau diagnosis** — if coverage has stopped increasing, check for: `[Exclude]` attributes hiding large code sections, tests that execute code but assert nothing (inflated coverage without verification), or integration code that needs external dependencies (databases, file system).
Do not project a CRAP reduction from an arbitrary target without showing the
assumption. Recalculate with the stated projected method coverage.
**AI-generated test quality** — coverage delta alone is insufficient. Flag methods where CRAP score is still above threshold after coverage increased (tests may be happy-path only), and methods covered by a single test with no branch variation.
---
Do not generate tests during analysis. Recommend focused test cases;
implementation is a separate follow-up.
## Style
- **Keep risk hotspots prominent and immediately after the summary section** — developers should find the highest-risk methods quickly
- **Quantify recommendations** — "adding 3 tests for `ProcessOrder` would cut the CRAP score from 48 to ~6"
- **Be direct** — skip preamble, get to the table
- **Emoji for visual scanning in generated output** (defined in `references/output-format.md`):
| Symbol | Meaning |
|--------|---------|
| 🔥 | hotspots |
| 📋 | gaps |
| 💡 | recommendations |
| 📁 | reports |
| ✅ | passing |
| ❌ | failing |
| ⚠️ | warning |
| 🔴 | HIGH priority |
| 🟡 | MED priority |
| 🟢 | LOW priority |
- **Always use Unicode emoji in generated output** — never shortcodes like `:x:` or `:fire:`
- Lead with the risk verdict, not setup narration.
- Quantify recommendations only from actual line/coverage evidence.
- Use one compact hotspot table. Do not append the full report template unless
the user requested a report.
@@ -1,6 +1,10 @@
# Output Format
Copy the template below **verbatim** for all fixed elements (headings, table headers, emoji, symbols). Only replace `<placeholder>` values with actual data. Do not substitute emoji with text equivalents, do not change `·` to `-`, do not change `×` to `x`, and do not drop section emoji prefixes.
Read and use this template only when the user explicitly requests a full
markdown coverage report. Normal interpretation, plateau, arithmetic, and
hotspot requests use the scaled response contract in `SKILL.md` instead.
For an explicit full report, copy the template below **verbatim** for all fixed elements (headings, table headers, emoji, symbols). Only replace `<placeholder>` values with actual data. Do not substitute emoji with text equivalents, do not change `·` to `-`, do not change `×` to `x`, and do not drop section emoji prefixes.
```markdown
# Coverage Analysis - <ProjectName>
@@ -25,7 +29,7 @@ Copy the template below **verbatim** for all fixed elements (headings, table hea
> Coverage collected from **<N> of <M> test project(s)**.
> Outputs saved to: `<coverageDir>/` (markdown summary + raw Cobertura XML).
> *If Phase 5 ran:* HTML/CSV reports also at `<coverageDir>/reports/`.
> *If HTML/CSV generation ran:* additional reports are at `<coverageDir>/reports/`.
If any coverage provider package was added to test projects, include this note after the summary:
@@ -84,4 +88,4 @@ Files below the line or branch coverage threshold, ordered by uncovered lines de
| CSV data | `<coverageDir>/reports/Summary.csv` *or* `Not generated` |
```
If ReportGenerator (Phase 5) has not run, mark the HTML/Text/GitHub-markdown/CSV rows as `Not generated (optional — request HTML reports to enable)`. Do not invent paths for files that have not been produced. For **Raw Cobertura XML**, list the actual XML file path(s) used in analysis (for from-scratch runs this is typically under `<coverageDir>/raw/`; for existing-data runs this may be under `TestResults/` or another user-supplied location).
If ReportGenerator has not run, mark the HTML/Text/GitHub-markdown/CSV rows as `Not generated (optional — request HTML reports to enable)`. Do not invent paths for files that have not been produced. For **Raw Cobertura XML**, list the actual XML file path(s) used in analysis (for from-scratch runs this is typically under `<coverageDir>/raw/`; for existing-data runs this may be under `TestResults/` or another user-supplied location).
@@ -1,8 +1,9 @@
# Coverage Analysis — ReportGenerator HTML/CSV reports (Phase 5)
# Coverage Analysis — ReportGenerator HTML/CSV reports
Read this file **only** when the user explicitly asked for HTML/CSV reports and the Phase 4 summary has already been delivered.
Read this file **only** when the user explicitly asked for HTML/CSV reports and
the direct coverage/risk answer has already been delivered.
## Step 6: Verify or install ReportGenerator (only if running Phase 5)
## Verify or install ReportGenerator
```powershell
$rgAvailable = $false
@@ -51,4 +52,6 @@ if ($rgAvailable) {
}
```
After Phase 5 completes successfully, you may follow up with a short message pointing the user to the generated HTML report (one paragraph, no need to repeat the summary).
After report generation completes successfully, you may follow up with a short
message pointing the user to the generated HTML report (one paragraph, no need
to repeat the summary).
@@ -1,6 +1,8 @@
# Coverage Analysis — setup and discovery
Read this file at the start of Phase 1. It carries the discovery, output-directory and existing-report probes referenced by `SKILL.md`.
Read this file only when the user did not supply usable coverage evidence and
the request requires workspace discovery or fresh collection. Do not read or run
these probes for a supplied excerpt or valid Cobertura path.
## Step 1: Locate the solution or project
@@ -175,15 +177,15 @@ Write-Host "EXISTING_COBERTURA_COUNT:$($coberturaFiles.Count)"
$coberturaFiles | ForEach-Object { Write-Host "EXISTING_COBERTURA:$($_.FullName)" }
```
- If `EXISTING_COBERTURA_COUNT` > 0 → **skip Phase 2 entirely** and pass these paths to the Phase 3 scripts.
- If `EXISTING_COBERTURA_COUNT` > 0 → skip fresh collection and analyze these paths.
- If `EXISTING_COBERTURA_COUNT` == 0 and all test projects are SDK-style → run
Phase 2 to generate fresh coverage.
the collection workflow in `test-execution.md`.
- If `EXISTING_COBERTURA_COUNT` == 0 and only classic/packages.config projects
exist → use a repository-owned coverage command that emits Cobertura;
otherwise stop with the message above.
- If `EXISTING_COBERTURA_COUNT` == 0 and both classic and SDK-style projects
exist → continue with Phase 2 for `SDK_TEST_PROJECTS` only and mark the result
partial until classic-project Cobertura is available.
exist → collect only for `SDK_TEST_PROJECTS` and mark the result partial until
classic-project Cobertura is available.
## Step 2c: Recommend ignoring `TestResults/`
@@ -1,6 +1,8 @@
# Coverage Analysis — test execution (Phase 2)
# Coverage Analysis — fresh test execution
Read this file **only** when Phase 1 found no existing Cobertura XML (`EXISTING_COBERTURA_COUNT:0`) and fresh coverage must be produced. When a report already exists, skip straight to Phase 3.
Read this file **only** when discovery found no existing Cobertura XML
(`EXISTING_COBERTURA_COUNT:0`) and fresh coverage must be produced. When a report
already exists, return to the existing-data path in `SKILL.md`.
This automatic provider workflow is for `SDK_TEST_PROJECTS` only. Exclude every
`CLASSIC_TEST_PROJECT` before provider detection, package addition, restore, or