Add experimental CRAP score skill for dotnet-test (#413)

2026-09-20 09:49:54 +08:00 · 2026-03-20 16:29:58 +01:00
parent 00c1e82006
commit 9adebf86a8
8 changed files with 564 additions and 0 deletions
@@ -0,0 +1,151 @@
+---
+name: exp-crap-score
+description: >
+  Calculates CRAP (Change Risk Anti-Patterns) score for .NET methods, classes,
+  or files. Use when the user asks to assess test quality, identify risky
+  untested code, compute CRAP scores, or evaluate whether complex methods have
+  sufficient test coverage. Requires code coverage data (Cobertura XML) and
+  cyclomatic complexity analysis.
+  DO NOT USE FOR: writing tests, general test execution unrelated to coverage/CRAP
+  analysis, or general code coverage reporting without CRAP context.
+---
+
+# CRAP Score Analysis
+
+Calculate CRAP (Change Risk Anti-Patterns) scores for .NET methods to identify code that is both complex and undertested.
+
+## Background
+
+The CRAP score combines **cyclomatic complexity** and **code coverage** into a single metric:
+
+$$\text{CRAP}(m) = \text{comp}(m)^2 \times (1 - \text{cov}(m))^3 + \text{comp}(m)$$
+
+Where:
+- $\text{comp}(m)$ = cyclomatic complexity of method $m$
+- $\text{cov}(m)$ = code coverage ratio (0.0 to 1.0) of method $m$
+
+| CRAP Score | Risk Level | Interpretation |
+|------------|------------|----------------|
+| < 5        | Low        | Simple and well-tested |
+| 5–15       | Moderate   | Acceptable for most code |
+| 15–30      | High       | Needs more tests or simplification |
+| > 30       | Critical   | Refactor and add coverage urgently |
+
+A method with 100% coverage has CRAP = complexity (the minimum). A method with 0% coverage has CRAP = complexity² + complexity.
+
+## When to Use
+
+- User wants to assess which methods are risky due to low coverage and high complexity
+- User asks for CRAP score of specific methods, classes, or files
+- User wants to prioritize which code to test next
+- User wants to evaluate test quality beyond simple coverage percentages
+
+## When Not to Use
+
+- User just wants to run tests (use `run-tests` skill)
+- User wants to write new tests (use `writing-mstest-tests` skill or general coding assistance)
+- User only wants a coverage percentage without complexity analysis
+
+## Inputs
+
+| Input | Required | Description |
+|-------|----------|-------------|
+| Target scope | Yes | Method name, class name, or file path to analyze |
+| Test project path | No | Path to the test project. Defaults to discovering test projects in the solution. |
+| Source project path | No | Path to the source project under analysis |
+
+## Workflow
+
+### Step 1: Collect code coverage data
+
+Check the test project's `.csproj` for the coverage package, then run the appropriate command:
+
+| Coverage Package | Command | Output Location |
+|---|---|---|
+| `coverlet.collector` | `dotnet test --collect:"XPlat Code Coverage" --results-directory ./TestResults` | Typically under `TestResults/<guid>/coverage.cobertura.xml`. Search recursively under the results directory (for example, `TestResults/**/coverage.cobertura.xml`) or use any explicit coverage path the user provides. |
+| `Microsoft.Testing.Extensions.CodeCoverage` (.NET 9) | `dotnet test -- --coverage --coverage-output-format cobertura --coverage-output ./TestResults` | `--coverage-output` path |
+| `Microsoft.Testing.Extensions.CodeCoverage` (.NET 10+) | `dotnet test --coverage --coverage-output-format cobertura --coverage-output ./TestResults` | `--coverage-output` path |
+
+### Step 2: Compute cyclomatic complexity
+
+Analyze the target source files to determine cyclomatic complexity per method. Count the following decision points (each adds 1 to the base complexity of 1):
+
+| Construct | Example |
+|-----------|---------|
+| `if` | `if (x > 0)` |
+| `else if` | `else if (y < 0)` |
+| `case` (each) | `case 1:` |
+| `for` | `for (int i = 0; ...)` |
+| `foreach` | `foreach (var item in list)` |
+| `while` | `while (running)` |
+| `do...while` | `do { } while (cond)` |
+| `catch` (each) | `catch (Exception ex)` |
+| `&&` | `if (a && b)` |
+| `\|\|` (OR) | `if (a \|\| b)` |
+| `??` | `value ?? fallback` |
+| `?.` | `obj?.Method()` |
+| `? :` (ternary) | `x > 0 ? a : b` |
+| Pattern match arm | `x is > 0 and < 10` |
+
+Base complexity is 1 for every method. Each decision point adds 1.
+
+When analyzing, read the source file and count these constructs per method. Report the breakdown.
+
+### Step 3: Extract per-method coverage from Cobertura XML
+
+Parse the Cobertura XML to find each method's `line-rate` attribute under the target `<class>` element. If `line-rate` is not available at method level, compute it from the `<lines>` elements:
+
+$$\text{cov}(m) = \frac{\text{lines with hits} > 0}{\text{total lines}}$$
+
+Method names in Cobertura may differ from source (async methods, lambdas). Match by line ranges when names don't align.
+
+### Step 4: Calculate CRAP scores
+
+For each method in scope, apply the formula:
+
+$$\text{CRAP}(m) = \text{comp}(m)^2 \times (1 - \text{cov}(m))^3 + \text{comp}(m)$$
+
+### Step 5: Present results
+
+Present a sorted table (highest CRAP first):
+
+```
+| Method                          | Complexity | Coverage | CRAP Score | Risk     |
+|---------------------------------|------------|----------|------------|----------|
+| OrderService.ProcessOrder       | 12         | 45%      | 28.4       | High     |
+| OrderService.ValidateItems      | 8          | 90%      | 8.1        | Moderate |
+| OrderService.CalculateTotal     | 3          | 100%     | 3.0        | Low      |
+```
+
+Include:
+- **Summary**: total methods analyzed, how many in each risk category
+- **Top offenders**: methods with CRAP > 30, with specific recommendations
+- **Quick wins**: methods with high complexity but where small coverage improvements would drop the score significantly
+
+### Step 6: Provide actionable recommendations
+
+For high-CRAP methods, suggest one or both:
+
+1. **Add tests** — identify uncovered branches and suggest specific test cases
+2. **Reduce complexity** — suggest extract-method refactoring for deeply nested logic
+
+Calculate the **coverage needed** to bring a method below a CRAP threshold of 15:
+
+$$\text{cov}_{\text{needed}} = 1 - \left(\frac{15 - \text{comp}}{\text{comp}^2}\right)^{1/3}$$
+
+This formula only applies when $\text{comp} < 15$. When $\text{comp} \geq 15$, the minimum possible CRAP score (at 100% coverage) is $\text{comp}$ itself, which already meets or exceeds the threshold. In that case, **coverage alone cannot bring the CRAP score below the threshold** — the method must be refactored to reduce its cyclomatic complexity first.
+
+Report this as: "To bring `ProcessOrder` (complexity 12) below CRAP 15, increase coverage from 45% to at least 72%." For methods where complexity alone exceeds the threshold, report: "`ComplexMethod` (complexity 18) cannot reach CRAP < 15 through testing alone — reduce complexity by extracting sub-methods."
+
+## Validation
+
+- Verify that coverage data was collected successfully (Cobertura XML exists and contains data)
+- Cross-check that method names in coverage data match the source code
+- Confirm CRAP scores by spot-checking the formula on one method manually
+- Ensure a 100%-covered method's CRAP equals its complexity exactly
+
+## Common Pitfalls
+
+- **Stale coverage data**: Always regenerate coverage before computing CRAP scores. Old coverage files will produce misleading results.
+- **Method name mismatches**: Cobertura XML may use mangled/compiler-generated names for async methods, lambdas, or local functions. Match by line ranges when names don't align.
+- **Generated code**: Exclude auto-generated files (e.g., `*.Designer.cs`, `*.g.cs`) from analysis unless explicitly requested.
@@ -0,0 +1,65 @@
+scenarios:
+  - name: "Calculate CRAP score for a single method with partial coverage"
+    prompt: "Calculate the CRAP score for the ProcessOrder method in OrderService.cs. I already have coverage data in TestResults/coverage.cobertura.xml."
+    setup:
+      files:
+        - path: "OrderService.cs"
+          source: "fixtures/partial-coverage/OrderService.cs"
+        - path: "TestResults/coverage.cobertura.xml"
+          source: "fixtures/partial-coverage/coverage.cobertura.xml"
+        - path: "OrderServiceTests.csproj"
+          source: "fixtures/partial-coverage/OrderServiceTests.csproj"
+    assertions:
+      - type: "output_matches"
+        pattern: "CRAP"
+      - type: "output_matches"
+        pattern: "complexity|cyclomatic"
+    rubric:
+      - "Reads the source file to compute cyclomatic complexity of ProcessOrder"
+      - "Parses the Cobertura XML to extract coverage data for the method"
+      - "Applies the CRAP formula correctly: comp^2 * (1 - cov)^3 + comp"
+      - "Identifies ProcessOrder as high risk due to moderate complexity and low coverage"
+      - "Provides a recommendation to add tests or reduce complexity"
+    timeout: 180
+
+  - name: "Identify riskiest methods across a file"
+    prompt: "Which methods in OrderService.cs are the riskiest? I want to know where to focus my testing effort. Coverage data is in TestResults/coverage.cobertura.xml."
+    setup:
+      files:
+        - path: "OrderService.cs"
+          source: "fixtures/partial-coverage/OrderService.cs"
+        - path: "TestResults/coverage.cobertura.xml"
+          source: "fixtures/partial-coverage/coverage.cobertura.xml"
+        - path: "OrderServiceTests.csproj"
+          source: "fixtures/partial-coverage/OrderServiceTests.csproj"
+    assertions:
+      - type: "output_matches"
+        pattern: "CRAP"
+    rubric:
+      - "Analyzes all methods in the file, not just one"
+      - "Produces a sorted table or ranking by CRAP score"
+      - "Correctly identifies the most complex and least-covered methods as highest risk"
+      - "Suggests specific actions for the top offenders"
+    timeout: 180
+
+  - name: "Generate coverage then compute CRAP score"
+    prompt: "I want to know the CRAP scores for my OrderService class. I haven't run coverage yet."
+    setup:
+      files:
+        - path: "OrderService.cs"
+          source: "fixtures/no-coverage/OrderService.cs"
+        - path: "OrderServiceTests.cs"
+          source: "fixtures/no-coverage/OrderServiceTests.cs"
+        - path: "TestProject.csproj"
+          source: "fixtures/no-coverage/TestProject.csproj"
+    assertions:
+      - type: "output_matches"
+        pattern: "dotnet test"
+      - type: "output_matches"
+        pattern: "coverage"
+    rubric:
+      - "Runs dotnet test with code coverage collection to generate Cobertura data before analysis"
+      - "Extracts per-method coverage from the generated Cobertura XML"
+      - "Computes CRAP scores using both cyclomatic complexity and code coverage for each method in OrderService"
+      - "Provides risk categorization and actionable recommendations for the riskiest methods"
+    timeout: 360
@@ -0,0 +1,80 @@
+namespace MyApp;
+
+public class OrderService
+{
+    private readonly IOrderRepository _repository;
+
+    public OrderService(IOrderRepository repository)
+    {
+        _repository = repository;
+    }
+
+    // Complexity: 7 (if, if, ||, foreach, if, if)
+    public OrderResult ProcessOrder(Order order)
+    {
+        if (order == null)
+            throw new ArgumentNullException(nameof(order));
+
+        if (order.Items == null || order.Items.Count == 0)
+            return new OrderResult { Success = false, Error = "No items" };
+
+        decimal total = 0;
+        foreach (var item in order.Items)
+        {
+            if (item.Quantity <= 0)
+                return new OrderResult { Success = false, Error = "Invalid quantity" };
+            total += item.Price * item.Quantity;
+        }
+
+        if (total <= 0)
+            return new OrderResult { Success = false, Error = "Invalid total" };
+
+        order.Total = total;
+        order.Status = OrderStatus.Confirmed;
+        _repository.Save(order);
+
+        return new OrderResult { Success = true, OrderId = order.Id, Total = total };
+    }
+
+    // Complexity: 2
+    public void CancelOrder(int orderId)
+    {
+        var order = _repository.GetById(orderId);
+        if (order != null)
+        {
+            order.Status = OrderStatus.Cancelled;
+            _repository.Save(order);
+        }
+    }
+}
+
+public class Order
+{
+    public int Id { get; set; }
+    public List<OrderItem> Items { get; set; } = new();
+    public decimal Total { get; set; }
+    public OrderStatus Status { get; set; }
+}
+
+public class OrderItem
+{
+    public string Name { get; set; } = "";
+    public decimal Price { get; set; }
+    public int Quantity { get; set; }
+}
+
+public class OrderResult
+{
+    public bool Success { get; set; }
+    public string? Error { get; set; }
+    public int OrderId { get; set; }
+    public decimal Total { get; set; }
+}
+
+public enum OrderStatus { Pending, Confirmed, Shipped, Cancelled }
+
+public interface IOrderRepository
+{
+    Order? GetById(int id);
+    void Save(Order order);
+}
@@ -0,0 +1,38 @@
+using Microsoft.VisualStudio.TestTools.UnitTesting;
+using MyApp;
+using NSubstitute;
+
+namespace MyApp.Tests;
+
+[TestClass]
+public sealed class OrderServiceTests
+{
+    [TestMethod]
+    public void ProcessOrder_WithValidItems_ReturnsSuccess()
+    {
+        var repo = Substitute.For<IOrderRepository>();
+        var service = new OrderService(repo);
+        var order = new Order
+        {
+            Id = 1,
+            Items = new List<OrderItem>
+            {
+                new() { Name = "Widget", Price = 10m, Quantity = 2 }
+            }
+        };
+
+        var result = service.ProcessOrder(order);
+
+        Assert.IsTrue(result.Success);
+        Assert.AreEqual(20m, result.Total);
+    }
+
+    [TestMethod]
+    public void ProcessOrder_WithNullOrder_ThrowsArgumentNull()
+    {
+        var repo = Substitute.For<IOrderRepository>();
+        var service = new OrderService(repo);
+
+        Assert.ThrowsException<ArgumentNullException>(() => service.ProcessOrder(null!));
+    }
+}
@@ -0,0 +1,13 @@
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <TargetFramework>net9.0</TargetFramework>
+    <Nullable>enable</Nullable>
+    <IsPackable>false</IsPackable>
+  </PropertyGroup>
+  <ItemGroup>
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
+    <PackageReference Include="MSTest" Version="3.8.2" />
+    <PackageReference Include="coverlet.collector" Version="6.0.4" />
+    <PackageReference Include="NSubstitute" Version="5.3.0" />
+  </ItemGroup>
+</Project>
@@ -0,0 +1,135 @@
+namespace MyApp;
+
+public class OrderService
+{
+    private readonly IOrderRepository _repository;
+    private readonly IPaymentGateway _paymentGateway;
+
+    public OrderService(IOrderRepository repository, IPaymentGateway paymentGateway)
+    {
+        _repository = repository;
+        _paymentGateway = paymentGateway;
+    }
+
+    // Complexity: 10 (if, if, ||, for, if, ||, ??, if, if)
+    public OrderResult ProcessOrder(Order order)
+    {
+        if (order == null)
+            throw new ArgumentNullException(nameof(order));
+
+        if (order.Items == null || order.Items.Count == 0)
+            return new OrderResult { Success = false, Error = "No items in order" };
+
+        decimal total = 0;
+        for (int i = 0; i < order.Items.Count; i++)
+        {
+            var item = order.Items[i];
+            if (item.Quantity <= 0 || item.Price < 0)
+                return new OrderResult { Success = false, Error = $"Invalid item at index {i}" };
+
+            var discount = item.Discount ?? 0m;
+            total += (item.Price - discount) * item.Quantity;
+        }
+
+        if (total <= 0)
+            return new OrderResult { Success = false, Error = "Order total must be positive" };
+
+        var payment = _paymentGateway.Charge(order.CustomerId, total);
+        if (!payment.Succeeded)
+            return new OrderResult { Success = false, Error = payment.ErrorMessage };
+
+        order.Total = total;
+        order.Status = OrderStatus.Confirmed;
+        _repository.Save(order);
+
+        return new OrderResult { Success = true, OrderId = order.Id, Total = total };
+    }
+
+    // Complexity: 6 (if, ||, foreach, if, &&)
+    public decimal CalculateTotal(List<OrderItem> items)
+    {
+        if (items == null || items.Count == 0)
+            return 0;
+
+        decimal total = 0;
+        foreach (var item in items)
+        {
+            if (item.Quantity > 0 && item.Price >= 0)
+                total += item.Price * item.Quantity;
+        }
+        return total;
+    }
+
+    // Complexity: 7 (if, 4 switch arms [_ default excluded], ?:)
+    public string GetOrderStatus(int orderId)
+    {
+        var order = _repository.GetById(orderId);
+        if (order == null)
+            return "Not Found";
+
+        return order.Status switch
+        {
+            OrderStatus.Pending => "Pending",
+            OrderStatus.Confirmed => "Confirmed",
+            OrderStatus.Shipped => order.TrackingNumber != null ? "Shipped (Tracked)" : "Shipped",
+            OrderStatus.Cancelled => "Cancelled",
+            _ => "Unknown"
+        };
+    }
+
+    // Complexity: 3 (if, &&)
+    public void CancelOrder(int orderId)
+    {
+        var order = _repository.GetById(orderId);
+        if (order != null && order.Status != OrderStatus.Shipped)
+        {
+            order.Status = OrderStatus.Cancelled;
+            _repository.Save(order);
+        }
+    }
+}
+
+public class Order
+{
+    public int Id { get; set; }
+    public string CustomerId { get; set; } = "";
+    public List<OrderItem> Items { get; set; } = new();
+    public decimal Total { get; set; }
+    public OrderStatus Status { get; set; }
+    public string? TrackingNumber { get; set; }
+}
+
+public class OrderItem
+{
+    public string Name { get; set; } = "";
+    public decimal Price { get; set; }
+    public int Quantity { get; set; }
+    public decimal? Discount { get; set; }
+}
+
+public class OrderResult
+{
+    public bool Success { get; set; }
+    public string? Error { get; set; }
+    public int OrderId { get; set; }
+    public decimal Total { get; set; }
+}
+
+public enum OrderStatus { Pending, Confirmed, Shipped, Cancelled }
+
+public interface IOrderRepository
+{
+    Order? GetById(int id);
+    void Save(Order order);
+}
+
+public interface IPaymentGateway
+{
+    PaymentResult Charge(string customerId, decimal amount);
+}
+
+public class PaymentResult
+{
+    public bool Succeeded { get; set; }
+    public string? ErrorMessage { get; set; }
+}
@@ -0,0 +1,12 @@
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <TargetFramework>net9.0</TargetFramework>
+    <Nullable>enable</Nullable>
+    <IsPackable>false</IsPackable>
+  </PropertyGroup>
+  <ItemGroup>
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
+    <PackageReference Include="MSTest" Version="3.8.2" />
+    <PackageReference Include="coverlet.collector" Version="6.0.4" />
+  </ItemGroup>
+</Project>
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="utf-8"?>
+<coverage line-rate="0.65" branch-rate="0.50" version="1.0" timestamp="1700000000">
+  <packages>
+    <package name="MyApp" line-rate="0.65" branch-rate="0.50" complexity="26">
+      <classes>
+        <class name="MyApp.OrderService" filename="OrderService.cs" line-rate="0.65" branch-rate="0.50" complexity="26">
+          <methods>
+            <method name="ProcessOrder" signature="(MyApp.Order)" line-rate="0.45" branch-rate="0.30" complexity="10">
+              <lines>
+                <line number="17" hits="5"/>
+                <line number="18" hits="5"/>
+                <line number="20" hits="5"/>
+                <line number="21" hits="2"/>
+                <line number="23" hits="3"/>
+                <line number="24" hits="3"/>
+                <line number="26" hits="3"/>
+                <line number="27" hits="0"/>
+                <line number="28" hits="0"/>
+                <line number="30" hits="3"/>
+                <line number="31" hits="3"/>
+                <line number="34" hits="0"/>
+                <line number="35" hits="0"/>
+                <line number="37" hits="3"/>
+                <line number="38" hits="0"/>
+                <line number="39" hits="0"/>
+                <line number="41" hits="3"/>
+                <line number="42" hits="3"/>
+                <line number="43" hits="3"/>
+                <line number="45" hits="3"/>
+              </lines>
+            </method>
+            <method name="CalculateTotal" signature="(System.Collections.Generic.List`1&lt;MyApp.OrderItem&gt;)" line-rate="1.0" branch-rate="1.0" complexity="6">
+              <lines>
+                <line number="51" hits="10"/>
+                <line number="52" hits="10"/>
+                <line number="54" hits="8"/>
+                <line number="55" hits="8"/>
+                <line number="57" hits="8"/>
+                <line number="58" hits="6"/>
+                <line number="60" hits="8"/>
+              </lines>
+            </method>
+            <method name="GetOrderStatus" signature="(System.Int32)" line-rate="0.60" branch-rate="0.40" complexity="7">
+              <lines>
+                <line number="65" hits="4"/>
+                <line number="66" hits="4"/>
+                <line number="67" hits="1"/>
+                <line number="69" hits="3"/>
+                <line number="72" hits="2"/>
+                <line number="73" hits="1"/>
+                <line number="74" hits="0"/>
+                <line number="75" hits="0"/>
+                <line number="76" hits="0"/>
+              </lines>
+            </method>
+            <method name="CancelOrder" signature="(System.Int32)" line-rate="0.80" branch-rate="0.50" complexity="3">
+              <lines>
+                <line number="82" hits="3"/>
+                <line number="83" hits="3"/>
+                <line number="85" hits="2"/>
+                <line number="86" hits="2"/>
+                <line number="87" hits="1"/>
+              </lines>
+            </method>
+          </methods>
+        </class>
+      </classes>
+    </package>
+  </packages>
+</coverage>