mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
9792d1808022673eff93f9dceb62c390afd5c129
357 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amaury Levé
|
9792d18080 |
Add markdownlint analysis for local dev and CI (#534)
* Add markdownlint analysis for local dev and CI - Add .markdownlint.jsonc with rule configuration - Add .markdownlint-cli2.jsonc with ignore patterns - Add package.json with markdownlint-cli2 and lint scripts - Add GitHub Actions workflow running on PRs touching *.md files - Add VS Code extension recommendation for vscode-markdownlint * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review comments: use markdownlint-cli2-action, merge configs, lint only changed files - Replace manual Node.js + npm setup with DavidAnson/markdownlint-cli2-action (pinned to SHA) - Use tj-actions/changed-files to lint only PR-changed markdown files - Merge .markdownlint.jsonc into .markdownlint-cli2.jsonc (single config file) - Remove package.json and package-lock.json (no longer needed) --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d7b80a806b |
Fix SDK 9.0.200 global.json pinning causing eval timeouts (#542)
Add rollForward: latestFeature to all global.json entries that pin SDK 9.0.200 so that any 9.0.xxx SDK satisfies the requirement. Without this, machines with only 9.0.3xx installed fail all dotnet commands during evaluation, causing the model to burn its entire timeout budget on failed bash retries. |
||
|
Amaury Levé
|
c18d4260de |
Fix timeout in MSTest.Sdk v3-to-v4 eval scenario 9 (#544)
The fixture csproj had MSTest.Sdk/3.8.0 while the prompt claimed the user already upgraded to 4.1.0. This mismatch caused the model to edit the csproj and run dotnet build, triggering slow MSTest SDK NuGet downloads that exceeded the 240s timeout on CI. - Update fixture csproj to MSTest.Sdk/4.1.0 to match the prompt - Add specific CS error codes to the prompt so the model provides direct fixes instead of building - Update rubric item to reflect the project already being on 4.1.0 |
||
|
Amaury Levé
|
7791735b60 |
Add test-quality-auditor hub agent for .NET test suite analysis (#541)
* Add test-quality-auditor hub agent for .NET test suite analysis Introduces a new test-quality-auditor agent that triages test quality concerns to specialized analysis skills: assertion quality, test smells, anti-patterns, mock usage, test gaps, maintainability, coverage risk, and test tagging. Includes a comprehensive audit pipeline that chains multiple skills for broad health checks. * Address Copilot review: clarify diagnostic role and coverage routing - Reword intro to acknowledge file-modifying workflows (test tagging) are only used with explicit user confirmation - Clarify coverage-analysis vs crap-score: prefer coverage-analysis for project-wide analysis, reserve crap-score for targeted method/class scope |
||
|
Amaury Levé
|
f9d7faf0ee |
Add test-migration hub agent for .NET test framework and platform migrations (#540)
* Add test-migration hub agent for .NET test framework and platform migrations Introduces a new test-migration agent that auto-detects the current test framework, version, and platform, then routes to the appropriate migration skill (MSTest v1/v2→v3, v3→v4, xUnit v2→v3, VSTest→MTP). Handles multi-step migrations with commit boundaries between steps. * Align framework detection with platform-detection skill Address Copilot review: MSTest requires metapackage, Sdk, or both TestFramework+TestAdapter (not either alone). Add missing xUnit MTP package variants. |
||
|
Jan Krivanek
|
05e2868b00 | Reintroduce copilot sessions storing (#530) | ||
|
Jan Krivanek
|
e294085cf0 | Automate GH AW updates (#536) | ||
|
Amaury Levé
|
9f20782332 | Add *.lscache to .gitignore and untrack cached files (#539) | ||
|
Amaury Levé
|
54cdb7ebca |
Add skill-coverage PR workflow with slash command support (#538)
* Add skill-coverage PR workflow with slash command support - Auto-triggers on PRs that change plugins/** or tests/** - Detects specific plugin/skill pairs from the PR diff - Runs Measure-SkillCoverage.ps1 and posts a coverage report as a PR comment - Supports /skill-coverage slash command on PR comments: /skill-coverage (runs on changed skills in the PR) /skill-coverage <plugin> (runs on all skills in the plugin) /skill-coverage <plugin> <skill> (runs on a specific skill) * Address Copilot review: security, permissions, and robustness fixes - Add permission check: require write access before running slash command - Tighten /skill-coverage matching to avoid false triggers on similar commands - Add issues: write permission (PR comments use Issues API) - Use refs/pull/<num>/head for checkout to support fork PRs - Remove unused \ variable - Fix multi-line JSON parsing with Out-String * Address PR #538 review: fork PR handling, reactions, and sourcing note - Skip PR comment posting for fork PRs (read-only token); coverage still runs in step summary - Add continue-on-error to reaction step to avoid 403 failures - Add header comment noting issue_comment trigger sources from default branch - Add continue-on-error to no-changes comment step * Split into analyze + comment jobs; fix pagination and dedup Security: - Split into two jobs: analyze (contents: read only) and comment (write only) - Analyze job checks out base branch for trusted tooling, PR head into a separate git worktree - untrusted PR code is never executed - Comment job only reads an uploaded artifact, no PR code checkout Fixes: - Use github.paginate() for listComments to handle PRs with >100 comments - Deduplicate targets: plugin-wide changes replace per-skill targets for that plugin instead of producing duplicates |
||
|
Jan Krivanek
|
6ebb5616b2 | Support checking skills and agents at single invocation (#533) | ||
|
Jan Krivanek
|
74cf19f605 | Update gh-aw (#532) | ||
|
Amaury Levé
|
dd37cd3bb6 |
Add skill coverage measurement tool (#509)
Introduce Measure-SkillCoverage.ps1 under eng/skill-coverage/ that measures how much of a SKILL.md's teaching content is verified by eval.yaml test scenarios — analogous to code coverage for skill files. The script extracts testable 'coverage points' from four structural sections of a skill file (Validation checklist, Common Pitfalls, Workflow Steps, and code patterns from fenced blocks), then cross-references them against eval.yaml assertions and rubric criteria using regex matching and keyword overlap heuristics. Features: - Per-skill and aggregate (-All) analysis across all plugins - Table output with colored pass/fail and per-category percentages - JSON output for machine consumption (single object or array) - Distinguishes deterministic assertions from rubric-only coverage - MinCoverage threshold for CI gating - Tracks source line numbers for all coverage point categories |
||
|
Jan Krivanek
|
9f2c53f68e |
Fix the DevOps health dashboard grooming (#529)
* Improve the groomer * Cleanup the daily health issue |
||
|
Artur Spychaj
|
3dbe832796 |
Add 'dotnet sln add' step to code-testing-implementer (#522)
* Add 'dotnet sln add' step to code-testing-implementer When a new test project is created, register it with the solution file using 'dotnet sln add' so that 'dotnet test <solution>' discovers the tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: improve dotnet sln add guidance - Use solution identified in research/plan rather than searching - Account for .slnf solution filter scenarios - Prefer 'dotnet test --solution' over 'dotnet test <solution>' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move dotnet sln add guidance to dotnet.md extension The implementer is polyglot, so 'dotnet sln add' doesn't belong there. Replace with a generic 'register with build system' step that defers to extensions/, and add the .NET-specific detail (sln/slnx/slnf handling) to extensions/dotnet.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: fix solution registration guidance in dotnet.md - Include .slnf in 'don't substitute' warning alongside .sln/.slnx - Qualify '--solution' flag as SDK 10+/MTP-only; fall back to positional form for older SDKs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
7079f6c3ae |
Bump the github-actions-dependencies group with 3 updates (#526)
Bumps the github-actions-dependencies group with 3 updates: [actions/github-script](https://github.com/actions/github-script), [actions/cache](https://github.com/actions/cache) and [actions/upload-artifact](https://github.com/actions/upload-artifact). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/ed597411d8f924073f98dfc5c65a23a2325f34cd...3a2844b7e9c422d3c10d287c895573f7108da1b3) Updates `actions/cache` from 5.0.4 to 5.0.5 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/668228422ae6a00e4ad889ee87cd7109ec5666a7...27d5ce7f107fe9357f9df03efb73ab90386fccae) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/bbbca2ddaa5d8feaa63e36b76fdaad77386f024f...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/cache dependency-version: 5.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
665062de79 |
Bump the microsoft-and-system group with 4 updates (#527)
Bumps Microsoft.Bcl.Memory from 10.0.5 to 10.0.6 Bumps Microsoft.Data.Sqlite from 10.0.5 to 10.0.6 Bumps Microsoft.Extensions.FileSystemGlobbing from 10.0.5 to 10.0.6 Bumps System.CommandLine from 2.0.5 to 2.0.6 --- updated-dependencies: - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jan Krivanek
|
e6f416848c | Fix the repo path traversal check (#525) | ||
|
Jan Krivanek
|
292a4bcfa3 |
Add --allow-repo-traversal and remove body line count checks (#524)
* Add validator options * Address PR review: validate options and add tests - Reject --allow-repo-traversal with --plugin mode (plugins must be portable) - Validate --max-declaration-lines and --max-agent-lines are positive integers - Add unit tests for MaxDeclarationLines override in SkillProfiler - Add unit tests for AllowRepoTraversal suppression and depth check preservation - Add unit tests for MaxAgentLines override and fallback to MaxDeclarationLines * Remove body line count checks and --max-declaration-lines/--max-agent-lines options Line count checks are redundant with BPE token-based complexity classification, which already provides better guidance. Keep --allow-repo-traversal only. * Address review: remove unused CheckOptions from AgentProfiler, fix traversal test - Remove unused CheckOptions parameter from AgentProfiler.AnalyzeAgent and RunAgentsCheckCore since line checks were removed - Fix AllowRepoTraversalSuppressesParentTraversalError test to use a non-deep path (../SKILL.md) and assert no file-ref errors at all |
||
|
Jan Krivanek
|
70e2d3b333 |
Simplify the focus of devops health workflows (#521)
* Simplify the focus of devops health workflows * Updated gh-aw and recompiled * Fix scope wording to include resource usage per review feedback |
||
|
Amaury Levé
|
7e6954f9a9 |
Add testability migration skills and agent for dotnet-test (#518)
* Add testability migration skills and agent for dotnet-test Introduces three skills and one orchestrator agent to help developers incrementally replace hard-to-test static dependencies with injectable abstractions: - detect-static-dependencies: scans C# code for DateTime.Now, File.*, Environment.*, HttpClient, Console.*, Process.* and ranks by frequency - generate-testability-wrappers: generates wrapper interfaces or guides adoption of built-in abstractions (TimeProvider, IHttpClientFactory, System.IO.Abstractions) - migrate-static-to-wrapper: mechanical codemod-style bulk replacement of static call sites with wrapper calls, scoped per project/namespace - testability-migration agent: orchestrates the Detect-Generate-Migrate pipeline end-to-end Includes eval.yaml tests and fixture projects for all three skills and the agent. * Fix activation, token overhead, and eval timeouts - Expand skill descriptions with more trigger phrases for migrate-static-to-wrapper and generate-testability-wrappers to fix NOT ACTIVATED issues - Collapse verbose replacement mapping tables and constructor examples to reduce token overhead (~30% fewer tokens) - Increase generate-testability-wrappers eval timeouts from 120s to 240s to prevent timeout-impacted scoring - Soften detect-static-dependencies scenario 2 rubric to reduce run-to-run variance in plugin mode - Increase migrate-static-to-wrapper non-activation timeout |
||
|
Jan Krivanek
|
5a6e8ba82a | Update GitHub.Copilot.SDK to version 0.2.2 (#520) | ||
|
Devin Breshears
|
08ff14b009 |
Add Reviewer Teams (#323)
* Add Reviewer Teams * Add the org * Keep original msbuild owners alongside reviewer team Agent-Logs-Url: https://github.com/dotnet/skills/sessions/0f052a87-406b-4ec3-aa67-57c2d11c64b4 Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> * Update .github/CODEOWNERS Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Copilot
|
7ab8006c18 |
Fix markdownlint violations in dotnet-test plugin (#513)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com> |
||
|
Milos Kotlar
|
038dd4f087 |
Add Apple crash symbolication skill (#201)
* Add SKILL.md for iOS crash symbolication process Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/ios-crash-symbolication/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add automation script, tests, and review fixes for ios-crash-symbolication - Add Symbolicate-Crash.ps1 (664 lines): parses .ips JSON, searches local dSYMs (SDK packs, NuGet cache, user paths), verifies UUIDs via dwarfdump, batch-symbolicates with atos, identifies runtime version - Update SKILL.md: add Automation Script and Runtime Version Identification sections, fix atos -o to point inside dSYM bundle, add dwarfdump and MAUI to frontmatter, wrap steps in Workflow heading for consistency with Android sibling, fix misleading 'rebuild' guidance per review feedback (Rolf) - Add eval.yaml with 7 test scenarios: Mono crash, CoreCLR crash, no .NET frames, NativeAOT, multi-library UUIDs, ASI field priority, and Android tombstone rejection - Add 5 .ips test fixture files (two-part JSON format) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add iOS crash log and evaluation scenarios for symbolication tests * Remove unused file * Add CODEOWNERS entry for iOS crash symbolication * Rename ios-crash-symbolication to apple-crash-symbolication Address PR feedback to support all Apple platforms (tvOS, Mac Catalyst, macOS) not just iOS: - Add $appleRids array covering ios, tvos, maccatalyst, and osx RIDs - Refactor Find-Dsym and Find-RuntimeVersion to search all platform packs - Rewrite SKILL.md for orchestration focus and reduced token budget - Extract domain knowledge to references/ips-crash-format.md - Tune eval.yaml: outcome-based rubrics, broad assertions, overfit 0.26 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix script robustness: base address parsing and null-safe property access - Simplify base address cast to [uint64] without manual hex prefix stripping - Use PSObject.Properties check before accessing lastExceptionBacktrace Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dSYM-path rubric items to CoreCLR and ASI scenarios Add rubric items testing for skill-specific domain knowledge (SDK pack paths, NuGet cache directories) that baseline agents cannot provide. This creates the quality delta needed to pass the 10% improvement threshold while keeping overfit at 0.12 (Low). 3-run validation: 30.1% improvement, 5/7 scenarios positive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor code structure for improved readability and maintainability * Fix crash symbolication script for real .ips format and improve eval scenarios - Fix Symbolicate-Crash.ps1 strict-mode bugs with real .ips crash logs: safe property access for image name/path (sentinel entries), thread name (most threads unnamed), and single-element array unwrapping (.Count) - Add SKILL.md efficiency guidance: resolve script path from skill directory (no find /), start with -ParseOnly, don't run broad filesystem searches - Reframe eval scenarios for platform-independent evaluation (parse/analyze instead of requiring macOS-only atos/dwarfdump), tighten rubric to test skill-specific knowledge (NuGet package name, all .NET binary images) Validation results (3 runs, claude-opus-4.6): Scenario 1 (parse frames): 3.3 → 4.0 (+0.7) ✅ Scenario 2 (investigate crash): 3.3 → 4.0 (+0.7) ✅ Scenario 3 (reject Android): 3.3 → 4.0 (+0.7) ✅ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Auto-fallback to parse-only output when atos is unavailable On Linux (CI), atos and xcrun don't exist. Previously the script would error out after completing all parsing, losing the results. Now it detects the missing tool and falls back to ParseOnly output automatically, ensuring the agent always gets structured parse data regardless of platform. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move apple-crash-symbolication to dotnet-diag plugin Relocate skill from plugins/dotnet to plugins/dotnet-diag and tests from tests/dotnet to tests/dotnet-diag. Update CODEOWNERS accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve apple-crash-symbolication: macOS symbols, bug fixes, training log - Fix JSON case-conflict parsing (vmRegionInfo/vmregioninfo duplicate keys) - Fix strict-mode safe access for optional asi field - Expand Step 4 with macOS-specific symbol package guidance (.symbols NuGet) - Add .dwarf to .dSYM conversion instructions - Add src/coreclr/ to validation paths - Soften stop signals to allow crash analysis and deeper investigation - Add macOS Symbol Packages and JSON Parsing Gotchas to reference doc - Create training log documenting session findings - Add .github/skills/ project-local copy for CLI testing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: automated version extraction and symbol acquisition Script improvements: - Preserve full image paths from crash log (previously discarded by GetFileName) - Add Get-RuntimeVersionFromPath: extracts .NET version from image paths (e.g., .../shared/Microsoft.NETCore.App/10.0.4/libcoreclr.dylib) - Add Get-RidFromPath: infers RID from path or crash metadata (OS/CPU) - Path-based version detection as fast primary method, UUID matching as fallback - Emit copy-pasteable symbol acquisition commands when dSYMs are missing - Show .NET version in ParseOnly library listing SKILL.md updates: - Step 2: document automated version detection and acquisition commands - Step 4: script now prints ready-to-run download/conversion commands Training log: record session 2 findings (5 issues, script + SKILL.md changes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add symbol server anti-pattern dotnet-symbol and msdl.microsoft.com do not serve macOS dSYM/DWARF symbols — only Windows PDBs and Linux ELF debug info. NuGet packages are the only public source. Added anti-pattern to SKILL.md (both copies) and reference doc to prevent wasted tool calls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: fix symbol server guidance — dotnet-symbol works for macOS dotnet-symbol --symbols <binary> successfully downloads .dwarf debug symbols for macOS Mach-O binaries from msdl.microsoft.com. Previous commit incorrectly claimed this didn't work. Replaced anti-pattern with positive guidance in SKILL.md (both copies) and reference doc. Added training log entry documenting the correction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Promote dotnet-symbol as preferred macOS symbol acquisition method - Script: macOS acquisition block now shows Option A (dotnet-symbol, preferred) and Option B (.symbols NuGet, fallback). Fixed .dwarf filename doubling bug in cp command. - SKILL.md (both copies): Step 2 updated for dotnet-symbol preference, Step 4 reordered with dotnet-symbol as #2 and NuGet symbols as #3. - Reference doc: Restructured macOS Symbol Packages section with Preferred/Fallback subsections and shared .dwarf→.dSYM conversion. - Training log: Session 4 entry documenting the promotion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add automated symbol server download for macOS crash symbolication - Add Get-DebugSymbols function: downloads .dwarf files from Microsoft symbol server using Mach-O UUID (mirrors android tombstone approach) - Add Convert-DwarfToDsym function: creates .dSYM bundle from .dwarf with UUID verification via dwarfdump - New params: -SymbolCacheDir, -SymbolServerUrl, -SkipSymbolDownload - Wire download+conversion into main flow after local dSYM search - Refactor manual acquisition guidance as fallback-only - Update both SKILL.md copies: frontmatter, Step 4, new flags - Update ips-crash-format.md: automated download as primary method - Add training log session 5 URL pattern: https://msdl.microsoft.com/download/symbols/_.dwarf/mach-uuid-sym-{UUID}/_.dwarf Verified end-to-end: 391/391 .NET frames symbolicated with clean cache. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add triage order guidance to Step 3 Step 3 now instructs the agent to explain the faulting mechanism (frames #0-#1) before examining cross-thread context. This addresses a misdiagnosis where GC activity on neighboring threads was mistaken for causation when the actual root cause (_sigtramp -> NULL signal handler) was visible in the crashing thread's first two frames. Training log updated with session retrospective and corrected the original crash description from GC race to NULL signal handler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove stale .github/skills/apple-crash-symbolication remnant Skill lives in plugins/dotnet-diag/skills/apple-crash-symbolication/ since the plugin restructuring. The old .github/skills/ copy was a leftover. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains.txt The reference scanner CI check fails because the apple-crash-symbolication skill references https://libimobiledevice.org/ which is not in the allowed domains list. This domain hosts the libimobiledevice project, a community library for communicating with iOS devices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Delete eng/reference-scanner/known-domains.txt * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains allowlist The apple-crash-symbolication SKILL.md references libimobiledevice.org for the idevicecrashreport tool. Add the domain to the known-domains file to fix the skill-check CI reference validation failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restructure apple-crash-symbolication for analysis-first approach The skill was getting ❌ verdicts on all eval scenarios because it directed the LLM to 'run the script' instead of teaching crash analysis domain knowledge. This mirrors the problem where quality decreased from 3.0 to 2.7 in the parse scenario. Restructure to follow the Android sibling's proven pattern: - Lead with parsing (.ips two-part JSON format, key fields) - Teach .NET library identification (inline library table) - Teach crash interpretation (asi, faulting thread, exception) - Teach atos command construction (concrete examples) - Teach dSYM search paths (ordered list with commands) - Move automation script to optional section at the end - Move crash log retrieval to a separate section (not Step 1) Trim the reference doc to avoid duplication, keeping only macOS symbol distribution differences and supported RID list. Token budget: ~2,100 tokens (within 800-2,500 optimal range). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refine prompts and assertions in eval.yaml for iOS crash symbolication scenarios * Add format verification guard and fix overfitting in reject scenario - SKILL.md: Add explicit format check at start of Step 1 to verify .ips JSON before proceeding; add 'Wrong file format' stop signal - eval.yaml: Remove direct skill reference from scenario 3 prompt to reduce overfitting (was triggering skill activation in plugin mode) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix scenario 3: provide actual Android tombstone file for rejection test - eval.yaml: Replace invalid 'extra_files' with correct 'files' syntax (extra_files was silently ignored, so crash_android.txt was never copied) - eval.yaml: Remove copy_test_files for scenario 3 to avoid copying ios_crash.ips which tempts the agent in plugin mode - Add tombstone_sample.txt to test directory (copy from android sibling) - SKILL.md: Mention ndk-stack/addr2line in wrong-format stop signal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/apple-crash-symbolication/tombstone_sample.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback: simulator RIDs, perf, safety, accuracy fixes - Add simulator RIDs (iossimulator, tvossimulator) to script search list - Build hashtable for O(1) image lookups in Get-ThreadFrames - Remove -UseBasicParsing (unnecessary in pwsh 7+) - Include UUID in Convert-DwarfToDsym cache path; sanitize library name - Make version regex greedy to capture pre-release suffixes - Improve format detection error message for non-.ips files - Guard xcrun fallback with Get-Command check - Add .dwarf-to-.dSYM conversion guidance for macOS manual fallback - Escape .ips in eval.yaml regex assertion - Fix UUID note in reference doc (normalize, not assume lowercase) - Fix symbol server example to use crash-log image name Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 2: case-sensitive replace, dedup, path fix, redact PII - Use -creplace with guard for vmregioninfo duplicate key handling - Use Sort-Object -Unique for proper library deduplication - Fix macOS fallback to create .dSYM bundles under symbols-out/ - Redact device identifiers in test fixture .ips file Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Prevent dSYM cache poisoning: validate UUID on cache hit, remove on mismatch - Convert-DwarfToDsym now verifies cached dSYM UUID before reusing - On UUID mismatch during download, remove the bad cached bundle so subsequent runs can retry cleanly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cache validation: normalize UUID comparison, clean up dwarf + bundle on mismatch - Use Format-Uuid to normalize dwarfdump output before comparing to already-normalized \ in Convert-DwarfToDsym cache check - On UUID mismatch after download, remove the entire .dSYM bundle directory and the cached .dwarf file to prevent repeat failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames null handling: use List[object] to preserve null entries PowerShell array += \ silently drops the element, breaking index alignment between results and input addresses. Switch to List[object].Add() which correctly preserves null entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dSYM bundle root traversal, qualify version-from-path note - Walk up directory tree until *.dSYM is found instead of going up only 2 levels (which lands at Contents/Resources, not the bundle) - SKILL.md: note that version-in-path only works on macOS shared framework installs; iOS paths don't embed the runtime version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove incorrect .github/skills path reference from training log The .github/skills/ directory doesn't exist in this repo. The file was already listed under its actual plugins/dotnet-diag/ path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Detect simulator for RID inference in manual fallback guidance Detect CoreSimulator in image paths to use iossimulator-/tvossimulator- RIDs for simulator crashes, avoiding UUID mismatches from wrong runtime packs. Also handle arm64e CPU type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames return type and scope issue in simulator detection - Return \.ToArray() instead of ,\ to avoid wrapping the list in a single-element array - Use \.usedImages instead of undefined \ in RID inference block Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify ParseOnly .NET Libraries section shows only frame-relevant images Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix YAML escape in assertion pattern (use single quotes for regex backslash) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten stop signal to prevent Android symbolication spillover The apple-crash-symbolication skill's stop signal for wrong file formats was naming specific Android tools (ndk-stack, addr2line) and the android-tombstone-symbolication skill. This caused models to learn about Android symbolication from the Apple skill and then actually execute it, resulting in 4-5x token bloat and completion regression on the Android rejection eval scenario. Remove tool suggestions from the stop signal — just say 'stop, don't symbolicate.' The Android skill handles its own routing when loaded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify scenario 3 rubric to focus on positive skill knowledge Replace 4 rubric items (including 2 'Did NOT' items) with 2 items that test positive knowledge the skill provides. The 'Did NOT' items gave credit to baseline responses that also don't attempt iOS workflow (because they don't know about it), minimizing the quality delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix nuget.org domain reference: drop www. prefix to match known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge upstream main and use api.nuget.org v3 endpoint Main changed known-domains.txt to use path-scoped entries (nuget.org/account/trustedpublishing instead of bare nuget.org). Switch the symbols download URL to the v3 flatcontainer endpoint on api.nuget.org which is in the allowlist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .gitignore merge conflict Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Steve Pfister <stpfiste@microsoft.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Jan Krivanek
|
a331d0994e | Fix the devops health workflows after gh-aw update (#511) | ||
|
Jan Krivanek
|
ba4ff9569c | Fix session visualisation filtering (#498) | ||
|
Amaury Levé
|
93dc33da86 |
Stabilize and unify some test skills (#501)
* Deduplicate test skill references and clarify skill boundaries - Move platform-detection.md and filter-syntax.md to plugins/dotnet-test/shared/, removing 3 identical copies of each from run-tests, mtp-hot-reload, and migrate-vstest-to-mtp reference directories. - Move dotnet.md from exp-test-smell-detection/extensions/ to shared/ as dotnet-test-frameworks.md in both dotnet-test and dotnet-experimental plugins. Update exp-assertion-quality, exp-test-boilerplate-detection, exp-test-tagging, and test-anti-patterns to reference the shared file instead of inlining framework detection tables. - Differentiate test-anti-patterns (quick pragmatic review) from exp-test-smell-detection (deep formal audit with academic taxonomy) by updating descriptions and cross-referencing each other in When Not to Use sections. - Update skill-validator to allow ../../shared/ file references while still blocking other parent-directory traversals. Add tests for the new rule. * Switch from shared/ directories to hidden reference skills Replace the plugin-level shared/ directories with non-invocable reference skills (user-invocable: false) that other skills reference by name. - Create platform-detection, filter-syntax, and dotnet-test-frameworks as hidden skills under plugins/dotnet-test/skills/. These contain the detection tables and syntax references previously duplicated across run-tests, mtp-hot-reload, and migrate-vstest-to-mtp. - Create exp-dotnet-test-frameworks as a hidden skill under plugins/dotnet-experimental/skills/ for the experimental test analysis skills (exp-test-smell-detection, exp-assertion-quality, etc.). - Update all consuming skills to reference these by skill name in backtick notation instead of file links. - Revert the skill-validator ../../shared/ exception — no longer needed since all references now use the standard skill name mechanism. * Merge exp-test-boilerplate-detection into exp-test-maintainability exp-test-maintainability was only 6 calibration rules with no workflow. exp-test-boilerplate-detection had the full 5-category detection workflow, examples, calibration, and validation. Merge the boilerplate content into exp-test-maintainability (the broader, more user-facing name) and add the two unique maintainability rules (DisplayName guidance, DataRow vs DynamicData preference) to Category 3. - Replace exp-test-maintainability SKILL.md with the merged content - Move test fixtures from exp-test-boilerplate-detection to exp-test-maintainability - Merge eval.yaml scenarios (4 total: 2 from each original skill) - Delete exp-test-boilerplate-detection skill and tests - Update all cross-references in exp-test-smell-detection, dotnet-test-frameworks, exp-dotnet-test-frameworks, and CODEOWNERS * Add cross-references to test-anti-patterns for deep mock and duplication analysis Point users to exp-mock-usage-analysis from the Over-mocking entry and to exp-test-maintainability from the Duplicate tests entry. * Add exp-dotnet-test-frameworks to CODEOWNERS * Improve run-tests SDK 10 MTP detection for blame-hang scenario Inline the critical SDK 10 detection signal (global.json test.runner) directly in Step 1 instead of deferring entirely to the platform-detection skill. This makes the distinction between SDK 10 (no -- separator) and SDK 8/9 (requires -- separator) more prominent. Add a quick detection summary table, strengthen the Common Pitfalls entry for SDK 10 with a blame-hang-timeout example, and keep the platform-detection skill reference for the full detection logic. * Improve skill activation keywords in descriptions - exp-test-maintainability: Add 'suggest a better test structure', 'consolidate similar test methods', 'convert copy-paste tests to data-driven parameterized tests' to match prompts like 'each new case needs a whole new method, suggest a better structure'. - test-anti-patterns: Add 'review tests', 'find test problems', 'check test quality', 'audit tests for common mistakes' to match review-style prompts that don't use the word 'anti-pattern'. - run-tests: Add 'hang timeout', 'blame-hang', 'blame-crash', 'TUnit' to match SDK 10 blame scenarios and TUnit filter scenarios that were intermittently not activating. |
||
|
dependabot[bot]
|
17f0323582 |
Bump the all-other-nuget group with 1 update (#504)
Bumps GitHub.Copilot.SDK from 0.2.0 to 0.2.1 --- updated-dependencies: - dependency-name: GitHub.Copilot.SDK dependency-version: 0.2.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-other-nuget - dependency-name: GitHub.Copilot.SDK dependency-version: 0.2.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-other-nuget ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
25e3b5fd0c |
Bump the github-actions-dependencies group with 2 updates (#503)
Bumps the github-actions-dependencies group with 2 updates: [actions/cache](https://github.com/actions/cache) and [actions/setup-node](https://github.com/actions/setup-node). Updates `actions/cache` from 4.2.3 to 5.0.4 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v4.2.3...668228422ae6a00e4ad889ee87cd7109ec5666a7) Updates `actions/setup-node` from 4.4.0 to 6.3.0 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/49933ea5288caeca8642d1e84afbd3f7d6820020...53b83947a5a98c8d113130e565377fae1a50d02f) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 5.0.4 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/setup-node dependency-version: 6.3.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Cathy Sullivan
|
42e29303b4 |
Add coverage-analysis skill to dotnet-test plugin (#499)
* Add coverage-analysis skill to dotnet-test plugin Adds an automated, project-wide code coverage and CRAP score analysis skill for .NET projects. Complements the existing crap-score skill (which is manual/targeted for single methods) with an end-to-end pipeline: - Auto-detects solution structure and test projects - Detects coverage provider (Microsoft.Testing.Extensions.CodeCoverage or Coverlet) per test project and selects the correct dotnet test arguments - Runs coverage collection, generates reports via ReportGenerator - Calculates CRAP scores per method via bundled PowerShell scripts - Surfaces risk hotspots ranked by danger (complex + uncovered code) - Produces actionable recommendations with projected score improvements Boundary with crap-score: coverage-analysis is project-wide and automated; crap-score is targeted and LLM-directed. The frontmatter descriptions cross-reference each other so the skill router disambiguates correctly. Includes eval.yaml with 3 scenarios (existing data analysis, from-scratch coverage collection, coverage plateau diagnosis) and shared buildable fixtures. Fixes #457 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR review: coverage output path and scoped tool install - Add --coverage-output $rawDir to all MS CodeCoverage dotnet test commands (unified, .NET 9, .NET 10+, and mixed-project modes) so Cobertura XML lands in the expected directory for downstream search - Replace dotnet tool install --global with --tool-path scoped to the coverage output directory to avoid mutating the user environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Cobertura XML fixture line numbers to match StudentService.cs FindById starts at line 101 (was 88), Search starts at line 109 (was 96). Updated both partial-coverage and plateau fixtures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wrap Get-ChildItem pipelines in @() for single-result array safety PowerShell returns a scalar FileInfo (not array) when a pipeline yields exactly one result, making .Count return null. Wrapping in @() ensures the result is always an array so .Count works reliably for the common single-test-project case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove CODEOWNERS change - not needed for this PR The existing /plugins/dotnet-test/ and /tests/dotnet-test/ wildcards already route reviews to @dotnet/dotnet-testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: .csproj fallback, cross-platform paths, .sln header - Add .csproj fallback discovery in Step 1 when no .sln found (emit ENTRY_TYPE:Project) to match documented behavior - Fix ProjectReference backslash to forward slash in test fixture .csproj - Remove leading blank line from ContosoUniversity.sln fixture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add plateau/stuck trigger language to frontmatter description The skill was NOT ACTIVATED for the 'Coverage plateau diagnosis' eval scenario because the frontmatter description (used by the skill router) lacked plateau-related keywords. The prompt 'coverage is stuck' and 'what's blocking me' didn't match the existing description text. Add 'stuck', 'plateaued', and 'blocking improvement' to the frontmatter Use-when sentence so the skill router activates for plateau diagnosis prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make CRAP scores mandatory, add existing-data shortcut path The plateau eval scenario scored 1/5 on CRAP scores across all modes because the LLM shortcut the workflow when existing Cobertura XML was provided — it read the XML directly and gave a quick diagnostic answer without computing CRAP scores or following the output template. - Add existing-data guidance at top of Workflow section: skip test execution but still run CRAP score computation (Steps 5-6) - Add mandatory CRAP rule to guidelines.md: Risk Hotspots table is required in every output regardless of analysis mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
3b16e4e4c8 |
docs: document token count reporting in skill-validator (#497)
* docs: document token count reporting in skill-validator Add a 'Getting the token count for a skill' section to the README explaining how to use 'check --verbose' to see BPE token counts and complexity tiers. Expand the Run metrics table in InvestigatingResults.md with the inputTokens, outputTokens, cache, and judge token fields available in results.json. Update the example analysis script to print input/output tokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: address review feedback on token count documentation - Fix tier table: replace 'Optimal sweet spot' with 'Recommended tier (sweet spot: 800–2,500)' to align with earlier statement - Fix tokenEstimate description to accurately reflect implementation behavior (cache fallback when inputTokens is zero) Agent-Logs-Url: https://github.com/dotnet/skills/sessions/dd5ded07-a968-4574-bfe9-1602b4434f07 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
070f8b4648 |
Add pats rotation to agentic workflows (#500)
* Copy PATs rotation from dotnet/core * Recompile agentic workflows with gh-aw v0.67.1 Remove unsupported 'failure' safe-output (dropped in gh-aw v0.67.1). Regenerate lock files with the new compiler version. No functional changes. * Add Copilot PAT pool rotation to agentic workflows Use the select-copilot-pat action to randomly select from a pool of COPILOT_GITHUB_TOKEN / COPILOT_GITHUB_TOKEN_2..8 secrets, distributing Copilot API load across multiple PATs. This is a stop-gap until organization-level billing is available for Agentic Workflows. See .github/actions/select-copilot-pat/README.md. |
||
|
Mukund Raghav Sharma (Moko)
|
665c39e957 |
Add configuring-opentelemetry-dotnet skill (#268)
* Add configuring-opentelemetry-dotnet skill (+8.1% eval, near-miss) Teaches OpenTelemetry SDK configuration in ASP.NET Core: package selection, AddOpenTelemetry with tracing/metrics/logging, custom ActivitySource and Meter creation, OTLP exporter setup, and context propagation. Eval results: +8.1% improvement (threshold: 10%, near-miss) Includes eval.yaml with OTel setup scenario + negative test. * Sharpen eval.yaml: add custom-spans-invisible pain-point and name-matching rubric emphasis * Migrate configuring-opentelemetry-dotnet to plugins/ directory structure * Move OpenTelemetry skill to new dotnet-aspnet plugin, address review feedback - Move skill from plugins/dotnet-diag to new plugins/dotnet-aspnet plugin - Create dotnet-aspnet plugin.json - Fix SKILL.md frontmatter: remove code fence wrapper so YAML metadata parses - Fix undefined 'order' variable in ProcessOrderAsync example - Move SqlClient/Runtime instrumentation to clearly marked optional sections - Add missing using directives (System.Diagnostics, OpenTelemetry.Context.Propagation) - Handle null Activity.Current in context propagation snippet - Use IMeterFactory via DI instead of static Meter per official guidance - Add OpenTelemetry.Instrumentation.Http to eval rubric package list - Simplify eval prompt to be more natural/generalized - Update CODEOWNERS: add dotnet-aspnet plugin entries, remove old dotnet-diag entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Configure metrics OTLP exporter endpoint explicitly to match tracing config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address round 2 review: logging endpoint, marketplace, RecordException using, ActivitySource in propagation, HttpClient pitfall - Configure logging OTLP exporter endpoint explicitly for consistency - Add dotnet-aspnet to both marketplace.json files - Add 'using OpenTelemetry.Trace' for RecordException extension method - Declare ActivitySource in context propagation snippet - Fix HttpClient instrumentation pitfall (works with new HttpClient() too) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify logging package source and add ActivitySource registration reminder Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add @dotnet/aspnet team to skill CODEOWNERS entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dotnet-aspnet to README table, make propagation ActivitySource static Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add expect_activation:false, IMeterFactory using, clarify OTLP exporter scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OtlpExporter using directive, clarify OTLP package covers logging Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Prometheus from OTLP claim, add OTLP exporter assertion to eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify Jaeger is traces-only, document logging package provenance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval: increase timeout, reject tools, rewrite prompt as explanation-only - Scenario 1: rewrite prompt to ask for code explanation (not file modification), add reject_tools to prevent bash/edit overhead, increase timeout 120s->360s, update rubric wording - Scenario 2: add reject_tools to eliminate tool-overhead noise Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve OTel skill evals and address review feedback - Remove redundant per-skill CODEOWNERS lines (ViktorHofer feedback) - Add SetResourceBuilder + ParseStateValues to SKILL.md Step 3 (logging provider gap) - Rewrite eval scenarios: 2 targeted positive scenarios testing real skill gaps - Remove noisy negative activation scenario (high variance, -21.5% drag) - Eval results: +14.2% improvement (threshold: 10%), overfitting 0.10 (low) Scenario 1: 3.0/5 -> 5.0/5, Scenario 2: 3.8/5 -> 5.0/5 * Add propagate-trace-context scenario to OTel evals (+25.6%) Third scenario tests trace context propagation across message queues, where the base model scores 3.2/5 and skill-enhanced scores 5.0/5. Removed metrics and debug scenarios that baseline already aced (5.0/5) which only added token cost without measurable improvement. Final: 3 scenarios, +25.6% improvement, overfitting 0.09 (low) * Remove duplicate dotnet-aspnet CODEOWNERS entries (L69-70 shadowed by L86-87) * Use @dotnet/aspnet team alias in CODEOWNERS for wider reach * Addressed feedback --------- Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Jan Krivanek
|
d861bbf62e |
Add AGENTVIZ session replay integration (#494)
* Add AGENTVIZ session replay integration - Add workflow_dispatch trigger to evaluation.yml - Add --keep-sessions to skill-validator evaluate step - Add publish-session-data job (mirrors publish-token-data) - Add replay link to PR comments (comment-on-pr) - Add AGENTVIZ SPA build/deploy to deploy-dashboard job - Add setup-node step to deploy-dashboard - Add per-plugin Sessions Visualisation links to dashboard.js - Create build-replay-sessions.ps1 (manifest generation from sessions.db) - Create purge-replay-sessions.ps1 (7-day retention management) * Address code review feedback on PR #494 - Use [IO.Path]::PathSeparator instead of hardcoded ';' for cross-platform PATH - Compute cutoff date in UTC for correct retention comparisons - Precompute ID HashSet before merge loop to avoid O(n^2) - Pin actions/setup-node to commit SHA (49933ea5...#v4) - Pin AGENTVIZ clone to commit SHA with verification - Skip npm ci + build when deployed commit SHA matches pinned SHA * Remove hardcoded AGENTVIZ SHA; use cache + zero-clone checks - Resolve AGENTVIZ target SHA via git ls-remote (no clone) - Read deployed SHA via curl from raw.githubusercontent.com (no clone) - Cache build output keyed by commit SHA (skip npm ci+build on hit) - Only clone AGENTVIZ repo on cache miss when SPA needs rebuild * Address round 2 code review feedback on PR #494 - Compare scheduled dir dates at day granularity (dirDate.Date vs cutoffDate.Date) - Fix useTempDir null-safe path comparison via GetFullPath + null guard - Use UTC for dateTag in build-replay-sessions.ps1 (consistent with purge) - Fix cache/deploy path mismatch: deploy from /tmp/agentviz-dist (cache path) - Deterministic clone: full clone + git checkout TARGET_SHA (fail on mismatch) - URL-encode manifest param in PR comment link (jq @uri) - Derive manifest generated timestamp from newest session mtime to avoid churn |
||
|
David Ortinau
|
c4b6819dda |
Add 7 core .NET MAUI development skills to dotnet-maui plugin (#456)
* Add 7 core .NET MAUI development skills to dotnet-maui plugin Add maui-shell-navigation, maui-collectionview, maui-data-binding, maui-dependency-injection, maui-app-lifecycle, maui-safe-area, and maui-theming skills with eval.yaml tests and reference docs. Update plugin.json, marketplace.json (both copies), and CODEOWNERS. Resolves #455 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cross-skill references to non-existent skills Replace 10 dangling references to skills not in this repo (maui-deep-linking, maui-platform-invoke, maui-unit-testing, maui-maps, maui-animations, maui-accessibility, maui-app-icons-splash, maui-bootstrap-theme, maui-localization) with plain-English guidance or references to skills that exist in this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix code correctness bugs and remaining cross-references Address PR review feedback and skeptic review findings: Code correctness: - Fix BackgroundColor="Primary" (XAML crash) → use {StaticResource Primary} - Fix MergedDictionaries.Clear() (nukes all styles) → remove-then-add pattern - Fix HorizontalStackLayout Fill (no effect) → use Grid with ColumnDefinitions - Fix AncestorType with ViewModel type → use ContentPage (visual element) - Fix Frame usage → replace with Border (recommended control) - Remove redundant MultiBinding StringFormat="{}{0}" - Fix OnStart reference → use CreateWindow for theme restore - Soften lifecycle OnStart/OnSleep characterization (compat shims, not absent) - Qualify ResourceDictionary code-behind requirement (only for x:Class) Cross-references: - Remove all remaining body-text refs to non-existent skills - Replace with Microsoft Learn doc links or generic guidance Eval: - Relax SafeAreaRegions assertion to accept SafeAreaEdges or SafeAreaRegions - Relax theming rubric for ResourceDictionary code-behind Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore Clear() theme pattern with clarifying comment The Clear()+Add() pattern is intentional — theme dictionaries are designed to be the only merged dictionaries in this architecture. Added comment explaining the assumption and what to do if non-theme dictionaries exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix second-pass review findings: API accuracy and eval robustness Code correctness: - ScrollView SafeAreaEdges="All" → "Container" (ScrollView only honors Container and None; All silently fails for keyboard avoidance) - DI CreateWindow example: add comment noting AppShell must be registered - AddHttpClient: add comment noting Microsoft.Extensions.Http NuGet required - MultiBinding: remove redundant StringFormat from reference doc (matches SKILL.md) - RelativeSource Self table: Path=Width → Path=WidthRequest (Width is read-only -1 before layout; WidthRequest is the correct bindable property) - CollectionView StackLayout warning: clarify it refers to BindableLayout on StackLayout vs CollectionView ItemsLayout types - SafeAreaEdges: fix "attached property" → "property" (it's a regular property) - ios:NavigationPage example: add xmlns comment for ios namespace Eval assertions: - DI Scoped regex: accept "doesn't"/"lacks"/"without" not just "no" - CollectionView ViewCell: match negation in either direction around ViewCell - Lifecycle events: require 3+ distinct events instead of any single one Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix final PR review comments - safe-area eval: tighten Container assertion to require SafeAreaEdges context - DI: merge #else fallback into single compilable #if block - lifecycle: behaviour → behavior (American English consistency) - theming: macOS Catalyst minimum 10.14 → 10.15 (Catalina required) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
948a9a52ef |
Add exp-mock-usage-analysis skill and evaluation (#483)
* Add exp-mock-usage-analysis skill and evaluation New experimental skill that detects unused, redundant, or unnecessary mocks in .NET test suites. Covers Moq, NSubstitute, FakeItEasy, and manual test doubles. - SKILL.md with 6-step workflow: gather code, classify mocks, detect anti-patterns, incorporate runtime data, generate recommendations, report findings - eval.yaml with 10 scenarios: trivial type mocking, unused setups, redundant configs, stable utility mocks, well-placed mocks (no false positives), NSubstitute support, FakeItEasy support, setup sprawl, and 2 non-activation boundary tests - Fixture files for each scenario with production + test code - CODEOWNERS entry for @dotnet/dotnet-testing * Remove no-headroom scenarios, increase NSubstitute timeout Remove 4 scenarios where baseline already scores 5.0/5 (model handles these well without skill guidance): - Detect mocking of DTOs, records, and enums - Recognize well-placed mocks without inventing false positives - Decline when asked about non-mock test anti-patterns - Decline mock framework migration request Increase NSubstitute scenario timeout from 120s to 180s to avoid baseline timeouts impacting scoring. Remaining 6 scenarios all show positive skill impact (3→5, 3→4, etc). * Document areas the model handles well without the skill Add a subsection under 'When Not to Use' listing mock-related tasks where the base model already scores 5.0/5 without skill guidance: - Mocking trivial types (DTOs, records, enums) - Recognizing well-placed mocks (no false positives) * Fix md * Improve |
||
|
Amaury Levé
|
d5ca4c6c01 |
Add exp-test-smell-detection skill and evaluation (#491)
* Add exp-test-smell-detection skill and evaluation - Language-agnostic core skill detecting 10 test smells (conditional logic, mystery guest, sleepy test, assertion-free, eager test, magic numbers, sensitive equality, exception handling, general fixture, ignored tests) - .NET extension with framework-specific detection tables (MSTest/xUnit/NUnit/TUnit) - Reference catalog with 19 smells from testsmells.org research - Evaluation with 4 scenarios: smelly suite, clean suite, integration tests, non-activation - CODEOWNERS entry for @dotnet/dotnet-testing * Improve integration test fixture and non-activation timeout - Add real smells to integration test fixture (Thread.Sleep, conditional logic, assertion-free test) mixed with legitimate integration patterns - Update rubric to test nuanced distinction between real smells and acceptable integration test patterns - Increase non-activation scenario timeout from 120s to 300s to prevent timeout-related scoring penalties * Fix test projects * Add domains for testsmells * Remove extra domains |
||
|
Dan Moseley
|
165e6439fe |
Tighten known-domains allowlist (#496)
* Tighten known-domains allowlist Narrow broad domain entries to specific paths/subdomains actually used: - aka.ms -> aka.ms/dotnet-trace, aka.ms/perfcollect - nuget.org -> api.nuget.org, www.nuget.org/account/trustedpublishing - npmjs.com/package/@modelcontextprotocol -> pinned to inspector/v/0.21.1 Remove unused domains: fluentassertions.com, speedscope.app Update skill content to match: pin MCP Inspector link, remove FluentAssertions link, remove speedscope hyperlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * updated mcp skills codeowner * Refine scoped known-domain entries Remove the remaining nugettest.org publish example and drop the nugettest.org allowlist entry. Also normalize path-scoped entries to omit a hardcoded www prefix so both host forms match the existing ReferenceScanner logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Leslie Richardson <lerich@microsoft.com> |
||
|
Amaury Levé
|
eb0f98eba6 |
Add exp-test-gap-analysis skill for pseudo-mutation test gap detection (#490)
New experimental skill that performs pseudo-mutation analysis on .NET production code to find gaps in existing test suites. Covers boundary, boolean/logic, return value, exception removal, arithmetic, and null-check mutation categories. Includes eval.yaml with 4 scenarios (boundary gaps, logic gaps, well-tested code recognition, non-activation) and fixture projects. |
||
|
Youssef Victor
|
ae9e2b4302 |
Add SKILL for xunit.v3 migration (#484)
* Add SKILL for xunit.v3 migration * eval.yaml, to be reviewed * Add run_command_and_assert to eval.yaml for dotnet test validation Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8e1539d5-327e-4216-9a01-5aa062e53608 Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix eval * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix eval.yml * Update eval * Update skill * Increase timeout * Update eval.yaml * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/dotnet-test/migrate-xunit-to-xunit-v3/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
59d4485e14 |
Improve dotnet-template-engine skills and eval suite, add template-validation skill (#480)
* Improve dotnet-template-engine eval quality and plugin activation - Replace useless 'Create project with specific framework' test (vanilla=10/10 in all 30 runs, zero skill delta) with harder scenarios that vanilla cannot trivially solve - Expand eval suite from 5 to 10 scenarios to reduce overfitting signal and improve coverage of untested capabilities (dry-run, search, install, template creation from existing project, ambiguous intent resolution) - Add file_not_exists assertion to dry-run scenario to verify projects are not actually created during preview - Add output_matches assertions with specific patterns (e.g. 'individual', 'blazor') instead of relying solely on output_contains - Increase all timeouts from 120s to 180s to reduce MCP cold-start timeouts - Enrich plugin.json description with activation keywords (console app, class library, web API, Blazor, MAUI, framework versions) to fix Plugin mode activation failures (19/21 not-activated events were from one scenario) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add template-validation skill with eval scenarios New skill encoding template.json validation rules: required fields, identity format, shortName conflicts with CLI commands, symbol type/datatype validation, choice parameter default checking, computed/generated symbol completeness, parameter prefix collisions, source condition syntax, post-action requirements, constraint configuration, and tag recommendations. Two eval scenarios: - Validate a template with 8+ deliberate errors (missing identity, CLI- conflicting shortName, invalid defaults, incomplete symbols, missing actionId) - Validate a correct template and confirm it's ready to publish Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - Add ./test-output to dry-run prompt so file_not_exists assertion checks the correct location - Add missing expect_tools: skill to template-authoring validate scenario - Add 'using Xunit;' to seeded test file so the setup project is compilable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten assertions and replace low-delta scenario - Replace low-delta 'worker service search' (vanilla solves trivially) with 'create multi-project solution' (two projects + project reference — vanilla struggles with multi-step orchestration) - Fix gRPC search assertion: replace trivial output_contains 'template' with specific pattern matching gRPC service/package names - Fix validation restore assertion: require 'shortName' context near 'restore' to prevent false pass from post-action description - Fix correct-template assertion: replace fragile output_not_contains with positive assertion that agent mentioned specific fields it checked - Fix create-from-existing assertion: replace pass-anything pattern with assertion requiring template.json / sourceName / .template.config mention Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 2) - Change datatype 'integer' to 'int' in eval fixture and SKILL.md to align with actual template engine schema - Tighten correct-template assertion: require unambiguous success signal (e.g. 'no errors', 'template is valid') instead of permissive pattern that matches 'not ready' or 'pass' in other contexts - Replace output_not_contains 'missing required' (fails on 'no missing required fields') with positive assertion checking field names - Remove int/integer dual listing from valid datatypes in SKILL.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval failures from CI run 4 scenarios failed; fixes: - Web API + AOT: increase timeout 180s -> 300s (auth scaffolding + NuGet restore needs more time) - Dry-run preview: rephrase prompt to explicitly mention 'template' for better skill activation matching - Multi-project solution: replace with simpler 'create xUnit test project with net8.0' — the multi-project scenario gave vanilla a HIGHER score (4.7) than skilled (4.3) because the skill added overhead without helping - Create-from-existing: increase timeout 180s -> 300s (template generation burned 256K tokens and timed out) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dishonest eval adjustments - Revert dry-run prompt to natural language — don't coach activation by injecting 'template' keyword. If skill doesn't activate, that's a real signal about activation quality. - Replace recycled 'xUnit + net8.0' test (same ceiling problem as the classlib + net9.0 test we removed) with 'Blazor + interactive rendering' which tests parameter knowledge vanilla is less likely to have (render mode selection, blazor vs blazorserver vs blazorwasm template choice). - Keep timeout increases (300s for web API + auth, 300s for create-from- existing) as these address real infrastructure constraints. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move dry-run scenario from template-instantiation to template-discovery The dry-run scenario failed with 'NOT ACTIVATED' because the platform couldn't match a preview/dry-run prompt to the template-instantiation skill. This is correct behavior — previewing what files a template produces is a discovery operation, not instantiation. The template-discovery SKILL.md explicitly covers dry-run in Step 4: 'Use dotnet new <template> --dry-run to show what files a template would create without writing anything to disk.' This is an objective reclassification, not prompt tuning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill content causing eval failures template-instantiation: Add parameter reference table with common parameter combinations (auth values, interactivity modes, AOT flag) for webapi, blazor, grpc, worker templates. The web API + AOT scenario scored 2.0 for both vanilla and skilled because the skill mentioned --auth and --aot but never explained what values they accept. Now the agent gets actionable knowledge. template-authoring: Trim verbose content to reduce token overhead. The create-from-existing scenario burned 256K tokens (vs 185K vanilla) and timed out because the agent read packaging/distribution steps it didn't need. Condensed Step 1 example, removed Step 5 (packaging), compressed Step 4 (testing). Reduces skill size by ~30%. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 3) - Web API+AOT: replace output_matches for AOT with file_contains checking for PublishAot in the generated .csproj — proves the parameter was actually passed, not just mentioned in text - Validation multi-error: add assertions for bool default (enableLogging/ yes), int default (maxRetries/abc), computed missing value, generated missing generator — now enforces 7 of 8 rubric items via assertions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 4) - Fix YAML double-quote escaping for regex word boundaries - Fix worker template example to include --aot flag - Fix webapi example to show --auth Individual --aot together - Fix grammar: Available Symbol Generators Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove web API+AOT and Blazor eval scenarios Web API+AOT consistently degraded quality (run 1: 2.0->2.0, run 2: 3.0->2.7). The skill adds reading overhead on execution tasks without improving the result. Blazor hit ceiling (5.0->5.0) — vanilla already creates Blazor apps perfectly, so the scenario contributes zero signal. template-instantiation now has 1 honest scenario (console app, +1.0 delta). The skill's value is modest for execution tasks; the PR's strength is in discovery (+2.3 to +3.7), validation (+1.7 to +2.0), and authoring (+2.0) where knowledge matters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 5) - Tighten bool/int validation assertions to require explicit error wording (invalid, not valid, must be) alongside field/value — prevents passing when agent just echoes the JSON without diagnosing the issue - Add actionId assertion to enforce detection of missing postAction actionId (was in rubric but not in assertions) - Use single-quoted YAML for patterns with backslash sequences Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
0417c8ec62 |
Bump actions/cache in the github-actions-dependencies group (#492)
Bumps the github-actions-dependencies group with 1 update: [actions/cache](https://github.com/actions/cache). Updates `actions/cache` from 4.2.3 to 5.0.4 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/5a3ec84eff668545956fd18022155c47e93e2684...668228422ae6a00e4ad889ee87cd7109ec5666a7) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 5.0.4 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jan Krivanek
|
498755bc0d |
Make the infra eval gate faster (#489)
* Make the infra eval gate faster * Address PR feedback: handle infra+skill overlap, detect evaluation.yml changes - When both infra and skill changes exist in a PR, evaluate the changed skills normally instead of the random subset (fixes Copilot review). - Expand infra detection to include .github/workflows/evaluation.yml changes in all three detection sites (pr-status, fork-pr-status, discover). |
||
|
Youssef Victor
|
b8e3092b37 |
Add missing interpolations in Reporter.cs (#485)
* Add missing interpolations in Reporter.cs * Update Reporter.cs * Update eng/skill-validator/src/Evaluate/Reporter.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Use Ansi constants directly instead of interpolated strings in FormatMetricWithDelta and FormatPctDelta Agent-Logs-Url: https://github.com/dotnet/skills/sessions/aca02d03-103f-47d0-b76f-199b02c45e37 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Update eng/skill-validator/src/Evaluate/Reporter.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Refactor color assignment for score display * Update eng/skill-validator/src/Evaluate/Reporter.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix non-interpolated pTimeout strings in ReportScenarioDetail Agent-Logs-Url: https://github.com/dotnet/skills/sessions/801b8d2e-ee34-42f0-8210-f0f3d76839cd Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Update eng/skill-validator/src/Evaluate/Reporter.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
167b7c3772 |
Fix evaluation cache key: compute explicitly instead of cache-primary-key (#488)
The pinned actions/cache SHA (5a3ec84, v4) does not expose the cache-primary-key output. This caused the evaluate job to fail with 'Input required and not supplied: key' because the build-validator job output was always empty. Fix: compute the cache key in a dedicated step and pass it as a job output, instead of relying on an output that doesn't exist at this action version. Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
187c2e7c31 |
Refactor plugin creation instructions in CONTRIBUTING.md (#481)
Removed redundant instructions for creating a new plugin from the 'Plugin organization' section. |
||
|
Viktor Hofer
|
3fff1f2573 |
Skill-validator: perf, cancellation, and reliability improvements (#482)
* Skill-validator: perf, cancellation, and reliability improvements Performance: - Cache skill-validator-dist.tar.gz in CI (evaluation.yml and skill-check.yml) to skip .NET SDK setup, build, and tar on cache hit. Evaluate and comment-on-pr jobs restore from cache with artifact download fallback. - Replace SemaphoreSlim with Lock in SessionDatabase for lighter-weight synchronous locking. Cancellation: - Thread CancellationToken from System.CommandLine entry point through the entire evaluate call chain: Run -> EvaluateTarget -> EvaluateSkill/EvaluateAgent -> ExecuteScenario/ExecuteAgentScenario -> ExecuteRun/ExecuteAgentRun, plus EvaluateSkillNoise -> ExecuteNoiseTest. - Add CancellationToken parameter to AgentRunner.RunAgent, Judge.JudgeRun, PairwiseJudge.Judge, and OverfittingJudge.Analyze, forwarding to RetryHelper and LlmSession.SendAsync respectively. - Pass CancellationToken to all ConcurrencyLimiter.RunAsync call sites so semaphore waits are cancellable. - Re-throw OperationCanceledException at the top-level skill evaluation loop so Ctrl+C / CI cancellation propagates instead of being swallowed as a skill failure. Reliability: - Smart timeline truncation in Judge and PairwiseJudge: error events (session.error, runner.error) are always preserved during truncation regardless of position, with a summary marker showing event type breakdown. - Add jitter (±25%) to RetryHelper exponential backoff to prevent thundering herd on rate-limit recovery. - Add CancellationToken parameter to ConcurrencyLimiter.RunAsync<T>. - Add ConcurrencyLimiter tests (6 tests) and RetryHelper jitter tests (2 tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback - RetryHelper: re-clamp delay after jitter so it never exceeds the remaining budget or MaxSingleDelayMs. - Judge/PairwiseJudge: handle edge case where error count alone exceeds MaxTimelineEvents by trimming errors with a summary instead of unconditionally appending all of them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add high-variance detection and refactor FailureKind to enum - Add Statistics.CoefficientOfVariation() for measuring score consistency - Flag scenarios with CV > 0.5 as high-variance in ScenarioComparison - Show console warning and markdown footnote for high-variance scenarios - Include high-variance scenario names in Comparator verdict reason - Refactor FailureKind from magic strings to strongly-typed enum with JsonStringEnumMemberName attributes for backwards-compatible JSON output - Add 6 new CV unit tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Mukund Raghav Sharma (Moko)
|
c92faa7800 |
Add minimal-api-file-upload skill (#264)
* Move minimal-api-file-upload skill to aspnetcore plugin with 3 eval scenarios Per repo restructuring feedback, ASP.NET Core specific skills should be under the aspnetcore plugin rather than the dotnet plugin. * Increase eval timeouts to 180s to avoid timeout-related scoring penalties * Generalized and addressed feedback * Improve eval scenarios: remove noisy tests, add dual-limit and security review scenarios * Add CODEOWNERS for aspnetcore plugin * Fix eval: enrich prompts for skill activation, bump security review timeout to 240s * Drop size-limit scenario (baseline already 5.0), keep 3 high-signal scenarios * Reframe security review as file upload fix to improve skill activation * Review * Contribution guide followed * Updated name * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-aspnet/skills/minimal-api-file-upload/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR #264 review feedback from Brennan and Copilot - Rename plugin from aspnetcore to dotnet-aspnet per repo conventions - Fix streaming filename: remove user-controlled extension, use bare GUID - Remove explicit sanitization code (Option B) per Brennan's feedback - Add case-insensitive MIME type comparison - Validate ContentType matches magic bytes (reject mismatches) - Fix stream rewind + CopyToAsync: copy from already-opened stream - Reword streaming section to accurately describe buffering vs chunks - Update marketplace.json, README.md, CODEOWNERS for dotnet-aspnet rename --------- Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
db83be92f7 | Ensure create-skill mentions codeowners (#478) | ||
|
Jan Krivanek
|
e4670b33a1 |
[PoC] Code testing agent + tests PoC (#433)
* Add code testing agent agents + tests PoC * Remove AssertionEvaluator.cs change (moved to dev/jankrivanek/agents-evals) |