mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
YuliiaKovalova
b39a041ed0
* Strengthen dotnet-template-engine skills per cross-family eval (#898) Cross-family skill eval (issue #898) flagged 5 of 6 skills as STRENGTHEN ("mostly ties - too generic/non-prescriptive") and template-instantiation as KEEP-POLISH. Invocation is already ~100% and token overhead is a penalty, so these edits add decisive, opinionated OUTPUT requirements (not volume). All skills remain in the recommended "detailed" tier. - template-validation: require a one-line verdict header + single findings table, and a concrete fix for every ERROR/WARNING. - template-smart-defaults: require a "Defaults applied" log + the exact single dotnet new command line (only flags actually passed). - template-comparison: require a decisive Recommendation verdict and add opinionated decision shortcuts for canonical template pairs. - template-discovery: lead the answer with a ready-to-run command line. - template-authoring: require a complete template.json + a conventions-preserved confirmation table. - template-instantiation: add a lean "match the workspace, then stop" directive to cut the excess turns/tokens flagged for the Opus gap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: remove template-specific AOT claims, narrow authoring validation routing - template-smart-defaults: reword the AOT note, rules row, and pitfall so they no longer claim --aot is unavailable on webapi/webapp (dotnet new webapi --aot exists since .NET 8 and template-instantiation references it). Now say --aot is present only where dotnet new <template> --help lists it, without naming specific templates as unsupported. - template-authoring: narrow the frontmatter DO NOT USE to standalone validation (use template-validation) and align the When-to-Use bullet, so it no longer contradicts the body, which validates as a step during authoring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: keep Defaults-applied Source strictly user|rule The sample row used 'rule (AOT)', which made the Source column look free-form. Set it to 'rule' and moved the AOT context into the Why column, matching the guidance that Source is strictly user or rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: tighten discovery description, make comparison AOT row conditional - template-discovery: simplify the redundant 'two or more templates against each other' phrasing to 'comparing templates side by side' and rewrap so each '(use <skill>)' stays with its clause. - template-comparison: reword the example AOT row so it is conditional on what 'dotnet new <template> --help' exposes instead of asserting webapi has no --aot (which contradicted template-instantiation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: align find/compare routing between frontmatter and body - template-discovery: narrow the frontmatter exclusion to a *detailed* side-by-side comparison (use template-comparison), matching the body which supports light comparison but routes detailed comparisons away. - template-instantiation: split the frontmatter 'finding or comparing' exclusion into finding (template-discovery) vs detailed comparison (template-comparison), and align the When-Not-to-Use bullet accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: mention validation in template-authoring description The description said only 'Guides creation' while the skill validates the template it authors (intro + Step 2). Restored 'creation and validation' and added a USE FOR entry for validating the template.json you are authoring, while keeping standalone validation routed to template-validation in DO NOT USE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: mention high-level compare in discovery; clarify defaults log scope - template-discovery: description now says 'find, inspect, and compare (at a high level)' so it isn't under-scoped vs the body (detailed side-by-side still routes to template-comparison via DO NOT USE). - template-smart-defaults: clarify the 'Defaults applied' log has one row per parameter covering both preserved user values (Source=user) and rule-applied defaults (Source=rule). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: symmetric AOT cell + clarify clean verdict - template-comparison: make the webapp AOT cell reference 'dotnet new webapp --help' so it is symmetric with the webapi cell and unambiguous. - template-validation: the 'Ready to publish' verdict now says '0 errors, 0 warnings' and notes optional suggestions may still apply, instead of 'clean'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: allow line:col location for JSON parse errors Widened the findings-table Location column to 'JSON path or line:col' and added an example parse-error row, so the required table can represent the JSON parse errors that Step 2 asks to report with a line number. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Improve template-discovery resilience: answer before any dotnet new call Eval investigation (PR #904 results, run 29503063336) showed template-discovery scoring 1.0-2.0 on Opus because sessions died with '(no output)' after a 'persistence'/'mutex' error from concurrent dotnet new calls. When the session survived, the same content scored 4.0-4.3. The failing runs invoked the skill then ran 'dotnet new --help' as the first/last action and the mutex error left the turn empty. Make answer-first an imperative rule: write the recommendation and ready-to-run command from the intent mapping BEFORE running any dotnet new command, and never end the turn on a dotnet new call. This makes the answer survive the mutex error (and gives users a faster answer). Skill stays in the 'detailed' token tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Iterate on eval quality: shadow-verb shortName warning + discovery resilience From PR #904 eval run 29509140219: - template-validation & template-authoring: add a WARNING tier for shortNames that *shadow* a well-known top-level dotnet verb (build/run/test/publish/pack/clean/ restore). These do NOT technically conflict with dotnet new (correctness kept), but 'dotnet new build' is easily confused with 'dotnet build', so it warrants a warning + a distinctive-name suggestion. This is a genuine authoring footgun and also resolves the 'Validate a template.json file' scenario where the agent was penalized for correctly saying shortName 'build' is fine. - template-discovery: require the pre-CLI answer to be complete and standalone (never end on a 'let me confirm from the CLI…' teaser, which the mutex error was truncating), and add a dry-run-failure fallback (describe expected files from template knowledge) so preview requests never return empty. Skill stays 'detailed'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Rubber-duck follow-up: revert eval-gaming, fix real defects Independent review + empirical checks (dotnet new build/restore/list --help; EvaluateCommand.cs Task.WhenAll) confirmed: - The mutex '(no output)' deaths come from the validator running Baseline+Isolated +Plugin concurrently, not cross-skill parallelism — eval.yaml can't fix it. - build/restore/test do NOT conflict with dotnet new (only real subcommands like 'list' do), so the shortName-shadow WARNING added last commit was appeasing a factually-wrong rubric (and the 'restore' entry contradicted the validation eval). Changes: - Revert the shortName-shadow WARNING in template-validation and template-authoring (the skills were already correct; do not game an incorrect rubric). - template-discovery: fix a real mapping bug — 'windows auth' mapped to SingleOrg; it is --auth Windows. Split azure ad/entra (SingleOrg) from windows auth (Windows). - template-discovery: tighten the dry-run-failure fallback to give only a representative structure and forbid inventing specific parameter values/choices/ file paths (avoids confident hallucination), keeping the 'answer-first, never end on a teaser' resilience. Skill stays in the detailed token tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: fix validation findings-table example (real subcommand + line:col) - Use a real dotnet new subcommand ('list') in the ERROR example instead of 'restore' (which does not conflict), matching the corrected shortName rules. - Make the parse-error location example '12:5' match the 'line:col' column header. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: consistency fixes across three skills - template-authoring: add ExcludeAssets to the conventions-preserved table (the workflow above lists it, so include it to avoid implying partial carry-over). - template-discovery: add SingleOrg to the example --auth choices so options aren't under-reported. - template-validation: broaden the concrete-fix requirement to allow a specific edit instruction (e.g. 'remove the trailing comma'), matching the parse-error example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Fix incorrect eval rubrics: shortName 'build'/'restore' are not dotnet new conflicts The 'Validate a template.json' (authoring) and 'Validate template with multiple errors' (validation) scenarios used shortName 'build'/'restore' and asserted they conflict with dotnet new. Verified empirically they do NOT (dotnet new build/ restore --help treat them as template-name lookups; only real subcommands like 'list'/'search' intercept). The skills correctly report this, so the wrong rubrics were penalizing correct behavior and causing regressions. Change both fixtures to shortName 'list' (a real dotnet new subcommand that genuinely conflicts) and update the rubric/assertion wording accordingly, so the eval tests correct behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * template-discovery: faithful dry-run output + built-in package guidance Fresh eval (run 29581297999) confirmed the earlier rubric fix worked (authoring Validate 4.3->5.0). Two repeatable clean-output defects remained in template-discovery (seen in surviving Plugin runs, not just mutex deaths): - Preview: the agent summarized/regrouped the dry-run output and invented files (e.g. ReconnectModal), and didn't explain key files. Now: present the actual dry-run file list faithfully (no summarizing/inventing) and give a one-line purpose for key entry points. - Search: for a built-in template the agent never clearly answered 'which package to install'. Now: name the exact package, or say 'no install needed - ships with the SDK' for built-in templates. Compressed the resilience note to keep the skill in the detailed token tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Reduce eval overfitting: outcome-focused rubrics + de-cued prompts The overfitting judge flagged moderate overfit on smart-defaults (0.45), comparison (0.34), and instantiation (0.28). Per its per-element analysis (assertions were all 'broad'/fine), make the rubrics/prompts outcome-focused: - Drop every 'The agent used the template-* skill' rubric item (tests activation, not outcome — expect_tools already enforces skill usage as a hard gate). - Reword skill-vocabulary items (e.g. 'native AOT needs a recent AOT-capable TFM', 'smart defaults only fill gaps') and skill-method items ('verify against dotnet new --help', 'inspected via --help', 'logged each applied default', 'verified the project was created') into result-focused criteria that any valid approach satisfies. - De-cue the three smart-defaults prompts that mirrored the skill's 'Required Output B' verbatim ('output the exact single dotnet new command line ... only the flags you will actually pass'); they still elicit a concrete command so the negative assertions remain meaningful. - Drop the blazor prompt's 'Inspect both with dotnet new --help' methodology cue. Assertions and expect_tools are unchanged, so activation and correctness are still verified — the eval now measures outcomes instead of parroting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * template-smart-defaults: defer --help to creation time (trim advice-only cost) Eval run 29586558625 showed the advice-only AOT scenario tie at 4.0 but the skilled run cost 45.7k tokens / 3 turns vs baseline 12.7k / 1 turn — driven entirely by running 'dotnet new worker --help' (feeding ~30k tokens of help output into context) just to advise, which flipped the weighted score negative. Make --help confirmation a creation-time step: for advice-only requests, answer from the rules and note names are confirmed at creation, rather than spending a --help call to advise on well-known parameters. The quality wins come from the decisive output (defaults log + exact command), not from --help, so quality holds while cost drops. Addresses the #898 TRIM-COST concern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: workspace-framework precedence + sync eval.vally.yaml - Precedence conflict (instantiation 'mirror workspace framework' vs smart-defaults 'latest AOT-capable when --framework unset'): treat the workspace framework discovered in Step 2 as an explicit --framework value that smart-defaults must not override; deviate only when incompatible with a requested feature (then flag it). Added a matching note to the smart-defaults AOT rule (a framework pinned by the workspace/global.json counts as set). - Synced eval.vally.yaml (cross-family harness) with the eval.yaml changes that had drifted: outcome-focused rubrics + de-cued prompts for smart-defaults/comparison/ instantiation, and the shortName build/restore -> list fixture+rubric fix for authoring/validation (including the referenced fixture template.json files). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Assert required skill outputs with broad checks (address review) Reviewer flagged that the outcome-focused rubric rewrite left several scenarios without any assertion of the skill's required outputs. Add broad (non-overfitting) assertions in eval.yaml and eval.vally.yaml: - smart-defaults (all 4 scenarios): assert a 'dotnet new' command line is present (output B) and a markdown table is present (the 'Defaults applied' log, output A). - comparison (webapi/webapp, blazor): assert a recommendation is present. - validation (multiple errors): assert a verdict summary (Not ready / N errors) and a findings table are present. - authoring (validate): assert the agent flagged that shortName 'list' conflicts. These are broad presence checks (any valid phrasing passes), so they verify the required output without re-introducing the overfitting the rubric rewrite removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>