mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
5b4d76e8b45dc195bc1b5dd3a860c78e98a47954
307 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Abhitej John
|
1105254fa7 |
Run secondary-judge adapter from trusted staged copy
The dual-judge re-score invoked node eng/vally-adapter/adapt.mjs from the repo checkout, which a fork PR could tamper with. Route it through the trusted staged copy (\/trusted-validator-src/...) like the primary adapt.mjs call, restoring the fork-checkout trust boundary that test_fork_checkout_is_blocked_and_adapter_code_is_trusted enforces. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf |
||
|
Abhitej John
|
0a34b56326 |
Swap default executor to gpt-5.6-luna; judges to gpt-5.6-sol
Per review: replace gpt-5.5 in the default set. The default (and full) executor profile now runs claude-sonnet-4.6 + gpt-5.6-luna, and every GPT-family cross-family JUDGE role that previously used gpt-5.5 now uses gpt-5.6-sol. gpt-5.6-luna's own leg is judged cross-family by claude-opus-4.8 (judge2 claude-sonnet-4.6 on the dual-judge cadence), mirroring the prior gpt-5.5 executor route. All profile models keep a cross-family, non-self judge. Also address PR review comments: - Fix log typo "entr(ies)" -> "entries". - Reword the judge-comparison plugin-key comment to match the code (_.Directory.Parent is the plugin), removing the misleading "parent-of-parent" phrasing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf |
||
|
Abhitej John
|
dc8c397923 |
Merge origin/main into abhitejjohn-cautious-meme
Resolve semantic conflict in evaluation-run.yml: main centralized model resolution into the val-models step (outputs model/judge-model). Layer the cross-family matrix override into that step so it also honors ENTRY_MODEL/ENTRY_JUDGE/ENTRY_JUDGE2 and emits a judge2 output; downstream steps consume the outputs. Removes the now-duplicate inline read_override block from the Run vally step. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf |
||
|
Jan Krivanek
|
1a64002677 |
chore: complete gh-aw upgrade
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
ee1cffae72 |
MMR fixes: anchor /evaluate flags; qualify judge-comparison key by plugin
Multi-model review (Opus-4.8, GPT-5.6, rubber-duck) surfaced two agreed, in-scope issues: - The /evaluate flag regexes matched anywhere as substrings, so a malformed near-miss (--fuller, --opus480, --midnight, --newer-model) silently selected an expensive profile. Anchor each flag on token boundaries ((?:^|\\s) before, (?=\\s|\$) after) so near-misses fall through to default; \\s boundaries still recognize a flag anywhere in a multi-line review body. - judge-comparison pairing keyed verdicts on model|skillName with no plugin qualifier, so two plugins sharing a skill name on the same executor collided and one was dropped from judge-comparison.json and the agreement denominator. Derive the plugin from the results.json path (<plugin>/<skill>/results.json) and key on plugin|model|skill; also emit plugin in each comparison record. actionlint clean; both fixes covered by parser/keying simulations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf |
||
|
Abhitej John
|
8532795214 |
Add /evaluate --opus48 PR-comment flag
The opus48 profile was reachable only via the Thursday schedule and the workflow_dispatch dropdown. Add a --opus48 comment flag so it matches the other opt-in tiers (--mid, --newer, --full) and can be requested from a PR. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf |
||
|
Abhitej John
|
6da3fad774 |
Capture symmetric judge stats for apples-to-apples comparison
judge-comparison.json now records the same stat set for both the primary (opus-4.8) and second (sonnet-4.6) judge per (model, skill): passed, regressed, conclusive, underpowered, meanScore, winRate, trialCount -- not just the pass/regressed booleans. Clarifies in the step comment that this data is experiment-only: scheduled dual-judge cadence only (no-op on PRs), excluded from components.json, and never posted to any PR comment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf |
||
|
Abhitej John
|
17ac1a4f76 |
Remove legacy single profile; default is now the floor
The single (one-model, self-judged claude-opus-4.6) profile was the only non-cross-family path left. Every trigger now resolves to a cross-family profile: manual dispatch and the PR gate default to the two DEFAULT models (sonnet-4.6 + gpt-5.5), heavier tiers stay opt-in. Drops the dropdown option, the --single comment flag, the single special-cases, and the now unused EVAL_PR_NUMBER env. /evaluate --single degrades gracefully to default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf |
||
|
Abhitej John
|
68d82f645c |
Rename opus profile to opus48 for specificity
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf |
||
|
Abhitej John
|
cb203295f2 |
Add cost-optimized cross-family cadence, judge experiment, and full PR mode
Repoint the default executor set to {sonnet-4.6, gpt-5.5} and add tiered
scheduled cadence so every model family runs at a regular-but-cost-optimized
interval:
- default (sonnet-4.6, gpt-5.5): Mon/Wed/Fri + every authorized PR
- mid (haiku-4.5, mai, gpt-5.3-codex): Tue/Sat
- opus (opus-4.8): Thu
- newer (gpt-5.6-sol, opus-5, sonnet-5): Sun
Heavy tiers replace the defaults on their day rather than co-running them.
PR gates run the default profile; an opt-in `full` profile (defaults + mid +
opus) is available via `/evaluate --full` with a warning that it is expensive.
Add an Opus-4.8 vs Sonnet-4.6 judge experiment: on scheduled runs the
GPT-executor legs carry a second Sonnet-4.6 judge, and a new step pairs the two
judges' verdicts into an isolated judge-comparison.json (kept out of the
dashboard components manifest).
Make the scheduled skip-guard per-profile by tagging scheduled runs
"schedule: <profile>" via run-name and matching on display_title, so one
profile no longer suppresses the others on an unchanged commit.
Also fix the token-usage and benchmark aggregation to group each plugin's
results by its own executor model before merging, so cross-family runs no
longer mislabel per-model rows.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
|
||
|
Amaury Levé
|
8b6b51ba59 |
Preflight Copilot PAT pool for evaluation rate limits (#992)
* Retry eval token selection on rate limits Probe shuffled Copilot PAT candidates before running Vally, fail over only on explicit rate-limit errors, and keep the remaining pool out of untrusted subprocesses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden eval token failover boundaries Skip token work for empty legs, probe the resolved agent and judge models, and execute validator and adapter code from isolated trusted workflow sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify failover test setup errors Report missing PyYAML and workflow-step shape failures with actionable diagnostics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align validator cache restore path Use the same archive path as skill-check while deleting any PR-provided file before cache restore. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden trusted evaluation setup Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d * Fix Vally runtime tool resolution Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d * Harden fork evaluation trust boundary Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d * Use trusted NuGet config for validator cache Keep the trusted cache producer consistent with the fallback validator build and invalidate caches when the evaluation NuGet configuration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d * Restore validator package sources Do not apply the evaluation-only dotnet-public mapping to the trusted validator build because it excludes the validator's analyzer package. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d * Unify Copilot rate-limit failover checks Use one superset classifier for preflight and post-run failures, and skip candidates whose availability probes time out. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d |
||
|
Amaury Levé
|
69fe56a74f |
Improve dotnet-test eval power and resilience (#989)
* Improve dotnet-test eval coverage and efficiency Address remaining high-confidence items from #899 by bounding the code-testing pipeline and adding eval coverage for grade-tests and find-untested-sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e430fee9-d3df-4ef5-85a4-745ae4b17046 * Fix dotnet-test eval activation and quality Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98 * Strengthen dotnet-test skill activation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98 * Improve dotnet-test eval signal and resilience Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Address eval review findings Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Remove non-measuring smell eval trials Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Fail evaluation on unexpected adapter errors Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Stabilize assertion quality evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Strengthen assertion quality routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Treat adapter SIGKILL as failure Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a --------- Copilot-Session: e430fee9-d3df-4ef5-85a4-745ae4b17046 Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98 Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a |
||
|
Amaury Levé
|
d94c40450e |
Make BinlogMcp feed-neutral and configure eval NuGet source (#984)
* Revert "Resolve binlog MCP from the dnceng dotnet-public feed"
This reverts commit
|
||
|
Jan Krivanek
|
4d25f17176 |
chore: recompile agentic workflows with gh-aw 0.84.3 (#985)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8fd9e116-87df-4682-8313-fd4a97aa1a07 |
||
|
Abhitej John
|
ccefe9e380 |
ci(eval): add opt-in cross-family model matrix to the eval pipeline
Wire the IMPACT-ANALYSIS.md §10 cross-family (multi-model) executor/judge dimension into the existing Vally eval workflows as a strictly OPT-IN feature. Every current trigger keeps byte-for-byte behavior: the default `single` profile performs no expansion, so entries stay model-less and the runner falls back to the experiment file's existing executor/judge. evaluation.yml (discover job): - New `matrix_profile` workflow_dispatch choice input (single|default|full|newer). - Resolve a profile from the event: workflow_dispatch input, `/evaluate` --full-matrix|--newer|--cross-family flags, or a scheduled cadence day. Event bodies arrive via env and are only regex-matched, never executed. - Expand each plugin/shard entry across the profile's executor models, attaching a cross-family primary judge (judge is never the same model as the executor) and, on scheduled dual-judge days, an optional within-family second judge. - Extend the strict per-entry allowlist to model/judge/judge2. evaluation-run.yml (reusable runner): - Validate the optional ENTRY_MODEL/ENTRY_JUDGE/ENTRY_JUDGE2 against the same allowlist; prefer the injected executor/judge when present. - Pass --model/--judge-model to gen-experiment.mjs, with a sed fallback that patches the static experiment file only when a cross-family model is active. - After the primary judge, optionally re-score the SAME transcripts with the second judge (cheap: re-runs only `vally compare`, not the executor) into a separate `vally-crossjudge/` root, uploaded as a distinct `vally-crossjudge-*` artifact that downstream `vally-results-*` consumers do not glob. gen-experiment.mjs: - Add --model/--judge-model that rewrite the copied base experiment's overrides.model/judge_model lines, with model-id shape validation. Validated: actionlint clean on both workflows; PowerShell parse of the new discover block; gen-experiment functional injection; bash substitution + sed fallback; and an assertion that judge != executor holds for every route. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fb19734-0288-416c-8c13-07b1a49dfde0 |
||
|
Amaury Levé
|
2c22b6c815 |
Fix the underpowered eval gate: score direction, not magnitude (#952) (#965)
The skill evaluation gate flipped ~27% of its verdicts between runs on byte-identical inputs. Reproducing all 22 verdicts from the A/A test in #952 from the formula alone isolates two independent defects. The gate scored the judge's *magnitude*. Trial scores are a five-point ordinal scale (much-better +1.0, slightly-better +0.4, equal 0, ...), and a Student-t interval over those reads the 0.4 -> 1.0 step as variance, so a skill is punished for winning more decisively. Four wins and three ties over seven trials passes at ciLow +0.031 when every win is "slightly-better" and fails at -0.021 when one is "much-better" - same record, better outcome, reversed verdict. coverage-analysis failed five consecutive runs while winning 100% of its trials, then passed on a sixth with the same 3W/0T/0L record. The gate had no minimum sample size. vally returns ciLow = mean below two trials, so a single lucky judgment passed outright; a neutral skill passes ~30% of the time at one trial, and 18 of 94 gated evals sit there. adapt.mjs now reads only each trial's winner and decides with an exact one-sided sign test at 5%. The verdict is a deterministic function of the win/tie/loss record, so identical records always produce identical results. Collapsing to direction alone is not enough: a t-interval over win/tie/loss disagrees with the exact test on 12 records up to n=10 and is the permissive side in every one, passing 4W/0T/0L, 4W/3T/0L and 6W/0T/1L at p = 0.0625. MIN_CREDIBLE_TRIALS = 5 falls out of the same test rather than being chosen: the sign test cannot reach 5% below five discordant trials, and discordant trials can never exceed counted trials. Below it no record can pass, so the verdict is reported as underpowered - never a pass, never a regression - which is what stops "won every trial, failed anyway" being rediagnosed each run. check_eval_quality.py enforces the same floor on specs before they run, counting trials as scenarios x defaults.runs. The 60 existing evals below it are grandfathered in a ledger that can only shrink: the gate errors on a stale, duplicated or unnecessary entry, and --base-ref rejects entries new relative to the base branch, so a PR cannot add a below-floor eval and exempt it in the same change. Renames are read from git and are not growth. The experiment file no longer sets `runs` in `overrides:`. Precedence is CLI > experiment overrides > eval defaults and the merge is a plain spread, so that line did not default anything - it overwrote every eval's own defaults.runs and made per-eval trial counts impossible to express. No eval declares `defaults:` today, so this changes nothing until one opts in, and it is what makes raising a thin eval possible at all. Also fixes two latent bugs found while testing: --vally paths containing a space were split into separate argv entries (which made the adapter suite unrunnable on Windows, where process.execPath is under "Program Files"), and importing adapt.mjs made it parse its importer's command line. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 29c41814-d553-40fd-b50c-6590c88c9540 |
||
|
Amaury Levé
|
f2eb897a12 |
Fix dotnet-test findings from the refreshed cross-family eval (#899) (#945)
* Fix dotnet-test findings from the refreshed cross-family eval (#899) Every change below is driven by judge evidence from the losing trials of the refreshed 5-family dotnet-test matrix (runs 30108473397 + recovery runs), not by style preference. Eval measurement fix — the "discovery" P2s were an artifact: - assertion-quality, test-gap-analysis, test-smell-detection, and test-tagging each have a decline stimulus with `constraints.reject_skills: ["*"]`, so the skill cannot activate there by construction. Without `expect_activation: false` the adapter counted those dormant runs as missed activations, which is exactly the 75-88% invocation rates reported in the scorecard. Annotating them (the convention already used by agent.test-quality-auditor) removes the false signal; the non-activations were the only ones observed for these skills. Skill fixes: - test-gap-analysis: baselines won by actually running the suite while the skill reasoned statically and reported survivors that the tests in fact kill. Added Step 4b: confirm every reported survivor by applying it, re-running the covering tests, and reverting; fall back to reasoning only when the suite cannot run, labelled unverified. Calibrated severity down for strong suites. - test-anti-patterns: baselines won on depth, not polish. Added a depth bar — account for every test in scope, give exact expected values in fixes, name the adjacent error-path/boundary gaps, and keep counts consistent. Trimmed three pitfall rows that duplicated the calibration step so the skill stays under the profiler's "comprehensive" threshold. - detect-static-dependencies: losses were all counting accuracy. One authoritative total (no findings parked outside it), classify by the resource touched rather than by the `static` keyword, exclude pure helpers such as Path.Combine from the needs-wrapping total, require file:line, and add the missing randomness/culture/serialization categories. - test-smell-detection: the calibration rule told models to downgrade Sleepy Test for integration tests, which is what lost both losing scenarios. Fixed sleeps now stay High in any category; Mystery Guest and Eager Test still downgrade. - crap-score: losses came from estimating coverage after collection failed. Added the dotnet-coverage/ReportGenerator recovery path and a hard rule never to publish a CRAP score built on assumed coverage. - coverage-analysis: answer the asked question first, reconcile every number against the script output, and list every below-threshold member instead of declaring one method the entire gap. - migrate-static-to-wrapper: migrate exactly what was requested (no adjacent DateTime.Now rewrites, respect intentional-use comments) and never report "build succeeded" when the build or restore failed. - code-testing-agent: quote each requirement verbatim in the evidence table so multi-condition requirements map to a test that covers the whole combination, and cite a clean run rather than a coverage attempt that exited non-zero. Validation: skill-validator check passes (20 skills, 10 agents); markdownlint clean; eval specs parse and the adapter now reports all four decline stimuli as expect-dormant. Refs #899 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa * Strengthen underpowered dotnet-test evals from the PR 945 eval run The PR eval reported 5 of 10 skills as "no credible improvement". Reproducing the gate arithmetic from the artifacts shows the dominant cause is statistical power, not skill quality. The gate is `mean > 0 AND ci_low > 0` with a t-based CI over per-trial scores, which reduces to `sqrt(n) * (mean/sd) > t(n-1)`. The required mean/sd ratio is brutal at small n: n=2 -> 8.98 n=3 -> 2.48 n=4 -> 1.59 n=5 -> 1.24 n=6 -> 1.05 n=8 -> 0.84 Recomputing each reported CI from the per-trial scores reproduces the published numbers exactly, which confirms the mechanism: crap-score [0.4,1.0,0.4] n=3 CI [-0.261, 1.461] migrate-static-to-wrapper [1.0,0.4,0.4] n=3 CI [-0.261, 1.461] test-gap-analysis [0.4,0,0.4,0.4] n=4 CI [-0.018, 0.618] test-anti-patterns [0.4,0,0.4,0,0,0.4] n=6 CI [-0.030, 0.430] code-testing-agent [0,0.4] n=2 CI [-2.341, 2.741] crap-score and migrate-static-to-wrapper won 100% of their trials (3W/0T/0L) and still failed: at n=3 nothing short of three identically-sized wins can clear the gate. That is a property of a thin eval, not of the skill. Scenario counts are raised with discriminating cases, four of them by wiring up fixtures that were already committed but had no stimulus referencing them: - test-gap-analysis 4 -> 6, using the orphaned `report-quality` fixture (trivial auto-properties and an auto-generated .g.cs to skip, private helpers reachable only through the public API, and a deliberately weak Assert.IsTrue that cannot kill arithmetic mutations) and the orphaned `rust-error-propagation` fixture (an untested `?` propagation path and an untested `<=` boundary). - test-anti-patterns 6 -> 8, using the orphaned `pytest-mixed` fixture (which also checks the calibration rule that pytest's bare `assert` must not be flagged) and the orphaned `assertion-problems` fixture (which separates Critical false-confidence assertions from a Low-severity message nit). - crap-score 3 -> 6, with a new `refactor-required` fixture whose numbers are self-consistent: ApplySurcharges has complexity 13 behind a stale "Complexity: 4" comment (CRAP 28.4, needs 77.2% coverage), ClassifyAccount has complexity 17 so coverage alone can never reach CRAP < 15, and RoundToCurrency is 100% covered so its CRAP equals its complexity exactly. - migrate-static-to-wrapper 3 -> 5, adding a DateTimeKind-preservation scenario over the existing fixture and a new `static-helper` fixture where a static class must gain an ambient TimeProvider seam without breaking its callers. - code-testing-agent 2 -> 3, with a compact C# fixture that must extend an existing suite to the untested method only. This eval stays the weakest: each scenario is expensive, so raising `runs` is a better lever than adding more heavyweight scenarios. Verification: - every eval spec parses and all 254 fixture references resolve - the three new fixtures compile; the shipping-quotes fixture restores, builds and its three seed tests pass under `dotnet test` in a clean workspace - skill-validator check passes (20 skills, 10 agents) - markdownlint clean Refs #899 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa * Address review feedback on fixture and counting wording - BillableWeightTests: the ZeroOrNegative test only asserted the zero case, so its name overstated what it covered. Made it data-driven over 0 and -1 so the name matches the assertions. This matters more than usual here: the file is the seed suite for a test-quality eval, and a misleading test name is exactly what these skills are supposed to flag. - detect-static-dependencies: the Step 3 lead-in said to count each "static call pattern", which contradicted the rule immediately below it that instance members reaching the same untestable resource must also be counted. Reworded to "call site" and made the instance-member inclusion explicit. Verified: the fixture restores, builds and now passes 4 tests (was 3); skill-validator check passes; markdownlint clean. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa |
||
|
Abhitej John
|
e93e52b7db |
Improve /evaluate visibility and run one evaluation per PR (#961)
* Fix /evaluate not-found path capturing gh api error body When a commented SHA does not resolve, 'gh api commits/<sha>' can emit an error body onto stdout, so FULL_SHA held a JSON payload instead of being empty. That slipped past the not-found check and surfaced the raw API error in the reachability message. Accept only a real 40-char hex object id; anything else is treated as not found (fail-closed, clean message). Found while validating the gates on a dummy PR after #956 merged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 29ef3316-14f6-4dc1-a838-40f8be09f915 * Add /evaluate PR visibility and per-PR single-flight Tightens the automated evaluation workflow so a maintainer's /evaluate always gets a clear, prompt response on the PR and a PR is evaluated one run at a time. - Give every /evaluate trigger a unique concurrency group so its gate job always starts and can respond, instead of being starved behind an in-flight evaluation (the cause of a /evaluate sometimes going quiet). - Acknowledge authorized comment triggers early with a reaction, right after the write-access check, regardless of outcome. - Add a best-effort single-flight in the gate: defer to an in-flight run on the same commit (posting a link to it) and supersede an older run on a stale commit. Sibling identity is keyed on (head repository, head branch) so it can never affect a run from an unrelated repository, and a numeric run-id tiebreak makes the race deterministic. It is not a security control -- each run stays bound to its own validated commit. - Cap cheap gate fan-out per PR with a job-level concurrency group so a burst of /evaluate cannot spin up unbounded runners; the expensive evaluation jobs are excluded and are never interrupted by it. - Restore workflow-level cancel-in-progress: true (now only affects the lightweight status/noop groups). Harden the gate step under set -euo pipefail so a transient runs/jobs API error fails open rather than aborting the gate. gate gains actions: write, used only after the write-access check to supersede a stale run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 29ef3316-14f6-4dc1-a838-40f8be09f915 * Harden /evaluate single-flight and clear stale acknowledgements Review follow-ups on the gate's single-flight step: * Narrow sibling selection. `(head repository, head branch)` is not a PR identity, so restrict candidates to this workflow's evaluation entry points and, when the run payload lists associated pull requests, require this PR's number to be among them. That drops `schedule`/`push` runs and the `pull_request` status-only runs (whose jobs are still undecided in the first moments of a run and would otherwise read as a real evaluation), and stops one head branch feeding two PRs with different bases from cross-cancelling. Fork runs report an empty PR list and still fall back to the repo/branch key. * Fix the jq event filter. `[...] | index(.event)` evaluates `.event` against the array, not the run, so it errored out; with the surrounding `|| true` the candidate list would silently come back empty and disable single-flight entirely. Use an explicit `or` chain instead. * Never let an archaeological run kill head coverage. `/evaluate <old-sha>` produced a newer run id, so it superseded (cancelled) the in-flight evaluation of the PR head and left that commit's required check pending with nothing left to resolve it. A run bound to a non-head commit now neither cancels, nor yields to, a run covering the head. Run-id ordering still breaks every symmetric race, so exactly one run survives. * Clear the eyes reaction on gate-terminal paths. The acknowledgement now fires for every authorized `/evaluate`, but `report-status` (which removes it) only runs when `should_eval == 'true'` -- so bare `/evaluate` guidance, an unresolvable commit, and defer/yield left the comment marked in-progress forever. * Paginate the jobs lookup, matching `eval_run_exists_for_head`, and document the gate concurrency trade-off (GitHub keeps one pending job per group, so the middle request of a same-PR burst misses its own ack). Validated with actionlint and a bash test covering the decision matrix, both symmetric races, and the jq filter against run-payload fixtures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 449babf9-dc33-4164-b226-e7bdc0318e49 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Amaury Levé <amauryleve@microsoft.com> Copilot-Session: 29ef3316-14f6-4dc1-a838-40f8be09f915 Copilot-Session: 449babf9-dc33-4164-b226-e7bdc0318e49 |
||
|
AbhitejJohn
|
618d19de8d |
Address review: robust SHA parse + accurate dispatch doc
Parse '/evaluate <sha>' from issue comments with a bash regex instead of a sed BRE, avoiding word-boundary escape ambiguity across sed implementations and matching the validation used elsewhere in the gate. Reject over-length hex strings to the guidance path (fail-safe). Clarify the workflow_dispatch docs: the PR head travels in the head_sha input, while a dispatched run's github.sha is the default-branch tip. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 29ef3316-14f6-4dc1-a838-40f8be09f915 |
||
|
Abhitej John
|
9d08a0a31d |
Bind PR evaluation to the reviewed commit
Harden the /evaluate pipeline in evaluation.yml so a run is always tied to one specific reviewed commit captured when the trigger fires, instead of resolving the live branch head downstream. Every job (gate, discover, evaluate, report-status) now consumes the single gate-bound SHA. - Add a pull_request_review trigger (recommended path): the run binds to review.commit_id, so no SHA needs to be typed. Files changed -> Review changes -> Submit review with /evaluate in the body. - Comment path (/evaluate <sha>) now requires an explicit SHA that must belong to the PR; a bare /evaluate posts copy-paste-ready guidance and points to the review flow. - Label and workflow_dispatch paths bind to the head carried in the event payload / passed by the triage worker. - Warn (log + PR comment) when the bound commit is not the current head. - Only write-access triggers (label, dispatch) may cancel an in-progress run; comment/review triggers queue so an authorized run finishes. - Reachability check tolerates PRs >250 commits via a compare-ancestry fallback, staying fail-closed. - discover fails closed if the bound commit can't be checked out. - Pass inputs.plugin and actor logins via env instead of inlining. - Update docs/design/pr-triage-workflows.md (4 entry points, commit binding). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 29ef3316-14f6-4dc1-a838-40f8be09f915 |
||
|
Abhitej John
|
f09fd590c4 |
Merge pull request #932 from dotnet/abhitejjohn-bridge-eval-report-gap
Enrich Vally evaluation PR comment (legend, reason, overfit, activation, per-scenario detail) |
||
|
Abhitej John
|
9f7d1576bd |
Address PR review: rank-order high-priority details; explicit non-success fallback
- consolidate.mjs: sort the high-priority detail set by rank so failing (❌) blocks are budgeted before inconclusive (⚠️) ones; previously skill-name order could let a ⚠️ block consume budget a later ❌ block needed. - evaluation.yml: add an explicit branch for a non-success evaluate result (failure/cancelled) so the 'produced no results' guidance is reserved for a successful job that emitted zero verdicts. (Copilot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
15c336709e |
Fold Close Stale PRs into deterministic pr-triage stale sweep (#928)
Replace the agentic close-stale-prs.agent.md with a deterministic stale-PR sweep hosted in pr-triage-batch.yml. The new stale-sweep job runs .github/scripts/pr-stale-sweep.sh weekly (cron 17 4 * * 1) and on manual dispatch (stale_sweep=true), warning about and closing PRs with no non-bot activity for 30/37 days. Preserves the original policy (no-stale and maestro exemptions, drafts included, non-bot activity timer) without model calls or token cost. Deletes close-stale-prs.agent.md and its compiled lock file, and updates docs/design/pr-triage-workflows.md. Fixes #915 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
0af65ec64c |
Bridge eval report gap: enrich Vally PR comment
Surface signals that previously only appeared in the step summary or not at all, in eng/vally-adapter/consolidate.mjs (both simple and full formats): - Add a column legend/glossary beneath the table. - Add Overfit column from overfittingResult.severity (Low/Moderate/High icons, mirroring the old Reporter.cs FormatOverfitCell). - Add Skills Loaded column from scenarios' skillActivationIsolated/Plugin, with a warning when an expected activation is missing. - Add a per-skill collapsible <details> block rendering the verdict reason and a per-scenario preference table (mirrors evaluation-run.yml). - Details blocks placed after the table so they don't break table rows. evaluation.yml: replace the misleading generic "Evaluation failed" fallback with a message explaining the likely real cause (evaluate completed but produced zero verdicts, often a transient LLM-session auth failure) and telling the user to re-post /evaluate, distinct from the upstream-skipped case. Docs: document the PR comment format in eng/vally-adapter/InvestigatingResults.md and sync the Skills Loaded activation symptom in the skill-validator guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
78054c1161 |
Migrate LLM evals to the Vally harness (#877)
* Make Vally the sole LLM eval engine, retiring skill-validator evaluate Collapses the parallel skill-validator + Vally eval pipeline into a single Vally-only path while preserving every capability skill-validator provided: PR gate, PR comment, dotnet/skills-data historical push, and the dashboard. The skill-validator `check` linter is retained (skill-check.yml) pending microsoft/vally #463. - evaluation.yml: delete build-validator and the skill-validator `evaluate` job; single `evaluate:` job now uses vally-evaluation.yml. Forward the COPILOT_PAT_0..9 pool onto it (from #911) so the reusable workflow's token gate is satisfied. Rewire comment-on-pr / report-status / publish-* / deploy-dashboard onto `evaluate` and vally-results-* artifacts. - vally-evaluation.yml: pin @microsoft/vally-cli@0.9.0; keep the #911 prepare guard; write the generated per-plugin experiment to $GITHUB_WORKSPACE so relative eval globs resolve. - adapt.mjs: reconcile #887's runCompareWithRetry + conclusive/unmatched hardening with the consolidation helpers (nonActivation, compareByStim, roleToDashboard, etc.). Add consolidate.mjs and gen-experiment.mjs. - build-replay-sessions.ps1: read Vally executor-session-logs/events.jsonl instead of skill-validator session output. - Rename all tests/*/*/eval.vally.yaml to eval.yaml (Vally format is now the only eval format); author Vally evals for the four skills that lacked one. - CONTRIBUTING.md + InvestigatingResults: point contributors at the Vally harness; drop non-public links. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix setup-local-sdk eval for Vally schema The hand-authored setup-local-sdk eval used skill-validator grader/environment schema that Vally rejects at validation (no stimuli executed): - file-contains/file-not-contains graders require 'value:' not 'substring:' (output-contains correctly keeps 'substring:'). - environment.files entries require 'src:'/'dest:' (copy a fixture) rather than inline 'path:'/'content:'. Moved the global.json body into fixtures/global.json. Verified with 'vally lint --eval-spec' (0 errors) across all 97 eval.yaml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix markdownlint violations in migration docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback on eval migration - Consolidate run-vally-evals.sh into a single neutral entrypoint eng/run-skill-evals.sh (deletes the thin wrapper; adapt.mjs stays in eng/vally-adapter/ since CI invokes it directly) - Rename reusable workflow vally-evaluation.yml -> evaluation-run.yml and update all references/self-trigger regexes - Rename local output dir vally-results -> eval-results - Bake prerequisite preflight (Node 20+, GITHUB_TOKEN/gh) into the runner - Simplify CONTRIBUTING "Running tests locally" prerequisites - Scrub harness name from consumer-facing surfaces Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reinstate overfitting detection under the Vally harness Overfitting detection previously ran inside skill-validator's `evaluate` command, which the Vally migration replaced, so it silently stopped running. Reinstate it as a standalone step wired into the Vally pipeline: - Add `skill-validator overfitting` command that discovers skills, parses each eval.yaml, runs the existing OverfittingJudge, and emits [{plugin, skill, overfittingResult}] JSON (per-skill failures non-fatal, bounded parallelism). - Add a Vally-format parser bridge (ParseEvalConfigFlexible) so the judge handles the current stimuli/graders eval.yaml format; the legacy scenarios-only parser rejected all 98 evals, so the judge never ran. - adapt.mjs: merge overfitting results onto each verdict via --overfitting <file> (keyed by plugin/skill); output is byte-identical when the flag is absent. - evaluation-run.yml: build skill-validator (shared cache with check), run the judge per leg reading GITHUB_TOKEN, and pass --overfitting to adapt.mjs so the dashboard/skills-data consume it unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reconcile #919 Vally eval configs into the sole eval.yaml convention PR #919 landed on main adding fresh Vally specs for three coverage-gap skills (setup-local-sdk, convert-blazor-server-to-webapp, dotnet-webapi) under the interim eval.vally.yaml filename. This branch collapses to a single Vally spec named eval.yaml, so the merge left each skill with both my migrated eval.yaml and #919's newer eval.vally.yaml (a git-clean but semantic duplicate the pipeline would not discover). Adopt #919's reviewed Vally content as the canonical eval.yaml for all three skills and drop the redundant eval.vally.yaml files, preserving the 'sole eval.yaml, zero eval.vally.yaml' invariant. Content is byte-identical to #919's blobs ( |
||
|
dependabot[bot]
|
1e2fc4f10e |
Bump the github-actions-dependencies group across 1 directory with 5 updates (#918)
Bumps the github-actions-dependencies group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `6.0.2` | `7.0.0` | | [actions/cache/restore](https://github.com/actions/cache) | `5.0.5` | `6.1.0` | | [actions/cache](https://github.com/actions/cache) | `5.0.5` | `6.1.0` | | [actions/setup-dotnet](https://github.com/actions/setup-dotnet) | `5.2.0` | `6.0.0` | | [actions/setup-node](https://github.com/actions/setup-node) | `6.4.0` | `7.0.0` | Updates `actions/checkout` from 6.0.2 to 7.0.0 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v6.0.2...9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0) Updates `actions/cache/restore` from 5.0.5 to 6.1.0 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v5.0.5...55cc8345863c7cc4c66a329aec7e433d2d1c52a9) Updates `actions/cache` from 5.0.5 to 6.1.0 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/27d5ce7f107fe9357f9df03efb73ab90386fccae...55cc8345863c7cc4c66a329aec7e433d2d1c52a9) Updates `actions/setup-dotnet` from 5.2.0 to 6.0.0 - [Release notes](https://github.com/actions/setup-dotnet/releases) - [Commits](https://github.com/actions/setup-dotnet/compare/c2fa09f4bde5ebb9d1777cf28262a3eb3db3ced7...a98b56852c35b8e3190ac28c8c2271da59106c68) Updates `actions/setup-node` from 6.4.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e...820762786026740c76f36085b0efc47a31fe5020) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/cache/restore dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/cache dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/setup-dotnet dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/setup-node dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Amaury Levé
|
4df4da469a |
Upgrade agentic workflows and fix stale PR cleanup (#916)
* chore: upgrade gh-aw runtime * fix: paginate stale pull request cleanup Upgrade the generated agentic workflow assets and ensure stale PR discovery includes every result page and draft pull requests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 968a22c2-327f-4d26-8f86-1c59bcd323ea * Improve dotnet-test eval coverage and efficiency Address remaining high-confidence items from #899 by bounding the code-testing pipeline and adding eval coverage for grade-tests and find-untested-sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e430fee9-d3df-4ef5-85a4-745ae4b17046 * fix: address agentic workflow review Pin the Copilot setup checkout action and include the cutoff date in stale PR search results. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 968a22c2-327f-4d26-8f86-1c59bcd323ea * Improve test migration skill guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5e19d263-02a6-45b1-9cfb-424fa4d10863 * fix: add fixture namespace imports Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a464e6e4-3e45-41fe-b17d-887c8cb8a448 * test: assert GetOrderById grade Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a464e6e4-3e45-41fe-b17d-887c8cb8a448 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Art Leonard
|
ab72985132 |
Harden vally comparison reliability (#887)
* Harden vally comparison reliability Retry transient comparison errors once, surface unmatched trajectories, and mark degraded reports as inconclusive. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 33efeadb-dcc9-4005-a63f-c5d832b06f45 * Address vally review feedback Harden test diagnostics, pass workflow paths as arguments, and clarify the shadow-result schema. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 33efeadb-dcc9-4005-a63f-c5d832b06f45 * Move Vally adapter guidance Keep shadow evaluation reliability documentation with its implementation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 33efeadb-dcc9-4005-a63f-c5d832b06f45 --------- Co-authored-by: Art Leonard <artl@Arts-MacBook-Pro-M5.local> |
||
|
Abhitej John
|
6f53aaf841 |
fix(ci): unblock vally eval on the reusable PR path (#911)
* Fix vally eval on the reusable PR path: skip spurious prepare, forward PAT pool The PR-evaluation path (evaluation.yml -> reusable vally-evaluation.yml) could not start vally. Two independent problems, both fixed here in code: 1. Spurious `prepare` run. The `prepare` job guarded only on `github.event_name == 'workflow_dispatch'`, but that event name propagates into workflow_call, so prepare ran during the reusable call, read empty dispatch inputs, and threw "Invalid plugin ''". Now additionally guarded on `inputs.entries == ''` so prepare runs only for a direct manual dispatch and is skipped whenever the caller supplies `entries`. 2. Empty COPILOT_PAT_0..9 pool. A reusable workflow does not inherit the caller's org/repo secrets, and the `copilot-pat-pool` environment holds no environment secrets, so `secrets.COPILOT_PAT_*` read empty across the workflow_call boundary and the vally jobs failed with "No copilot-pat-pool entries are configured". The caller (a top-level job) CAN read the existing org-level pool -- the same pool the scheduled validate-pat-pool workflow reads -- so evaluation.yml now forwards it via an explicit named `secrets:` mapping of the 10 tokens. This mirrors how skill-validator sources its own token set in-workflow and needs no admin configuration. Deliberately NOT `secrets: inherit`, which would forward every repo/org secret into the reusable workflow and reverse #868's blast-radius reduction; only these 10 named tokens cross the boundary. Validated with actionlint 1.7.7 (exit 0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
ca1b47371f |
Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop (#813)
* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop WHY Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a plugin's version directly from its checked-in manifest. With no versioning discipline, a plugin's behavior can change while its advertised version stays flat, so clients never learn to re-pull, and there is no human-readable signal of what changed. We want correct, current versions in the repo with minimal manual work and without bloating the marketplace clone. WHAT - Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin owns a version.json whose pathFilters exclude the generated manifests and the version.json itself, so version height tracks real content changes only. - The computed version is materialized into the checked-in manifests (plugin.json and .codex-plugin/plugin.json) so every consumer reads a current value with no build step on their side. - eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the set of changed plugins from a git diff, computes each version with nbgv (predicting the squash-merge height for PRs), and either reports or stamps. AUTOMATIONS (two, low-touch by design) - /version-bump: an admin/maintainer comments the command on a PR and the affected plugins are stamped on the PR branch. Gated on collaborator permission (admin/write/maintain); forks are rejected before any privileged step. No other PRs are auto-modified. - weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any drift on main, opens/updates a single bot PR, and explains the per-plugin reason. This self-heals anything that merged without a bump. We deliberately did NOT auto-edit contributor PRs or add a noisy advisory comment bot; maintainers stay in control and the signal stays clean. SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro) - Supply chain (High, both models): dotnet tool restore would have honored a nuget.config authored in the PR tree, letting an attacker remap the nbgv package source to a malicious feed and run code in the privileged contents:write context. Mitigated with a trusted eng/version/nuget.config (clear + nuget.org-only + packageSourceMapping), overlaid from main and used via --configfile so PR-supplied configs are ignored. No nuget.config is tracked in the repo today, so this path was genuinely exploitable. - TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather than the mutable branch name; a racing push fails non-fast-forward, which is the safe outcome. - Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement string) so a "$"-bearing version cannot re-expand, plus a strict major.minor.patch guard that throws on a malformed base, leaving manifests untouched. - A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped. VERIFIED End-to-end against a real NBGV git harness: content-scoped predict, base-only bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard, and --configfile restore (exit 0). actionlint passes on both workflows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review feedback - Add missing plugins/dotnet-test-migration/version.json so it participates in versioning (it was the only plugin without one; manifests are at 0.1.0). - CONTRIBUTING: the two manifests are not byte-identical; say the version is duplicated across two manifest files instead. - weekly-version-sync: include version.json in commit attribution so a base-only bump is explained rather than showing 'no attributable commits'. - Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI failures are diagnosable, while keeping stdout clean for JSON parsing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Aditya Mandaleeka
|
52ba152ed6 | Update to Vally 0.7.0 (#854) | ||
|
Abhitej John
|
80baddd6c4 |
Harden vally evaluation and source tokens from copilot-pat-pool (#868)
* Harden vally evaluation against command injection; source tokens from copilot-pat-pool Close a command-injection vector where fork-PR-controlled path segments flowed into a bash run: body via matrix.entry.plugin interpolation in the token-bearing job. - Allowlist-validate matrix entries (plugin/name/skills_path) before they enter the matrix in evaluation.yml, and re-validate defense-in-depth as the first step of the vally-evaluate job. - Pass all PR-controlled values through env: vars and reference them quoted; no raw expression interpolation remains in any run: body. - Build workflow_dispatch entries with jq --arg instead of string interpolation. - Replace secrets: inherit with explicit secret passing to reduce blast radius. - Source Copilot tokens from the copilot-pat-pool environment (COPILOT_PAT_0..9) via the job environment, with COPILOT_GITHUB_TOKEN as an empty-pool fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address Copilot review: reject empty skills_path and quote dirname - Fail the matrix-entry validation when skills_path is empty (never a legitimate state for vally entries) instead of silently skipping the loop. - Quote the inner $(dirname ...) in EVAL_NAME to avoid word-splitting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address jeffhandley review: drop PAT fallback, explain regex anchors - Remove the COPILOT_GITHUB_TOKEN fallback secret from vally-evaluation.yml (workflow_call.secrets, the token-selection step, and evaluation.yml's caller-side secrets passing). An empty copilot-pat-pool is now a hard configuration failure instead of falling back to a separate secret. - Add short comments explaining why bash's ^/$ anchors are safe here without the \A/\z equivalents used in the PowerShell discover job. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden regex validation against locale-dependent bracket ranges Add LC_ALL=C before the allowlist regex checks in both the prepare job and the Validate matrix entry step. POSIX bracket ranges like [A-Za-z] can behave unexpectedly under non-C locale collation; forcing the C locale makes the anchor/character-class behavior deterministic regardless of the runner's default locale. Found via cross-model (GPT-5.5) review of the prior fixup commit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address review: constrain vally by skills_path, reject empty skills_path - vally: select eval specs per matrix leg's skills_path so sharded plugins run only their own skills (no N-times re-run / PAT-pool token waste) - evaluation.yml: hard-fail on empty skills_path in the matrix allowlist - vally: quote inner dirname substitution in the Write summary step Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reject whitespace-only skills_path in matrix validation A value like a single space passed the bash `-z` empty check and then word-split to zero loop iterations, letting an effectively-empty skills_path slip through. Count word-split segments instead so empty and whitespace-only both hard-fail. Mirror the intent on the PowerShell discover side with a trim-based emptiness filter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Bind skills_path segments to the matrix entry's plugin The skills_path allowlist previously validated only the generic shape (plugins/<any>/skills[/<skill>]); it did not require each segment to belong to the entry's own plugin. A mismatched entry (plugin=foo + skills_path=plugins/bar/skills) would pass, and downstream steps assume the plugins/$PLUGIN/skills prefix. Add a literal (non-regex) prefix binding in both validators: - vally-evaluation.yml: [[ seg != prefix && seg != prefix/* ]] with skills_prefix="plugins/$ENTRY_PLUGIN/skills". - evaluation.yml: case-sensitive -cne plus Ordinal StartsWith against "plugins/$($e.plugin)/skills". Literal comparison avoids regex-metachar pitfalls from '.'/'-' in plugin names; -cne/Ordinal keeps it case-sensitive to match Linux path semantics. Legitimate entries are always same-plugin, so this rejects no valid input. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
042972d411 |
Reduce evaluation fan-out to avoid Copilot rate limits (#872)
* Reduce evaluation fan-out to avoid Copilot rate limits Group PR evaluation entries by plugin and preserve existing executionShard bucketing so large PRs stop launching one skill-validator job per changed skill. Cap PR matrix concurrency and keep PR grouped legs at parallel-skills=1 so each assigned Copilot token stays below the rate-limit threshold. Keep vally on the proven reusable-workflow matrix pattern, cap its concurrency, and only synthesize manual entries when inputs.entries is empty so workflow_call runs use the caller-provided grouped entries. Also skip agent.* and eval-less skills when constructing vally --eval-filter arguments. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make vally full runs work before the experiment file lands PR 872 is intentionally based on main only, but main does not have the new repo-root dotnet-skills.experiment.yaml yet. The vally workflow was hard-coded to that file, so full branch-level runs from PR 872 failed before starting any trials. When the repo file is absent, synthesize an equivalent temporary experiment file in the runner and use that for read_override() and vally experiment run. That keeps PR 872 self-contained while preserving the same per-skill skilled-vs-baseline experiment shape. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
134917c42c |
fix: reinforce no-script-execution constraint in devops-health-check (#845)
* fix: reinforce no-script-execution constraint in devops-health-check The agent was hallucinating Python script creation (run_analysis2.py) and attempting to execute it, which was correctly blocked by the bash allowlist security policy. Strengthen the efficiency guideline to explicitly state that python/python3/node are NOT in the bash allowlist and will be blocked, directing the agent to use jq instead. Fixes #841 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: narrow wording to 'general-purpose language runtimes' per review Addresses review feedback: 'any script interpreter' was too broad since jq/sed are technically interpreters but ARE in the allowlist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jeff Handley
|
11c5dfc738 |
Require environment input to pat_pool shared workflow. (#832)
* Require environment input to pat_pool shared workflow. - Updates the PAT pool shared workflow to latest version - Updates the validate-pat-pool workflow to match other repos - Updates all agentic workflows to use the updated PAT pool import with the standard secret names used across other repos - The new version of the import does not require the 'needs' workaround but the issue-triage workflow's 'roles: all' configuration requires a different workaround to ensure the pre_activation job exists for the pat_pool job to be able to depend on it (necessary for ordering). * Update markdown bullet style * Apply Copilot PR feedback Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Bump versions to match gh-aw --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
9b50865f24 |
build: recompile gh-aw workflows to v0.79.8 (follow-up to #658) (#814)
PR #658 bumped .github/aw/actions-lock.json to v0.79.8 but its automation could not recompile the .lock.yml files, because the GITHUB_TOKEN it runs under lacks the 'workflow' scope. PR #815 then shipped prompt fixes but likewise did not re-pin the runtime. As a result main is in a split state: actions-lock.json declares v0.79.8 while all 8 compiled workflows still pin and execute the v0.77.5 runtime (setup@v0.77.5). Run 'gh aw compile' (CLI pinned to v0.79.8) so the deployed workflows actually run the version main already declares, eliminating the declared-vs-deployed drift. This only re-pins the runtime and regenerates compiled output (the 8 .lock.yml files, the generated agentics-maintenance.yml, and the compile-managed dependabot entry); it makes no prompt/behaviour changes, so #815's groom and health-check fixes are preserved verbatim (groom body_hash unchanged at a705abb0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d656edc4f2 |
fix(devops-health): stop recurring groom node_id and health-check tool/token failures (#815)
* fix(devops-health): stop recurring groom node_id and health-check tool/token failures Groom (fixes 780/792/797): MCP issue_read(get_comments) sometimes omits node_id, which hide-comment requires. The agent was reporting missing_data and failing the run. Make node_id best-effort: skip hiding comments without a node_id and continue; the weekly cleanup workflow removes them by age. Health check (fixes 810/791/788): the agent attempted sed and git commands (incl. manual git plumbing on the cache-memory dir) that the bash allowlist denied, looping on the denials and exhausting the 25M effective-token budget. Add sed and git to the allowlist, and add guidelines to not git-manage cache-memory and not retry denied commands. Recompiled lock files with pinned gh-aw v0.77.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(groom): clarify skipped-comment count goes in noop summary only Address PR review: Step 6 only emits a summary via noop when no other safe-output calls were made, so 'note in the Step 6 summary' was misleading. Track the skipped count internally and include it in the noop message only when that summary is emitted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
01b4875ac2 |
Add authoring-github-workflows skill + actionlint CI gate (prevent workflow-YAML breakage) (#760)
* Fix evaluation.yml: quote run-name so '#' is not parsed as a YAML comment The run-name added in #746 is an unquoted plain scalar containing ` #{0}`. In YAML, a space followed by '#' starts a comment, so everything from '#{0} ...' onward was stripped, leaving an unterminated ${{ }} expression. The file still parses as YAML (yaml.safe_load succeeds) but GitHub Actions rejects it ("This run likely failed because of a workflow file issue"), which broke every evaluation run on main after #746 merged. Wrapping the value in double quotes keeps the full expression intact; verified with actionlint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add authoring-github-workflows skill + actionlint CI gate Prevents the class of bug that broke evaluation on main (#746): an unquoted `${{ }}` workflow expression containing `#`, which YAML treats as a comment and silently truncates, producing a file that parses as YAML but that GitHub Actions refuses to run ("This run likely failed because of a workflow file issue", no jobs started). Two layers: - Knowledge: .agents/skills/authoring-github-workflows/SKILL.md teaches the `#`-as-comment trap and other quoting rules, and to validate with actionlint (plain YAML linters accept the truncated form). - Enforcement: .github/workflows/actionlint.yml runs a pinned actionlint on hand-authored workflows for any PR/push touching .github/workflows. Generated gh-aw files (DO NOT EDIT headers) are skipped; shellcheck/pyflakes are off so the gate focuses on workflow/expression errors. .github/actionlint.yaml declares the repo's custom runner labels (ubuntu-slim, windows-11-arm). Verified locally: actionlint exits 0 on all hand-authored workflows, and exits 1 on the original unquoted run-name, confirming the gate catches it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix markdownlint MD038: remove spaces inside code spans in SKILL.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments: actionlint config trigger, curl -f, syntactic-vs-semantic scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden actionlint workflow: pin checkout to SHA, disable persisted creds, verify actionlint checksum Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
c8be21018f |
Fix evaluation.yml: quote run-name so '#' is not parsed as a YAML comment (#759)
The run-name added in #746 is an unquoted plain scalar containing ` #{0}`. In YAML, a space followed by '#' starts a comment, so everything from '#{0} ...' onward was stripped, leaving an unterminated ${{ }} expression. The file still parses as YAML (yaml.safe_load succeeds) but GitHub Actions rejects it ("This run likely failed because of a workflow file issue"), which broke every evaluation run on main after #746 merged. Wrapping the value in double quotes keeps the full expression intact; verified with actionlint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
0c0f6f0263 |
Fix PR-triage eval trigger: dispatch evaluation.yml instead of bot label (#746)
The triage worker added the 'evaluate-now' label via GITHUB_TOKEN, but label events emitted by GITHUB_TOKEN do not start workflows (GitHub's recursion guard), so evaluation.yml's pull_request_target:[labeled] entry point never fired for the bot (repro: PR #745). workflow_dispatch and repository_dispatch are the only token-initiated events exempt from that guard. The worker now dispatches evaluation.yml directly via 'gh workflow run' with a pr_number input, routed through the existing gate job so the path is identical to /evaluate. A dispatched run's head_sha is the default branch (not the PR head), so idempotency now matches the deterministic run name 'Evaluate PR #<n> @ <sha7>'. The 'evaluate-now' label remains a valid human entry point. Worker granted actions:write for 'gh workflow run'. |
||
|
Abhitej John
|
8b7867331c |
Pin github/gh-aw-actions to commit SHA and add pinning guard (#749)
* Pin github/gh-aw-actions to commit SHA and add pinning guard The agentic workflows referenced github/gh-aw-actions/setup and setup-cli by mutable tag (@v0.77.5) because .github/aw/actions-lock.json still carried a stale pre-migration entry (github/gh-aw/actions/setup@v0.71.5) with no entry for the action the compiled workflows actually use. With no matching lock entry, gh aw compile fell back to emitting the bare tag. Refresh the lock with SHA-pinned entries for github/gh-aw-actions/setup@v0.77.5 and setup-cli@v0.77.5 (commit 3ea13c02...), and regenerate the workflows via gh aw compile so every uses: ref, decorative comment, and manifest sha is pinned. This mirrors how dotnet/msbuild pins the same action and unblocks enabling org-level "require actions pinned to a full-length commit SHA". Also add enforce-action-pinning.yml, a self-contained PR/push check that fails any workflow referencing an action by tag/branch. The agentic workflows are schedule/issue-triggered and never run on PRs, so neither /evaluate nor the runtime org policy gates them pre-merge; this check does. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Remove enforce-action-pinning guard workflow Drop the CI guard in favor of relying on the org/repo 'Require actions pinned to a full-length commit SHA' setting, per review feedback. The lock-file fix and SHA pins remain the root-cause fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
118cc69319 |
Issue Triage: process issues from all authors (#741)
Lower the GitHub MCP integrity filter (min-integrity: none, allowed-repos: public) so the triage agent can read issue bodies/comments from external contributors and read-only org members instead of getting a [Filtered] placeholder. Harden the now-less-filtered path: disable issue-body edits on update-issue (body: false) and add an Untrusted content prompt section. Recompiled lock file. |
||
|
Jan Krivanek
|
78f3f755e7 |
Migrate Copilot PAT rotation to shared pat_pool import (#743)
Adopt the shared workflow import pattern from dotnet/runtime PR #127946, replacing the per-workflow select-copilot-pat action + custom job (shipped in #736) with a reusable shared/pat_pool.md import. What changed: - Add .github/workflows/shared/pat_pool.md: an import that defines a `pat_pool` job (inline bash, no separate action) exposing a `pat_number` output, plus an import-schema mapping COPILOT_PAT_0..7 to this repo's pool secrets (COPILOT_GITHUB_TOKEN, COPILOT_GITHUB_TOKEN_2..8). - Add .github/workflows/shared/pat_pool.README.md documenting the pattern. - Convert all 8 agentic workflows to `imports: - shared/pat_pool.md` + `engine.env` `case(needs.pat_pool.outputs.pat_number ...)`. - Delete the now-unused .github/actions/select-copilot-pat action. - Add .github/workflows/validate-pat-pool.yml: a daily standalone workflow that validates each pool PAT with a Copilot CLI request and summarizes pool health. Wiring note (adaptation from runtime): consuming workflows declare `on.needs: [pat_pool]` instead of runtime's `needs: [pre_activation]` + `on.permissions: {}`. This wires pat_pool ahead of the pre_activation and activation jobs so the selected PAT is validated by the activation job and used by the agent, and it works for `roles: all` workflows (issue-triage), which do not produce a pre_activation job for the runtime workaround to attach to. Compiled with gh-aw v0.77.5. Verified end-to-end with a temporary test-pat-rotation workflow (since removed): a run selected token #2 of the 3-token pool and the agent job observed pat_number='2', confirming the rotated PAT reaches the agent. |
||
|
Jan Krivanek
|
cc0621e2ac |
Fix Copilot PAT rotation for gh-aw v0.77.5 runtime (#736)
* Fix Copilot PAT rotation for gh-aw v0.77.5 runtime
The PAT-rotation stop-gap wired the rotated token into engine.env via needs.pre_activation.outputs.copilot_pat_number. Because the compiled agent job depends only on 'activation' (not the built-in 'pre_activation'), that needs reference evaluated to an empty string in the agent job, so the case() fell through to the default COPILOT_GITHUB_TOKEN and rotation never reached the agent. gh-aw v0.77.5 surfaces this as a compiler warning.
Replace the pre_activation step-injection with a 'select_copilot_pat' custom job wired via on.needs. As a user-defined job referenced in engine.env, the compiler makes it a direct dependency of the agent job, so needs.select_copilot_pat.outputs.copilot_pat_number resolves correctly in both the activation and agent jobs. Same action, same secret pool, same case() expression. Recompiled all workflows with gh-aw v0.77.5.
* Add temporary test-pat-rotation workflow to validate rotation
Non-destructive pull_request-triggered workflow that selects a pool token and asserts, in the agent job, that needs.select_copilot_pat.outputs.copilot_pat_number is non-empty (the exact value that was silently empty with the old pre_activation wiring). To be removed after validation.
* Work around gh-aw v0.77.5 invalid-YAML rendering of top-level if
gh-aw v0.77.5 emits the top-level frontmatter `if:` on the built-in
pre_activation job WITHOUT a ${{ }} wrapper. When the condition starts with
`!` (the fork guards), the emitted `if: !(...)` is invalid YAML (a leading `!`
starts a YAML tag), which GitHub rejects as a workflow-file startup failure.
v0.68.3 wrapped it (valid); v0.77.5 does not. Confirmed independent of the
PAT-rotation change via a minimal probe.
Wrap the fork-guard conditions in parentheses so the emitted scalar starts
with `(` instead of `!` (semantically identical). Affects close-stale-prs,
devops-health-check, devops-health-groom, markdown-linter, pr-malicious-scan.
* Update select-copilot-pat README for the custom-job + on.needs pattern
* Remove temporary test-pat-rotation workflow (rotation validated)
|
||
|
Jan Krivanek
|
7bd6628e91 |
fix(devops-health): resilient dashboard issue discovery (#737)
The health check located its dashboard issue solely via the devops-health label search. When GitHub silently dropped the pinned dashboard (#288) from its issue search/list index, the workflow could not find it, created a duplicate (#695), and abandoned the pinned issue -- leaving the pinned dashboard stale for days. Step 4.1 now resolves the dashboard by a cached issue number first (updating it directly by number, which works even when the issue is missing from search), persists that number to cache-memory every run, falls back to label + pinned-issue lookup, and consolidates duplicates. Documents the new health-dashboard-issue cache key. Lock file unchanged: the prompt body is runtime-imported from these .md files. |
||
|
Copilot
|
2d163bb600 |
Rename ASP.NET Core plugin from dotnet-aspnet to dotnet-aspnetcore (#711)
* Initial plan * Rename dotnet-aspnet plugin to dotnet-aspnetcore --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
01050ea83c |
Fix pr-malicious-scan: repeat-spam + integrity-filter blocks (#722)
* Fix malicious-scan repeat-spam + integrity-filter blocks Root causes (observed on PR #237): 1. Agent's emitted add_comment body did not include the HTML marker line (<!-- pr-malicious-scan:fingerprint=... -->), so both the orchestrator's pre-dispatch check and the agent's own Step 1 idempotency lookup failed to find a prior scan for the same head SHA. Result: hourly re-dispatch. 2. The github MCP tools (pull_request_read, list_pull_requests, search_pull_requests) are blocked by the gh-aw integrity filter on PRs from non-approved authors -- exactly the population this scanner targets. Result: 'Integrity filter blocked N items' notes in every comment. Fixes: - pr-malicious-scan.agent.md: drop the github MCP toolset, add 'gh' to the bash allowlist, and instruct the agent to use 'gh api' for all PR data reads (PAT-authenticated, not subject to the integrity filter). - Strengthen Step 5: the HTML marker MUST be the first line of the comment body. Add a defense-in-depth note that the orchestrator also accepts the visible-body sentinel. - pr-triage-batch.yml + pr-triage-act.sh: match prior scans by EITHER the HTML marker OR the visible-body sentinel ('Automated diff scan' + backticked sha7), so a missing marker on a previously-emitted comment no longer triggers re-dispatch. Workflow disabled remotely while this lands. * Orchestrator-only dispatch + integrity-filter opt-out Replace the per-push pull_request_target trigger and the gh-api workaround with the documented gh-aw pattern: - pr-malicious-scan.agent.md: drop pull_request_target; trigger only via workflow_dispatch from the orchestrator. Restore the github MCP toolset with min-integrity: none (the documented level for spam-detection / analytics workflows; safe-outputs still gates every mutation). Drop the 'gh' bash hack and visible-body sentinel requirements. - pr-triage-batch.yml: orchestrator now posts a deterministic '<!-- pr-malicious-scan:dispatched=SHORT --> ' comment BEFORE calling gh workflow run. That comment is the source of truth for 'a scan has been initiated for this head SHA' and survives every agent-side failure mode (PAT outage, integrity block, dropped HTML marker). Dedup matches either that orchestrator marker OR the agent's own fingerprint marker. - pr-triage-act.sh: drop the visible-body-sentinel fallback; match the orchestrator dispatched marker plus the agent fingerprint marker. Validated: gh aw compile clean; bash -n clean for both worker script and orchestrator embedded script; markdownlint clean; dedup query and POST api tested live against PR #713. |
||
|
Jan Krivanek
|
d9e4a5e113 |
PR triage workflows: orchestrator, worker, and evaluate-now label (#716)
* PR triage workflows: orchestrator, worker, and evaluate-now label Implements docs/design/pr-triage-workflows.md: - pr-triage-batch.yml: hourly orchestrator that classifies open PRs - pr-triage.yml + pr-triage-act.sh: per-PR worker (state recompute, label reconciliation, eval-trigger, ping comments with cool-down) - evaluation.yml: gate job now also handles pull_request_target [labeled] with the evaluate-now label as a second entry point alongside /evaluate * Add temporary push triggers for testing pr-triage workflows * test: live-run pr-triage worker once * test: re-run worker for cool-down check * fix: age gate uses created_at and applies only before first ping * Remove temporary test triggers and inline test marker * Add pr-malicious-scan agent workflow; replace design doc with brief overview - New: .github/workflows/pr-malicious-scan.agent.md + compiled .lock.yml. Static diff scanner for external (non-trusted) PR contributors. Triggers on pull_request_target [opened/synchronize/reopened] and workflow_dispatch. Surfaces findings as code-scanning alerts plus a single maintainer-ping comment per head SHA when high-severity / workflow-tamper / supply-chain hits. Never executes PR head code. - docs/design/pr-triage-workflows.md replaced with a brief overview + diagram. The full implementation plan is kept locally as docs/design/pr-triage-workflows-plan.md (gitignored). - pr-triage-batch.yml's existing dispatch-scanner branch now resolves to the new scanner; orchestrator unchanged. * Fix markdownlint MD038 (pipe inside code span) in malicious-scan agent |
||
|
Jan Krivanek
|
19f024ba2d |
Redirect session-data reads/writes to dotnet/skills-data repo (#717)
The dashboard-session-data branch on this repo had grown to ~480MB and was bloating clones. Session data has been migrated to the standalone dotnet/skills-data repository (branch of the same name). This PR redirects: - dashboard.js -> reads manifest from dotnet/skills-data - evaluation.yml PR-comment link -> points at dotnet/skills-data - evaluation.yml publish-session-data job -> clones/pushes to dotnet/skills-data using a new SKILLS_DATA_TOKEN secret (fine-grained PAT with contents:write on the data repo) Also adds --depth 1 to the deploy clone so future runs do not re-import history. AGENTVIZ replay UI requires no changes; it accepts an arbitrary manifest URL via query string. raw.githubusercontent.com serves the new repo with Access-Control-Allow-Origin: * so the dotnet.github.io/skills/replay page can load it cross-origin. The old branch on dotnet/skills is left in place for now and will be deleted manually after one successful run end-to-end. |
||
|
Jan Krivanek
|
510cbe1a40 | Adjust issues triage workflow (#714) |