mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Fix skill activation for polyglot scenarios in PR #709
Both polyglot scenarios (pytest + Flask, Vitest + TS) scored 5.0/5 quality but reported `⚠️ NOT ACTIVATED` because the SDK's skill router did not load `code-testing-agent` into the agent's context: `activated: false, detectedSkills: []` in skillActivationIsolated / skillActivationPlugin. The .NET ContosoUniversity scenario activates fine because its prompt has stronger `project-wide, multi-file ... high coverage` framing and SKILL.md already covers .NET keywords. Fixes per `eng/skill-validator/src/docs/InvestigatingResults.md` section "Skill not activated": - SKILL.md description: add framework-specific keywords (pytest, Flask/Django, Vitest, Jest, Mocha, JUnit, Node libraries, API, package, project-wide, multi-file) so the router has explicit hooks for polyglot prompts. Trimmed verbose `DO NOT USE FOR` clauses to stay under the 1,024-char skill spec limit (now 1,019 chars). - eval.yaml + eval.vally.yaml: rewrite the two polyglot prompts to mirror the .NET scenario's framing — add `project-wide, multi-file test generation task across the ... layers` and `achieve high coverage`, soften the library-prescriptive bullets (Mock(spec=...), vi.fn()) into capability-level requirements (`mocked or stubbed`, `mock or hand-written stub`). Rubric items unchanged — they remain flexible enough to grade either mock style. Validated locally: `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` => ✅ All checks passed (23 skill(s), 11 agent(s), 1 plugin(s)). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Amaury Levé
@@ -1,20 +1,19 @@
|
||||
---
|
||||
name: code-testing-agent
|
||||
description: >-
|
||||
Generates and writes new unit tests for any programming language using a
|
||||
Research-Plan-Implement pipeline. Use when asked to generate tests,
|
||||
write unit tests, add tests, improve test coverage, create test
|
||||
project, achieve high coverage, comprehensive tests, or asked to
|
||||
scaffold a new test project for an app, service, or library. Supports
|
||||
C#, TypeScript, JavaScript, Python, Go, Rust, Java, and more. Orchestrates
|
||||
the code-testing-generator sub-agent through research, planning, and
|
||||
implementation phases so tests compile, pass, and follow project
|
||||
conventions. DO NOT USE FOR: running existing tests or test filters
|
||||
(use run-tests); diagnosing coverage plateaus or project-wide
|
||||
coverage/CRAP analysis without writing tests (use coverage-analysis);
|
||||
targeted method/class CRAP scores (use crap-score); MSTest assertion
|
||||
guidance, MSTest test pattern modernization, or fixing existing MSTest test
|
||||
code (use writing-mstest-tests).
|
||||
Generates and writes new unit tests for any programming language. Use when
|
||||
asked to generate tests, write unit tests, add tests, improve test
|
||||
coverage, create test project, achieve high coverage, comprehensive
|
||||
tests, or scaffold a new test project for an app, service, library,
|
||||
API, or package — including project-wide or multi-file test
|
||||
generation tasks. Supports C#/.NET, Python (pytest, Flask/Django apps),
|
||||
TypeScript/JavaScript (Vitest, Jest, Mocha, Node libraries), Go,
|
||||
Rust, Java (JUnit), and more. Orchestrates the code-testing-generator
|
||||
sub-agent through research, planning, and implementation phases so
|
||||
tests compile, pass, and follow project conventions. DO NOT USE FOR:
|
||||
running existing tests (use run-tests); coverage/CRAP analysis without
|
||||
writing tests (use coverage-analysis or crap-score); MSTest pattern
|
||||
modernization or fixing existing MSTest code (use writing-mstest-tests).
|
||||
license: MIT
|
||||
---
|
||||
|
||||
|
||||
@@ -48,22 +48,29 @@ stimuli:
|
||||
|
||||
- name: Generate pytest tests for the Flask tasks API (Python polyglot)
|
||||
prompt: |
|
||||
I have a small Python Flask application under fixtures/python-flask-tasks/.
|
||||
It exposes a /tasks REST API backed by a TaskService and an in-memory
|
||||
repository (see fixtures/python-flask-tasks/README.md for the layout).
|
||||
There are no tests yet — the tests/ directory is empty.
|
||||
I have a Python Flask web application under fixtures/python-flask-tasks/
|
||||
— a small /tasks REST API with a service layer (TaskService with an
|
||||
injected `now` clock), an in-memory repository implementing a Protocol,
|
||||
and a Flask blueprint exposing the HTTP surface (see
|
||||
fixtures/python-flask-tasks/README.md for the layout). There are no
|
||||
tests yet — the tests/ directory is empty. This is a project-wide,
|
||||
multi-file test generation task across the service, repository, and
|
||||
routes layers.
|
||||
|
||||
Please generate a comprehensive pytest suite under
|
||||
fixtures/python-flask-tasks/tests/ that:
|
||||
- Unit-tests TaskService with the repository mocked
|
||||
(unittest.mock.Mock(spec=TaskRepository)) and an injected `now`
|
||||
callable, covering happy path, validation errors (empty title,
|
||||
title >200 chars), not-found, and already-done.
|
||||
- Integration-tests the Flask blueprint using
|
||||
`create_app(service=...).test_client()` — no real network ports.
|
||||
- Asserts both HTTP status codes and JSON response bodies for the
|
||||
POST /tasks, GET /tasks, GET /tasks/<id>, and POST /tasks/<id>/complete
|
||||
endpoints, plus 400/404/409 error cases.
|
||||
Please scaffold a comprehensive pytest suite under
|
||||
fixtures/python-flask-tasks/tests/ and achieve high code coverage
|
||||
across the tasks_api package. The suite should:
|
||||
- Unit-test TaskService with the repository mocked or stubbed (not
|
||||
the real in-memory implementation), and an injected `now`
|
||||
callable so completion timestamps are deterministic. Cover the
|
||||
happy path, validation errors (empty title, oversized title),
|
||||
not-found, and already-done cases.
|
||||
- Integration-test the Flask blueprint in-process (Flask exposes a
|
||||
test client for this — no real network ports).
|
||||
- Assert both HTTP status codes and JSON response bodies for the
|
||||
POST /tasks, GET /tasks, GET /tasks/<id>, and POST
|
||||
/tasks/<id>/complete endpoints, including 400/404/409 error
|
||||
cases.
|
||||
|
||||
Tests should pass with: `python -m pip install -e ".[test]"` then
|
||||
`python -m pytest` from the fixtures/python-flask-tasks/ directory.
|
||||
@@ -95,24 +102,27 @@ stimuli:
|
||||
|
||||
- name: Generate Vitest tests for the shopping-cart library (TypeScript polyglot)
|
||||
prompt: |
|
||||
I have a small TypeScript shopping-cart library under
|
||||
fixtures/typescript-vitest-cart/ (see its README.md for the layout). It
|
||||
exposes a Cart class with an injectable DiscountPolicy seam, plus
|
||||
NoDiscountPolicy and PercentageDiscountPolicy implementations. There are
|
||||
no tests yet — the tests/ directory is empty.
|
||||
I have a TypeScript shopping-cart library under
|
||||
fixtures/typescript-vitest-cart/ — a small package with a Cart class,
|
||||
an injectable DiscountPolicy seam, and NoDiscountPolicy /
|
||||
PercentageDiscountPolicy implementations (see its README.md for the
|
||||
layout). There are no tests yet — the tests/ directory is empty. This
|
||||
is a project-wide, multi-file test generation task across the cart
|
||||
and pricing modules.
|
||||
|
||||
Please generate a comprehensive Vitest suite under
|
||||
Please scaffold a comprehensive Vitest suite under
|
||||
fixtures/typescript-vitest-cart/tests/ (one or more `*.test.ts` files)
|
||||
that:
|
||||
- Mocks DiscountPolicy with `vi.fn()` (or a hand-rolled stub
|
||||
implementing the interface) when testing Cart, to isolate Cart from
|
||||
its discount collaborator.
|
||||
- Covers Cart.add merge semantics for repeated product ids, add's
|
||||
and achieve high code coverage across the src/ modules. The suite
|
||||
should:
|
||||
- Use a mock or hand-written stub for DiscountPolicy when testing
|
||||
Cart, so Cart is exercised in isolation from its discount
|
||||
collaborator rather than only through the real policy classes.
|
||||
- Cover Cart.add merge semantics for repeated product ids, add's
|
||||
guard against non-positive / non-integer quantities, updateQuantity
|
||||
(including 0 removing the line and unknown-id throwing), remove
|
||||
returning false for unknown ids, and totals() clamping the discount
|
||||
when the policy returns a value greater than the subtotal.
|
||||
- Includes boundary tests for PercentageDiscountPolicy's constructor
|
||||
- Include boundary tests for PercentageDiscountPolicy's constructor
|
||||
(rejects out-of-range percent) and computeDiscountCents (rounds
|
||||
down, returns 0 for non-positive subtotals).
|
||||
|
||||
|
||||
@@ -52,22 +52,29 @@ scenarios:
|
||||
|
||||
- name: "Generate pytest tests for the Flask tasks API (Python polyglot)"
|
||||
prompt: |
|
||||
I have a small Python Flask application under fixtures/python-flask-tasks/.
|
||||
It exposes a /tasks REST API backed by a TaskService and an in-memory
|
||||
repository (see fixtures/python-flask-tasks/README.md for the layout).
|
||||
There are no tests yet — the tests/ directory is empty.
|
||||
I have a Python Flask web application under fixtures/python-flask-tasks/
|
||||
— a small /tasks REST API with a service layer (TaskService with an
|
||||
injected `now` clock), an in-memory repository implementing a Protocol,
|
||||
and a Flask blueprint exposing the HTTP surface (see
|
||||
fixtures/python-flask-tasks/README.md for the layout). There are no
|
||||
tests yet — the tests/ directory is empty. This is a project-wide,
|
||||
multi-file test generation task across the service, repository, and
|
||||
routes layers.
|
||||
|
||||
Please generate a comprehensive pytest suite under
|
||||
fixtures/python-flask-tasks/tests/ that:
|
||||
- Unit-tests TaskService with the repository mocked
|
||||
(unittest.mock.Mock(spec=TaskRepository)) and an injected `now`
|
||||
callable, covering happy path, validation errors (empty title,
|
||||
title >200 chars), not-found, and already-done.
|
||||
- Integration-tests the Flask blueprint using
|
||||
`create_app(service=...).test_client()` — no real network ports.
|
||||
- Asserts both HTTP status codes and JSON response bodies for the
|
||||
POST /tasks, GET /tasks, GET /tasks/<id>, and POST /tasks/<id>/complete
|
||||
endpoints, plus 400/404/409 error cases.
|
||||
Please scaffold a comprehensive pytest suite under
|
||||
fixtures/python-flask-tasks/tests/ and achieve high code coverage
|
||||
across the tasks_api package. The suite should:
|
||||
- Unit-test TaskService with the repository mocked or stubbed (not
|
||||
the real in-memory implementation), and an injected `now`
|
||||
callable so completion timestamps are deterministic. Cover the
|
||||
happy path, validation errors (empty title, oversized title),
|
||||
not-found, and already-done cases.
|
||||
- Integration-test the Flask blueprint in-process (Flask exposes a
|
||||
test client for this — no real network ports).
|
||||
- Assert both HTTP status codes and JSON response bodies for the
|
||||
POST /tasks, GET /tasks, GET /tasks/<id>, and POST
|
||||
/tasks/<id>/complete endpoints, including 400/404/409 error
|
||||
cases.
|
||||
|
||||
Tests should pass with: `python -m pip install -e ".[test]"` then
|
||||
`python -m pytest` from the fixtures/python-flask-tasks/ directory.
|
||||
@@ -101,24 +108,27 @@ scenarios:
|
||||
|
||||
- name: "Generate Vitest tests for the shopping-cart library (TypeScript polyglot)"
|
||||
prompt: |
|
||||
I have a small TypeScript shopping-cart library under
|
||||
fixtures/typescript-vitest-cart/ (see its README.md for the layout). It
|
||||
exposes a Cart class with an injectable DiscountPolicy seam, plus
|
||||
NoDiscountPolicy and PercentageDiscountPolicy implementations. There are
|
||||
no tests yet — the tests/ directory is empty.
|
||||
I have a TypeScript shopping-cart library under
|
||||
fixtures/typescript-vitest-cart/ — a small package with a Cart class,
|
||||
an injectable DiscountPolicy seam, and NoDiscountPolicy /
|
||||
PercentageDiscountPolicy implementations (see its README.md for the
|
||||
layout). There are no tests yet — the tests/ directory is empty. This
|
||||
is a project-wide, multi-file test generation task across the cart
|
||||
and pricing modules.
|
||||
|
||||
Please generate a comprehensive Vitest suite under
|
||||
Please scaffold a comprehensive Vitest suite under
|
||||
fixtures/typescript-vitest-cart/tests/ (one or more `*.test.ts` files)
|
||||
that:
|
||||
- Mocks DiscountPolicy with `vi.fn()` (or a hand-rolled stub
|
||||
implementing the interface) when testing Cart, to isolate Cart from
|
||||
its discount collaborator.
|
||||
- Covers Cart.add merge semantics for repeated product ids, add's
|
||||
and achieve high code coverage across the src/ modules. The suite
|
||||
should:
|
||||
- Use a mock or hand-written stub for DiscountPolicy when testing
|
||||
Cart, so Cart is exercised in isolation from its discount
|
||||
collaborator rather than only through the real policy classes.
|
||||
- Cover Cart.add merge semantics for repeated product ids, add's
|
||||
guard against non-positive / non-integer quantities, updateQuantity
|
||||
(including 0 removing the line and unknown-id throwing), remove
|
||||
returning false for unknown ids, and totals() clamping the discount
|
||||
when the policy returns a value greater than the subtotal.
|
||||
- Includes boundary tests for PercentageDiscountPolicy's constructor
|
||||
- Include boundary tests for PercentageDiscountPolicy's constructor
|
||||
(rejects out-of-range percent) and computeDiscountCents (rounds
|
||||
down, returns 0 for non-positive subtotals).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user