mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Keep test agent state out of commits (#1108)
* Keep test agent state out of commits Move broad test-generation pipeline state to host scratch storage, worktree-specific Git metadata, or OS temp, and enforce the exclusion in evals. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Clarify absolute test agent state path Use Git's explicit absolute path formatting in both test-generation entry points. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Prune Git metadata from test agent eval guards Avoid scanning nested repositories and align the remaining TESTAGENT_DIR placeholder with the documented format. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Clarify test agent command handoff Require callers to provide exact commands, excerpts, or absolute TESTAGENT_DIR document paths to command-running sub-agents. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Reject all repository-local testagent entries Match .testagent by name regardless of whether it is a directory, file, or symlink while continuing to prune Git metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Verify external test agent artifacts Restore broad-run artifact checks at the Git metadata path and pass the researched lint command and state directory to the linter agent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Make testagent eval guards constant time Check only the forbidden workspace-root path, including broken symlinks, instead of recursively traversing dependency trees. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Broaden comprehensive test generation Treat explicit requirements as the floor for broad suites and add mutation-relevant equivalence-partition and invariant coverage without test-count padding. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Fix external artifact grader quoting Run state checks directly in the harness shell so TESTAGENT_DIR expands after assignment, with an isolated command probe covering valid and forbidden states. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Run broad skill evals in Git worktrees Initialize the seven broad evaluation roots as Git repositories so TESTAGENT_DIR resolves deterministically and external artifacts remain verifiable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Clarify non-stageable test agent state Describe the real invariant across the pipeline: state may live under .git metadata but must never be version-controlled workspace content or appear in git status. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Standardize intermediate test state contract Use one TESTAGENT_DIR placeholder, clearer intermediate-state terminology, and detect stageable research, plan, or status files regardless of directory name. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Use one Git root in workspace integrity eval Baseline the fixture from the evaluation root so stageable intermediate-state files remain visible to the directory-independent guard. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Prune Vitest dependencies from state scan Exclude node_modules through per-eval Git metadata so stageable state detection remains fast without modifying fixture content. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Strengthen focused intermediate-state guards Separate shell execution, reject Git-metadata files on focused runs, include ignored state files, and prune node_modules with a pathspec exclusion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 --------- Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
This commit is contained in:
Amaury Levé
@@ -26,7 +26,9 @@ Run the appropriate build command and report success or failure with error detai
|
||||
|
||||
If not provided, check in order:
|
||||
|
||||
1. `.testagent/research.md` or `.testagent/plan.md` for Commands section
|
||||
1. The exact command or relevant Commands excerpt supplied by the caller; if
|
||||
the caller instead supplies a document, it must provide its absolute
|
||||
`<TESTAGENT_DIR>/research.md` or `<TESTAGENT_DIR>/plan.md` path
|
||||
2. Project files:
|
||||
- SDK-style `*.csproj` / `*.sln` → `dotnet build`
|
||||
- Classic non-SDK `*.csproj` / `*.sln` → repository-documented MSBuild command
|
||||
|
||||
@@ -35,7 +35,24 @@ You coordinate test generation using the Research-Plan-Implement (RPI) pipeline.
|
||||
|
||||
Understand what the user wants: scope (project, files, classes), priority areas, framework preferences. If clear, proceed directly. If the user provides no details or a very basic prompt (e.g., "generate tests"), use [unit-test-generation.prompt.md](../skills/code-testing-agent/unit-test-generation.prompt.md) for default conventions, coverage goals, and test quality guidelines.
|
||||
|
||||
Before writing code, read the language-specific base extension. Reuse it for the whole run; sub-agents must not independently reload the same reference unless they need a section that was not captured in `.testagent/research.md`.
|
||||
Before writing code, read the language-specific base extension. Reuse it for the whole run; sub-agents must not independently reload the same reference unless they need a section that was not captured in the research document.
|
||||
|
||||
For Single pass and Iterative strategies, resolve one absolute
|
||||
`<TESTAGENT_DIR>`
|
||||
before invoking any sub-agent:
|
||||
|
||||
1. Prefer a host-provided session artifact or scratch directory when one is
|
||||
available.
|
||||
2. Otherwise, in a Git worktree run
|
||||
`git rev-parse --path-format=absolute --git-path testagent`. This returns a
|
||||
path in worktree-specific Git metadata, which cannot be staged or committed.
|
||||
3. Outside Git, create a unique directory under the operating system's
|
||||
temporary directory.
|
||||
|
||||
Create the resolved directory and pass its absolute path explicitly in every
|
||||
sub-agent prompt. Never create `<TESTAGENT_DIR>` or any intermediate state file
|
||||
in version-controlled workspace content, and never modify `.gitignore` to hide
|
||||
them.
|
||||
|
||||
Create a **requirement checklist** from the request before choosing a strategy.
|
||||
Preserve each explicit behavior, layer, collaborator seam, boundary case,
|
||||
@@ -43,7 +60,7 @@ integration, coverage threshold, and required artifact as a separate item. For
|
||||
example, "mock the repository in service tests", "exercise SQLite in memory",
|
||||
and "cover pagination boundaries" are three independently verifiable
|
||||
requirements. Direct strategy keeps this checklist in context; delegated
|
||||
strategies record it in `.testagent/research.md`.
|
||||
strategies record it in `<TESTAGENT_DIR>/research.md`.
|
||||
|
||||
### Step 2: Choose Execution Strategy
|
||||
|
||||
@@ -53,7 +70,7 @@ Based on the request scope, pick exactly one strategy and follow it:
|
||||
| ---------- | ------------- | ------------ |
|
||||
| **Direct** | A small, self-contained request (e.g., tests for a single function or class) that you can complete without sub-agents | Follow the codebase conventions on test file structure, naming, style, and testing approaches. Reuse existing test projects and test files when possible — if the code under test already has tests, add new tests to the same file or test project. Only create a new test file when no canonical file is named or discoverable for the symbol under test. Write the tests immediately. **Run them right away** — if any test fails, read the production code, fix the assertion, and re-run before writing more tests. Skip Steps 3-5 (research, plan, implement sub-agents). Then proceed to Steps 6-9 for validation and reporting — **Direct skips only the sub-agents, never the Step 7 pre-completion gate** (which still runs per its own threshold in Step 7 — i.e. for any non-trivial addition: ≥5 tests, or any request that enumerates behaviors/scenarios to verify). |
|
||||
| **Single pass** | A moderate scope (couple projects or modules) that a single Research → Plan → Implement cycle can cover | Execute Steps 3-8 once, then proceed to Step 9. |
|
||||
| **Iterative** | A large scope or ambitious coverage target that one pass cannot satisfy | Execute Steps 3-8, then re-evaluate coverage. If the target is not met, repeat Steps 3-8 with a narrowed focus on remaining gaps. Use unique names for each iteration's `.testagent/` documents (e.g., `research-2.md`, `plan-2.md`) so earlier results are not overwritten. Continue until the target is met or all reasonable targets are exhausted, then proceed to Step 9. |
|
||||
| **Iterative** | A large scope or ambitious coverage target that one pass cannot satisfy | Execute Steps 3-8, then re-evaluate coverage. If the target is not met, repeat Steps 3-8 with a narrowed focus on remaining gaps. Use unique names for each iteration's documents in `<TESTAGENT_DIR>` (e.g., `research-2.md`, `plan-2.md`) so earlier results are not overwritten. Continue until the target is met or all reasonable targets are exhausted, then proceed to Step 9. |
|
||||
|
||||
**Default to Direct** unless the user asks for a project/package-wide suite or
|
||||
the scope explicitly spans multiple files or modules. Most test generation
|
||||
@@ -87,25 +104,25 @@ Delegate to the `code-testing-researcher` subagent with this task:
|
||||
```text
|
||||
runSubagent({
|
||||
agent: "code-testing-researcher",
|
||||
prompt: "Research [REQUESTED SCOPE] at [PATH] for test generation. Produce a bounded target inventory, existing test conventions, source-to-test pairs, dependencies only for those targets, and exact build/test/discovery commands. Do not inventory unrelated source files."
|
||||
prompt: "Research [REQUESTED SCOPE] at [PATH] for test generation. Write the research document to <TESTAGENT_DIR>/research.md. Produce a bounded target inventory, existing test conventions, source-to-test pairs, dependencies only for those targets, and exact build/test/discovery commands. Do not inventory unrelated source files."
|
||||
})
|
||||
```
|
||||
|
||||
Output: `.testagent/research.md`
|
||||
Output: `<TESTAGENT_DIR>/research.md`
|
||||
|
||||
### Step 4: Planning Phase
|
||||
|
||||
Delegate to the `code-testing-planner` subagent with this task:
|
||||
|
||||
> Create a test implementation plan based on .testagent/research.md. Create phased approach with specific files and test cases.
|
||||
> Create a test implementation plan based on `<TESTAGENT_DIR>/research.md`. Write it to `<TESTAGENT_DIR>/plan.md`. Create a phased approach with specific files and test cases.
|
||||
|
||||
Output: `.testagent/plan.md`
|
||||
Output: `<TESTAGENT_DIR>/plan.md`
|
||||
|
||||
### Step 5: Implementation Phase
|
||||
|
||||
Execute each phase by delegating to the `code-testing-implementer` subagent — once per phase, sequentially. For each phase, delegate with this task:
|
||||
|
||||
> Implement Phase N from .testagent/plan.md: [phase description]. Ensure tests compile and pass.
|
||||
> Implement Phase N from `<TESTAGENT_DIR>/plan.md`: [phase description]. Use `<TESTAGENT_DIR>/research.md` for commands and conventions. Ensure tests compile and pass.
|
||||
|
||||
### Step 6: Final Build Validation
|
||||
|
||||
@@ -152,19 +169,29 @@ Additional self-review heuristics (still required, even when running the skills)
|
||||
|
||||
### Step 8: Coverage Gap Iteration
|
||||
|
||||
After the previous phases complete, use the target inventory already recorded in `.testagent/research.md` and the files reported by implementers. Do not rescan or reread the workspace.
|
||||
After the previous phases complete, use the target inventory already recorded in `<TESTAGENT_DIR>/research.md` and the files reported by implementers. Do not rescan or reread the workspace.
|
||||
|
||||
1. Compare the requirement checklist and bounded target inventory with the implemented tests.
|
||||
2. Inspect the generated test bodies for evidence of every checklist item. A covered line does not prove that a requested collaborator was mocked, a concrete result was asserted, or a boundary/property combination was exercised.
|
||||
3. If the user requested a measurable coverage target, collect coverage once and prioritize only gaps inside the requested scope.
|
||||
4. Add tests for any unaddressed checklist item before adding optional cases merely to raise test count.
|
||||
5. Stop only when every feasible checklist item is covered and the stated target is met; do not recursively expand into unrelated files.
|
||||
6. If this step added or modified tests, re-run the full Step 7 pre-completion gate (`test-gap-analysis` + `assertion-quality` + prompt-scenario coverage) on those tests before reporting completion.
|
||||
4. Add tests for any unaddressed checklist item first.
|
||||
5. For Single pass and Iterative strategies, treat that checklist as the floor.
|
||||
Sweep each bounded target API for still-unproved observable equivalence
|
||||
partitions and invariants: identity/empty/singleton/interior inputs, exact
|
||||
and immediately adjacent boundaries, invalid partitions, and ordering,
|
||||
monotonicity, rollover, capacity, truncation, or state properties implied by
|
||||
the implementation. Add one mutation-relevant case per distinct partition;
|
||||
consolidate sibling inputs in parameterized or table-driven tests.
|
||||
6. Stop only when every feasible checklist item and distinct behavioral
|
||||
partition is covered and the stated target is met. Do not recursively expand
|
||||
into unrelated files or add equivalent cases merely to raise test count.
|
||||
7. If this step added or modified tests, re-run the full Step 7 pre-completion gate (`test-gap-analysis` + `assertion-quality` + prompt-scenario coverage) on those tests before reporting completion.
|
||||
|
||||
For Single pass and Iterative strategies, write `.testagent/status.md` after
|
||||
For Single pass and Iterative strategies, write `<TESTAGENT_DIR>/status.md` after
|
||||
the final review and validation. Record the completed checklist, commands and
|
||||
results, quality findings, fixes, and any explicit blockers. Direct strategy
|
||||
keeps this evidence in the final response and must not create `.testagent/`.
|
||||
keeps this evidence in the final response and must not create intermediate
|
||||
state files.
|
||||
|
||||
### Step 9: Report Results
|
||||
|
||||
@@ -208,11 +235,12 @@ Use a language example from `code-testing-extensions` only when no existing test
|
||||
|
||||
## State Management
|
||||
|
||||
All state is stored in `.testagent/` folder:
|
||||
All delegated intermediate state files are stored in the resolved,
|
||||
non-stageable `<TESTAGENT_DIR>`:
|
||||
|
||||
- `.testagent/research.md` — Research findings
|
||||
- `.testagent/plan.md` — Implementation plan
|
||||
- `.testagent/status.md` — Final quality review, fixes, and validation status
|
||||
- `<TESTAGENT_DIR>/research.md` — Research findings
|
||||
- `<TESTAGENT_DIR>/plan.md` — Implementation plan
|
||||
- `<TESTAGENT_DIR>/status.md` — Final quality review, fixes, and validation status
|
||||
|
||||
## Rules
|
||||
|
||||
@@ -224,9 +252,9 @@ All state is stored in `.testagent/` folder:
|
||||
6. **Scoped builds during phases, full build at the end** — build specific test projects during implementation for speed; run a full-workspace non-incremental build after all phases to catch cross-project errors
|
||||
7. **No environment-dependent tests** — mock all external dependencies; never call external URLs, bind ports, or depend on timing
|
||||
8. **Fix assertions, don't skip tests** — when tests fail, read production code and fix the expected value; never `[Ignore]` or `[Skip]`
|
||||
9. **Retain `.testagent/` through completion** — keep the research, plan, and final status available as auditable pipeline evidence. Do not delete them automatically; if the repository should not commit agent state, advise the user to add `.testagent/` to `.gitignore` after reporting the result.
|
||||
9. **Keep intermediate state files out of commits** — retain research, plan, and final status in `<TESTAGENT_DIR>` through completion, but never place `<TESTAGENT_DIR>` or its files in version-controlled workspace content, stage them, or modify `.gitignore` to hide them. Before reporting, inspect the working-tree changes and confirm they contain only requested deliverables and required manifest edits.
|
||||
10. **Read language extensions first** — always call the `code-testing-extensions` skill and read the relevant extension file before writing any code; it contains critical project registration and build validation steps
|
||||
11. **Always validate** — final build, final test, coverage-gap review, and reporting are mandatory for ALL strategies including Direct; never skip final validation. The pre-completion self-review gate from Step 7 (`test-gap-analysis` + `assertion-quality` skills, plus the prompt-scenario coverage check) is mandatory for every non-trivial test addition and may be skipped only for trivially small tasks (fewer than 5 generated tests *and* no behaviors specified in the prompt), per Step 7
|
||||
12. **Preserve existing tests** — never delete or overwrite existing test files; create new files or append to existing ones
|
||||
13. **Never mutate version control** — your only outputs are additive test files plus minimal build-manifest edits to register a new test project. Any command that reverts, restores, resets, stashes, or cleans the tree, or deletes tracked files, is out of scope — even when the workspace looks broken or incomplete.
|
||||
14. **Bound context and reuse findings** — scope every search to the user's requested files/modules, read only the source and existing tests needed for the next implementation phase, and reuse `.testagent/research.md` instead of repeating workspace discovery.
|
||||
14. **Bound context and reuse findings** — scope every search to the user's requested files/modules, read only the source and existing tests needed for the next implementation phase, and reuse `<TESTAGENT_DIR>/research.md` instead of repeating workspace discovery.
|
||||
|
||||
@@ -30,8 +30,10 @@ Given a phase from the plan, write all the test files for that phase and ensure
|
||||
|
||||
### 1. Read the Plan and Research
|
||||
|
||||
- Read only the current phase from `.testagent/plan.md`
|
||||
- Read the command, convention, and target entries needed for that phase from `.testagent/research.md`
|
||||
- Read only the current phase from the caller-provided absolute
|
||||
`<TESTAGENT_DIR>/plan.md` path
|
||||
- Read the command, convention, and target entries needed for that phase from
|
||||
`<TESTAGENT_DIR>/research.md`
|
||||
- Identify which phase you're implementing
|
||||
|
||||
### 2. Read Source Files and Validate References
|
||||
@@ -62,6 +64,10 @@ For each test file in your phase:
|
||||
- Create the test file with appropriate structure
|
||||
- Follow the project's testing patterns
|
||||
- Include tests for: happy path, edge cases (empty, null, boundary), error conditions
|
||||
- Treat the plan's explicit requirements as a floor. For broad/comprehensive
|
||||
scope, add one mutation-relevant case for every distinct observable
|
||||
equivalence partition or invariant in the target implementation that the plan
|
||||
missed; parameterize sibling inputs instead of duplicating test structure.
|
||||
- Mock all external dependencies — never call external URLs, bind ports, or depend on timing
|
||||
|
||||
#### Edit boundaries (cross-language invariants)
|
||||
@@ -69,7 +75,11 @@ For each test file in your phase:
|
||||
These rules apply to every language and override any pattern an existing test file may suggest. They keep generated changes additive so reviewers, CI gates, and test-quality benchmarks treat your output as a clean test addition rather than a refactor:
|
||||
|
||||
- **Existing test files are append-only.** When growing an existing test file, insert new test methods/cases at the end of the relevant class/describe-block/module. Do not reformat, reorder, rename, or remove any existing line — even whitespace-only churn counts as a destructive edit.
|
||||
- **Do not modify non-test source files.** If a class, method, or symbol is hard to test (sealed, internal, no seam, tightly coupled), record the gap in `.testagent/plan.md` as a follow-up. Do not edit production code to make it testable as part of test generation — that is the scope of the `testability-migration` agent, not this one.
|
||||
- **Do not modify non-test source files.** If a class, method, or symbol is hard to test (sealed, internal, no seam, tightly coupled), record the gap in `<TESTAGENT_DIR>/plan.md` as a follow-up. Do not edit production code to make it testable as part of test generation — that is the scope of the `testability-migration` agent, not this one.
|
||||
- **Keep intermediate state files non-stageable.** Use only the absolute
|
||||
`<TESTAGENT_DIR>` supplied by the caller for research, plan, or status
|
||||
updates. Never place `<TESTAGENT_DIR>` or its files in version-controlled
|
||||
workspace content or stage them.
|
||||
- **Never revert or clean the working tree.** Do not run `git checkout`, `git restore`, `git reset`, `git clean`, `git stash`, `git rm`, or delete tracked files. Generate tests against the workspace exactly as delivered, even if the source looks synthetic, deleted, gutted, or incomplete — that state is intentional, not corruption.
|
||||
- **Prefer new test files over edits to existing ones** when both options are equally valid (e.g., a new feature, a separate concern, or any case where the existing file isn't strictly required). A new file is always purely additive.
|
||||
- **One exception**: build-system manifests (`.csproj`/`.sln`/`packages.config`/`pom.xml`/`build.gradle`/`Cargo.toml`/`package.json`/etc.) may be edited when registering a new test file/project or adding a missing test dependency. Keep these edits minimal and limited to the registration/dependency change. Never convert `packages.config` to `PackageReference`, convert a classic project to SDK style, or upgrade the test stack unless the user explicitly requested that migration.
|
||||
@@ -80,13 +90,16 @@ Coverage alone gives false confidence — every test must *pin down behavior* so
|
||||
|
||||
### 5. Verify with Build
|
||||
|
||||
Call the `code-testing-builder` sub-agent to compile. Build only the specific test project, not the full solution.
|
||||
Call the `code-testing-builder` sub-agent to compile, passing the exact build
|
||||
command and absolute `<TESTAGENT_DIR>`. Build only the specific test project,
|
||||
not the full solution.
|
||||
|
||||
If build fails: call `code-testing-fixer`, rebuild, retry up to 3 times.
|
||||
|
||||
### 6. Verify with Tests
|
||||
|
||||
Call the `code-testing-tester` sub-agent to run tests.
|
||||
Call the `code-testing-tester` sub-agent to run tests, passing the exact test
|
||||
command and absolute `<TESTAGENT_DIR>`.
|
||||
|
||||
If tests fail:
|
||||
|
||||
@@ -115,7 +128,8 @@ If your language extension has no "Harness Discovery Check" section, use the can
|
||||
|
||||
### 8. Format Code (Optional)
|
||||
|
||||
If a lint command is available, call the `code-testing-linter` sub-agent.
|
||||
If a lint command is available, call the `code-testing-linter` sub-agent,
|
||||
passing the exact lint command and absolute `<TESTAGENT_DIR>`.
|
||||
|
||||
### 9. Report Results
|
||||
|
||||
|
||||
@@ -24,7 +24,9 @@ Run the appropriate lint/format command to fix code style issues.
|
||||
|
||||
If not provided, check in order:
|
||||
|
||||
1. `.testagent/research.md` or `.testagent/plan.md` for Commands section
|
||||
1. The exact command or relevant Commands excerpt supplied by the caller; if
|
||||
the caller instead supplies a document, it must provide its absolute
|
||||
`<TESTAGENT_DIR>/research.md` or `<TESTAGENT_DIR>/plan.md` path
|
||||
2. Project files:
|
||||
- `*.csproj` / `*.sln` → `dotnet format`
|
||||
- `package.json` → `npm run lint:fix` or `npm run format`
|
||||
|
||||
@@ -3,7 +3,7 @@ description: >-
|
||||
Creates structured test implementation plans from research findings.
|
||||
|
||||
Use when: organizing tests into phases, prioritizing test generation,
|
||||
creating .testagent/plan.md from research.
|
||||
creating the pipeline plan document from research.
|
||||
name: code-testing-planner
|
||||
user-invocable: false
|
||||
tools: ["skill", "read", "search", "edit", "execute", "Skill", "Read", "Glob", "Grep", "Edit", "Write", "Bash", "read_file", "replace", "write_file", "glob", "grep_search", "run_shell_command"]
|
||||
@@ -22,7 +22,9 @@ Read the research document and create a phased implementation plan that will gui
|
||||
|
||||
### 1. Read the Research
|
||||
|
||||
Read the target inventory, command section, dependency summary, and testing conventions from `.testagent/research.md`. Do not reread repository files during planning.
|
||||
Read the target inventory, command section, dependency summary, and testing
|
||||
conventions from the absolute `<TESTAGENT_DIR>/research.md` path provided by the
|
||||
caller. Do not reread repository files during planning.
|
||||
|
||||
- Project structure and language
|
||||
- Files that need tests
|
||||
@@ -68,12 +70,17 @@ For each file in each phase, specify:
|
||||
- Test class/module name
|
||||
- Methods/functions to test
|
||||
- Key test scenarios (happy path, edge cases, errors)
|
||||
- For broad/comprehensive scope, one mutation-relevant case for each observable
|
||||
equivalence partition or invariant discovered in the source, including useful
|
||||
identity/empty/singleton/interior cases, exact and adjacent boundaries, and
|
||||
ordering, rollover, capacity, truncation, or state properties not named by
|
||||
the prompt. Group sibling inputs in parameterized or table-driven tests.
|
||||
|
||||
**Important**: When adding new tests, they MUST go into the existing test project that already tests the target code. Do not create a separate test project unnecessarily. If no existing test project covers the target, create a new one.
|
||||
|
||||
### 5. Generate Plan Document
|
||||
|
||||
Create `.testagent/plan.md` with this structure:
|
||||
Create `<TESTAGENT_DIR>/plan.md` with this structure:
|
||||
|
||||
```markdown
|
||||
# Test Implementation Plan
|
||||
@@ -139,4 +146,7 @@ Only consult a language example when research found no existing tests and the ba
|
||||
|
||||
## Output
|
||||
|
||||
Write the plan document to `.testagent/plan.md` in the workspace root.
|
||||
Write the plan document to the absolute `<TESTAGENT_DIR>/plan.md` path provided
|
||||
by the caller. `<TESTAGENT_DIR>` must be non-stageable host scratch storage,
|
||||
Git metadata, or OS temp. Never place it or its files in version-controlled
|
||||
workspace content.
|
||||
|
||||
@@ -3,7 +3,7 @@ description: >-
|
||||
Analyzes codebases to understand structure, testing patterns, and testability.
|
||||
|
||||
Use when: researching project structure, identifying source files to test,
|
||||
discovering test frameworks and build commands, producing .testagent/research.md.
|
||||
discovering test frameworks and build commands, producing the pipeline research document.
|
||||
name: code-testing-researcher
|
||||
user-invocable: false
|
||||
tools: ["skill", "read", "search", "edit", "execute", "Skill", "Read", "Glob", "Grep", "Edit", "Write", "Bash", "read_file", "replace", "write_file", "glob", "grep_search", "run_shell_command"]
|
||||
@@ -96,7 +96,7 @@ Search for commands in:
|
||||
- `README.md` instructions
|
||||
- Project files
|
||||
|
||||
Identify **two** test commands and record both in `.testagent/research.md`:
|
||||
Identify **two** test commands and record both in the caller-provided research document:
|
||||
|
||||
1. **Scoped test command** — what the implementer should run during fix cycles (e.g., `dotnet test <test.csproj>` for SDK-style .NET, the repository's MSBuild + VSTest/MSTest command for classic .NET, `bundle exec rspec spec/foo_spec.rb`, `Invoke-Pester -Path ./Tests/Foo.Tests.ps1`). Optimized for speed and locality.
|
||||
2. **Harness-equivalent discovery command** — what a generic CI/benchmark verifier would run from the repo root with no args (e.g., `dotnet test <solution> --list-tests` for SDK-style .NET, the checked-in runner/discovery command for classic .NET, `bundle exec rspec --dry-run`, `Invoke-Pester` with default config, `pytest --collect-only -q`). This is the command the implementer's "Verify Harness Discovery" step uses to confirm new tests are visible to outside tooling. Call the `code-testing-extensions` skill and consult the "Harness Discovery Check" section of the relevant language extension.
|
||||
@@ -121,7 +121,7 @@ Before manually pairing source ↔ test files in C#, Python, TypeScript/JavaScri
|
||||
|
||||
### 8. Generate Research Document
|
||||
|
||||
Create `.testagent/research.md` with this structure:
|
||||
Create `<TESTAGENT_DIR>/research.md` with this structure:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
@@ -189,6 +189,9 @@ For each test project found, list:
|
||||
|
||||
## Output
|
||||
|
||||
Write the research document to `.testagent/research.md` in the workspace root.
|
||||
Write the research document to the absolute `<TESTAGENT_DIR>/research.md` path
|
||||
provided by the caller. `<TESTAGENT_DIR>` must be non-stageable host scratch
|
||||
storage, Git metadata, or OS temp. Never place `<TESTAGENT_DIR>` or its files in
|
||||
version-controlled workspace content.
|
||||
|
||||
Only consult a language example when no representative tests exist and the base extension does not establish the needed convention.
|
||||
|
||||
@@ -26,7 +26,9 @@ Run the appropriate test command and report pass/fail with details.
|
||||
|
||||
If not provided, check in order:
|
||||
|
||||
1. `.testagent/research.md` or `.testagent/plan.md` for Commands section
|
||||
1. The exact command or relevant Commands excerpt supplied by the caller; if
|
||||
the caller instead supplies a document, it must provide its absolute
|
||||
`<TESTAGENT_DIR>/research.md` or `<TESTAGENT_DIR>/plan.md` path
|
||||
2. Project files:
|
||||
- SDK-style `*.csproj` with Test SDK → `dotnet test`
|
||||
- Classic non-SDK `*.csproj` / `packages.config` → repository-documented VSTest, MSTest, or custom runner command
|
||||
|
||||
@@ -22,21 +22,26 @@ An AI-powered skill that generates comprehensive, workable unit tests for any pr
|
||||
Classify scope **before editing**:
|
||||
|
||||
- **Broad** (a project/package-wide suite, or multiple production
|
||||
files/modules): create
|
||||
`.testagent/research.md` and `.testagent/plan.md` before implementation, then
|
||||
`.testagent/status.md` after the final test-quality review. If these files are
|
||||
absent, the broad workflow is incomplete.
|
||||
files/modules): create `research.md` and `plan.md` in a resolved
|
||||
non-stageable `<TESTAGENT_DIR>` before implementation, then `status.md` there
|
||||
after the final test-quality review. If these files are absent, the broad
|
||||
workflow is incomplete.
|
||||
- **Focused** (the user explicitly limits work to one function/class/file or one
|
||||
missing method): do not create `.testagent/` artifacts or fan out to multiple
|
||||
missing method): do not create intermediate state files or fan out to multiple
|
||||
agents. A sparse project-wide request remains broad even when only one source
|
||||
module is present.
|
||||
|
||||
For either scope, run the narrowest relevant test command to a clean exit and
|
||||
finish with a compact `Requirement | Evidence` table. Each requested behavior
|
||||
must cite an exact test name; validation rows cite the successful command.
|
||||
For focused work, "no `.testagent/` artifacts" changes only the process, not the
|
||||
For focused work, "no intermediate state files" changes only the process, not the
|
||||
final evidence contract.
|
||||
|
||||
Intermediate state files are internal working data, never deliverables. Keep
|
||||
`<TESTAGENT_DIR>` non-stageable, never place it or its files in
|
||||
version-controlled workspace content, and never modify `.gitignore` to hide
|
||||
them.
|
||||
|
||||
Treat completeness as a requirement matrix, not a test-count target. Give every
|
||||
independently requested state, boundary, error path, or interaction its own
|
||||
concrete assertion. Combine cases only when one execution genuinely proves the
|
||||
@@ -48,6 +53,17 @@ prove composition, but do not substitute for the requested module-level
|
||||
coverage. Judge breadth by the behavior matrix, never by matching or exceeding a
|
||||
raw test count.
|
||||
|
||||
For a **broad or comprehensive** request, the explicit matrix is the floor, not
|
||||
the ceiling. After satisfying it, inspect each target API for observable
|
||||
equivalence partitions and invariants that the prompt did not name: identity,
|
||||
empty, singleton and representative interior inputs; exact boundaries plus an
|
||||
immediately adjacent value; invalid partitions; and ordering, monotonicity,
|
||||
rollover, capacity, truncation, or state invariants implied by the implementation.
|
||||
Add one mutation-relevant case per distinct partition not already proved, using
|
||||
parameterized or table-driven cases for siblings. Stop when remaining inputs
|
||||
exercise the same branch and invariant, not merely when the explicit checklist
|
||||
is complete; never add cases only to raise the count.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when you need to:
|
||||
@@ -114,8 +130,8 @@ request costs turns and tool calls without improving the tests.
|
||||
|
||||
| Scope | What it looks like | How to run it |
|
||||
| --- | --- | --- |
|
||||
| **Focused** | One function, class, or file; "tests for X only"; extending an existing suite with the missing cases | Skip the `.testagent/` artifacts and the sub-agent fan-out. Keep the requirement checklist in your head (or in the final table), read only the target and one neighbouring test for conventions, write the tests, run the narrowest test command, review your own assertions inline. |
|
||||
| **Broad** | A project, package, or module set; "comprehensive suite"; a coverage threshold to clear across several files | Run the full Research → Plan → Implement pipeline in Step 3, with the `.testagent/` artifacts and the completion contract below. |
|
||||
| **Focused** | One function, class, or file; "tests for X only"; extending an existing suite with the missing cases | Skip intermediate state files and the sub-agent fan-out. Keep the requirement checklist in your head (or in the final table), read only the target and one neighbouring test for conventions, write the tests, run the narrowest test command, review your own assertions inline. |
|
||||
| **Broad** | A project, package, or module set; "comprehensive suite"; a coverage threshold to clear across several files | Run the full Research → Plan → Implement pipeline in Step 3, with intermediate state files under `<TESTAGENT_DIR>` and the completion contract below. |
|
||||
|
||||
When in doubt, start focused and escalate only if the request turns out to span
|
||||
several files. Escalating costs one extra pass; running the broad pipeline on a
|
||||
@@ -143,15 +159,30 @@ Generate unit tests for [path or description of what to test], following the [un
|
||||
The Test Generator will manage the entire pipeline automatically.
|
||||
|
||||
If `code-testing-generator` is unavailable, do not skip the workflow. Execute the
|
||||
same Research → Plan → Implement sequence inline, create the `.testagent/`
|
||||
artifacts described below, and apply the same completion contract.
|
||||
same Research → Plan → Implement sequence inline, resolve `<TESTAGENT_DIR>` as
|
||||
described below, create the intermediate state files there, and apply the same
|
||||
completion contract.
|
||||
|
||||
For broad scope, resolve one absolute `<TESTAGENT_DIR>` before creating
|
||||
intermediate state files:
|
||||
|
||||
1. Prefer a host-provided session artifact or scratch directory.
|
||||
2. Otherwise, in a Git worktree run
|
||||
`git rev-parse --path-format=absolute --git-path testagent`; this returns a
|
||||
path in worktree-specific Git metadata that cannot be staged.
|
||||
3. Outside Git, create a unique directory under the operating system's
|
||||
temporary directory.
|
||||
|
||||
Pass the absolute directory to every pipeline agent. The path may be inside the
|
||||
repository's `.git` metadata directory, but it must not be version-controlled
|
||||
workspace content, appear in `git status`, or be stageable.
|
||||
|
||||
### Step 4: Execute with bounded context
|
||||
|
||||
For multi-file requests:
|
||||
|
||||
1. Turn every explicit user requirement into a checklist before implementation. Include requested layers, collaborators to mock, boundary cases, integrations, coverage thresholds, and report artifacts. Copy multi-condition requirements verbatim — they must each map to one test that exercises the whole combination.
|
||||
2. Research only the requested module or project and write the checklist plus a compact target inventory to `.testagent/research.md`.
|
||||
2. Research only the requested module or project and write the checklist plus a compact target inventory to `<TESTAGENT_DIR>/research.md`.
|
||||
3. Reuse manifests, symbol references, and deterministic pairing tools instead of reading every source and test file.
|
||||
4. For multi-file scopes in C#, Python, TypeScript/JavaScript, Go, Java, Rust, or Ruby, run `find-untested-sources` once and consume its pairing and suggested-path output; do not repeat that discovery manually.
|
||||
5. Plan each target file once, then implement phases sequentially. Map every checklist item to at least one concrete test or explain why it is blocked.
|
||||
@@ -167,13 +198,13 @@ For multi-file requests:
|
||||
|
||||
Every scope must satisfy points 3–5 below. Points 1 and 2 are the **broad-scope**
|
||||
artifacts: on a focused request the same reasoning happens inline and no
|
||||
`.testagent/` files are written.
|
||||
intermediate state files are written.
|
||||
|
||||
Do not report completion until all of these are true:
|
||||
|
||||
1. *(broad scope)* `.testagent/research.md` records the bounded target
|
||||
1. *(broad scope)* `<TESTAGENT_DIR>/research.md` records the bounded target
|
||||
inventory, existing test conventions, and the acceptance checklist.
|
||||
2. *(broad scope)* `.testagent/plan.md` maps each checklist item to a planned
|
||||
2. *(broad scope)* `<TESTAGENT_DIR>/plan.md` maps each checklist item to a planned
|
||||
test or an explicit blocker.
|
||||
3. Generated tests compile and pass with the narrowest relevant test command.
|
||||
4. Every explicit user requirement is backed by a concrete test and assertion.
|
||||
@@ -187,12 +218,15 @@ Do not report completion until all of these are true:
|
||||
A passing suite with fewer tests is not automatically weaker: judge
|
||||
completeness by whether every independently requested behavior has direct,
|
||||
nonredundant evidence, not by raw test volume.
|
||||
For broad/comprehensive scope, also verify that every observable equivalence
|
||||
partition and invariant discovered in the bounded target APIs has one
|
||||
mutation-relevant case, even when the prompt did not name it.
|
||||
When the request names multiple modules, verify that each module's own
|
||||
non-trivial public behavior has direct test evidence in addition to any
|
||||
end-to-end composition test.
|
||||
5. Review the generated tests for behavior gaps and weak assertions. On a broad
|
||||
scope, invoke `test-gap-analysis` and `assertion-quality` when available and
|
||||
record the findings and fixes in `.testagent/status.md`. On a focused scope,
|
||||
record the findings and fixes in `<TESTAGENT_DIR>/status.md`. On a focused scope,
|
||||
do the equivalent review inline — re-read each generated assertion against
|
||||
the source — without spawning extra passes.
|
||||
|
||||
@@ -215,16 +249,21 @@ thresholds were requested, the per-module coverage table from a run that exited
|
||||
0. If the last coverage run exited non-zero, fix it and re-run before reporting;
|
||||
never infer threshold clearance from a failed or partial run.
|
||||
|
||||
Before reporting, inspect the final working-tree changes and confirm that
|
||||
`research.md`, `plan.md`, `status.md`, and any other intermediate state files are
|
||||
not among the changes intended for commit.
|
||||
|
||||
## State Management
|
||||
|
||||
Broad-scope runs store pipeline state in the `.testagent/` folder. A focused
|
||||
request does not create these files:
|
||||
Broad-scope runs store intermediate state files in a non-stageable
|
||||
`<TESTAGENT_DIR>` backed by host scratch storage, Git metadata, or OS temp. A
|
||||
focused request does not create these files:
|
||||
|
||||
| File | Purpose |
|
||||
| ------------------------ | ---------------------------- |
|
||||
| `.testagent/research.md` | Codebase analysis results |
|
||||
| `.testagent/plan.md` | Phased implementation plan |
|
||||
| `.testagent/status.md` | Final quality review and fixes |
|
||||
| `<TESTAGENT_DIR>/research.md` | Codebase analysis results |
|
||||
| `<TESTAGENT_DIR>/plan.md` | Phased implementation plan |
|
||||
| `<TESTAGENT_DIR>/status.md` | Final quality review and fixes |
|
||||
|
||||
## Agent Reference
|
||||
|
||||
@@ -254,7 +293,10 @@ execution as blocked rather than substituting `dotnet test`.
|
||||
|
||||
### Tests don't compile
|
||||
|
||||
The `code-testing-fixer` agent will attempt to resolve compilation errors. Check `.testagent/plan.md` for the expected test structure. Call the `code-testing-extensions` skill and read the language-specific extension file for error code references (e.g., `dotnet.md` for .NET).
|
||||
The `code-testing-fixer` agent will attempt to resolve compilation errors. Check
|
||||
`<TESTAGENT_DIR>/plan.md` for the expected test structure. Call the
|
||||
`code-testing-extensions` skill and read the language-specific extension file
|
||||
for error code references (e.g., `dotnet.md` for .NET).
|
||||
|
||||
### Tests fail
|
||||
|
||||
|
||||
@@ -72,7 +72,7 @@ void InvoiceService::mark_as_paid(int id) {
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
|
||||
@@ -56,7 +56,7 @@ public class InvoiceService(IInvoiceRepository repository)
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
@@ -112,7 +112,7 @@ What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
|
||||
## Sample Plan Output
|
||||
|
||||
What `code-testing-planner` produces in `.testagent/plan.md`:
|
||||
What `code-testing-planner` produces in `<TESTAGENT_DIR>/plan.md`:
|
||||
|
||||
```markdown
|
||||
# Test Implementation Plan
|
||||
|
||||
@@ -101,7 +101,7 @@ not part of the test assembly and must never be reported as generated coverage.
|
||||
|
||||
A new `.csproj` is **invisible** to `dotnet test <solution>`, to `dotnet test` run from the repo root, and to any CI/benchmark harness until it is added to the solution. Run `dotnet sln add` *immediately* after creating the project as part of Step 3 ("Register Test Project with Build System") — do not defer it to a later step.
|
||||
|
||||
1. Use the exact solution or solution-filter target identified in `.testagent/research.md` or `.testagent/plan.md` — do not search for or substitute a different `.sln`, `.slnx`, or `.slnf` target.
|
||||
1. Use the exact solution or solution-filter target identified in the research or plan document under `<TESTAGENT_DIR>` — do not search for or substitute a different `.sln`, `.slnx`, or `.slnf` target.
|
||||
2. If that target is a `.sln` or `.slnx`, run `dotnet sln <solution> add <test-project.csproj>`.
|
||||
3. If the target is a `.slnf` (solution filter), also ensure the new project is included in the filter; adding only to the underlying `.sln` may not be enough for test discovery.
|
||||
4. Skip this if the project is already included in the solution or solution filter used for testing.
|
||||
@@ -112,7 +112,7 @@ A new `.csproj` is **invisible** to `dotnet test <solution>`, to `dotnet test` r
|
||||
Before reporting success, run the **harness-equivalent** discovery command from the repo root and confirm the test count went up by at least the number of tests you generated. The harness (CI, msbench, coverage tools) does not know which `.csproj` you targeted — it runs the solution-level command, so a test that passes via `dotnet test MyProject.Tests.csproj` is still worthless if `dotnet test <solution> --list-tests` doesn't enumerate it.
|
||||
|
||||
```bash
|
||||
# From repo root, against the solution identified in .testagent/research.md
|
||||
# From repo root, against the solution identified in <TESTAGENT_DIR>/research.md
|
||||
dotnet test <solution> --list-tests --no-build 2>&1 | grep -c '^ [A-Za-z]'
|
||||
```
|
||||
|
||||
|
||||
@@ -80,7 +80,7 @@ func (s *InvoiceService) MarkAsPaid(ctx context.Context, id int) error {
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
|
||||
@@ -75,7 +75,7 @@ public class InvoiceService {
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
|
||||
@@ -57,7 +57,7 @@ class InvoiceService(
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
|
||||
+1
-1
@@ -72,7 +72,7 @@ Export-ModuleMember -Function Get-InvoiceTotal, Get-InvoiceById, Set-InvoicePaid
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
|
||||
@@ -67,7 +67,7 @@ def _utcnow():
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
@@ -121,7 +121,7 @@ What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
|
||||
## Sample Plan Output
|
||||
|
||||
What `code-testing-planner` produces in `.testagent/plan.md`:
|
||||
What `code-testing-planner` produces in `<TESTAGENT_DIR>/plan.md`:
|
||||
|
||||
```markdown
|
||||
# Test Implementation Plan
|
||||
|
||||
@@ -60,7 +60,7 @@ end
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
|
||||
@@ -75,7 +75,7 @@ where
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
|
||||
+2
-2
@@ -71,7 +71,7 @@ function roundTo2(n: number): number {
|
||||
|
||||
## Sample Research Output
|
||||
|
||||
What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
What `code-testing-researcher` produces in `<TESTAGENT_DIR>/research.md`:
|
||||
|
||||
```markdown
|
||||
# Test Generation Research
|
||||
@@ -126,7 +126,7 @@ What `code-testing-researcher` produces in `.testagent/research.md`:
|
||||
|
||||
## Sample Plan Output
|
||||
|
||||
What `code-testing-planner` produces in `.testagent/plan.md`:
|
||||
What `code-testing-planner` produces in `<TESTAGENT_DIR>/plan.md`:
|
||||
|
||||
```markdown
|
||||
# Test Implementation Plan
|
||||
|
||||
Reference in New Issue
Block a user