diff --git a/plugins/dotnet-test/agents/code-testing-builder.agent.md b/plugins/dotnet-test/agents/code-testing-builder.agent.md index d7a2d13f..dd8c229c 100644 --- a/plugins/dotnet-test/agents/code-testing-builder.agent.md +++ b/plugins/dotnet-test/agents/code-testing-builder.agent.md @@ -26,7 +26,9 @@ Run the appropriate build command and report success or failure with error detai If not provided, check in order: -1. `.testagent/research.md` or `.testagent/plan.md` for Commands section +1. The exact command or relevant Commands excerpt supplied by the caller; if + the caller instead supplies a document, it must provide its absolute + `/research.md` or `/plan.md` path 2. Project files: - SDK-style `*.csproj` / `*.sln` → `dotnet build` - Classic non-SDK `*.csproj` / `*.sln` → repository-documented MSBuild command diff --git a/plugins/dotnet-test/agents/code-testing-generator.agent.md b/plugins/dotnet-test/agents/code-testing-generator.agent.md index fd53a78a..b19d8df7 100644 --- a/plugins/dotnet-test/agents/code-testing-generator.agent.md +++ b/plugins/dotnet-test/agents/code-testing-generator.agent.md @@ -35,7 +35,24 @@ You coordinate test generation using the Research-Plan-Implement (RPI) pipeline. Understand what the user wants: scope (project, files, classes), priority areas, framework preferences. If clear, proceed directly. If the user provides no details or a very basic prompt (e.g., "generate tests"), use [unit-test-generation.prompt.md](../skills/code-testing-agent/unit-test-generation.prompt.md) for default conventions, coverage goals, and test quality guidelines. -Before writing code, read the language-specific base extension. Reuse it for the whole run; sub-agents must not independently reload the same reference unless they need a section that was not captured in `.testagent/research.md`. +Before writing code, read the language-specific base extension. Reuse it for the whole run; sub-agents must not independently reload the same reference unless they need a section that was not captured in the research document. + +For Single pass and Iterative strategies, resolve one absolute +`` +before invoking any sub-agent: + +1. Prefer a host-provided session artifact or scratch directory when one is + available. +2. Otherwise, in a Git worktree run + `git rev-parse --path-format=absolute --git-path testagent`. This returns a + path in worktree-specific Git metadata, which cannot be staged or committed. +3. Outside Git, create a unique directory under the operating system's + temporary directory. + +Create the resolved directory and pass its absolute path explicitly in every +sub-agent prompt. Never create `` or any intermediate state file +in version-controlled workspace content, and never modify `.gitignore` to hide +them. Create a **requirement checklist** from the request before choosing a strategy. Preserve each explicit behavior, layer, collaborator seam, boundary case, @@ -43,7 +60,7 @@ integration, coverage threshold, and required artifact as a separate item. For example, "mock the repository in service tests", "exercise SQLite in memory", and "cover pagination boundaries" are three independently verifiable requirements. Direct strategy keeps this checklist in context; delegated -strategies record it in `.testagent/research.md`. +strategies record it in `/research.md`. ### Step 2: Choose Execution Strategy @@ -53,7 +70,7 @@ Based on the request scope, pick exactly one strategy and follow it: | ---------- | ------------- | ------------ | | **Direct** | A small, self-contained request (e.g., tests for a single function or class) that you can complete without sub-agents | Follow the codebase conventions on test file structure, naming, style, and testing approaches. Reuse existing test projects and test files when possible — if the code under test already has tests, add new tests to the same file or test project. Only create a new test file when no canonical file is named or discoverable for the symbol under test. Write the tests immediately. **Run them right away** — if any test fails, read the production code, fix the assertion, and re-run before writing more tests. Skip Steps 3-5 (research, plan, implement sub-agents). Then proceed to Steps 6-9 for validation and reporting — **Direct skips only the sub-agents, never the Step 7 pre-completion gate** (which still runs per its own threshold in Step 7 — i.e. for any non-trivial addition: ≥5 tests, or any request that enumerates behaviors/scenarios to verify). | | **Single pass** | A moderate scope (couple projects or modules) that a single Research → Plan → Implement cycle can cover | Execute Steps 3-8 once, then proceed to Step 9. | -| **Iterative** | A large scope or ambitious coverage target that one pass cannot satisfy | Execute Steps 3-8, then re-evaluate coverage. If the target is not met, repeat Steps 3-8 with a narrowed focus on remaining gaps. Use unique names for each iteration's `.testagent/` documents (e.g., `research-2.md`, `plan-2.md`) so earlier results are not overwritten. Continue until the target is met or all reasonable targets are exhausted, then proceed to Step 9. | +| **Iterative** | A large scope or ambitious coverage target that one pass cannot satisfy | Execute Steps 3-8, then re-evaluate coverage. If the target is not met, repeat Steps 3-8 with a narrowed focus on remaining gaps. Use unique names for each iteration's documents in `` (e.g., `research-2.md`, `plan-2.md`) so earlier results are not overwritten. Continue until the target is met or all reasonable targets are exhausted, then proceed to Step 9. | **Default to Direct** unless the user asks for a project/package-wide suite or the scope explicitly spans multiple files or modules. Most test generation @@ -87,25 +104,25 @@ Delegate to the `code-testing-researcher` subagent with this task: ```text runSubagent({ agent: "code-testing-researcher", - prompt: "Research [REQUESTED SCOPE] at [PATH] for test generation. Produce a bounded target inventory, existing test conventions, source-to-test pairs, dependencies only for those targets, and exact build/test/discovery commands. Do not inventory unrelated source files." + prompt: "Research [REQUESTED SCOPE] at [PATH] for test generation. Write the research document to /research.md. Produce a bounded target inventory, existing test conventions, source-to-test pairs, dependencies only for those targets, and exact build/test/discovery commands. Do not inventory unrelated source files." }) ``` -Output: `.testagent/research.md` +Output: `/research.md` ### Step 4: Planning Phase Delegate to the `code-testing-planner` subagent with this task: -> Create a test implementation plan based on .testagent/research.md. Create phased approach with specific files and test cases. +> Create a test implementation plan based on `/research.md`. Write it to `/plan.md`. Create a phased approach with specific files and test cases. -Output: `.testagent/plan.md` +Output: `/plan.md` ### Step 5: Implementation Phase Execute each phase by delegating to the `code-testing-implementer` subagent — once per phase, sequentially. For each phase, delegate with this task: -> Implement Phase N from .testagent/plan.md: [phase description]. Ensure tests compile and pass. +> Implement Phase N from `/plan.md`: [phase description]. Use `/research.md` for commands and conventions. Ensure tests compile and pass. ### Step 6: Final Build Validation @@ -152,19 +169,29 @@ Additional self-review heuristics (still required, even when running the skills) ### Step 8: Coverage Gap Iteration -After the previous phases complete, use the target inventory already recorded in `.testagent/research.md` and the files reported by implementers. Do not rescan or reread the workspace. +After the previous phases complete, use the target inventory already recorded in `/research.md` and the files reported by implementers. Do not rescan or reread the workspace. 1. Compare the requirement checklist and bounded target inventory with the implemented tests. 2. Inspect the generated test bodies for evidence of every checklist item. A covered line does not prove that a requested collaborator was mocked, a concrete result was asserted, or a boundary/property combination was exercised. 3. If the user requested a measurable coverage target, collect coverage once and prioritize only gaps inside the requested scope. -4. Add tests for any unaddressed checklist item before adding optional cases merely to raise test count. -5. Stop only when every feasible checklist item is covered and the stated target is met; do not recursively expand into unrelated files. -6. If this step added or modified tests, re-run the full Step 7 pre-completion gate (`test-gap-analysis` + `assertion-quality` + prompt-scenario coverage) on those tests before reporting completion. +4. Add tests for any unaddressed checklist item first. +5. For Single pass and Iterative strategies, treat that checklist as the floor. + Sweep each bounded target API for still-unproved observable equivalence + partitions and invariants: identity/empty/singleton/interior inputs, exact + and immediately adjacent boundaries, invalid partitions, and ordering, + monotonicity, rollover, capacity, truncation, or state properties implied by + the implementation. Add one mutation-relevant case per distinct partition; + consolidate sibling inputs in parameterized or table-driven tests. +6. Stop only when every feasible checklist item and distinct behavioral + partition is covered and the stated target is met. Do not recursively expand + into unrelated files or add equivalent cases merely to raise test count. +7. If this step added or modified tests, re-run the full Step 7 pre-completion gate (`test-gap-analysis` + `assertion-quality` + prompt-scenario coverage) on those tests before reporting completion. -For Single pass and Iterative strategies, write `.testagent/status.md` after +For Single pass and Iterative strategies, write `/status.md` after the final review and validation. Record the completed checklist, commands and results, quality findings, fixes, and any explicit blockers. Direct strategy -keeps this evidence in the final response and must not create `.testagent/`. +keeps this evidence in the final response and must not create intermediate +state files. ### Step 9: Report Results @@ -208,11 +235,12 @@ Use a language example from `code-testing-extensions` only when no existing test ## State Management -All state is stored in `.testagent/` folder: +All delegated intermediate state files are stored in the resolved, +non-stageable ``: -- `.testagent/research.md` — Research findings -- `.testagent/plan.md` — Implementation plan -- `.testagent/status.md` — Final quality review, fixes, and validation status +- `/research.md` — Research findings +- `/plan.md` — Implementation plan +- `/status.md` — Final quality review, fixes, and validation status ## Rules @@ -224,9 +252,9 @@ All state is stored in `.testagent/` folder: 6. **Scoped builds during phases, full build at the end** — build specific test projects during implementation for speed; run a full-workspace non-incremental build after all phases to catch cross-project errors 7. **No environment-dependent tests** — mock all external dependencies; never call external URLs, bind ports, or depend on timing 8. **Fix assertions, don't skip tests** — when tests fail, read production code and fix the expected value; never `[Ignore]` or `[Skip]` -9. **Retain `.testagent/` through completion** — keep the research, plan, and final status available as auditable pipeline evidence. Do not delete them automatically; if the repository should not commit agent state, advise the user to add `.testagent/` to `.gitignore` after reporting the result. +9. **Keep intermediate state files out of commits** — retain research, plan, and final status in `` through completion, but never place `` or its files in version-controlled workspace content, stage them, or modify `.gitignore` to hide them. Before reporting, inspect the working-tree changes and confirm they contain only requested deliverables and required manifest edits. 10. **Read language extensions first** — always call the `code-testing-extensions` skill and read the relevant extension file before writing any code; it contains critical project registration and build validation steps 11. **Always validate** — final build, final test, coverage-gap review, and reporting are mandatory for ALL strategies including Direct; never skip final validation. The pre-completion self-review gate from Step 7 (`test-gap-analysis` + `assertion-quality` skills, plus the prompt-scenario coverage check) is mandatory for every non-trivial test addition and may be skipped only for trivially small tasks (fewer than 5 generated tests *and* no behaviors specified in the prompt), per Step 7 12. **Preserve existing tests** — never delete or overwrite existing test files; create new files or append to existing ones 13. **Never mutate version control** — your only outputs are additive test files plus minimal build-manifest edits to register a new test project. Any command that reverts, restores, resets, stashes, or cleans the tree, or deletes tracked files, is out of scope — even when the workspace looks broken or incomplete. -14. **Bound context and reuse findings** — scope every search to the user's requested files/modules, read only the source and existing tests needed for the next implementation phase, and reuse `.testagent/research.md` instead of repeating workspace discovery. +14. **Bound context and reuse findings** — scope every search to the user's requested files/modules, read only the source and existing tests needed for the next implementation phase, and reuse `/research.md` instead of repeating workspace discovery. diff --git a/plugins/dotnet-test/agents/code-testing-implementer.agent.md b/plugins/dotnet-test/agents/code-testing-implementer.agent.md index 24f7eb31..15de31c4 100644 --- a/plugins/dotnet-test/agents/code-testing-implementer.agent.md +++ b/plugins/dotnet-test/agents/code-testing-implementer.agent.md @@ -30,8 +30,10 @@ Given a phase from the plan, write all the test files for that phase and ensure ### 1. Read the Plan and Research -- Read only the current phase from `.testagent/plan.md` -- Read the command, convention, and target entries needed for that phase from `.testagent/research.md` +- Read only the current phase from the caller-provided absolute + `/plan.md` path +- Read the command, convention, and target entries needed for that phase from + `/research.md` - Identify which phase you're implementing ### 2. Read Source Files and Validate References @@ -62,6 +64,10 @@ For each test file in your phase: - Create the test file with appropriate structure - Follow the project's testing patterns - Include tests for: happy path, edge cases (empty, null, boundary), error conditions +- Treat the plan's explicit requirements as a floor. For broad/comprehensive + scope, add one mutation-relevant case for every distinct observable + equivalence partition or invariant in the target implementation that the plan + missed; parameterize sibling inputs instead of duplicating test structure. - Mock all external dependencies — never call external URLs, bind ports, or depend on timing #### Edit boundaries (cross-language invariants) @@ -69,7 +75,11 @@ For each test file in your phase: These rules apply to every language and override any pattern an existing test file may suggest. They keep generated changes additive so reviewers, CI gates, and test-quality benchmarks treat your output as a clean test addition rather than a refactor: - **Existing test files are append-only.** When growing an existing test file, insert new test methods/cases at the end of the relevant class/describe-block/module. Do not reformat, reorder, rename, or remove any existing line — even whitespace-only churn counts as a destructive edit. -- **Do not modify non-test source files.** If a class, method, or symbol is hard to test (sealed, internal, no seam, tightly coupled), record the gap in `.testagent/plan.md` as a follow-up. Do not edit production code to make it testable as part of test generation — that is the scope of the `testability-migration` agent, not this one. +- **Do not modify non-test source files.** If a class, method, or symbol is hard to test (sealed, internal, no seam, tightly coupled), record the gap in `/plan.md` as a follow-up. Do not edit production code to make it testable as part of test generation — that is the scope of the `testability-migration` agent, not this one. +- **Keep intermediate state files non-stageable.** Use only the absolute + `` supplied by the caller for research, plan, or status + updates. Never place `` or its files in version-controlled + workspace content or stage them. - **Never revert or clean the working tree.** Do not run `git checkout`, `git restore`, `git reset`, `git clean`, `git stash`, `git rm`, or delete tracked files. Generate tests against the workspace exactly as delivered, even if the source looks synthetic, deleted, gutted, or incomplete — that state is intentional, not corruption. - **Prefer new test files over edits to existing ones** when both options are equally valid (e.g., a new feature, a separate concern, or any case where the existing file isn't strictly required). A new file is always purely additive. - **One exception**: build-system manifests (`.csproj`/`.sln`/`packages.config`/`pom.xml`/`build.gradle`/`Cargo.toml`/`package.json`/etc.) may be edited when registering a new test file/project or adding a missing test dependency. Keep these edits minimal and limited to the registration/dependency change. Never convert `packages.config` to `PackageReference`, convert a classic project to SDK style, or upgrade the test stack unless the user explicitly requested that migration. @@ -80,13 +90,16 @@ Coverage alone gives false confidence — every test must *pin down behavior* so ### 5. Verify with Build -Call the `code-testing-builder` sub-agent to compile. Build only the specific test project, not the full solution. +Call the `code-testing-builder` sub-agent to compile, passing the exact build +command and absolute ``. Build only the specific test project, +not the full solution. If build fails: call `code-testing-fixer`, rebuild, retry up to 3 times. ### 6. Verify with Tests -Call the `code-testing-tester` sub-agent to run tests. +Call the `code-testing-tester` sub-agent to run tests, passing the exact test +command and absolute ``. If tests fail: @@ -115,7 +128,8 @@ If your language extension has no "Harness Discovery Check" section, use the can ### 8. Format Code (Optional) -If a lint command is available, call the `code-testing-linter` sub-agent. +If a lint command is available, call the `code-testing-linter` sub-agent, +passing the exact lint command and absolute ``. ### 9. Report Results diff --git a/plugins/dotnet-test/agents/code-testing-linter.agent.md b/plugins/dotnet-test/agents/code-testing-linter.agent.md index d81a733b..84cf7144 100644 --- a/plugins/dotnet-test/agents/code-testing-linter.agent.md +++ b/plugins/dotnet-test/agents/code-testing-linter.agent.md @@ -24,7 +24,9 @@ Run the appropriate lint/format command to fix code style issues. If not provided, check in order: -1. `.testagent/research.md` or `.testagent/plan.md` for Commands section +1. The exact command or relevant Commands excerpt supplied by the caller; if + the caller instead supplies a document, it must provide its absolute + `/research.md` or `/plan.md` path 2. Project files: - `*.csproj` / `*.sln` → `dotnet format` - `package.json` → `npm run lint:fix` or `npm run format` diff --git a/plugins/dotnet-test/agents/code-testing-planner.agent.md b/plugins/dotnet-test/agents/code-testing-planner.agent.md index 965e7631..ead105b8 100644 --- a/plugins/dotnet-test/agents/code-testing-planner.agent.md +++ b/plugins/dotnet-test/agents/code-testing-planner.agent.md @@ -3,7 +3,7 @@ description: >- Creates structured test implementation plans from research findings. Use when: organizing tests into phases, prioritizing test generation, - creating .testagent/plan.md from research. + creating the pipeline plan document from research. name: code-testing-planner user-invocable: false tools: ["skill", "read", "search", "edit", "execute", "Skill", "Read", "Glob", "Grep", "Edit", "Write", "Bash", "read_file", "replace", "write_file", "glob", "grep_search", "run_shell_command"] @@ -22,7 +22,9 @@ Read the research document and create a phased implementation plan that will gui ### 1. Read the Research -Read the target inventory, command section, dependency summary, and testing conventions from `.testagent/research.md`. Do not reread repository files during planning. +Read the target inventory, command section, dependency summary, and testing +conventions from the absolute `/research.md` path provided by the +caller. Do not reread repository files during planning. - Project structure and language - Files that need tests @@ -68,12 +70,17 @@ For each file in each phase, specify: - Test class/module name - Methods/functions to test - Key test scenarios (happy path, edge cases, errors) +- For broad/comprehensive scope, one mutation-relevant case for each observable + equivalence partition or invariant discovered in the source, including useful + identity/empty/singleton/interior cases, exact and adjacent boundaries, and + ordering, rollover, capacity, truncation, or state properties not named by + the prompt. Group sibling inputs in parameterized or table-driven tests. **Important**: When adding new tests, they MUST go into the existing test project that already tests the target code. Do not create a separate test project unnecessarily. If no existing test project covers the target, create a new one. ### 5. Generate Plan Document -Create `.testagent/plan.md` with this structure: +Create `/plan.md` with this structure: ```markdown # Test Implementation Plan @@ -139,4 +146,7 @@ Only consult a language example when research found no existing tests and the ba ## Output -Write the plan document to `.testagent/plan.md` in the workspace root. +Write the plan document to the absolute `/plan.md` path provided +by the caller. `` must be non-stageable host scratch storage, +Git metadata, or OS temp. Never place it or its files in version-controlled +workspace content. diff --git a/plugins/dotnet-test/agents/code-testing-researcher.agent.md b/plugins/dotnet-test/agents/code-testing-researcher.agent.md index 0ccad7ba..d426ca89 100644 --- a/plugins/dotnet-test/agents/code-testing-researcher.agent.md +++ b/plugins/dotnet-test/agents/code-testing-researcher.agent.md @@ -3,7 +3,7 @@ description: >- Analyzes codebases to understand structure, testing patterns, and testability. Use when: researching project structure, identifying source files to test, - discovering test frameworks and build commands, producing .testagent/research.md. + discovering test frameworks and build commands, producing the pipeline research document. name: code-testing-researcher user-invocable: false tools: ["skill", "read", "search", "edit", "execute", "Skill", "Read", "Glob", "Grep", "Edit", "Write", "Bash", "read_file", "replace", "write_file", "glob", "grep_search", "run_shell_command"] @@ -96,7 +96,7 @@ Search for commands in: - `README.md` instructions - Project files -Identify **two** test commands and record both in `.testagent/research.md`: +Identify **two** test commands and record both in the caller-provided research document: 1. **Scoped test command** — what the implementer should run during fix cycles (e.g., `dotnet test ` for SDK-style .NET, the repository's MSBuild + VSTest/MSTest command for classic .NET, `bundle exec rspec spec/foo_spec.rb`, `Invoke-Pester -Path ./Tests/Foo.Tests.ps1`). Optimized for speed and locality. 2. **Harness-equivalent discovery command** — what a generic CI/benchmark verifier would run from the repo root with no args (e.g., `dotnet test --list-tests` for SDK-style .NET, the checked-in runner/discovery command for classic .NET, `bundle exec rspec --dry-run`, `Invoke-Pester` with default config, `pytest --collect-only -q`). This is the command the implementer's "Verify Harness Discovery" step uses to confirm new tests are visible to outside tooling. Call the `code-testing-extensions` skill and consult the "Harness Discovery Check" section of the relevant language extension. @@ -121,7 +121,7 @@ Before manually pairing source ↔ test files in C#, Python, TypeScript/JavaScri ### 8. Generate Research Document -Create `.testagent/research.md` with this structure: +Create `/research.md` with this structure: ```markdown # Test Generation Research @@ -189,6 +189,9 @@ For each test project found, list: ## Output -Write the research document to `.testagent/research.md` in the workspace root. +Write the research document to the absolute `/research.md` path +provided by the caller. `` must be non-stageable host scratch +storage, Git metadata, or OS temp. Never place `` or its files in +version-controlled workspace content. Only consult a language example when no representative tests exist and the base extension does not establish the needed convention. diff --git a/plugins/dotnet-test/agents/code-testing-tester.agent.md b/plugins/dotnet-test/agents/code-testing-tester.agent.md index 37b206bd..23612053 100644 --- a/plugins/dotnet-test/agents/code-testing-tester.agent.md +++ b/plugins/dotnet-test/agents/code-testing-tester.agent.md @@ -26,7 +26,9 @@ Run the appropriate test command and report pass/fail with details. If not provided, check in order: -1. `.testagent/research.md` or `.testagent/plan.md` for Commands section +1. The exact command or relevant Commands excerpt supplied by the caller; if + the caller instead supplies a document, it must provide its absolute + `/research.md` or `/plan.md` path 2. Project files: - SDK-style `*.csproj` with Test SDK → `dotnet test` - Classic non-SDK `*.csproj` / `packages.config` → repository-documented VSTest, MSTest, or custom runner command diff --git a/plugins/dotnet-test/skills/code-testing-agent/SKILL.md b/plugins/dotnet-test/skills/code-testing-agent/SKILL.md index 74ac6728..f6f27438 100644 --- a/plugins/dotnet-test/skills/code-testing-agent/SKILL.md +++ b/plugins/dotnet-test/skills/code-testing-agent/SKILL.md @@ -22,21 +22,26 @@ An AI-powered skill that generates comprehensive, workable unit tests for any pr Classify scope **before editing**: - **Broad** (a project/package-wide suite, or multiple production - files/modules): create - `.testagent/research.md` and `.testagent/plan.md` before implementation, then - `.testagent/status.md` after the final test-quality review. If these files are - absent, the broad workflow is incomplete. + files/modules): create `research.md` and `plan.md` in a resolved + non-stageable `` before implementation, then `status.md` there + after the final test-quality review. If these files are absent, the broad + workflow is incomplete. - **Focused** (the user explicitly limits work to one function/class/file or one - missing method): do not create `.testagent/` artifacts or fan out to multiple + missing method): do not create intermediate state files or fan out to multiple agents. A sparse project-wide request remains broad even when only one source module is present. For either scope, run the narrowest relevant test command to a clean exit and finish with a compact `Requirement | Evidence` table. Each requested behavior must cite an exact test name; validation rows cite the successful command. -For focused work, "no `.testagent/` artifacts" changes only the process, not the +For focused work, "no intermediate state files" changes only the process, not the final evidence contract. +Intermediate state files are internal working data, never deliverables. Keep +`` non-stageable, never place it or its files in +version-controlled workspace content, and never modify `.gitignore` to hide +them. + Treat completeness as a requirement matrix, not a test-count target. Give every independently requested state, boundary, error path, or interaction its own concrete assertion. Combine cases only when one execution genuinely proves the @@ -48,6 +53,17 @@ prove composition, but do not substitute for the requested module-level coverage. Judge breadth by the behavior matrix, never by matching or exceeding a raw test count. +For a **broad or comprehensive** request, the explicit matrix is the floor, not +the ceiling. After satisfying it, inspect each target API for observable +equivalence partitions and invariants that the prompt did not name: identity, +empty, singleton and representative interior inputs; exact boundaries plus an +immediately adjacent value; invalid partitions; and ordering, monotonicity, +rollover, capacity, truncation, or state invariants implied by the implementation. +Add one mutation-relevant case per distinct partition not already proved, using +parameterized or table-driven cases for siblings. Stop when remaining inputs +exercise the same branch and invariant, not merely when the explicit checklist +is complete; never add cases only to raise the count. + ## When to Use This Skill Use this skill when you need to: @@ -114,8 +130,8 @@ request costs turns and tool calls without improving the tests. | Scope | What it looks like | How to run it | | --- | --- | --- | -| **Focused** | One function, class, or file; "tests for X only"; extending an existing suite with the missing cases | Skip the `.testagent/` artifacts and the sub-agent fan-out. Keep the requirement checklist in your head (or in the final table), read only the target and one neighbouring test for conventions, write the tests, run the narrowest test command, review your own assertions inline. | -| **Broad** | A project, package, or module set; "comprehensive suite"; a coverage threshold to clear across several files | Run the full Research → Plan → Implement pipeline in Step 3, with the `.testagent/` artifacts and the completion contract below. | +| **Focused** | One function, class, or file; "tests for X only"; extending an existing suite with the missing cases | Skip intermediate state files and the sub-agent fan-out. Keep the requirement checklist in your head (or in the final table), read only the target and one neighbouring test for conventions, write the tests, run the narrowest test command, review your own assertions inline. | +| **Broad** | A project, package, or module set; "comprehensive suite"; a coverage threshold to clear across several files | Run the full Research → Plan → Implement pipeline in Step 3, with intermediate state files under `` and the completion contract below. | When in doubt, start focused and escalate only if the request turns out to span several files. Escalating costs one extra pass; running the broad pipeline on a @@ -143,15 +159,30 @@ Generate unit tests for [path or description of what to test], following the [un The Test Generator will manage the entire pipeline automatically. If `code-testing-generator` is unavailable, do not skip the workflow. Execute the -same Research → Plan → Implement sequence inline, create the `.testagent/` -artifacts described below, and apply the same completion contract. +same Research → Plan → Implement sequence inline, resolve `` as +described below, create the intermediate state files there, and apply the same +completion contract. + +For broad scope, resolve one absolute `` before creating +intermediate state files: + +1. Prefer a host-provided session artifact or scratch directory. +2. Otherwise, in a Git worktree run + `git rev-parse --path-format=absolute --git-path testagent`; this returns a + path in worktree-specific Git metadata that cannot be staged. +3. Outside Git, create a unique directory under the operating system's + temporary directory. + +Pass the absolute directory to every pipeline agent. The path may be inside the +repository's `.git` metadata directory, but it must not be version-controlled +workspace content, appear in `git status`, or be stageable. ### Step 4: Execute with bounded context For multi-file requests: 1. Turn every explicit user requirement into a checklist before implementation. Include requested layers, collaborators to mock, boundary cases, integrations, coverage thresholds, and report artifacts. Copy multi-condition requirements verbatim — they must each map to one test that exercises the whole combination. -2. Research only the requested module or project and write the checklist plus a compact target inventory to `.testagent/research.md`. +2. Research only the requested module or project and write the checklist plus a compact target inventory to `/research.md`. 3. Reuse manifests, symbol references, and deterministic pairing tools instead of reading every source and test file. 4. For multi-file scopes in C#, Python, TypeScript/JavaScript, Go, Java, Rust, or Ruby, run `find-untested-sources` once and consume its pairing and suggested-path output; do not repeat that discovery manually. 5. Plan each target file once, then implement phases sequentially. Map every checklist item to at least one concrete test or explain why it is blocked. @@ -167,13 +198,13 @@ For multi-file requests: Every scope must satisfy points 3–5 below. Points 1 and 2 are the **broad-scope** artifacts: on a focused request the same reasoning happens inline and no -`.testagent/` files are written. +intermediate state files are written. Do not report completion until all of these are true: -1. *(broad scope)* `.testagent/research.md` records the bounded target +1. *(broad scope)* `/research.md` records the bounded target inventory, existing test conventions, and the acceptance checklist. -2. *(broad scope)* `.testagent/plan.md` maps each checklist item to a planned +2. *(broad scope)* `/plan.md` maps each checklist item to a planned test or an explicit blocker. 3. Generated tests compile and pass with the narrowest relevant test command. 4. Every explicit user requirement is backed by a concrete test and assertion. @@ -187,12 +218,15 @@ Do not report completion until all of these are true: A passing suite with fewer tests is not automatically weaker: judge completeness by whether every independently requested behavior has direct, nonredundant evidence, not by raw test volume. + For broad/comprehensive scope, also verify that every observable equivalence + partition and invariant discovered in the bounded target APIs has one + mutation-relevant case, even when the prompt did not name it. When the request names multiple modules, verify that each module's own non-trivial public behavior has direct test evidence in addition to any end-to-end composition test. 5. Review the generated tests for behavior gaps and weak assertions. On a broad scope, invoke `test-gap-analysis` and `assertion-quality` when available and - record the findings and fixes in `.testagent/status.md`. On a focused scope, + record the findings and fixes in `/status.md`. On a focused scope, do the equivalent review inline — re-read each generated assertion against the source — without spawning extra passes. @@ -215,16 +249,21 @@ thresholds were requested, the per-module coverage table from a run that exited 0. If the last coverage run exited non-zero, fix it and re-run before reporting; never infer threshold clearance from a failed or partial run. +Before reporting, inspect the final working-tree changes and confirm that +`research.md`, `plan.md`, `status.md`, and any other intermediate state files are +not among the changes intended for commit. + ## State Management -Broad-scope runs store pipeline state in the `.testagent/` folder. A focused -request does not create these files: +Broad-scope runs store intermediate state files in a non-stageable +`` backed by host scratch storage, Git metadata, or OS temp. A +focused request does not create these files: | File | Purpose | | ------------------------ | ---------------------------- | -| `.testagent/research.md` | Codebase analysis results | -| `.testagent/plan.md` | Phased implementation plan | -| `.testagent/status.md` | Final quality review and fixes | +| `/research.md` | Codebase analysis results | +| `/plan.md` | Phased implementation plan | +| `/status.md` | Final quality review and fixes | ## Agent Reference @@ -254,7 +293,10 @@ execution as blocked rather than substituting `dotnet test`. ### Tests don't compile -The `code-testing-fixer` agent will attempt to resolve compilation errors. Check `.testagent/plan.md` for the expected test structure. Call the `code-testing-extensions` skill and read the language-specific extension file for error code references (e.g., `dotnet.md` for .NET). +The `code-testing-fixer` agent will attempt to resolve compilation errors. Check +`/plan.md` for the expected test structure. Call the +`code-testing-extensions` skill and read the language-specific extension file +for error code references (e.g., `dotnet.md` for .NET). ### Tests fail diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/cpp-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/cpp-examples.md index 186fe9b1..78f76505 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/cpp-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/cpp-examples.md @@ -72,7 +72,7 @@ void InvoiceService::mark_as_paid(int id) { ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet-examples.md index 4eed0a92..8e766021 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet-examples.md @@ -56,7 +56,7 @@ public class InvoiceService(IInvoiceRepository repository) ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research @@ -112,7 +112,7 @@ What `code-testing-researcher` produces in `.testagent/research.md`: ## Sample Plan Output -What `code-testing-planner` produces in `.testagent/plan.md`: +What `code-testing-planner` produces in `/plan.md`: ```markdown # Test Implementation Plan diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet.md index c0ca63c4..b0664719 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet.md @@ -101,7 +101,7 @@ not part of the test assembly and must never be reported as generated coverage. A new `.csproj` is **invisible** to `dotnet test `, to `dotnet test` run from the repo root, and to any CI/benchmark harness until it is added to the solution. Run `dotnet sln add` *immediately* after creating the project as part of Step 3 ("Register Test Project with Build System") — do not defer it to a later step. -1. Use the exact solution or solution-filter target identified in `.testagent/research.md` or `.testagent/plan.md` — do not search for or substitute a different `.sln`, `.slnx`, or `.slnf` target. +1. Use the exact solution or solution-filter target identified in the research or plan document under `` — do not search for or substitute a different `.sln`, `.slnx`, or `.slnf` target. 2. If that target is a `.sln` or `.slnx`, run `dotnet sln add `. 3. If the target is a `.slnf` (solution filter), also ensure the new project is included in the filter; adding only to the underlying `.sln` may not be enough for test discovery. 4. Skip this if the project is already included in the solution or solution filter used for testing. @@ -112,7 +112,7 @@ A new `.csproj` is **invisible** to `dotnet test `, to `dotnet test` r Before reporting success, run the **harness-equivalent** discovery command from the repo root and confirm the test count went up by at least the number of tests you generated. The harness (CI, msbench, coverage tools) does not know which `.csproj` you targeted — it runs the solution-level command, so a test that passes via `dotnet test MyProject.Tests.csproj` is still worthless if `dotnet test --list-tests` doesn't enumerate it. ```bash -# From repo root, against the solution identified in .testagent/research.md +# From repo root, against the solution identified in /research.md dotnet test --list-tests --no-build 2>&1 | grep -c '^ [A-Za-z]' ``` diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/go-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/go-examples.md index 236f3b8f..6e03a05d 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/go-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/go-examples.md @@ -80,7 +80,7 @@ func (s *InvoiceService) MarkAsPaid(ctx context.Context, id int) error { ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/java-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/java-examples.md index e920a4cb..2fe130f7 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/java-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/java-examples.md @@ -75,7 +75,7 @@ public class InvoiceService { ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/kotlin-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/kotlin-examples.md index a496c13b..2497db26 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/kotlin-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/kotlin-examples.md @@ -57,7 +57,7 @@ class InvoiceService( ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell-examples.md index 119db68a..178c13a1 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell-examples.md @@ -72,7 +72,7 @@ Export-ModuleMember -Function Get-InvoiceTotal, Get-InvoiceById, Set-InvoicePaid ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/python-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/python-examples.md index 6c276015..54995c10 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/python-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/python-examples.md @@ -67,7 +67,7 @@ def _utcnow(): ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research @@ -121,7 +121,7 @@ What `code-testing-researcher` produces in `.testagent/research.md`: ## Sample Plan Output -What `code-testing-planner` produces in `.testagent/plan.md`: +What `code-testing-planner` produces in `/plan.md`: ```markdown # Test Implementation Plan diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby-examples.md index b00c04ac..5b89e8f6 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby-examples.md @@ -60,7 +60,7 @@ end ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/rust-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/rust-examples.md index e34e2979..447f5b84 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/rust-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/rust-examples.md @@ -75,7 +75,7 @@ where ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/typescript-examples.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/typescript-examples.md index 85cd7107..6639d0a7 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/typescript-examples.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/typescript-examples.md @@ -71,7 +71,7 @@ function roundTo2(n: number): number { ## Sample Research Output -What `code-testing-researcher` produces in `.testagent/research.md`: +What `code-testing-researcher` produces in `/research.md`: ```markdown # Test Generation Research @@ -126,7 +126,7 @@ What `code-testing-researcher` produces in `.testagent/research.md`: ## Sample Plan Output -What `code-testing-planner` produces in `.testagent/plan.md`: +What `code-testing-planner` produces in `/plan.md`: ```markdown # Test Implementation Plan diff --git a/tests/dotnet-test/code-testing-agent/eval.yaml b/tests/dotnet-test/code-testing-agent/eval.yaml index 58082de8..27544092 100644 --- a/tests/dotnet-test/code-testing-agent/eval.yaml +++ b/tests/dotnet-test/code-testing-agent/eval.yaml @@ -35,6 +35,8 @@ stimuli: files: - src: fixtures/python-multimodule dest: fixtures/python-multimodule + commands: + - git init -q graders: - type: run-command config: @@ -49,12 +51,11 @@ stimuli: - type: output-matches config: pattern: \|\s*Requirement\s*\|\s*Evidence\s*\| - - type: file-exists + - type: run-command config: - path: .testagent/research.md - - type: file-exists - config: - path: .testagent/plan.md + command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test -f "$state_dir/research.md" && test -f "$state_dir/plan.md" && test -f "$state_dir/status.md" && test -z "$(git ls-files --cached --others -- ':(glob)**/research.md' ':(glob)**/plan.md' ':(glob)**/status.md' ':(exclude,glob)**/node_modules/**')" + expected_exit_code: 0 + timeout: 1m - type: prompt rubric: - Generated passing tests under the configured tests directory for all three modules @@ -62,7 +63,7 @@ stimuli: - Asserted concrete averages, peaks, percentiles, and slugs rather than existence-only results - Kept the work project-wide but bounded to the three supplied modules - Mapped each requested behavior to named test evidence - - Created the broad-scope research, plan, and status artifacts with the target inventory, acceptance mapping, and final quality review + - Kept broad-scope intermediate state files non-stageable so only requested deliverables remain commit candidates - name: Generate project-wide tests for a classic MSTest library prompt: | @@ -85,6 +86,7 @@ stimuli: dest: fixtures/classic-mstest commands: - rm -f fixtures/classic-mstest/tests/DiscountServiceBoundaryTests.cs fixtures/classic-mstest/tests/TieredDiscountPolicyTests.cs && cp fixtures/classic-mstest/tests/Discounts.Tests.csproj.pristine fixtures/classic-mstest/tests/Discounts.Tests.csproj && rm fixtures/classic-mstest/tests/Discounts.Tests.csproj.pristine && mkdir -p .eval-baseline && cp fixtures/classic-mstest/tests/DiscountServiceTests.cs .eval-baseline/DiscountServiceTests.cs && cp fixtures/classic-mstest/tests/packages.config .eval-baseline/packages.config + - git init -q graders: - type: file-exists config: @@ -148,12 +150,11 @@ stimuli: - type: output-matches config: pattern: \|\s*Requirement\s*\|\s*Evidence\s*\| - - type: file-exists + - type: run-command config: - path: .testagent/research.md - - type: file-exists - config: - path: .testagent/plan.md + command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test -f "$state_dir/research.md" && test -f "$state_dir/plan.md" && test -f "$state_dir/status.md" && test -z "$(git ls-files --cached --others -- ':(glob)**/research.md' ':(glob)**/plan.md' ':(glob)**/status.md' ':(exclude,glob)**/node_modules/**')" + expected_exit_code: 0 + timeout: 1m - type: prompt rubric: - Added both requested test files and registered each exactly once in the classic project @@ -161,7 +162,7 @@ stimuli: - Used Assert.ThrowsException for exception paths, avoiding both Assert.Throws and Assert.ThrowsExactly, which are unavailable in MSTest 3.5.2 - Covered every named DiscountService and TieredDiscountPolicy boundary with concrete expected values - Mapped each requested behavior and build-registration requirement to evidence - - Created the broad-scope research, plan, and status artifacts without modernizing the classic project + - Kept broad-scope intermediate state files non-stageable without modernizing the classic project - name: Generate a project-wide Go suite across collaborating packages prompt: | @@ -182,6 +183,8 @@ stimuli: files: - src: fixtures/go-multipackage dest: fixtures/go-multipackage + commands: + - git init -q graders: - type: run-command config: @@ -196,12 +199,11 @@ stimuli: - type: output-matches config: pattern: \|\s*Requirement\s*\|\s*Evidence\s*\| - - type: file-exists + - type: run-command config: - path: .testagent/research.md - - type: file-exists - config: - path: .testagent/plan.md + command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test -f "$state_dir/research.md" && test -f "$state_dir/plan.md" && test -f "$state_dir/status.md" && test -z "$(git ls-files --cached --others -- ':(glob)**/research.md' ':(glob)**/plan.md' ':(glob)**/status.md' ':(exclude,glob)**/node_modules/**')" + expected_exit_code: 0 + timeout: 1m - type: prompt rubric: - Generated passing tests in all three packages @@ -210,7 +212,7 @@ stimuli: - Covered each collaborator error and verified short-circuit behavior when discount calculation fails - Covered shipping equality and overflow boundaries - Mapped every requested behavior to named test evidence - - Created the broad-scope research, plan, and status artifacts with the package inventory, test mapping, and final quality review + - Kept broad-scope intermediate state files non-stageable # The fixture is intentionally committed with a larger project and then # stripped down in setup. The agent must test the remaining source without @@ -229,7 +231,7 @@ stimuli: - src: fixtures/python-workspace-integrity dest: fixtures/python-workspace-integrity commands: - - cd fixtures/python-workspace-integrity && git init -q && git config maintenance.auto false && git config gc.auto 0 && git -c user.email=eval@example.com -c user.name=eval add -A && git -c user.email=eval@example.com -c user.name=eval commit -qm baseline && rm -rf core io tests_real + - git init -q && git config maintenance.auto false && git config gc.auto 0 && git -c user.email=eval@example.com -c user.name=eval add fixtures/python-workspace-integrity && git -c user.email=eval@example.com -c user.name=eval commit -qm baseline && rm -rf fixtures/python-workspace-integrity/core fixtures/python-workspace-integrity/io fixtures/python-workspace-integrity/tests_real graders: - type: run-command config: @@ -249,12 +251,11 @@ stimuli: - type: output-matches config: pattern: \|\s*Requirement\s*\|\s*Evidence\s*\| - - type: file-exists + - type: run-command config: - path: .testagent/research.md - - type: file-exists - config: - path: .testagent/plan.md + command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test -f "$state_dir/research.md" && test -f "$state_dir/plan.md" && test -f "$state_dir/status.md" && test -z "$(git ls-files --cached --others -- ':(glob)**/research.md' ':(glob)**/plan.md' ':(glob)**/status.md' ':(exclude,glob)**/node_modules/**')" + expected_exit_code: 0 + timeout: 1m - type: prompt rubric: - Generated passing pytest tests for the synthstr module under fixtures/python-workspace-integrity/tests/, @@ -267,6 +268,7 @@ stimuli: command, rather than being covered by a broad list of tested areas - Once every requested behavior is covered, evaluates suite quality by meaningful, nonredundant cases rather than rewarding a higher raw test count + - Kept internal intermediate state files non-stageable - name: Generate a layered Vitest suite for an async shopping cart prompt: | @@ -284,6 +286,8 @@ stimuli: files: - src: fixtures/typescript-vitest-cart dest: fixtures/typescript-vitest-cart + commands: + - git init -q graders: - type: run-command config: @@ -298,19 +302,18 @@ stimuli: - type: output-matches config: pattern: \|\s*Requirement\s*\|\s*Evidence\s*\| - - type: file-exists + - type: run-command config: - path: .testagent/research.md - - type: file-exists - config: - path: .testagent/plan.md + command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test -f "$state_dir/research.md" && test -f "$state_dir/plan.md" && test -f "$state_dir/status.md" && test -z "$(git ls-files --cached --others -- ':(glob)**/research.md' ':(glob)**/plan.md' ':(glob)**/status.md' ':(exclude,glob)**/node_modules/**')" + expected_exit_code: 0 + timeout: 1m - type: prompt rubric: - Generated passing Vitest tests for every production module without real I/O - Covered sum and chain discount behavior plus discounted-subtotal tax and shipping composition - Proved snapshot isolation, refreshed prices, partial-stock denial, collaborator failures, and InventoryError fields - Cleared the configured line, statement, function, and branch coverage thresholds - - Mapped each requested behavior to exact test evidence after the broad-scope research, plan, and quality review + - Mapped each requested behavior to exact test evidence after the broad-scope research, plan, and quality review while keeping intermediate state files non-stageable - name: Expand a healthy existing pytest suite to every ledger boundary prompt: | @@ -324,6 +327,8 @@ stimuli: files: - src: fixtures/failing-suite dest: fixtures/failing-suite + commands: + - git init -q graders: - type: run-command config: @@ -338,18 +343,17 @@ stimuli: - type: output-matches config: pattern: \|\s*Requirement\s*\|\s*Evidence\s*\| - - type: file-exists + - type: run-command config: - path: .testagent/research.md - - type: file-exists - config: - path: .testagent/plan.md + command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test -f "$state_dir/research.md" && test -f "$state_dir/plan.md" && test -f "$state_dir/status.md" && test -z "$(git ls-files --cached --others -- ':(glob)**/research.md' ':(glob)**/plan.md' ':(glob)**/status.md' ':(exclude,glob)**/node_modules/**')" + expected_exit_code: 0 + timeout: 1m - type: prompt rubric: - Preserved the existing tests and added passing tests for every requested ledger boundary - Used exact Decimal values around the overdraft limit rather than float approximations - Covered empty and multi-entry running balances plus invalid entry kinds - - Treated the project-wide existing-suite extension as broad scope and completed its research, plan, and final review + - Treated the project-wide existing-suite extension as broad scope, completed its research, plan, and final review, and kept intermediate state files non-stageable - Mapped each requested behavior and the successful pytest command to concrete evidence - Once empty, multi-entry, exact/inside/outside limit, positive-balance, negative-limit, and unknown-kind states all have direct assertions, evaluates completeness by those independent state transitions rather than raw test count @@ -368,6 +372,7 @@ stimuli: dest: fixtures/sdk-xunit-orders commands: - find fixtures/sdk-xunit-orders/tests -type f -name '*.cs' -delete + - git init -q graders: - type: run-command config: @@ -382,19 +387,18 @@ stimuli: - type: output-matches config: pattern: \|\s*Requirement\s*\|\s*Evidence\s*\| - - type: file-exists + - type: run-command config: - path: .testagent/research.md - - type: file-exists - config: - path: .testagent/plan.md + command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test -f "$state_dir/research.md" && test -f "$state_dir/plan.md" && test -f "$state_dir/status.md" && test -z "$(git ls-files --cached --others -- ':(glob)**/research.md' ':(glob)**/plan.md' ':(glob)**/status.md' ':(exclude,glob)**/node_modules/**')" + expected_exit_code: 0 + timeout: 1m - type: prompt rubric: - Generated passing xUnit v3 tests for both production types without modifying production code - Covered every validation and exact decimal pricing boundary - Distinguished reservation start and expiry boundary semantics with concrete instants - Followed the SDK-style project conventions and kept the work bounded to this library - - Completed the broad research, plan, quality review, and requirement-to-test evidence mapping + - Completed the broad research, plan, quality review, and requirement-to-test evidence mapping while keeping intermediate state files non-stageable - name: Add focused xUnit tests for one reservation class prompt: | @@ -409,6 +413,7 @@ stimuli: dest: fixtures/sdk-xunit-orders commands: - find fixtures/sdk-xunit-orders/tests -type f -name '*.cs' -delete && cp fixtures/sdk-xunit-orders/src/ReservationWindow.cs .eval-reservation-window.cs + - git init -q graders: - type: run-command config: @@ -427,8 +432,12 @@ stimuli: && grep -Rq 'ReservationWindow' fixtures/sdk-xunit-orders/tests --include='*.cs' && ! grep -Rq 'OrderPricing' fixtures/sdk-xunit-orders/tests - --include='*.cs' - && test -z \"$(find . -type d -name .testagent -print -quit)\"" + --include='*.cs'" + expected_exit_code: 0 + timeout: 1m + - type: run-command + config: + command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test ! -e "$state_dir/research.md" && test ! -L "$state_dir/research.md" && test ! -e "$state_dir/plan.md" && test ! -L "$state_dir/plan.md" && test ! -e "$state_dir/status.md" && test ! -L "$state_dir/status.md" && test -z "$(git ls-files --cached --others -- ':(glob)**/research.md' ':(glob)**/plan.md' ':(glob)**/status.md' ':(exclude,glob)**/node_modules/**')" expected_exit_code: 0 timeout: 1m - type: output-matches @@ -455,6 +464,8 @@ stimuli: files: - src: fixtures/python-single-function dest: fixtures/python-single-function + commands: + - git init -q graders: - type: run-command config: @@ -468,7 +479,7 @@ stimuli: timeout: 1m - type: run-command config: - command: sh -c "test -z \"$(find . -type d -name .testagent -print -quit)\"" + command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test ! -e "$state_dir/research.md" && test ! -L "$state_dir/research.md" && test ! -e "$state_dir/plan.md" && test ! -L "$state_dir/plan.md" && test ! -e "$state_dir/status.md" && test ! -L "$state_dir/status.md" && test -z "$(git ls-files --cached --others -- ':(glob)**/research.md' ':(glob)**/plan.md' ':(glob)**/status.md' ':(exclude,glob)**/node_modules/**')" expected_exit_code: 0 timeout: 1m - type: output-matches @@ -479,5 +490,5 @@ stimuli: - Generated passing pytest tests for slugify under fixtures/python-single-function/tests/ - Covered separator collapsing/trimming, both truncation paths, non-positive max_length, and empty output - Asserted concrete expected slugs rather than only checking that a string came back - - Kept work proportional to one function with no .testagent artifacts, multi-phase ceremony, or unrelated tests + - Kept work proportional to one function with no intermediate state files, multi-phase ceremony, or unrelated tests - Produced the Requirement | Evidence table with each behavior traced to a named test