mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Retire invalid filter-syntax direct eval (#1050)
* Retire invalid filter syntax eval Remove the direct identical-arm evaluation and document consumer-level coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align reference skill eval guidance Remove stale filter-syntax direct-eval guidance called out in review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Report reference skill coverage as N/A Avoid treating model-disabled skills without direct evals as uncovered. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea380edb-63a9-40dc-9162-938c4d7754dd --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea380edb-63a9-40dc-9162-938c4d7754dd
This commit is contained in:
Amaury Levé
@@ -52,12 +52,10 @@ floor therefore applies to **skill** evals only. Author agent evals for the
|
||||
scenario coverage and the deterministic graders, and run them as described in Step 10.
|
||||
|
||||
**Be careful with a skill that sets `disable-model-invocation: true`.** The model cannot invoke it,
|
||||
so any eval graded on the skill self-activating compares two identical arms and returns judge noise.
|
||||
The honest coverage for such skills is dependency-level — through the evals of the skills that load
|
||||
them, and through the plugin arm. Two here take the other route and grade the *answer* rather than
|
||||
activation: `tests/dotnet-test/filter-syntax/eval.yaml` and
|
||||
`tests/dotnet-test/platform-detection/eval.yaml`. Whether that produces a measurable gap for a skill
|
||||
the model cannot invoke is still unconfirmed, so read a real verdict before copying the pattern.
|
||||
so the skill is absent from the model-facing skilled arm and any direct eval compares two identical
|
||||
arms. Answer-content graders do not create a difference between those arms. The honest coverage for
|
||||
such skills is dependency-level — through the outcome evals of the skills that load them, and through
|
||||
the plugin arm.
|
||||
|
||||
### Step 2: Write the spec skeleton
|
||||
|
||||
@@ -342,7 +340,7 @@ For the official run, submit a PR review containing `/evaluate` so it binds to t
|
||||
| Timeout too short for code generation | Use ~360s; empty output fails every grader |
|
||||
| Duplicate YAML key left behind by an edit | It overwrites the next stimulus field by field — delete the stray block |
|
||||
| Duplicate stimulus names | Vally uses names as comparison identity — give every stimulus a stable, unique name |
|
||||
| Direct activation-graded eval for a `disable-model-invocation: true` skill | Cover it through a consumer skill, or grade the answer content as `filter-syntax` does |
|
||||
| Direct eval for a `disable-model-invocation: true` skill | Remove it and cover the reference through consumer outcomes |
|
||||
| Agent eval sized for the stimulus floor | `agent.*` evals get no verdict; size them for scenario coverage instead |
|
||||
| Agent eval "run" with `./eng/run-skill-evals.sh` | The glob drops it — use a widened `EXPERIMENT_FILE` |
|
||||
| Agent eval missing `environment.skills` | Declare the skills the agent routes to, or it cannot invoke them |
|
||||
|
||||
@@ -130,10 +130,10 @@ An eval that compares the skill against itself measures judge noise:
|
||||
- A dormancy guard (`expect_activation: false`) must **not** also set `constraints.reject_skills`.
|
||||
That makes the skilled arm skill-free, i.e. identical to baseline. Across four evals the same
|
||||
guard scored −0.4, +0.4, +0.4 and 0, twice costing a skill its pass.
|
||||
- A skill with `disable-model-invocation: true` cannot self-activate, so an eval graded on
|
||||
activation compares two identical arms. Cover it through a consumer skill, or grade the answer
|
||||
content instead, as `tests/dotnet-test/filter-syntax/eval.yaml` and
|
||||
`tests/dotnet-test/platform-detection/eval.yaml` do.
|
||||
- A skill with `disable-model-invocation: true` is absent from the model-facing skilled arm, so its
|
||||
direct eval compares two identical arms regardless of whether graders inspect activation or answer
|
||||
content. Cover it through consumer outcomes instead; for example, `filter-syntax` is covered by
|
||||
`run-tests`, `mtp-hot-reload`, and `migrate-vstest-to-mtp`.
|
||||
- A grader whose `config` is missing its required key enforces nothing, so the stimulus has one
|
||||
fewer assertion than it appears to.
|
||||
|
||||
|
||||
@@ -60,7 +60,7 @@ Consequences seen in real runs:
|
||||
| Symptom | Cause | Fix | Evidence |
|
||||
|---------|-------|-----|----------|
|
||||
| A dormancy guard scores randomly across runs | `expect_activation: false` combined with `constraints.reject_skills`, making the skilled arm skill-free and identical to baseline | Use `expect_activation: false` alone | PR #945, PR #953 |
|
||||
| A reference skill shows no improvement | `disable-model-invocation: true` means the model cannot self-activate it, so an activation-graded eval compares identical arms | Cover it through a consumer skill, or grade answer content as `filter-syntax` does | PR #971, PR #976, issue #899 |
|
||||
| A reference skill shows no improvement | `disable-model-invocation: true` keeps it out of the model-facing skilled arm, so any direct eval compares identical arms | Remove the direct eval and cover the reference through consumer outcomes | PR #971, PR #976, issue #899 |
|
||||
| An eval "passes" while the skill stopped emitting its signature output | No grader asserts the mandated shape | Add a grader for the exact contract (e.g. the `Recommendation:` line) | PR #904 |
|
||||
| Overfit score high, user value unclear | Rubric items reward using the skill, or prompts echo skill vocabulary | Drop them: the harness already reports activation separately, so a rubric never needs to. Keep rubric items outcome-shaped and de-cue the prompt | PR #904 |
|
||||
| Both arms produce the same kind of artifact and the judge falls back on comparing volume | The rubric rewards raw output instead of the property under test | Add anti-hijack criteria: do not invoke the skill, and do not reward quantity (number of tests, findings, or lines produced) | PR #945 |
|
||||
|
||||
@@ -237,6 +237,11 @@ jobs:
|
||||
|
||||
foreach ($r in $reports) {
|
||||
$allJson += $r
|
||||
if ($r.coverageMode -eq 'consumer') {
|
||||
$summary += "| :information_source: | ``$($r.plugin)`` | ``$($r.skill)`` | - | **N/A (reference-only)** |"
|
||||
continue
|
||||
}
|
||||
|
||||
$pct = $r.summary.percentage
|
||||
$icon = if ($pct -ge 80) { ':white_check_mark:' }
|
||||
elseif ($pct -ge 50) { ':warning:' }
|
||||
@@ -264,7 +269,7 @@ jobs:
|
||||
# Add uncovered items details if any
|
||||
$uncoveredDetails = @()
|
||||
foreach ($r in $allJson) {
|
||||
if ($r.uncovered.Count -gt 0) {
|
||||
if ($r.coverageMode -ne 'consumer' -and $r.uncovered.Count -gt 0) {
|
||||
$uncoveredDetails += "<details><summary>Uncovered: <code>$($r.plugin)/$($r.skill)</code></summary>"
|
||||
$uncoveredDetails += ''
|
||||
foreach ($u in $r.uncovered) {
|
||||
|
||||
+15
-23
@@ -361,10 +361,11 @@ carries zero evidence of impact.
|
||||
`<available_skills>` menu, so the model cannot reach it from a user prompt — a
|
||||
consumer skill or agent loads it by name. The experiment's `skilled` variant
|
||||
loads exactly one skill (`plugins/${eval.grandparent}/skills/${eval.parent}`),
|
||||
so a direct-activation eval for one of these would run an arm the model can
|
||||
never invoke: treatment equals control by construction and the head-to-head
|
||||
score is judge noise. That is the same defect failing check 7 exists to prevent,
|
||||
and adding such an eval would make the number worse, not better.
|
||||
so any direct eval for one of these would run an arm the model can never invoke:
|
||||
treatment equals control by construction and the head-to-head score is judge
|
||||
noise. Answer-content graders cannot create a difference between identical
|
||||
arms. That is the same defect failing check 7 exists to prevent, and adding
|
||||
such an eval would make the number worse, not better.
|
||||
|
||||
The honest coverage for these is **dependency-level**: they are exercised
|
||||
through the evals of the skills that load them (for example `run-tests` and
|
||||
@@ -374,26 +375,17 @@ analysis skills load `test-analysis-extensions`, and `code-testing-agent` loads
|
||||
loaded. Closing this properly needs harness support for declaring a dependency
|
||||
in the skilled variant, not a per-skill eval file.
|
||||
|
||||
**A reference skill that already has a direct eval is reported too, and more
|
||||
loudly.** The same argument cuts both ways: if the skilled arm cannot reach the
|
||||
skill, an eval sitting beside it does not measure the skill — it measures the
|
||||
judge comparing baseline to baseline and then labels the result a pass or a
|
||||
fail. That is worse than no eval, because no eval is visibly zero evidence
|
||||
whereas a fabricated verdict is counted in the plugin's pass rate. The gate
|
||||
originally skipped any skill that had an eval, which made the worse case the
|
||||
quieter one; it now names them.
|
||||
**A reference skill that has a direct eval is reported too, and more loudly.**
|
||||
The same argument cuts both ways: if the skilled arm cannot reach the skill, an
|
||||
eval sitting beside it does not measure the skill — it measures the judge
|
||||
comparing baseline to baseline and then labels the result a pass or a fail. That
|
||||
is worse than no eval, because no eval is visibly zero evidence whereas a
|
||||
fabricated verdict is counted in the plugin's pass rate. Remove the direct eval
|
||||
and preserve its scenarios through reachable consumer outcomes instead.
|
||||
|
||||
> **Two `dotnet-test` reference skills currently carry a direct eval:**
|
||||
> `filter-syntax` (added in #976) and `platform-detection` (added in #974).
|
||||
> Their stimuli are ordinary user requests ("one command that runs only the
|
||||
> integration tests but leaves out the slow ones"), so the intent was to grade
|
||||
> the answer on whether it carries the correct syntax rather than on whether the
|
||||
> skill self-activated. Whether that can produce a *measurable* gap over baseline
|
||||
> for a skill the model cannot invoke is still unconfirmed — the evaluation on
|
||||
> #976 landed during the PAT-pool outage and reported "no results", and no
|
||||
> cross-family run has covered either eval since. Read a real result before
|
||||
> copying the pattern to `code-testing-extensions` or `test-analysis-extensions`;
|
||||
> if the gap is zero, retire both evals rather than keep scoring noise.
|
||||
The current `dotnet-test` reference skills — `code-testing-extensions`,
|
||||
`filter-syntax`, and `test-analysis-extensions` — therefore have no direct eval.
|
||||
Their consumer coverage is documented in `plugins/dotnet-test/README.md`.
|
||||
|
||||
### Dormancy guard without an anti-hijack rubric item
|
||||
|
||||
|
||||
@@ -12,6 +12,10 @@
|
||||
This is analogous to code coverage for skill files: it answers "what parts of
|
||||
my skill's guidance are actually verified by eval scenarios?"
|
||||
|
||||
Skills with disable-model-invocation: true are reference-only and cannot have
|
||||
meaningful direct-eval coverage. They are reported with consumer coverage mode
|
||||
and an N/A percentage instead of a misleading zero.
|
||||
|
||||
.PARAMETER PluginName
|
||||
Plugin directory name (e.g., "dotnet-test").
|
||||
|
||||
@@ -88,6 +92,19 @@ function Get-CoveragePoints([string]$content) {
|
||||
@(Get-CodePatterns $content)
|
||||
}
|
||||
|
||||
function Test-ReferenceSkill([string]$content) {
|
||||
if (-not $content.StartsWith('---')) { return $false }
|
||||
|
||||
$frontmatter = [regex]::Match(
|
||||
$content,
|
||||
'\A---\s*\r?\n(?<body>.*?)\r?\n---(?:\r?\n|$)',
|
||||
[System.Text.RegularExpressions.RegexOptions]::Singleline
|
||||
)
|
||||
if (-not $frontmatter.Success) { return $false }
|
||||
|
||||
$frontmatter.Groups['body'].Value -match '(?m)^disable-model-invocation:\s*true\s*$'
|
||||
}
|
||||
|
||||
function Get-ValidationItems([string]$content) {
|
||||
$lineNum = 0
|
||||
$inValidation = $false
|
||||
@@ -765,6 +782,24 @@ function Format-JsonReport($results, $skillName, $pluginName, $scenarioCount, $e
|
||||
$report
|
||||
}
|
||||
|
||||
function Format-ReferenceJsonReport($skillName, $pluginName) {
|
||||
[ordered]@{
|
||||
skill = $skillName
|
||||
plugin = $pluginName
|
||||
coverageMode = 'consumer'
|
||||
scenarios = $null
|
||||
evidence = $null
|
||||
summary = [ordered]@{
|
||||
totalPoints = $null
|
||||
coveredPoints = $null
|
||||
rubricOnlyPoints = $null
|
||||
percentage = $null
|
||||
}
|
||||
categories = [ordered]@{}
|
||||
uncovered = @()
|
||||
}
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# Discovery & Main
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@@ -819,6 +854,21 @@ $jsonReports = @()
|
||||
|
||||
foreach ($pair in $pairs) {
|
||||
$skillContent = Get-Content -Raw $pair.SkillPath
|
||||
|
||||
if (Test-ReferenceSkill $skillContent) {
|
||||
if ($Format -eq 'Json') {
|
||||
$jsonReports += Format-ReferenceJsonReport $pair.SkillName $pair.PluginName
|
||||
}
|
||||
else {
|
||||
Write-Host ''
|
||||
Write-Host " $($pair.PluginName)/$($pair.SkillName)" -ForegroundColor Cyan
|
||||
Write-Host ' Reference-only skill: direct eval coverage is not applicable.' -ForegroundColor DarkCyan
|
||||
Write-Host ' Measure its guidance through reachable consumer outcomes.' -ForegroundColor DarkCyan
|
||||
Write-Host ''
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
$coveragePoints = @(Get-CoveragePoints $skillContent)
|
||||
|
||||
if ($coveragePoints.Count -eq 0) {
|
||||
|
||||
@@ -76,20 +76,21 @@ For non-.NET languages, use the native coverage tool: `coverage.py`/`pytest-cov`
|
||||
|
||||
Three reference skills (`code-testing-extensions`, `test-analysis-extensions`,
|
||||
and `filter-syntax`) set `disable-model-invocation: true`, so the CLI keeps them
|
||||
out of the model-facing skill menu and a consumer loads them by name. Two
|
||||
(`code-testing-extensions`, `test-analysis-extensions`) deliberately have no
|
||||
`tests/dotnet-test/<skill>/eval.yaml`: the experiment's skilled arm loads a
|
||||
single skill, which the model could never invoke here, so such an eval would
|
||||
compare two identical arms and score judge noise. They are measured through the
|
||||
evals of the skills that load them — the polyglot analysis skills and
|
||||
`grade-tests` for `test-analysis-extensions`, and `code-testing-agent` for
|
||||
`code-testing-extensions`.
|
||||
out of the model-facing skill menu and a consumer loads them by name. They
|
||||
deliberately have no direct `tests/dotnet-test/<skill>/eval.yaml`: the
|
||||
experiment's skilled arm loads a single skill, which the model could never
|
||||
invoke here, so such an eval would compare two identical arms and score judge
|
||||
noise. They are measured through consumer outcomes — the polyglot analysis
|
||||
skills and `grade-tests` for `test-analysis-extensions`, `code-testing-agent`
|
||||
for `code-testing-extensions`, and `run-tests`, `mtp-hot-reload`, and
|
||||
`migrate-vstest-to-mtp` for `filter-syntax`. The `run-tests` eval covers VSTest
|
||||
expressions, MTP argument passing, xUnit v3 native filters, and TUnit tree-node
|
||||
filters; the migration eval covers translating VSTest filters to xUnit v3's MTP
|
||||
syntax.
|
||||
|
||||
`platform-detection` is model-invocable because identifying a project's runner
|
||||
is also a direct user task; `run-tests` and migration skills still load it as
|
||||
shared detection guidance. `filter-syntax` remains reference-only. Its current
|
||||
direct eval cannot measure activation and is retained only until consumer-level
|
||||
coverage replaces it.
|
||||
shared detection guidance. `filter-syntax` remains reference-only.
|
||||
|
||||
## Agents
|
||||
|
||||
|
||||
@@ -1,172 +0,0 @@
|
||||
name: filter-syntax
|
||||
executionShard: b
|
||||
description: Evaluates the dotnet-test/filter-syntax skill
|
||||
type: capability
|
||||
config:
|
||||
timeout: 4m
|
||||
stimuli:
|
||||
- name: Select one category and exclude another on a VSTest project
|
||||
prompt: >
|
||||
Our CI job for Contoso.Billing.Tests currently runs the whole suite and it
|
||||
takes far too long. I want one command that runs only the integration
|
||||
tests but leaves out the ones we marked as slow. I'd rather not touch the
|
||||
test code or the project file to make this happen.
|
||||
environment:
|
||||
files:
|
||||
- src: ./fixtures/vstest-mstest
|
||||
dest: .
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: --filter
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: TestCategory
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: '!=|!~'
|
||||
- type: exit-success
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Gave a single command that both selects the integration tests and excludes the slow ones
|
||||
- Used a negated operator for the exclusion rather than proposing two separate runs
|
||||
- Combined the two conditions into one expression with the AND combinator
|
||||
- Did not propose editing the test source or the project file to achieve the selection
|
||||
constraints:
|
||||
reject_tools:
|
||||
- edit
|
||||
- create
|
||||
|
||||
- name: Pass a filter to a Microsoft.Testing.Platform project on the .NET 9 SDK
|
||||
prompt: >
|
||||
I ran `dotnet test --filter "TestCategory=Smoke"` on Contoso.Shipping.Tests and
|
||||
the SDK rejected the option instead of running my smoke tests. The project
|
||||
builds fine and the tests all run when I don't filter. What command should
|
||||
I actually be running here, and will it change when we move to a newer SDK?
|
||||
environment:
|
||||
files:
|
||||
- src: ./fixtures/mtp-nunit-sdk9
|
||||
dest: .
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: --\s+--filter
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: Category
|
||||
- type: exit-success
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Gave a command that forwards the filter to the test application rather than to the SDK command line itself
|
||||
- Attributed the rejected option to how this project is run, not to a malformed filter expression
|
||||
- Did not tell the user to abandon the property-and-operator expression for a different filter syntax, because this
|
||||
framework accepts it unchanged on this platform
|
||||
- Stated whether the extra separator is still required on a newer SDK
|
||||
constraints:
|
||||
reject_tools:
|
||||
- edit
|
||||
- create
|
||||
|
||||
- name: Filter xUnit v3 tests that do not accept the generic filter expression
|
||||
prompt: >
|
||||
Contoso.Catalog.Tests is on xUnit v3 and `--filter "FullyQualifiedName~SearchIntegrationTests"`
|
||||
does nothing — every test still runs. I need three things: run just the
|
||||
SearchIntegrationTests class, run everything tagged Smoke, and finally run
|
||||
only the Smoke-tagged tests under the Integration namespace. That last one
|
||||
has a constraint: our pipeline template only ever passes a single filter
|
||||
option and its value, so it cannot pass two options. What do I use?
|
||||
environment:
|
||||
files:
|
||||
- src: ./fixtures/mtp-xunit-v3
|
||||
dest: .
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: --filter-class
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: --filter-trait
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: --filter-query
|
||||
- type: exit-success
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Explained that this runner does not honour the generic property-expression filter, which is why nothing was excluded
|
||||
- Gave a working command for the single-class selection using the runner's own option
|
||||
- Gave a working command for the tag-based selection using the runner's own option
|
||||
- Expressed the namespace-and-tag selection with a single filter option, appending the tag selector to the method
|
||||
segment as `[traitName=traitValue]`
|
||||
- Commands account for the separator the installed SDK requires
|
||||
constraints:
|
||||
reject_tools:
|
||||
- edit
|
||||
- create
|
||||
|
||||
- name: Filter a TUnit suite down to one class and one property value
|
||||
prompt: >
|
||||
Contoso.Portal.Tests is a TUnit project. I want to know how to run just the
|
||||
LoginTests class, how to run a single test by name no matter which class it
|
||||
is in, how to run everything tagged with the Smoke category, and how to skip
|
||||
everything tagged Slow. Give me the exact commands.
|
||||
environment:
|
||||
files:
|
||||
- src: ./fixtures/tunit
|
||||
dest: .
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: --treenode-filter
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: /\*/
|
||||
- type: exit-success
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Used the path-shaped selector this framework provides rather than a property-expression filter
|
||||
- Used wildcards for the path segments the request does not constrain
|
||||
- Showed both the include-by-property and the exclude-by-property forms
|
||||
- Explained what each position in the path stands for, so the user can build further selections
|
||||
constraints:
|
||||
reject_tools:
|
||||
- edit
|
||||
- create
|
||||
|
||||
- name: Translate CI filter expressions after moving to xUnit v3
|
||||
prompt: |
|
||||
We are moving Contoso.Catalog.Tests from the old runner to xUnit v3 and our
|
||||
pipeline has these four filter arguments baked into different jobs:
|
||||
|
||||
1. `FullyQualifiedName~SearchIntegrationTests`
|
||||
2. `FullyQualifiedName=Contoso.Catalog.Tests.Integration.PricingIntegrationTests.Price_WithTax_IncludesVat`
|
||||
3. `Category=Smoke`
|
||||
4. `FullyQualifiedName~PricingIntegrationTests&Category=Smoke`
|
||||
|
||||
Give me the replacement for each job. Do not change any files.
|
||||
environment:
|
||||
files:
|
||||
- src: ./fixtures/mtp-xunit-v3
|
||||
dest: .
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: --filter-class
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: --filter-method
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: --filter-trait
|
||||
- type: exit-success
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Stated that the four expressions do not keep working unchanged on this runner
|
||||
- Turned the substring match on the class into a class selection with wildcards on both sides
|
||||
- Turned the exact fully-qualified name into an exact method selection
|
||||
- Turned the category condition into a tag selection carrying a name and a value
|
||||
- Translated the fourth expression, which combines a class condition and a tag condition, into a form this runner
|
||||
actually accepts
|
||||
constraints:
|
||||
reject_tools:
|
||||
- edit
|
||||
- create
|
||||
@@ -1,15 +0,0 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net9.0</TargetFramework>
|
||||
<IsPackable>false</IsPackable>
|
||||
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.13.0" />
|
||||
<PackageReference Include="NUnit" Version="4.2.2" />
|
||||
<PackageReference Include="NUnit3TestAdapter" Version="4.6.0" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -1,27 +0,0 @@
|
||||
using NUnit.Framework;
|
||||
|
||||
namespace Contoso.Shipping.Tests;
|
||||
|
||||
[TestFixture]
|
||||
public class RateCalculatorTests
|
||||
{
|
||||
[Test]
|
||||
[Category("Unit")]
|
||||
public void Rate_DomesticParcel_UsesFlatFee() { Assert.Pass(); }
|
||||
|
||||
[Test]
|
||||
[Category("Smoke")]
|
||||
public void Rate_InternationalParcel_AddsSurcharge() { Assert.Pass(); }
|
||||
}
|
||||
|
||||
[TestFixture]
|
||||
public class LabelPrinterTests
|
||||
{
|
||||
[Test]
|
||||
[Category("Smoke")]
|
||||
public void Print_ValidLabel_ReturnsPdfBytes() { Assert.Pass(); }
|
||||
|
||||
[Test]
|
||||
[Category("Slow")]
|
||||
public void Print_TenThousandLabels_CompletesWithinBudget() { Assert.Pass(); }
|
||||
}
|
||||
@@ -1,6 +0,0 @@
|
||||
{
|
||||
"sdk": {
|
||||
"version": "9.0.200",
|
||||
"rollForward": "latestFeature"
|
||||
}
|
||||
}
|
||||
@@ -1,16 +0,0 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net9.0</TargetFramework>
|
||||
<OutputType>Exe</OutputType>
|
||||
<IsPackable>false</IsPackable>
|
||||
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.13.0" />
|
||||
<PackageReference Include="xunit.v3" Version="1.0.1" />
|
||||
<PackageReference Include="xunit.runner.visualstudio" Version="3.0.1" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -1,25 +0,0 @@
|
||||
using Xunit;
|
||||
|
||||
namespace Contoso.Catalog.Tests.Integration;
|
||||
|
||||
public class SearchIntegrationTests
|
||||
{
|
||||
[Fact]
|
||||
[Trait("Category", "Smoke")]
|
||||
public void Search_KnownTerm_ReturnsHits() { Assert.True(true); }
|
||||
|
||||
[Fact]
|
||||
[Trait("Category", "Nightly")]
|
||||
public void Search_FullReindex_Completes() { Assert.True(true); }
|
||||
}
|
||||
|
||||
public class PricingIntegrationTests
|
||||
{
|
||||
[Fact]
|
||||
[Trait("Category", "Smoke")]
|
||||
public void Price_WithTax_IncludesVat() { Assert.True(true); }
|
||||
|
||||
[Fact]
|
||||
[Trait("Category", "Nightly")]
|
||||
public void Price_UnknownSku_Throws() { Assert.True(true); }
|
||||
}
|
||||
@@ -1,21 +0,0 @@
|
||||
using Xunit;
|
||||
|
||||
namespace Contoso.Catalog.Tests.Unit;
|
||||
|
||||
public class SearchQueryParserTests
|
||||
{
|
||||
[Fact]
|
||||
[Trait("Category", "Smoke")]
|
||||
public void Parse_SingleTerm_ReturnsTerm() { Assert.True(true); }
|
||||
|
||||
[Fact]
|
||||
[Trait("Category", "Regression")]
|
||||
public void Parse_UnbalancedQuotes_Throws() { Assert.True(true); }
|
||||
}
|
||||
|
||||
public class PriceFormatterTests
|
||||
{
|
||||
[Fact]
|
||||
[Trait("Category", "Smoke")]
|
||||
public void Format_WholeAmount_HasTwoDecimals() { Assert.True(true); }
|
||||
}
|
||||
@@ -1,6 +0,0 @@
|
||||
{
|
||||
"sdk": {
|
||||
"version": "9.0.200",
|
||||
"rollForward": "latestFeature"
|
||||
}
|
||||
}
|
||||
@@ -1,13 +0,0 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<OutputType>Exe</OutputType>
|
||||
<IsPackable>false</IsPackable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="TUnit" Version="1.45.8" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -1,26 +0,0 @@
|
||||
using System.Threading.Tasks;
|
||||
using TUnit.Core;
|
||||
|
||||
namespace Contoso.Portal.Tests.Api;
|
||||
|
||||
public class LoginTests
|
||||
{
|
||||
[Test]
|
||||
[Category("Smoke")]
|
||||
public async Task AcceptCookiesTest() { await Task.CompletedTask; }
|
||||
|
||||
[Test]
|
||||
[Category("Slow")]
|
||||
public async Task LoginWithExpiredPasswordTest() { await Task.CompletedTask; }
|
||||
}
|
||||
|
||||
public class SignupTests
|
||||
{
|
||||
[Test]
|
||||
[Category("Smoke")]
|
||||
public async Task SignupWithValidEmailTest() { await Task.CompletedTask; }
|
||||
|
||||
[Test]
|
||||
[Category("Slow")]
|
||||
public async Task SignupRejectsDuplicateEmailTest() { await Task.CompletedTask; }
|
||||
}
|
||||
@@ -1,33 +0,0 @@
|
||||
using Microsoft.VisualStudio.TestTools.UnitTesting;
|
||||
|
||||
namespace Contoso.Billing.Tests;
|
||||
|
||||
[TestClass]
|
||||
public class InvoiceTests
|
||||
{
|
||||
[TestMethod]
|
||||
[TestCategory("Unit")]
|
||||
public void CreateInvoice_ValidLines_ComputesTotal() { Assert.IsTrue(true); }
|
||||
|
||||
[TestMethod]
|
||||
[TestCategory("Integration")]
|
||||
public void CreateInvoice_PersistsToDatabase() { Assert.IsTrue(true); }
|
||||
|
||||
[TestMethod]
|
||||
[TestCategory("Integration")]
|
||||
[TestCategory("Slow")]
|
||||
public void ReconcileLedger_FullMonth_Balances() { Assert.IsTrue(true); }
|
||||
}
|
||||
|
||||
[TestClass]
|
||||
public class PaymentTests
|
||||
{
|
||||
[TestMethod]
|
||||
[TestCategory("Unit")]
|
||||
public void Charge_NegativeAmount_Throws() { Assert.IsTrue(true); }
|
||||
|
||||
[TestMethod]
|
||||
[TestCategory("Integration")]
|
||||
[TestCategory("Slow")]
|
||||
public void Charge_RealGateway_Succeeds() { Assert.IsTrue(true); }
|
||||
}
|
||||
@@ -1,14 +0,0 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net8.0</TargetFramework>
|
||||
<IsPackable>false</IsPackable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />
|
||||
<PackageReference Include="MSTest.TestAdapter" Version="3.6.3" />
|
||||
<PackageReference Include="MSTest.TestFramework" Version="3.6.3" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
Reference in New Issue
Block a user