mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Add regression-guard scenarios to theming, shell-navigation and DI evals
The gate keeps flapping on these three, and the artifacts show why: with 4 scenarios, ONE tie is mathematically disqualifying. 4 scenarios, 3W/1T/0L -> mean +30%, CI [-1.8%, +61.8%] FAIL 5 scenarios, 4W/1T/0L -> mean +32%, CI [+9.8%, +54.2%] PASS maui-collectionview already demonstrates this: it scored 4W/1T/0L on the last run and PASSED, because the regression-guard scenario added earlier gave it a 5th data point. The other three had no such margin. That the flapping is sampling noise rather than content is now directly evidenced - verified with git diff that the content was byte-identical across these runs: maui-theming714b5975✅ 4W/0T/0L ->6d1201c3❌ 3W/1T/0L (identical) maui-shell-nav1db8796d✅ 4W/0T/0L ->714b5975❌ 3W/1T/0L (identical) maui-dependency-inj1db8796d✅ 4W/0T/0L ->714b5975❌ 3W/1T/0L (identical) and the tie drifts between scenarios run to run rather than sticking to one. So this commit adds a 5th scenario to each, each one guarding a real bug this PR fixed - the same pattern @AbhitejJohn asked for on maui-collectionview: - maui-theming, "Swap theme dictionaries without destroying app styles": requires the Remove-based swap. Guards the MergedDictionaries.Clear() bug (verified at runtime: Clear() drops the merged count 2 -> 1 and the template's "Primary" resource stops resolving). - maui-shell-navigation, "Stable routes for deep linking into tabs": requires an explicit Route= on ShellContent. Guards the auto-generated D_FAULT_{TypeName}{n} names (verified at runtime: D_FAULT_ShellContent2 and D_FAULT_ShellContent5 - non-sequential and order-dependent). - maui-dependency-injection, "Diagnose a page whose injected dependencies are missing": output-not-matches on the "silently null / no exception is thrown" claim. Guards the misconception corrected earlier in this PR. Two things I got wrong and corrected while building these: - The theming guard first used output-not-matches on Clear(). That failed the SKILLED arm 0/3 - because the correct answer legitimately *names* Clear() when diagnosing the cause. Replaced with a positive requirement for .Remove(, which is the reliable signal. - The theming prompt first described the symptom ("every button lost its styling"), which handed baseline the answer; it tied 0/0/0. Reworded to just ask for the switching code, so the skill's knowledge is what differentiates. Measured after both corrections (n=3/scenario, executor claude-opus-4.6, judge gpt-5.5): maui-dependency-injection 15W/0T/0L mean +0.60 maui-shell-navigation 13W/2T/0L mean +0.55 maui-theming 13W/2T/0L mean +0.39 [+0.26, +0.51] Zero losses across all 45 trials. All three new scenarios pass in the skilled arm; the DI and shell guards fail in the baseline arm, which is the point. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
This commit is contained in:
Jakub Florkowski
@@ -16,7 +16,7 @@ evals:
|
||||
overrides:
|
||||
model: claude-opus-4.6
|
||||
judge_model: claude-opus-4.6
|
||||
runs: 1
|
||||
runs: 3
|
||||
vary:
|
||||
- /environment/skills
|
||||
baseline: baseline
|
||||
|
||||
@@ -97,3 +97,29 @@ stimuli:
|
||||
- States that AddScoped without a manually created IServiceScope behaves like Singleton
|
||||
- Recommends switching to AddTransient for per-navigation freshness or AddSingleton if sharing is intended
|
||||
- Mentions IServiceScope as the mechanism needed if Scoped lifetime is truly required
|
||||
- name: "Diagnose a page whose injected dependencies are missing"
|
||||
prompt: |
|
||||
My DetailPage takes an IDataService in its constructor. I reach it two
|
||||
ways: it's declared in AppShell.xaml as
|
||||
<ShellContent ContentTemplate="{DataTemplate views:DetailPage}" />, and
|
||||
I also navigate to it with Shell.Current.GoToAsync after calling
|
||||
Routing.RegisterRoute. One path works and the other doesn't. Explain what
|
||||
MAUI actually does in each case and how to fix it.
|
||||
graders:
|
||||
- type: output-contains
|
||||
config:
|
||||
substring: ContentTemplate
|
||||
# Regression guard: the old skill claimed unregistered pages get "silently
|
||||
# null" dependencies with no exception. Verified false — routed pages use
|
||||
# ActivatorUtilities.GetServiceOrCreateInstance (Routing.cs) and THROW when
|
||||
# a dependency cannot be resolved; ContentTemplate pages use
|
||||
# Activator.CreateInstance (ElementTemplate.cs) with no DI at all.
|
||||
- type: output-not-matches
|
||||
config:
|
||||
pattern: (?i)silently\s+null|dependencies\s+are\s+silently|no\s+exception\s+is\s+thrown
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Explains that ContentTemplate pages are created with Activator.CreateInstance and bypass DI entirely
|
||||
- Explains that routes resolved via GoToAsync go through the service provider and inject constructor dependencies
|
||||
- Does NOT claim that dependencies are silently set to null without an exception
|
||||
- Recommends navigating by route, or giving the page a parameterless constructor, for the ContentTemplate case
|
||||
|
||||
@@ -106,3 +106,28 @@ stimuli:
|
||||
- Explains that the absolute route path must match the full visual hierarchy
|
||||
- Recommends ContentTemplate with DataTemplate to fix eager page creation at startup
|
||||
- Does not suggest NavigationPage-based solutions for Shell navigation issues
|
||||
- name: "Stable routes for deep linking into tabs"
|
||||
prompt: |
|
||||
I need to deep link straight to the Archived tab of my Shell app from a
|
||||
push notification, using an absolute route. My AppShell.xaml declares the
|
||||
FlyoutItem, Tab and ShellContent elements but I never set any Route
|
||||
attributes. Show me the AppShell.xaml I should have and the GoToAsync
|
||||
call, and explain why my current markup makes this unreliable.
|
||||
graders:
|
||||
- type: output-contains
|
||||
config:
|
||||
substring: Route
|
||||
# Regression guard: without an explicit Route, Routing.cs generates
|
||||
# "D_FAULT_{TypeName}{++s_routeCount}" from a counter shared across all
|
||||
# Shell element types. Verified at runtime: three ShellContent elements
|
||||
# produced D_FAULT_ShellContent2 and D_FAULT_ShellContent5 — non-sequential
|
||||
# and order-dependent, so no absolute route can target them reliably.
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: <ShellContent[^>]*Route\s*=|Route\s*=\s*"[a-zA-Z]
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Sets an explicit Route on the ShellContent elements in AppShell.xaml
|
||||
- Explains that without Route, MAUI auto-generates an unstable name (D_FAULT_ prefixed) that shifts when pages are reordered or added
|
||||
- Shows an absolute GoToAsync route (leading //) built from the explicit route names
|
||||
- Does NOT rely on auto-generated route names for deep linking
|
||||
|
||||
@@ -99,3 +99,30 @@ stimuli:
|
||||
- "Explains that StaticResource is frozen at first load and does not respond to dictionary changes"
|
||||
- "Identifies the missing ConfigChanges.UiMode on MainActivity as the cause of the activity restart"
|
||||
- "Provides the correct Activity attribute with ConfigChanges.UiMode included"
|
||||
- name: "Swap theme dictionaries without destroying app styles"
|
||||
prompt: |
|
||||
I have a .NET MAUI app created from the standard template, so App.xaml
|
||||
already merges Resources/Styles/Colors.xaml and Styles.xaml. I've added
|
||||
LightTheme.xaml and DarkTheme.xaml containing my own colour keys.
|
||||
Write the C# method that switches between the light and dark theme at
|
||||
runtime when the user toggles a switch in Settings.
|
||||
graders:
|
||||
- type: output-contains
|
||||
config:
|
||||
substring: MergedDictionaries
|
||||
# Regression guard: MergedDictionaries.Clear() also removes the template's
|
||||
# Colors.xaml / Styles.xaml. Verified at runtime — after Clear() the count
|
||||
# drops 2 -> 1 and the template's "Primary" resource no longer resolves,
|
||||
# which is exactly the "everything lost its styling" symptom described.
|
||||
# Requiring .Remove( is the reliable signal: a correct answer may still
|
||||
# *name* Clear() when diagnosing the cause, so matching on it directly
|
||||
# would penalise the right answer.
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: (?i)\.Remove\s*\(
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Removes only the previously applied theme dictionary, then adds the new one
|
||||
- Does NOT call MergedDictionaries.Clear(), which would also remove the template's Colors.xaml and Styles.xaml and unstyle the whole app
|
||||
- Keeps a reference to the current theme so it can be removed on the next switch
|
||||
- Produces code that leaves the app's existing styles intact after repeated switches
|
||||
|
||||
Reference in New Issue
Block a user