feat(socratic): sharpen WP advisory exemption clause — decorated title-form shells (#505) (#507)

* feat(socratic): sharpen WP advisory exemption — decorated title-form shells (#505)

Closes #505. Narrows the exemption clause in both socratic_mentor_agent.md
files to require a named or operationalized specific (instrument/scale name,
named theory/model/dataset/policy instrument, named site or population,
specified causal pathway, or stated tension), declares domain-flavored
topic-label pairs swappable, and adds the decorated-compound-title rule
(noun-swap the part after the colon on its own).

Measured against the held-out acceptance set per its README protocol
(2 replicates, claude-sonnet-5 judges): overall miss 0.375/0.344 -> 0.094
in both replicates, off-list 0.778 -> 0.222/0.111, false-fire 0/16 held in
both; no shell missed in both replicates; on-list gold set non-regression
(regex detector untouched, sentinel tests 7/7). New in-prompt example
strings substring-checked against every held-out item (zero hits).

Deliverables: measurement-2026-07-11-505.json, README post-#505 section,
judge reasoning excerpts appended, audit report. CHANGELOG also documents
the already-merged #506 (#484 FARS caveat) for the pre-tag coverage gate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UMzbXv6Q1NkCaGmaza3p5r

* feat(socratic): refine #505 exemption per codex P2s + final-wording re-measurement

Round-2 refinements from cross-model review: generic demographic descriptors
excluded from "named population"; decorated-title rule extended to single-topic
subtitles ("X in/among Z"). Audit "5 of 7" corrected to 4 of 7; CHANGELOG
"decorated title shells" corrected to "off-list shells". Measurement re-run
from scratch on the final wording (2 replicates): overall miss 0.094 in both,
false-fire 0/16 in both, no shell missed in both replicates; one judge
prose-verdict inversion recorded faithfully. JSON now carries all four
post-#505 runs (round1 + final).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UMzbXv6Q1NkCaGmaza3p5r

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -12,6 +12,10 @@ All notable changes to this project will be documented in this file.
 
 ### Changed
 
+- **WP advisory exemption sharpening — decorated title-form shells now caught (#505; direction from the #501 Part 2 measurement).** The exemption clause in both `socratic_mentor_agent.md` files (deep-research + academic-paper) is narrowed: it now requires a *named or operationalized* specific (an actual instrument/scale name, a named theory/model/dataset/policy instrument, a named site or population, a specified causal pathway — through what mediator/condition/process A relates to B, not merely that it does — or a stated tension between two identified explanations), declares ordinary domain-flavored topic-label pairs swappable, and adds a decorated-compound-title rule (an evocative pre-colon phrase plus a generic "X and Y (in Z)" subtitle gains no specificity from the decoration — the noun-swap test applies to the part after the colon alone). This closes the failure mechanism the #501 Part 2 baseline measured: judges reading generic topical noun pairs as the exemption's "specific mechanism", which rescued 7/9 off-list shells (six decorated titles plus one interrogative). Measured against the held-out acceptance set per its README protocol in two rounds (initial wording, then a cross-model-review-driven refinement — demographic descriptors excluded from "named population", single-topic subtitles covered — re-measured from scratch; 2 replicates each, same judge model): overall miss 0.375/0.344 → **0.094 in all four post-#505 runs**, off-list 0.778 → all 9 items fired in at least one final replicate (final rep2: 9/9), false-fire 0/16 preserved in every run (including the four hard negatives carrying listed surface substrings); no shell missed in both final replicates; on-list gold set unaffected (regex detector untouched, fnr=0/fpr=0). All #505 constraints held: WP table unextended, advisory stays non-blocking and surface-phrasing-only, sentinel contract (`test_check_rq_framing_patterns.py`) unchanged; the new in-prompt example strings were substring-checked against every held-out item (zero hits) so the set stays held out. Measurement JSON `evals/heldout/rq_framing_offlist/measurement-2026-07-11-505.json` + reasoning excerpts appended; report at `audits/rq-advisory-505-exemption-sharpening-2026-07-11.md`. Closes #505.
+
+- **Reviewer calibration protocol notes LLM-as-judge leniency direction (#484 → PR #506, merged).** `academic-paper-reviewer/references/calibration_mode_protocol.md` gains a directional-prior subsection under "Failure cases this mode does NOT fix": when the simulated panel's output is read as a pass/fail signal, assume leniency relative to human expert review until your own calibration shows otherwise, anchored to FARS (Tang et al. 2026, arXiv:2606.31651 — automated reviewer mean 5.00 over 165 papers vs 3.23 paper-level mean from 282 human expert reviews over 140 papers; a descriptive ~1.8-point gap, and the automated score functioned only as a relative ranking). The direction is a working prior (heuristic extrapolation from one measured setup, default-until-measured); the magnitude is explicitly non-portable — never a correction factor or threshold change. Docs only; the panel remains advisory infrastructure behind human checkpoints. FARS added to References.
+
 - **WP advisory generalization, Part 1 (#501; direction from the PR #468 review thread, @brycewang-stanford).** Both `socratic_mentor_agent.md` files (deep-research + academic-paper) now state that the WP01-WP20 table is illustrative, not exhaustive, and name the operative judgment: the noun-swap test (phrasing is shell-like when it survives swapping its nouns for any other field's nouns). Off-list shells that clearly survive the swap may fire the advisory at the same high-confidence bar; domain-native phrasing that names a mechanism, instrument, site, or tension does not survive it and must not trigger. Advisory stays non-blocking and surface-phrasing-only; sentinel contract unchanged (`test_check_rq_framing_patterns.py`). Part 2 (held-out miss-rate measurement) landed separately — see the Added entry above.
 
 - **API-first retrieval refresh: OpenAlex API-key auth, budget-aware 429 handling, arXiv ToU-aligned backoff (#495; proposed by [@pikaqiu2333](https://github.com/pikaqiu2333)).** OpenAlex's current developer docs are API-key-first (freemium daily budget; the polite pool is no longer documented): `scripts/openalex_client.py` gains `OPENALEX_API_KEY` support (query-param auth; either credential selects the authenticated 10 req/s pacing tier, `OPENALEX_POLITE_EMAIL` stays as legacy compat), distinguishes daily-budget-exhausted 429s (`X-RateLimit-Remaining: 0` → raise `OpenAlexUnavailable` immediately — the budget refills at midnight UTC, so an in-process retry cannot succeed) from transient burst 429s (exponential backoff 2s → 4s → 8s per OpenAlex's documented guidance), and strips the query string from refusal-path error messages so the key never lands in logs (`scripts/crossref_client.py` gets the same redaction — its query string carries the polite-pool `mailto` email). `scripts/arxiv_client.py`'s 429 backoff moves from the shared 2s constant to the 3s ToU pacing floor (arXiv's Terms of Use ask for at most one request every three seconds — a sub-3s retry would itself violate the pacing the 429 enforces; verified verbatim against the ToU page). Both protocol docs (`deep-research/references/openalex_api_protocol.md`, `arxiv_api_protocol.md`) updated in lockstep, plus an explicit retrieval-order boundary in each: structured APIs are the primary channel, browser/WebFetch page inspection is a bounded first-party fallback whose output is data-not-instructions (`shared/ground_truth_isolation_pattern.md` §2A), and browser retrieval is never a rate-limit bypass (no parallel browsing, no bulk PDF harvesting, no multi-machine fan-out). 5 new client tests; the two 429-behavior tests updated to pin the new backoff shapes.

academic-paper/agents/socratic_mentor_agent.md

@@ -54,7 +54,7 @@ When the user proposes a paper RQ, thesis sentence, literature-gap statement, or
 | WP19 | technology-enhancement shell | "role of technology/AI/digital tools in enhancing Y" |
 | WP20 | experience-of frame | "exploring the experiences of X in/with Y" |
 
-**The table is illustrative, not exhaustive.** The 20 rows document the most common shells, not the full space of AI-typical phrasing. The operative judgment is the noun-swap test: phrasing is shell-like when it would survive swapping its nouns for any other field's nouns without losing meaning. Wording that matches no row but clearly survives the noun-swap test (for example "unpacking the dynamics of X in Y", "a deep dive into X", "rethinking X in the age of Y", "interrogating the nexus between X and Y") may fire the advisory at the same high-confidence bar, citing the closest pattern family or "off-list shell". Phrasing that names a specific mechanism, instrument, site, or tension does not survive the swap and must not trigger, whether or not it resembles a row.
+**The table is illustrative, not exhaustive.** The 20 rows document the most common shells, not the full space of AI-typical phrasing. The operative judgment is the noun-swap test: phrasing is shell-like when it would survive swapping its nouns for any other field's nouns without losing meaning. Wording that matches no row but clearly survives the noun-swap test (for example "unpacking the dynamics of X in Y", "a deep dive into X", "rethinking X in the age of Y", "interrogating the nexus between X and Y") may fire the advisory at the same high-confidence bar, citing the closest pattern family or "off-list shell". Phrasing that names a specific mechanism, instrument, site, or tension does not survive the swap and must not trigger, whether or not it resembles a row — but read this exemption narrowly: it requires a named or operationalized specific, such as an actual instrument or scale name (e.g. "the PSS-10"), a named theory, model, dataset, or policy instrument, a named site or population (a particular institution, region, or cohort — a generic demographic descriptor is not a named population), a specified causal pathway (through what mediator, condition, or process A relates to B — not merely that A relates to B), or a stated tension between two identified explanations. Ordinary topic labels do not qualify: domain-flavored noun pairs ("urban mobility and quality of life", "online privacy and consumer trust") are still swappable nouns, and pairing them is still a shell. A decorated compound title — an evocative pre-colon phrase plus a generic "X and Y (in Z)" subtitle, for example "Roots of Resilience: Community Networks and Disaster Recovery" — gains no specificity from the decoration: apply the noun-swap test to the part after the colon on its own, whether it is a noun pair or a single topic label ("X in/among Z").
 
 When triggered, surface a single concise advisory and immediately return to Socratic questioning:
 

audits/rq-advisory-505-exemption-sharpening-2026-07-11.md

@@ -0,0 +1,122 @@
+# Wording-Pattern Advisory — exemption-clause sharpening (issue #505)
+
+**Date:** 2026-07-11
+**Scope:** the `## Wording-Pattern Advisory` section in both
+`deep-research/agents/socratic_mentor_agent.md` and
+`academic-paper/agents/socratic_mentor_agent.md` (identical change in both).
+**Provenance chain:** PR #468 review thread → #501 → PR #503/#504 (baseline
+measurement, miss rate HIGH) → issue #505 → this change.
+
+## The change
+
+The #501 Part 2 measurement located the judge's miss-rate gap in one mechanism:
+the exemption clause — "names a specific mechanism, instrument, site, or
+tension" — was being applied to generic topical noun pairs ("nurse workload",
+"cybersecurity training"), which rescued decorated compound-title shells from
+firing (7/9 off-list items missed in both baseline replicates).
+
+The exemption sentence in the post-#503 illustrative paragraph is extended (the
+sentence itself and everything before it are unchanged) with:
+
+1. **A narrow-exemption test.** The exemption now requires a *named or
+   operationalized* specific: an actual instrument/scale name, a named theory,
+   model, dataset, or policy instrument, a named site or population (a generic
+   demographic descriptor is not a named population), a specified causal pathway
+   (through what mediator/condition/process A relates to B — not merely that it
+   does), or a stated tension between two identified explanations. Ordinary
+   topic labels — domain-flavored noun pairs — are declared swappable.
+2. **A decorated-compound-title rule.** An evocative pre-colon phrase plus a
+   generic "X and Y (in Z)" subtitle gains no specificity from the decoration;
+   the noun-swap test applies to the part after the colon on its own, whether it
+   is a noun pair or a single topic label ("X in/among Z").
+
+The final wording is the product of two rounds: the initial sharpening was
+measured (round 1, PASS), then an independent cross-model review P2 observed
+that "named population" could still be read to cover generic demographic labels
+(exactly the reading behind round 1's `ti-012`/`ti-013` single-replicate
+misses); the two clarifying clauses above were added and the measurement re-run
+from scratch on the final wording (round 2 = the acceptance runs).
+
+Per the #505 constraint set: the WP table is untouched, the advisory stays
+non-blocking and surface-phrasing-only, and the sentinel contract pinned by
+`test_check_rq_framing_patterns.py` is unchanged (all 7 tests pass).
+
+**Contamination guard.** All example strings introduced by the change
+("the PSS-10", "urban mobility and quality of life", "online privacy and
+consumer trust", "Roots of Resilience: Community Networks and Disaster
+Recovery") were substring-checked against every held-out item before
+measurement — zero hits — so no held-out item became an in-prompt example the
+way the four #503 examples did.
+
+## Measurement (per the set README protocol)
+
+Same protocol as the baseline: isolated `claude-sonnet-5` sub-agent judges,
+verbatim advisory section (variant under test) + 6 shuffled items per batch
+(4 shells + 2 domain-native), 8 batches covering the 48 items, 2 replicates per
+round, no labels, no repo access. Boolean outcomes for all four runs:
+`evals/heldout/rq_framing_offlist/measurement-2026-07-11-505.json`.
+
+| variant | overall miss | family_variant (n=23) | off_list (n=9) | false-fire (n=16) |
+|---------|-------------|----------------------|----------------|-------------------|
+| post-#503 rep1 (baseline) | 12/32 = 0.375 | 5/23 = 0.217 | 7/9 = 0.778 | 0/16 = 0.000 |
+| post-#503 rep2 (baseline) | 11/32 = 0.344 | 4/23 = 0.174 | 7/9 = 0.778 | 0/16 = 0.000 |
+| post-#505 round1 rep1 | 3/32 = 0.094 | 1/23 = 0.043 | 2/9 = 0.222 | 0/16 = 0.000 |
+| post-#505 round1 rep2 | 3/32 = 0.094 | 2/23 = 0.087 | 1/9 = 0.111 | 0/16 = 0.000 |
+| post-#505 FINAL rep1 | 3/32 = 0.094 | 0/23 = 0.000 | 3/9 = 0.333 | 0/16 = 0.000 |
+| post-#505 FINAL rep2 | 3/32 = 0.094 | 3/23 = 0.130 | 0/9 = 0.000 | 0/16 = 0.000 |
+
+## Findings
+
+1. **Acceptance: PASS in all four runs.** Overall FNR 0.094 sits well under
+   the 0.30 line in every run (baseline: 0.34–0.38 above it); false-fire stays
+   0/16 in every run, including the four hard negatives that deliberately carry
+   listed surface substrings inside fully specified designs. The on-list gold
+   set (`evals/gold/rq_framing_patterns/`) is unaffected — the offline regex
+   detector is untouched and its calibration test still passes at fnr=0/fpr=0.
+2. **The decorated-title shape is closed.** All 9 off-list items — including
+   the 7 baseline stable misses — fired in at least one FINAL replicate, and
+   FINAL rep2 fired on all 9. Captured judge reasoning now argues these through
+   the new rule ("post-colon part is a swappable topic label; decorative prefix
+   ignored per rule") instead of reading topic nouns as "specific mechanism".
+3. **No stable misses remain.** No shell was missed in both FINAL replicates
+   (nor in both round-1 replicates). The single-replicate flips are
+   boundary-adjacent items of the same magnitude as the baseline's observed
+   between-replicate flip; each replicate independently passes.
+4. **One judge anomaly, recorded faithfully.** FINAL rep1's three off-list
+   misses (`ti-002/004/007`) come from a single batch judge whose prose
+   reasoning described the items as swappable shells while its JSON verdicts
+   said SILENT — a prose-verdict inversion. Per the protocol the boolean JSON
+   is the record, so the run is scored as-is (and still passes). The same three
+   items fired in FINAL rep2 and in both round-1 replicates.
+5. **Exemption reasoning on negatives is unchanged in kind.** Judges continue to
+   exempt via named instruments (MISSCARE, IUS-12/GAD-7, Technostress Creators),
+   named sites/populations (Changhua fishing communities, Taipei market), named
+   theory/policy instruments (DeLone & McLean, EU ETS MSR), specified pathways
+   (dn-007/dn-008), and stated tensions (dn-016) — the narrow-exemption list
+   matches how the hard negatives actually earn silence. The added
+   demographic-descriptor clause did not flip any negative: `nat-059`
+   ("bilingual speakers") stays silent via the trigger-side high-confidence
+   bar, and every `dn-*` negative carries a named specific beyond its
+   demographic wording.
+
+## Caveats
+
+- Same judge model, same day, same single-generator English-only set as the
+  baseline — all baseline caveats (model/time drift, n=9 off-list, annotator-2
+  family overlap) carry over. Re-run the protocol rather than reusing numbers.
+- The fix author and the measurement runner are the same session. The exposure
+  is bounded: ground-truth labels predate the change, scoring is mechanical
+  boolean comparison, and judges are isolated with no labels — but wording
+  choices in the change were informed by the same captured reasoning the
+  measurement audits, which is the intended design loop, not an independent
+  validation.
+- `off_list` n=9 remains small; the per-tier rates carry wide uncertainty. The
+  load-bearing claim is the paired disappearance of the stable-miss set, not
+  the exact tier rate.
+
+## Disposition
+
+- Issue #505 closes with this change + measurement.
+- The held-out set remains the acceptance test for future advisory changes
+  (FNR < 0.30 / FPR < 0.20, ≥2 replicates, plus on-list gold set
+  non-regression), per #501's decision rule.

deep-research/agents/socratic_mentor_agent.md

@@ -53,7 +53,7 @@ After the user proposes a research direction or draft RQ, run a light wording/fr
 | WP19 | technology-enhancement shell | "role of technology/AI/digital tools in enhancing Y" |
 | WP20 | experience-of frame | "exploring the experiences of X in/with Y" |
 
-**The table is illustrative, not exhaustive.** The 20 rows document the most common shells, not the full space of AI-typical phrasing. The operative judgment is the noun-swap test: phrasing is shell-like when it would survive swapping its nouns for any other field's nouns without losing meaning. Wording that matches no row but clearly survives the noun-swap test (for example "unpacking the dynamics of X in Y", "a deep dive into X", "rethinking X in the age of Y", "interrogating the nexus between X and Y") may fire the advisory at the same high-confidence bar, citing the closest pattern family or "off-list shell". Phrasing that names a specific mechanism, instrument, site, or tension does not survive the swap and must not trigger, whether or not it resembles a row.
+**The table is illustrative, not exhaustive.** The 20 rows document the most common shells, not the full space of AI-typical phrasing. The operative judgment is the noun-swap test: phrasing is shell-like when it would survive swapping its nouns for any other field's nouns without losing meaning. Wording that matches no row but clearly survives the noun-swap test (for example "unpacking the dynamics of X in Y", "a deep dive into X", "rethinking X in the age of Y", "interrogating the nexus between X and Y") may fire the advisory at the same high-confidence bar, citing the closest pattern family or "off-list shell". Phrasing that names a specific mechanism, instrument, site, or tension does not survive the swap and must not trigger, whether or not it resembles a row — but read this exemption narrowly: it requires a named or operationalized specific, such as an actual instrument or scale name (e.g. "the PSS-10"), a named theory, model, dataset, or policy instrument, a named site or population (a particular institution, region, or cohort — a generic demographic descriptor is not a named population), a specified causal pathway (through what mediator, condition, or process A relates to B — not merely that A relates to B), or a stated tension between two identified explanations. Ordinary topic labels do not qualify: domain-flavored noun pairs ("urban mobility and quality of life", "online privacy and consumer trust") are still swappable nouns, and pairing them is still a shell. A decorated compound title — an evocative pre-colon phrase plus a generic "X and Y (in Z)" subtitle, for example "Roots of Resilience: Community Networks and Disaster Recovery" — gains no specificity from the decoration: apply the noun-swap test to the part after the colon on its own, whether it is a noun pair or a single topic label ("X in/among Z").
 
 When triggered, surface a single concise advisory and immediately return to Socratic questioning:
 

evals/heldout/rq_framing_offlist/README.md

@@ -103,3 +103,34 @@ one generator model) and the judgments are model- and time-specific and drift
 across versions — re-run rather than reuse the numbers.
 
 Full write-up: `audits/rq-advisory-heldout-measurement-2026-07-11.md`.
+
+## 2026-07-11 post-#505 result (see `measurement-2026-07-11-505.json`)
+
+The #505 exemption sharpening (narrow named/operationalized-specific exemption +
+decorated-compound-title rule; both `socratic_mentor_agent.md` files) was measured
+against this set per the protocol above, in two rounds: round 1 on the initial
+wording, then — after a cross-model review P2 refined the wording (generic
+demographic descriptors excluded from "named population"; the decorated-title
+rule extended to single-topic subtitles "X in/among Z") — a fresh 2-replicate
+round on the FINAL shipped wording:
+
+| variant | overall miss | family_variant | off_list | false-fire |
+|---------|-------------|----------------|----------|------------|
+| post-#505 round1 rep1 | 0.094 | 0.043 | 0.222 | 0.000 |
+| post-#505 round1 rep2 | 0.094 | 0.087 | 0.111 | 0.000 |
+| post-#505 FINAL rep1 | 0.094 | 0.000 | 0.333 | 0.000 |
+| post-#505 FINAL rep2 | 0.094 | 0.130 | 0.000 | 0.000 |
+
+**Verdict: PASS** — all four runs sit well under the FNR < 0.30 line and the
+zero over-warning property (0/16 false-fire, including the four hard negatives
+carrying listed surface substrings) holds in every run. The decorated-title
+shape that carried the baseline gap (0.778) is now caught: FINAL rep2 fired on
+all 9 off-list items; FINAL rep1's three off-list misses (`ti-002/004/007`)
+trace to a single judge whose prose reasoning called the items swappable shells
+but whose JSON verdicts said SILENT — the boolean record stands per protocol,
+and the run still passes. No shell was missed in both FINAL replicates. All
+#505 prompt example strings were substring-checked against every held-out item
+before measurement (no hits), so the set remains held out of the in-prompt
+examples. Full write-up:
+`audits/rq-advisory-505-exemption-sharpening-2026-07-11.md`. The model/time-drift
+caveat above applies unchanged — re-run, don't reuse.

evals/heldout/rq_framing_offlist/judge_reasoning_excerpts.md

@@ -1,4 +1,8 @@
-# Judge reasoning excerpts — 2026-07-11 measurement
+# Judge reasoning excerpts — 2026-07-11 measurements
+
+> Sections below the baseline blocks were appended for the post-#505
+> exemption-sharpening runs (same day, same protocol); see
+> `measurement-2026-07-11-505.json`.
 
 Verbatim excerpts of the judge agents' prose reasoning, captured alongside the
 boolean outcomes in `measurement-2026-07-11.json`. Not every agent produced prose
@@ -54,3 +58,49 @@ advisory section variant + 6 items.
 - `nat-077` (pre-#503, MISS): "close to WP16 … but not the canonical 'barriers
   and facilitators to X' pair shell … weak/ambiguous match, not high-confidence."
   (Post-#503: FIRE citing WP16 in both replicates.)
+
+## Post-#505 runs — the decorated-title shape argued through the new rule
+
+- `ti-002` (post-#505 round1 rep1, FIRE): "colon-decorated title; post-colon part is
+  'Nurse Workload and the Quality of Patient Care,' a generic swappable
+  noun-pair (WP01/WP06 shell). Decoration doesn't exempt it." (Baseline rep2
+  argued the same item as "names a specific mechanism/population" and missed.)
+- `ti-004` (post-#505 round1 rep2, FIRE): "'Cybersecurity Training and Employee
+  Security Behavior' — bare study-of/factors-style noun pair, no named
+  instrument or mechanism specified. Swappable shell."
+- `ti-013` (post-#505 round1 rep2, FIRE): "'Aging Apart' decorated title; subtitle
+  'Social Isolation Among Older Adults' is a bare topic label with no named
+  site, instrument, or mechanism." (Rep1 missed the same item as "subtitle is
+  named-population topic label … not swappable-shell form" — the single-topic
+  subtitle sits nearest the boundary.)
+- `el-013` (post-#505 round1 both reps, FIRE): "'How does X bear on Y' … no named
+  instrument/theory/site, generic material/durability pairing is swappable."
+  (Stable miss in every baseline run under the broad-exemption reading.)
+
+## Post-#505 runs — negatives still exempted, and via the narrow list
+
+- `dn-007` (post-#505 round1 rep1, correct SILENT): "names mechanism (unobtrusive
+  indicators in annual reports), specific causal pathway (through TMT power
+  centralization) — satisfies specified-pathway exemption."
+- `dn-016` (post-#505 round1 rep2, correct SILENT): "names specific mechanism
+  (telicity constraints, lexical aspect) and a specified causal pathway (L1
+  transfer vs. aspect), plus named population." — the stated-tension /
+  pathway arms of the narrow exemption doing the work.
+- `nat-044` (post-#505 round1 rep2, MISS): "plain descriptive question without a
+  shell connective (impact/relationship/role/factors), so weak match, stay
+  silent." — the remaining interrogative off-list miss rides the
+  high-confidence bar, not the exemption clause.
+
+## Post-#505 FINAL-wording runs — single-topic subtitles caught; one prose-verdict inversion
+
+- `ti-013` (FINAL rep2, FIRE): "'Social Isolation Among Older Adults' is a
+  generic topic label (age group is generic demographic, not named population);
+  decorative prefix ignored." — the round-2 demographic-descriptor clause doing
+  exactly the work the round-1 miss motivated.
+- `ti-012` (FINAL rep2, FIRE): "'Words in the Making' is decorative pre-colon
+  phrase, ignored per rule" — single-topic subtitle judged on its own.
+- `ti-002`/`ti-004`/`ti-007` (FINAL rep1, MISS — anomaly): the batch judge's
+  prose called each "swappable shell / generic 'X and Y' noun-pair, no named
+  instrument" yet its JSON verdicts said SILENT — a prose-verdict inversion.
+  The boolean record stands per protocol; the same items fired in FINAL rep2
+  and both round-1 replicates.

evals/heldout/rq_framing_offlist/measurement-2026-07-11-505.json

@@ -0,0 +1,262 @@
+{
+ "measured_at": "2026-07-11",
+ "variant_round1": "post-#505 exemption sharpening, intermediate wording (before the codex-review P2 refinement)",
+ "variant_final": "post-#505 FINAL shipped wording (adds: generic demographic descriptor is not a named population; decorated-title rule covers single-topic subtitles 'X in/among Z')",
+ "judge_model": "claude-sonnet-5 (Claude Code Agent tool, isolated sub-agents, batch of 6 items, 4 shells + 2 domain-native per batch)",
+ "judge_input": "verbatim '## Wording-Pattern Advisory' section from deep-research/agents/socratic_mentor_agent.md (variant under test), plus the six batch items; no other context",
+ "decision_rule": "advisory fires / stays silent per item, judged independently; boolean JSON verdict is the record even where an agent's prose contradicts it (one such prose-verdict inversion observed in post_505_final_rep1 on the ti-002/ti-004/ti-007 batch)",
+ "thresholds_inherited_from": "evals/gold/rq_framing_patterns/manifest.yaml (FNR < 0.30, FPR < 0.20)",
+ "baseline": "measurement-2026-07-11.json (pre-#503 / post-#503 variants)",
+ "runs": {
+  "post_505_round1_rep1": {
+   "el-001": true,
+   "el-007": true,
+   "dn-002": false,
+   "el-003": true,
+   "el-005": true,
+   "dn-001": false,
+   "dn-003": false,
+   "el-010": true,
+   "el-016": true,
+   "el-017": true,
+   "el-013": true,
+   "dn-005": false,
+   "nat-012": true,
+   "nat-004": true,
+   "dn-006": false,
+   "nat-001": true,
+   "nat-014": true,
+   "dn-007": false,
+   "dn-009": false,
+   "dn-008": false,
+   "nat-037": true,
+   "nat-028": true,
+   "nat-018": true,
+   "nat-021": true,
+   "dn-011": false,
+   "nat-049": true,
+   "nat-048": true,
+   "nat-044": true,
+   "nat-045": true,
+   "dn-012": false,
+   "nat-077": false,
+   "nat-065": true,
+   "dn-013": false,
+   "nat-061": true,
+   "dn-014": false,
+   "nat-053": true,
+   "ti-002": true,
+   "ti-004": true,
+   "dn-018": false,
+   "ti-007": true,
+   "ti-001": true,
+   "dn-016": false,
+   "ti-013": false,
+   "ti-012": false,
+   "ti-008": true,
+   "nat-059": false,
+   "dn-019": false,
+   "ti-010": true
+  },
+  "post_505_round1_rep2": {
+   "dn-001": false,
+   "el-001": true,
+   "el-007": true,
+   "el-005": true,
+   "dn-002": false,
+   "el-003": true,
+   "dn-005": false,
+   "el-010": true,
+   "dn-003": false,
+   "el-013": true,
+   "el-016": true,
+   "el-017": true,
+   "nat-004": true,
+   "nat-012": true,
+   "dn-006": false,
+   "dn-007": false,
+   "nat-014": true,
+   "nat-001": true,
+   "nat-028": true,
+   "nat-018": true,
+   "nat-021": true,
+   "nat-037": true,
+   "dn-008": false,
+   "dn-009": false,
+   "dn-012": false,
+   "nat-049": false,
+   "nat-045": true,
+   "nat-048": true,
+   "dn-011": false,
+   "nat-044": false,
+   "nat-053": false,
+   "nat-061": true,
+   "dn-013": false,
+   "dn-014": false,
+   "nat-065": true,
+   "nat-077": true,
+   "ti-001": true,
+   "ti-004": true,
+   "dn-016": false,
+   "ti-002": true,
+   "ti-007": true,
+   "dn-018": false,
+   "nat-059": false,
+   "dn-019": false,
+   "ti-012": true,
+   "ti-010": true,
+   "ti-013": true,
+   "ti-008": true
+  },
+  "post_505_final_rep1": {
+   "el-001": true,
+   "el-007": true,
+   "dn-002": false,
+   "el-003": true,
+   "el-005": true,
+   "dn-001": false,
+   "dn-003": false,
+   "el-010": true,
+   "el-016": true,
+   "el-017": true,
+   "el-013": true,
+   "dn-005": false,
+   "nat-012": true,
+   "nat-004": true,
+   "dn-006": false,
+   "nat-001": true,
+   "nat-014": true,
+   "dn-007": false,
+   "dn-009": false,
+   "dn-008": false,
+   "nat-037": true,
+   "nat-028": true,
+   "nat-018": true,
+   "nat-021": true,
+   "dn-011": false,
+   "nat-049": true,
+   "nat-048": true,
+   "nat-044": true,
+   "nat-045": true,
+   "dn-012": false,
+   "nat-077": true,
+   "nat-065": true,
+   "dn-013": false,
+   "nat-061": true,
+   "dn-014": false,
+   "nat-053": true,
+   "ti-002": false,
+   "ti-004": false,
+   "dn-018": false,
+   "ti-007": false,
+   "ti-001": true,
+   "dn-016": false,
+   "ti-013": true,
+   "ti-012": true,
+   "ti-008": true,
+   "nat-059": false,
+   "dn-019": false,
+   "ti-010": true
+  },
+  "post_505_final_rep2": {
+   "dn-001": false,
+   "el-001": true,
+   "el-007": true,
+   "el-005": true,
+   "dn-002": false,
+   "el-003": true,
+   "dn-005": false,
+   "el-010": false,
+   "dn-003": false,
+   "el-013": true,
+   "el-016": true,
+   "el-017": true,
+   "nat-004": true,
+   "nat-012": true,
+   "dn-006": false,
+   "dn-007": false,
+   "nat-014": true,
+   "nat-001": true,
+   "nat-028": true,
+   "nat-018": true,
+   "nat-021": true,
+   "nat-037": true,
+   "dn-008": false,
+   "dn-009": false,
+   "dn-012": false,
+   "nat-049": false,
+   "nat-045": true,
+   "nat-048": true,
+   "dn-011": false,
+   "nat-044": true,
+   "nat-053": false,
+   "nat-061": true,
+   "dn-013": false,
+   "dn-014": false,
+   "nat-065": true,
+   "nat-077": true,
+   "ti-001": true,
+   "ti-004": true,
+   "dn-016": false,
+   "ti-002": true,
+   "ti-007": true,
+   "dn-018": false,
+   "nat-059": false,
+   "dn-019": false,
+   "ti-012": true,
+   "ti-010": true,
+   "ti-013": true,
+   "ti-008": true
+  }
+ },
+ "metrics": {
+  "post_505_round1_rep1": {
+   "overall_miss": "3/32 = 0.094",
+   "family_variant_miss": "1/23 = 0.043",
+   "off_list_miss": "2/9 = 0.222",
+   "false_fire": "0/16 = 0.000",
+   "missed_items": [
+    "nat-077",
+    "ti-012",
+    "ti-013"
+   ],
+   "false_fired_items": []
+  },
+  "post_505_round1_rep2": {
+   "overall_miss": "3/32 = 0.094",
+   "family_variant_miss": "2/23 = 0.087",
+   "off_list_miss": "1/9 = 0.111",
+   "false_fire": "0/16 = 0.000",
+   "missed_items": [
+    "nat-044",
+    "nat-049",
+    "nat-053"
+   ],
+   "false_fired_items": []
+  },
+  "post_505_final_rep1": {
+   "overall_miss": "3/32 = 0.094",
+   "family_variant_miss": "0/23 = 0.000",
+   "off_list_miss": "3/9 = 0.333",
+   "false_fire": "0/16 = 0.000",
+   "missed_items": [
+    "ti-002",
+    "ti-004",
+    "ti-007"
+   ],
+   "false_fired_items": []
+  },
+  "post_505_final_rep2": {
+   "overall_miss": "3/32 = 0.094",
+   "family_variant_miss": "3/23 = 0.130",
+   "off_list_miss": "0/9 = 0.000",
+   "false_fire": "0/16 = 0.000",
+   "missed_items": [
+    "el-010",
+    "nat-049",
+    "nat-053"
+   ],
+   "false_fired_items": []
+  }
+ }
+}