Nick Nisi dc9dfb093c fix(workos): tighten auth guidance and risky eval reruns (#25) ...

* fix(workos): include required --org flag in portal generate-link examples

The "Not in the CLI" table in workos-management.md listed
`workos portal generate-link --intent=sso` and
`workos portal generate-link --intent=dsync` as the correct way to
reach the Admin Portal for connection creation. Both omit the
required --org flag and fail before opening the Portal. This is the
same hallucination shape the PR is trying to prevent agents from
producing — caught in review.

Add --org=<org_id> to both rows, matching the Quick Reference
entry at the top of the file.

* test(workos): tighten no-CLI assertion in connection-create evals

The first assertion in evals 8 and 9 was titled "States connection
creation is NOT in the CLI" but its needles included Admin Portal
and WorkOS Dashboard. An answer that only said "Use the Admin
Portal" without ever stating CLI support is absent passed the
assertion. Since the second assertion in each eval already checks
for the Admin Portal/Dashboard destination, the first was both
redundant and weakened.

Remove destination terms from the first assertion so it genuinely
checks for the explicit no-CLI signal.

* test(workos): improve risky eval coverage

Add targeted recipes for SSO, AuthKit, and RBAC cases that showed negative or noisy eval deltas. Tighten brittle flow-step expectations where the previous wording rewarded incidental prose ordering instead of the intended behavior.

Add multi-case eval filtering and an eval:risk helper so the latest triage cases can be rerun with more samples and saved outputs.

* Fix explicit triage handling in risk reruns

* fix: format:check script incorrectly using prettier

* chore: format files for oxfmt check

2026-04-26 09:06:35 -05:00

3.7 KiB

Raw Permalink Blame History

View Raw