19 Commits

Author SHA1 Message Date
Chris Banes 9d982e1d9f Add Android benchmark comparison skill (#69) 2026-09-11 19:46:22 +01:00
Chris Banes 91fc03562b Refine public developer documentation guidance (#68) 2026-09-11 19:37:49 +01:00
Chris Banes 2db11bb412 Add Kotlin library release skill (#66)
Add `release-kotlin-library` with an adapted Haze release helper. The
workflow reconciles an existing changelog against changes since the
previous release, resolves versions, and follows the repository's checks
and publishing mechanism. Release files are prepared and validated
before publication; completion requires artifact and remote Git
readback.

The helper separates preparation, publication, and recovery, loads
required credentials from `~/.env` without surfacing raw child output,
rejects unrelated staged work and divergent remote branches, and
preserves check-owned edits when preparation fails. README/router
integration and direct, novel, and no-change evaluation cases are
included. Versions are unchanged.

Validation: the full `npm test` suite passed (307 tests, one existing
Linux-only skip on macOS), followed by the final staged-file regression
test. After merging current main, all 40 affected tests, Markdown lint,
and validation of all 94 evaluation cases passed. Python compilation
also passed. Standards and spec reviews have no remaining blocking
findings. Publication tests use isolated repositories and synthetic
credentials; live publishing and the model evaluation benchmark were not
run.
2026-09-06 19:20:41 +01:00
Chris Banes f23bb5bd53 Clarify workflow authorization and verification reuse (#65) 2026-09-06 08:39:48 +01:00
Chris Banes 84c2c53a26 Document external skill providers (#64)
## Summary

- identify `implement` as an external workflow from `mattpocock/skills`
and make its current companion requirements actionable
- document every required, conditional, and optional external provider
used by the bundled workflows
- add missing-provider calibration cases for `implement-with-subagents`
and `run-github-project`

## Testing

- `npm run lint`
- `python3 evals/run.py validate --suite workflows-writing`
- `npm test`

Fixes #63
2026-09-04 09:25:46 +01:00
Chris Banes 7cd0ae127f Remove unsupported paths from skill frontmatter (#62)
## Summary

- remove the unsupported paths field from the Kotlin and Compose router
skill
- stop allowing paths in the repository frontmatter schema
- correct the README and add a regression assertion

## Validation

- npm run lint
- python3 -m unittest discover -s evals/tests -v (143 tests)
- git diff --check

Fixes #61
2026-09-03 16:34:04 +01:00
Chris Banes f3934dff32 Add Agent Plugins v1 manifest (#60)
## Summary

- add the portable Agent Plugins v1.0.0 root manifest
- keep release versioning and validation synchronized across the
portable, Claude, Codex, and OpenCode manifests
- align the repository skill schema with Agent Skills core fields while
retaining the existing Claude Code `disable-model-invocation` and
`paths` extensions

## Validation

- `npm run lint`
- `npm test` (280 tests passed, 1 skipped)
- `python3 evals/run.py validate` (79 cases)
- `python3 scripts/release.py validate-manifests 2026.9.2`
- canonical Agent Plugins v1.0.0 JSON Schema validation
- `git diff --check`
2026-09-02 17:15:32 +01:00
Chris Banes cff239a9a3 Teach concurrency skill to flag detached JVM work (#59)
## Summary

- Route raw `Thread` and `Executor` usage through the Kotlin concurrency
skill.
- Add structured migration guidance for suspend APIs, owned scopes and
dispatchers, interruptible blocking work, and executor-backed
dispatchers.
- Add direct, novel, and no-change evaluation coverage, then update the
documented Kotlin suite counts.

## Validation

- Skill quick validation passed.
- `npm run lint` passed.
- Evaluation corpus validation passed with 79 cases, including 22 Kotlin
cases.
- 22 focused CLI and Kotlin Gradle matrix tests passed.
- The direct fixture passed its Gradle red/green check.
- `git diff --check` passed.

Live behavioural model evaluation was not run; the planned forced and
automatic comparison would require 12 model calls.
2026-09-02 13:16:42 +01:00
Chris Banes 5e63396e8f Make advanced workflow skills explicit-only (#58)
## Summary
- make the four advanced workflow skills user-invokable only in both UI
metadata and SKILL.md frontmatter
- make automatic evaluation and reporting exclude explicit-only skills
while retaining forced coverage
- document and validate the frontmatter contract

## Validation
- npm test
- npm run evals:validate
- git diff --check

## Not run
- npm run lint (the required remark executable is unavailable in both
sandbox and host environments)
2026-09-02 11:37:15 +01:00
Chris Banes 0a83a9414f Restore skill evals and compare efficiency (#57)
## Summary

- restore evaluation behaviour after the recent skill simplification
changes
- capture subject token, tool-call, turn, elapsed-time, and retry
telemetry
- compare per-skill baseline and automatic efficiency with compact
values and percentage changes
- keep the result tables in the root and evaluation READMEs synchronized

## Validation

- npm test
- npm run lint
- npm run evals:validate
- git diff --check

The evaluation results are advisory and remain non-gating.
2026-08-27 16:14:58 +01:00
Chris Banes 305816026f Make shepherd fully verify CI failures (#56)
## Summary

- switch `shepherd` from narrow CI-driven repair loops to full locally
available CI-equivalent verification after a code-related failure
- batch all discovered repairs into one verified push and report checks
unavailable locally
- update the direct and novel evaluation cases plus the README

## Validation

- `npm run lint`
- `npm test` (250 tests, 1 skipped)
- `npm run evals:validate` (73 cases)
- `python3
/Users/chris/.codex/skills/.system/skill-creator/scripts/quick_validate.py
skills/shepherd`
- `git diff --check`
2026-08-27 11:28:36 +01:00
Chris Banes 8321e57057 Reduce skill instruction verbosity (#55) 2026-08-26 22:08:33 +01:00
Chris Banes 3b7eac4226 Document evaluation model provenance (#53)
## Summary

- identify `gpt-5.6-terra` with medium reasoning as the subject behind
the published evaluation scores
- identify `gpt-5.6-sol` with high reasoning as the judge
- link both official model pages and clarify that results are model- and
reasoning-specific

## Testing

- `npm run lint`
- `git diff --check`
2026-08-24 16:23:48 +01:00
Chris Banes f046bdbf0a Publish recertified Kotlin skill scores (#51)
## Summary

- publish the current-HEAD 100.0% Automatic result for kotlin-api-design
- publish the current-HEAD 100.0% Restraint result for
kotlin-concurrency-and-flow
- keep the result tables in README.md and evals/README.md synchronized

## Evaluation

- kotlin-api-design Automatic: 12/12 outcomes passed across four cases
and three repetitions
- kotlin-concurrency-and-flow Restraint: 3/3 forced and 3/3 automatic
controls passed
- zero objective failures, violations, forbidden actions, process
failures, or retries
- reported API routing precision was 100.0% and recall was 85.7%; both
routing gates passed

## Validation

- npm test (247 passed, 1 skipped)
- npm run lint
- npm run evals:validate (61 cases)
- git diff --check
2026-08-24 15:51:41 +01:00
Chris Banes 45ff6d67a8 Recertify changed Kotlin and Gradle skill evaluations (#50) 2026-08-23 12:14:33 +01:00
Chris Banes 8c87c8c28e Strengthen evaluation coverage and Gradle workflow evidence handling (#48)
## Summary

- Require evaluation coverage when adding, renaming, or removing skills.
- Simplify evaluation documentation and consolidate current score
reporting.
- Improve Gradle workflow evidence recovery for piped and logged
commands.
- Add deterministic tests for workflow recovery, write boundaries, and
real-wrapper preflight validation.
- Clarify `gradle-run` task selection and verification-reporting
requirements.

## Testing

Not run (not requested).
2026-08-21 16:04:36 +01:00
Chris Banes b439cb3d98 Expand evaluator to Kotlin and Gradle skills (#47)
## Summary

- generalize the advisory evaluator around suite-specific policies
- add a 19-case Kotlin/Gradle suite covering `gradle-run` and three
Kotlin skills
- add the deterministic Kotlin/JVM fixture, safety checks, reporting,
and regression coverage
- document the evaluated cohort and separately fingerprinted rubric
follow-up
- ignore generated Gradle caches and build output for all eval fixtures

## Testing

- `npm run evals:validate`
- `python3 evals/run.py validate --suite kotlin-gradle`
- `python3 -m unittest evals.tests.test_kotlin_gradle_matrix`
- `npm test`
- `npm run lint`
- `git diff --check`
2026-08-20 14:47:35 +01:00
Chris Banes ded78abbe5 Certify Compose skill evaluation results (#45) 2026-08-18 20:38:09 +01:00
Chris Banes 493f5c5444 Centralize evaluator policy and subject output validation (#42)
## Summary

- Remove per-case forbidden skill and action fields now covered by
global evaluator policy
- Centralize subject skill canonicalization and output validation in the
Codex harness
- Reuse the shared parsing and reporting helpers across experiment
execution
- Update evaluator documentation to describe global safety checks

## Testing

Not run (not requested)
2026-08-17 20:56:50 +01:00