skills/boshu2__agentops
1
0
Fork 0
mirror of https://github.com/boshu2/agentops.git synced 2026-09-14 15:08:13 +08:00
Code Issues Packages Projects Releases Wiki Activity

Converge retained WIP and harden evidence boundaries (#1065)

Browse Source 
Summary:
- lands the audited current WIP lanes and excludes stale/process-only
material
- hardens prune path confinement, probe-v3 evidence binding,
codebase-recon identity, handoff/release/reverse-engineer behavior, and
Codex prompt handling
- truth-labels static skill scoring and regenerates all owning
projections

Validation:
- fresh independent PASS on commit
427098ed10
- full Go suite and full Go race suite
- quick local release CI
- focused Bats, scenario/linkage, native-skill, reverse-engineer,
Cathedral, Ruff, Python ratchet, projection, and diff checks

Residual boundaries:
- final-basename ABA remains unclaimed
- live skill-probe coverage remains honestly 0/12
- release-only cross-build, SBOM, vulnerability, and release-evidence
checks are left to delivery CI
This commit is contained in:
Bo
2026-08-16 18:32:26 -04:00
committed by GitHub
parent 8694b07ebf
commit f3c6d0ecf2
139 changed files with 15437 additions and 1613 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference/skill-quality-rubric.md
+92 -21
View File
@@ -1,35 +1,104 @@
# Skill Quality Rubric
This advisory rubric identifies concrete hardening opportunities. It does not
change a deep-audit verdict and must not reward unnecessary references,
scripts, assets, self-tests, or subagent packets.
A good skill is conformant, safe for its declared effects, and demonstrably
improves the work it claims to help with. Package polish is useful evidence,
but it cannot compensate for a failed safety gate or missing behavioral proof.
## Scoring
This repository therefore reports three independent results. Do not collapse
them into one average:
Each category receives 0–3:
1. **Conformance gate:** the package satisfies the open Agent Skills contract,
the selected host profile, and repository structural checks.
2. **Safety gate:** every bundled file and declared effect has been reviewed;
privileges, writes, network access, credentials, destructive actions,
external content, approval points, bounds, and cleanup match the stated job.
3. **Effectiveness evidence:** representative tasks show correct activation and
better observable outcomes than the no-skill or previous-version baseline.
A failed gate is `FAIL`. A gate or required evidence layer that was not actually
checked is `NOT_PROVEN`, never an inferred pass. An overall quality `PASS`
requires both gates to pass and effectiveness level E2 or E3 below.
## Static package-readiness score
The existing deterministic scorer remains a cheap triage signal. It measures
visible package properties only; it evaluates neither the safety gate nor skill
effectiveness. Each category receives 0–3:
| Score | Meaning |
|---:|---|
| 0 | Missing and required for this skill's actual behavior |
| 0 | Missing and required for this skill's declared behavior |
| 1 | Present but weak, or warranted but absent |
| 2 | Solid for the skill's scope, including “not needed” |
| 2 | Solid for the declared scope, including justified “not needed” |
| 3 | Mechanically strong or unusually complete |
| Category | What good means |
| Category | What the static scorer looks for |
|---|---|
| Trigger quality | Description says what, when, and avoids obvious false positives. |
| Trigger quality | Description says what and when and includes an obvious false-positive boundary. |
| Kernel clarity | The bounded procedure and stop condition are easy to find. |
| Progressive disclosure | A concise kernel is self-contained; complex detail is linked. |
| Progressive disclosure | A concise kernel is self-contained; complex detail is directly linked and loaded only when needed. |
| Helper scripts | Repeated deterministic mechanics are scripted; judgment is not. |
| Validation | Evidence commands or artifacts prove executable behavior. |
| Validation | Commands or artifacts exist that can check executable behavior. |
| Self-test | Trigger or behavior examples exist when complexity warrants them. |
| Assets/templates | Reusable payloads exist only when the workflow actually needs them. |
| Subagents/roles | Delegation packets exist only for intentionally delegated work. |
| Safety boundaries | Mutation, authorization, and non-goals are explicit where relevant. |
| Packaging | The package is small, linked, mode-correct, and projection-safe. |
| Subagents/roles | AgentOps-specific delegation packets exist only for intentionally delegated work. This is not a portable Agent Skills quality criterion. |
| Safety boundaries | Boundary words are visible. This is a signal only, not the safety gate. |
| Packaging | The package is small, linked, host-profile-correct, and projection-safe. |
The maximum remains 30. Rating bands are C (0–10), B (11–20), A (21–26), and
S (27–30). A lower advisory rating is a review signal, not a ship blocker.
The maximum is 30. Static readiness bands are C (0–10), B (11–20), A (21–26),
and S (27–30). A lower band is a review signal, not a ship blocker; a high band
is not an effectiveness or safety claim.
## Effectiveness evidence levels
| Level | Required evidence |
|---|---|
| E0 | No skill-specific behavioral scenario or receipt was located. |
| E1 | A scenario, feature, or self-test exists, but there is no current baseline comparison proving outcome delta. |
| E2 | On the exact skill version, representative direct, indirect, incomplete-input, should-not-trigger, and edge cases pass against an explicit no-skill or prior-version baseline. Activation and output are graded separately. |
| E3 | E2 is repeatable as a regression suite across every intended model, host surface, and realistic installed-skill catalog; traces cover relevant tool calls, guardrails, and handoffs. |
Test observable behavior, not prose presence. Record the target model and host,
skill version or content digest, dataset, grader, baseline, treatment, failures,
and date. Structural validation can establish package conformance; it cannot
establish E2.
## Portable baseline and host profiles
The portable baseline is the [Agent Skills specification](https://agentskills.io/specification):
valid name/directory identity, a useful what-and-when description, relative
resource links, and progressive disclosure. Host rules are additional profiles,
not universal requirements. For example, Codex may impose a tighter metadata
budget or packaging surface than another compatible host.
For progressive disclosure, use the documented working limits: keep `SKILL.md`
under 500 lines and roughly 5,000 tokens, keep references one level deep, make
load conditions explicit, and add a table of contents to long reference files.
Split a skill when triggers, inputs, or success conditions materially differ.
## Current primary sources
- [OpenAI: Build skills for ChatGPT and Codex](https://learn.chatgpt.com/docs/build-skills)
covers focused jobs, front-loaded descriptions, explicit inputs/outputs,
progressive loading, and trigger testing.
- [OpenAI: Build skills for plugins](https://developers.openai.com/plugins/build/skills)
specifies direct, indirect, incomplete-input, should-not-trigger, and edge
cases and separates activation failures from output failures.
- [OpenAI API skills](https://developers.openai.com/api/docs/guides/tools-skills)
treats skill bundles as privileged, initially untrusted code and instructions
and requires bounded, approved high-impact actions.
- [OpenAI agent evaluations](https://developers.openai.com/api/docs/guides/agent-evals)
describes repeatable datasets, graders, and trace inspection.
- [Anthropic skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices)
recommends a no-skill baseline, representative scenarios, observable results,
concise kernels, and testing on every intended model.
- [Anthropic enterprise skills](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/enterprise)
covers full-bundle safety review and catalog-scale recall/coexistence tests.
- [GitHub Copilot agent skills](https://docs.github.com/en/enterprise-cloud@latest/copilot/how-tos/copilot-on-github/customize-copilot/customize-cloud-agent/add-skills)
emphasizes provenance, pinning, preview, and dry-run validation.
- [Microsoft Agent Framework skills](https://learn.microsoft.com/en-us/agent-framework/agents/skills)
covers sandboxing, resource limits, input allowlists, audit logs, and when a
deterministic high-side-effect workflow is a better fit than a skill.
## Required repository checks
@@ -55,10 +124,12 @@ assets or a dedicated delegation tree.
## Audit method
1. Read the complete `SKILL.md` and every linked resource.
2. Compare its declared trigger, boundaries, output, and evidence with actual
behavior.
3. Run the repository checks.
4. Score optional package features relative to demonstrated need.
5. Recommend the smallest change that removes a real defect.
2. Apply the conformance and safety gates without compensation.
3. Run the repository checks and record the static readiness score as triage.
4. Compare declared activation and output behavior with a baseline and grade
the effectiveness evidence E0–E3.
5. Test coexistence in the intended installed catalog and every target model or
host before claiming E3.
6. Recommend the smallest change that removes an observed defect.
Never add ceremony solely to raise a numeric score.