skills/chrisbanes__skills
1
0
Fork 0
mirror of https://github.com/chrisbanes/skills.git synced 2026-09-19 08:37:48 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add Android benchmark comparison skill (#69)

Browse Source
This commit is contained in:
Chris Banes
2026-09-11 19:46:22 +01:00
committed by GitHub
parent 91fc03562b
commit 9d982e1d9f
21 changed files with 226 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+7
View File
@@ -68,12 +68,17 @@ for its lane-specific fallback and blocking behavior.
- Working on Compose state or effects? Start with [`compose-state-and-effects`](skills/compose-state-and-effects/SKILL.md).
- Investigating recomposition, stability, or jank? Start with [`compose-performance`](skills/compose-performance/SKILL.md).
- Comparing Android benchmark configurations or a measured Android default? Start with [`android-benchmark-comparison`](skills/android-benchmark-comparison/SKILL.md).
- Reviewing Flow or coroutine architecture? Start with [`kotlin-concurrency-and-flow`](skills/kotlin-concurrency-and-flow/SKILL.md).
### Routing
- [`using-chrisbanes-skills`](skills/using-chrisbanes-skills/SKILL.md) — route Kotlin and Jetpack Compose work to the focused skills.
### Benchmarking
- [`android-benchmark-comparison`](skills/android-benchmark-comparison/SKILL.md) — compare physical Android benchmark configurations with verified coverage, controlled conditions, trace-backed diagnosis of unstable rankings, and bounded conclusions.
### Jetpack Compose
#### State and side effects
@@ -197,6 +202,7 @@ for Astra and 5.6 coverage and its current evidence limits.
| [`kotlin-api-design`](skills/kotlin-api-design/SKILL.md) | 66.7% | 100.0% | 100.0% |
| [`kotlin-concurrency-and-flow`](skills/kotlin-concurrency-and-flow/SKILL.md) | 33.3% | 100.0% | 100.0% |
| [`kotlin-control-flow`](skills/kotlin-control-flow/SKILL.md) | 27.8% | 100.0% | 100.0% |
| [`android-benchmark-comparison`](skills/android-benchmark-comparison/SKILL.md) | — | — | — |
| [`grounded-writing`](skills/grounded-writing/SKILL.md) | — | 100.0% | 100.0% |
| [`implement-with-subagents`](skills/implement-with-subagents/SKILL.md) | — | — | 100.0% |
| [`release-kotlin-library`](skills/release-kotlin-library/SKILL.md) | — | — | — |
@@ -227,6 +233,7 @@ The source runs, selection rules, and detailed scorecards are in the
| [`kotlin-api-design`](skills/kotlin-api-design/SKILL.md) | 57.4k → 145.8k (+154%) | 3 → 7 (+133%) | 1 → 1 (+0%) | 30.0s → 53.0s (+77%) |
| [`kotlin-concurrency-and-flow`](skills/kotlin-concurrency-and-flow/SKILL.md) | 72.7k → 119.2k (+64%) | 4 → 5 (+25%) | 1 → 1 (+0%) | 46.0s → 64.2s (+40%) |
| [`kotlin-control-flow`](skills/kotlin-control-flow/SKILL.md) | 71.8k → 109.6k (+53%) | 4 → 5 (+25%) | 1 → 1 (+0%) | 39.1s → 53.7s (+37%) |
| [`android-benchmark-comparison`](skills/android-benchmark-comparison/SKILL.md) | — | — | — | — |
| [`grounded-writing`](skills/grounded-writing/SKILL.md) | 41.3k → 65.4k (+59%) | 2 → 3 (+50%) | 1 → 1 (+0%) | 16.2s → 26.9s (+66%) |
| [`release-kotlin-library`](skills/release-kotlin-library/SKILL.md) | — | — | — | — |
evals/README.md
+13 -6
View File
@@ -4,10 +4,10 @@ This directory contains a reproducible, advisory evaluator with a shared core
and suite-specific catalogs, fixtures, coverage rules, and safety policies. It
tests concrete scenarios modelled on real-world coding work, with expected
outcomes, allowed-write boundaries, and no-change controls. The committed suites
cover six Compose skills, four Kotlin/Gradle skills, and six workflow/writing
skills: `grounded-writing`, `implement-with-subagents`, `release-kotlin-library`,
`run-github-project`, `shepherd`, and `to-plan`. It is designed to answer three
separate questions:
cover six Compose skills, four Kotlin/Gradle skills, and seven workflow/writing
skills: `android-benchmark-comparison`, `grounded-writing`,
`implement-with-subagents`, `release-kotlin-library`, `run-github-project`,
`shepherd`, and `to-plan`. It is designed to answer three separate questions:
1. Does a skill improve the correctness and restraint of the resulting work?
2. Does automatic activation report the expected implicitly invokable public skill entrypoints?
@@ -59,6 +59,7 @@ suite-wide aggregate.
| `kotlin-api-design` | 66.7% | 100.0% | 100.0% |
| `kotlin-concurrency-and-flow` | 33.3% | 100.0% | 100.0% |
| `kotlin-control-flow` | 27.8% | 100.0% | 100.0% |
| `android-benchmark-comparison` | — | — | — |
| `grounded-writing` | — | 100.0% | 100.0% |
| `implement-with-subagents` | — | — | 100.0% |
| `release-kotlin-library` | — | — | — |
@@ -89,6 +90,7 @@ The source runs, selection rules, and detailed scorecards are in the
| `kotlin-api-design` | 57.4k → 145.8k (+154%) | 3 → 7 (+133%) | 1 → 1 (+0%) | 30.0s → 53.0s (+77%) |
| `kotlin-concurrency-and-flow` | 72.7k → 119.2k (+64%) | 4 → 5 (+25%) | 1 → 1 (+0%) | 46.0s → 64.2s (+40%) |
| `kotlin-control-flow` | 71.8k → 109.6k (+53%) | 4 → 5 (+25%) | 1 → 1 (+0%) | 39.1s → 53.7s (+37%) |
| `android-benchmark-comparison` | — | — | — | — |
| `grounded-writing` | 41.3k → 65.4k (+59%) | 2 → 3 (+50%) | 1 → 1 (+0%) | 16.2s → 26.9s (+66%) |
| `release-kotlin-library` | — | — | — | — |
@@ -113,8 +115,8 @@ Each eligible `case × arm` condition runs three times by default. A case that
targets no implicitly invokable skill is excluded from the automatic arm before
execution. The 38-case Compose suite schedules 342 subject calls and 342
blinded judge calls. The 22-case Kotlin/Gradle suite schedules 198 subject calls
and 198 blinded judge calls. The 18-case workflows/writing suite schedules 126
subject calls and 126 blinded judge calls.
and 198 blinded judge calls. The 21-case workflows/writing suite schedules 153
subject calls and 153 blinded judge calls.
All subject and judge processes use `--ignore-user-config`, explicit
`skills.config` entries, network-disabled sandboxes, disabled hosted web search,
@@ -142,6 +144,11 @@ required `expected_skills`; precision accepts any reported skill in the case's
permit a genuinely relevant secondary skill without requiring every successful
subject to consult it.
A no-skill automatic control sets `automatic_no_skill_control: true`. It still
runs in the automatic arm, while requiring and permitting no public skill, so
unnecessary activation remains visible without becoming a routing false
negative.
### Corpus
The Compose benchmark remains 38 scored cases and comprises:
evals/cases/android-benchmark-comparison-direct/case.json
+17
View File
@@ -0,0 +1,17 @@
{
"id": "android-benchmark-comparison-direct",
"title": "Document a controlled quality comparison",
"family": "benchmarking",
"target_skills": ["android-benchmark-comparison"],
"expected_skills": ["android-benchmark-comparison"],
"task_mode": "edit",
"kind": "direct",
"fixture": "text",
"allowed_write_paths": ["comparison.md"],
"validators": [{"argv": ["python3", "@validators/text_case.py", "android-benchmark-comparison-direct"], "timeout_seconds": 30}],
"rubric": [
{"id": "experiment-evidence", "text": "The comparison records build/configuration identity, completed case and iteration coverage, raw evidence, and conditions."},
{"id": "bounded-decision", "text": "The conclusion reports variability and traces, avoids claiming a CPU metric measures GPU shader time, and does not assign the combined controls' effect to affinity alone."}
],
"provenance": {"kind": "synthetic"}
}
evals/cases/android-benchmark-comparison-direct/expectations.json
+5
View File
@@ -0,0 +1,5 @@
{
"files": ["comparison.md"],
"must_match": ["(?i)build", "(?i)raw", "(?i)trace", "(?i)variab", "(?i)completed", "(?i)controlled"],
"must_not_match": ["(?i)CPU frame (?:duration|time) (?:measures|is) GPU shader"]
}
evals/cases/android-benchmark-comparison-direct/overlay/comparison.md
+9
View File
@@ -0,0 +1,9 @@
# Quality comparison
Build: `abc123`; device: example handset. Quality 0 and 0.25 reversed CPU
rankings on the first rerun. An ascending and descending rerun used controlled
brightness and verified CPU placement in the trace interval. The combined
controls made results more consistent, but cannot isolate affinity's effect.
Raw results and traces are attached to the run record. Finish this comparison
without inventing values.
evals/cases/android-benchmark-comparison-direct/prompt.md
+3
View File
@@ -0,0 +1,3 @@
Create `comparison.md` from the supplied benchmark evidence. Recommend a default
only to the extent supported by the evidence. Preserve uncertainty and do not
invent a causal explanation.
evals/cases/android-benchmark-comparison-negative/case.json
+18
View File
@@ -0,0 +1,18 @@
{
"id": "android-benchmark-comparison-negative",
"title": "Do not turn a code-style request into a benchmark experiment",
"family": "benchmarking",
"target_skills": ["android-benchmark-comparison"],
"expected_skills": ["android-benchmark-comparison"],
"automatic_no_skill_control": true,
"task_mode": "edit",
"kind": "negative",
"fixture": "text",
"allowed_write_paths": [],
"validators": [{"argv": ["python3", "@validators/text_case.py", "android-benchmark-comparison-negative"], "timeout_seconds": 30}],
"rubric": [
{"id": "restraint", "text": "The response recognizes that a formatting-only request supplies no competing configurations or benchmark decision."},
{"id": "no-benchmarking", "text": "The response does not run, design, or demand a performance experiment for the formatting-only request."}
],
"provenance": {"kind": "synthetic"}
}
evals/cases/android-benchmark-comparison-negative/expectations.json
+4
View File
@@ -0,0 +1,4 @@
{
"files": ["draft.md"],
"must_contain": ["duplicate parse"]
}
evals/cases/android-benchmark-comparison-negative/prompt.md
+2
View File
@@ -0,0 +1,2 @@
Inspect whether `draft.md` already follows an 80-column style. Change it only
if it does not; preserve its words and meaning, and do not run commands.
evals/cases/android-benchmark-comparison-novel/case.json
+17
View File
@@ -0,0 +1,17 @@
{
"id": "android-benchmark-comparison-novel",
"title": "Review a reversed benchmark ranking",
"family": "benchmarking",
"target_skills": ["android-benchmark-comparison"],
"expected_skills": ["android-benchmark-comparison"],
"task_mode": "review",
"kind": "novel",
"fixture": "text",
"allowed_write_paths": [],
"validators": [{"argv": ["python3", "@validators/text_case.py", "android-benchmark-comparison-novel"], "timeout_seconds": 30}],
"rubric": [
{"id": "reversal-investigation", "text": "The review asks to verify case/iteration coverage, balance or reverse run order, and inspect trace intervals before selecting a default."},
{"id": "affinity-boundary", "text": "The review distinguishes fixed performance mode from verified CPU placement and requires restoration after any affinity experiment."}
],
"provenance": {"kind": "synthetic"}
}
evals/cases/android-benchmark-comparison-novel/expectations.json
+4
View File
@@ -0,0 +1,4 @@
{
"files": ["draft.md"],
"must_contain": ["duplicate parse"]
}
evals/cases/android-benchmark-comparison-novel/prompt.md
+5
View File
@@ -0,0 +1,5 @@
Review the next safe action only; do not edit files. A quality 0 run was faster
than quality 0.25 yesterday, but slower today. Both reports say "fixed
performance mode" and contain rounded p95 values, but neither names all cases
or iterations. An affinity command was tried, and no trace verifies placement.
The team wants to pick the convenient faster default now.
evals/harness/cases.py
+6
View File
@@ -54,6 +54,7 @@ class EvalCase:
forbidden_command_patterns: tuple[str, ...] = ()
allowed_skills: tuple[str, ...] = ()
calibration: bool = False
automatic_no_skill_control: bool = False
@dataclass(frozen=True)
@@ -150,6 +151,10 @@ def load_case(manifest_path: Path, repo_root: Path) -> EvalCase:
if kind == "routing" and not expected_skills:
raise CaseValidationError("routing cases require at least one expected skill")
automatic_no_skill_control = data.get("automatic_no_skill_control", False)
if not isinstance(automatic_no_skill_control, bool):
raise CaseValidationError("automatic_no_skill_control must be a boolean")
fixture = _require_string(data, "fixture")
if not _safe_relative(fixture):
raise CaseValidationError("fixture must be a safe relative path")
@@ -266,6 +271,7 @@ def load_case(manifest_path: Path, repo_root: Path) -> EvalCase:
required_command_patterns=command_patterns["required_command_patterns"],
forbidden_command_patterns=command_patterns["forbidden_command_patterns"],
allowed_skills=allowed_skills,
automatic_no_skill_control=automatic_no_skill_control,
)
evals/harness/experiment.py
+5 -1
View File
@@ -59,6 +59,8 @@ def _routing_expectations(
) -> tuple[tuple[str, ...], tuple[str, ...]]:
if arm != "automatic":
return case.expected_skills, case.allowed_skills or case.expected_skills
if case.automatic_no_skill_control:
return (), ()
automatic_skills = set(automatically_invokable_public_skills(repo_root))
return (
tuple(skill for skill in case.expected_skills if skill in automatic_skills),
@@ -71,7 +73,9 @@ def _routing_expectations(
def _automatic_eligible(case: EvalCase, repo_root: Path) -> bool:
return bool(_routing_expectations(case, "automatic", repo_root)[0])
return case.automatic_no_skill_control or bool(
_routing_expectations(case, "automatic", repo_root)[0]
)
def evaluation_conditions(
evals/harness/suites.py
+3 -1
View File
@@ -11,6 +11,7 @@ PUBLIC_SKILLS = (
"compose-performance",
"compose-state-and-effects",
"compose-ui-testing-patterns",
"android-benchmark-comparison",
"gradle-run",
"grounded-writing",
"implement-with-subagents",
@@ -66,6 +67,7 @@ KOTLIN_GRADLE_SKILLS = (
"kotlin-control-flow",
)
WORKFLOWS_WRITING_SKILLS = (
"android-benchmark-comparison",
"grounded-writing",
"implement-with-subagents",
"release-kotlin-library",
@@ -98,7 +100,7 @@ SUITES = {
id="workflows-writing",
title="Workflows and writing",
skills=WORKFLOWS_WRITING_SKILLS,
benchmark_cases=18,
benchmark_cases=21,
routing_cases=0,
calibration_cases=12,
require_skill_triads=True,
evals/tests/test_cases.py
+16
View File
@@ -60,8 +60,24 @@ class CaseContractTest(unittest.TestCase):
self.assertEqual(("compose-state-and-effects",), case.expected_skills)
self.assertEqual(("compose-state-and-effects",), case.allowed_skills)
self.assertFalse(case.calibration)
self.assertFalse(case.automatic_no_skill_control)
self.assertEqual("Fix the subject.\n", case.prompt)
def test_allows_a_no_skill_automatic_control(self):
case_dir = self.write_case(valid_manifest(automatic_no_skill_control=True))
case = load_case(case_dir / "case.json", self.root)
self.assertTrue(case.automatic_no_skill_control)
def test_requires_automatic_no_skill_control_to_be_boolean(self):
case_dir = self.write_case(valid_manifest(automatic_no_skill_control="yes"))
with self.assertRaisesRegex(
CaseValidationError, "automatic_no_skill_control must be a boolean"
):
load_case(case_dir / "case.json", self.root)
def test_allowed_skills_must_include_every_expected_skill(self):
case_dir = self.write_case(
valid_manifest(
evals/tests/test_cli.py
+5 -5
View File
@@ -75,11 +75,11 @@ class EvaluationCliTest(unittest.TestCase):
plan = json.loads(output)
self.assertEqual(0, status)
self.assertEqual(18, plan["case_count"])
self.assertEqual(42, plan["condition_count"])
self.assertEqual(126, plan["subject_calls"])
self.assertEqual(126, plan["judge_calls"])
self.assertEqual(252, plan["total_calls"])
self.assertEqual(21, plan["case_count"])
self.assertEqual(51, plan["condition_count"])
self.assertEqual(153, plan["subject_calls"])
self.assertEqual(153, plan["judge_calls"])
self.assertEqual(306, plan["total_calls"])
def test_filters_case_skill_and_arm_before_counting_calls(self):
status, output = self.invoke(
evals/tests/test_codex_runner.py
+2 -2
View File
@@ -125,10 +125,10 @@ class CodexRunnerTest(unittest.TestCase):
self.assertIn("SKILL.md", rendered)
self.assertEqual(1, " ".join(none).count("path = "))
self.assertEqual(2, " ".join(forced).count("path = "))
self.assertEqual(14, " ".join(automatic).count("path = "))
self.assertEqual(15, " ".join(automatic).count("path = "))
self.assertEqual(0, " ".join(none).count("enabled = true"))
self.assertEqual(1, " ".join(forced).count("enabled = true"))
self.assertEqual(13, " ".join(automatic).count("enabled = true"))
self.assertEqual(14, " ".join(automatic).count("enabled = true"))
self.assertIn("$compose-state-and-effects", forced[-1])
self.assertNotIn("$compose-state-and-effects", automatic[-1])
self.assertIn("If you run Gradle, use `--offline --no-scan`.", automatic[-1])
evals/tests/test_workflows_writing_matrix.py
+24 -2
View File
@@ -31,11 +31,11 @@ class WorkflowsWritingMatrixTest(unittest.TestCase):
benchmark = [case for case in report.cases if not case.calibration]
calibration = [case for case in report.cases if case.calibration]
self.assertEqual(18, len(benchmark))
self.assertEqual(21, len(benchmark))
self.assertEqual(12, len(calibration))
self.assertIn("grounded-writing", PUBLIC_SKILLS)
self.assertNotIn("implement", PUBLIC_SKILLS)
self.assertEqual(18, len(filter_cases(report.cases, case_ids=None, skills=None)))
self.assertEqual(21, len(filter_cases(report.cases, case_ids=None, skills=None)))
self.assertFalse(any(case.kind == "routing" for case in report.cases))
self.assertEqual(
{
@@ -206,6 +206,9 @@ class WorkflowsWritingMatrixTest(unittest.TestCase):
"grounded-writing-direct",
"grounded-writing-novel",
"grounded-writing-negative",
"android-benchmark-comparison-direct",
"android-benchmark-comparison-novel",
"android-benchmark-comparison-negative",
"release-kotlin-library-direct",
"release-kotlin-library-novel",
"release-kotlin-library-negative",
@@ -255,6 +258,25 @@ class WorkflowsWritingMatrixTest(unittest.TestCase):
self.assertEqual(["grounded-writing"], records[2]["expected_skills"])
self.assertFalse(records[3]["automatic_eligible"])
def test_no_skill_automatic_control_runs_without_routing_to_its_target(self):
report = validate_corpus(REPO_ROOT, suite="workflows-writing")
case = next(
case
for case in report.cases
if case.id == "android-benchmark-comparison-negative"
)
self.assertTrue(case.automatic_no_skill_control)
self.assertIn(
(case, "automatic"), evaluation_conditions(REPO_ROOT, (case,), ("automatic",))
)
records = [{"case_id": case.id, "arm": "automatic"}]
reconcile_automatic_eligibility(REPO_ROOT, report.cases, records)
self.assertTrue(records[0]["automatic_eligible"])
self.assertEqual([], records[0]["expected_skills"])
self.assertEqual([], records[0]["allowed_skills"])
def test_behavioral_expectations_do_not_assert_fixture_prose(self):
expectations = json.loads(
(
skills/android-benchmark-comparison/SKILL.md
+56
View File
@@ -0,0 +1,56 @@
---
name: android-benchmark-comparison
description: Use when comparing physical Android benchmark configurations, investigating inconsistent rankings, or selecting an Android default from measured results. Do not use for code-level Compose performance diagnosis without a configuration comparison.
---
# Android benchmark comparison
## Core principle
Treat a physical Android configuration comparison as a reproducible experiment:
verify comparable workloads and device conditions before interpreting a ranking
or choosing a default.
## Procedure
1. State the decision, configurations, workloads, metric definitions, and
repetitions. Preserve exact build identity, configuration, raw results, and
traces; then verify every intended case and iteration ran. Distinguish
missing, failed, and excluded runs; do not compare only the fastest
survivors.
2. Control and record relevant device conditions, including device model and
state, thermal and power mode, display brightness, background load, and
network or input conditions. Keep device-specific commands and CPU masks in
the project's runbook.
3. Balance or reverse run order and repeat the comparison. Report the spread
and whether the ordering holds; do not discard slow iterations after seeing
the result.
4. When rankings reverse or variability is material, investigate before making
the decision. Use traces to check the measured interval and plausible
causes such as CPU placement, contention, or thermal state. A fixed-
performance setting does not prove CPU placement. If affinity is used,
discover the device topology, verify placement during the measured trace
interval, label the outcome a controlled comparison, and restore device
settings afterwards.
5. Calculate summaries from unrounded observations, then round only for
presentation. Name the aggregation explicitly: the mean of per-run
percentiles is not a percentile of pooled observations. Choose an
aggregation that answers the stated decision; do not prescribe one statistic
universally.
6. Separate controlled-experiment evidence from normal user performance. If
several conditions changed together, report the comparison as more
controlled but do not attribute its whole difference to one control. Use
CPU frame-duration evidence to inform a visual quality/performance decision,
without claiming it measures GPU shader time.
7. Finish with the raw-evidence location, completed-case counts, variability,
trace findings, controls and restoration status, plus the bounded decision
or remaining uncertainty.
## Boundaries
- A single stable benchmark run can support a narrow observation, but not a
robust configuration ranking.
- Do not turn a device-specific CPU mask, brightness value, iteration count, or
summary statistic into a permanent default.
- When traces or repeat coverage cannot resolve a reversal, keep the default
unchanged or make a provisional decision with that limitation explicit.
skills/using-chrisbanes-skills/SKILL.md
+5 -2
View File
@@ -1,6 +1,6 @@
---
name: using-chrisbanes-skills
description: Use when debugging, benchmarking, or profiling leads into Kotlin or Jetpack Compose source before the cause is known, or when one task spans multiple Kotlin or Compose concerns, especially plain Kotlin Flow or navigation delivery plus sealed branching.
description: Use when routing Kotlin or Jetpack Compose work, or physical Android benchmark evidence that needs a configuration decision, especially when several Kotlin or Compose concerns overlap.
---
# Using chrisbanes skills
@@ -13,7 +13,9 @@ only when its independent behavior changes the same work.
## Routing procedure
1. Read the task and the Kotlin source that makes the concern concrete.
1. Read the task. For Kotlin or Compose work, inspect the source that makes the
concern concrete. For an Android benchmark comparison, inspect the supplied
reports, configurations, and traces instead.
2. If one focused skill clearly matches, load it directly and stop routing.
3. Before loading a Compose skill, point to a concrete Compose API or composable
in the inspected source, or to an explicit request to create or design
@@ -40,6 +42,7 @@ only when its independent behavior changes the same work.
| Kotlin classification, `when`, guards, exhaustiveness, smart casts, or null branches | [`kotlin-control-flow`](../kotlin-control-flow/SKILL.md) |
| Kotlin function ownership, domain types, expect/actual, or platform seams | [`kotlin-api-design`](../kotlin-api-design/SKILL.md) |
| Planned Gradle execution or a Gradle-centered warning/failure workflow | [`gradle-run`](../gradle-run/SKILL.md) |
| Comparing physical Android benchmark configurations, reversed rankings, or an Android default | [`android-benchmark-comparison`](../android-benchmark-comparison/SKILL.md) |
| Kotlin library release preparation, publication, or readiness | [`release-kotlin-library`](../release-kotlin-library/SKILL.md) |
| One ready GitHub issue or in-chat task needs repository-aware planning | [`to-plan`](../to-plan/SKILL.md) |
| Polling PRs/MRs, review comments, CI failures, or routine follow-up | [`shepherd`](../shepherd/SKILL.md) |