skills/trailofbits__skills
1
0
Fork 0
mirror of https://github.com/trailofbits/skills.git synced 2026-09-14 14:28:48 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
trailofbits__skills/plugins/constant-time-analysis/pyproject.toml
T
Joop van de Pol 9ea55c5987 Fix constant-time analyzer backends and trim the skill (#220) 
* Add per-language triage fixtures for the constant-time analyzer

One fixture per supported language, each pairing operations the analyzer
reports identically but that triage must separate: a division on a
private-key coefficient beside one on a public buffer length, and, where
the backend detects them, a weak RNG seeding a nonce beside the same call
jittering a retry delay.

Expected verdicts are deliberately absent from the fixture source. These
files double as evaluation input, and a verdict written next to the code
is a verdict the reviewer reads instead of deriving. A later commit adds
the manifest that records them.

Fixture shapes are dictated by what each toolchain emits: the Go helpers
are //go:noinline so each stays a distinct symbol, its main() takes input
from argv so the arithmetic is not constant-folded away, and the Swift
entry points use @_cdecl to keep symbol names readable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Configure ty and fix the typing weaknesses it reports

`ty check` reported 50 diagnostics, all pre-existing. Most were noise from
module resolution: the tests reach the modules via `sys.path.insert` and
import `analyzer` rather than `ct_analyzer.analyzer`, which ty cannot know
without being told. `tool.ty.environment extra-paths` tells it, dropping 32
unresolved-import reports and leaving the substantive ones visible.

The rest were real, if not yet bugs:

- Six compilers and several entry points annotated `list[str]`/`str` while
  defaulting to None. Now spelled `| None`.
- `get_compiler` was annotated to require a compiler name, but its body
  auto-detects from the language when the name is falsy, which is how every
  caller uses it.
- Each parser tracked functions as a bare dict, typing the values `str |
  int`, so the `functions[-1]["instructions"] += 1` counter present in all
  eight parsers read as string addition. A `ParsedFunction` TypedDict names
  the shape instead.

Three unresolved-import reports remain by construction: analyzer.py is
documented to run as a script as well as import as a package, so both the
relative and top-level spellings are needed and only one can resolve. Those
carry an inline ignore with the reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix analyzer backends that reported PASSED on vulnerable code

Auditing each backend against real toolchain output found six of them
skipping unparseable lines and reporting success. Every one was covered by
tests that replay hand-written listings, which is why the drift went
unnoticed; the tests added here drive the actual toolchains.

Python, on 3.11 and later:
- The BINARY_OP oparg map read 12 as `//`, but 12 is `^`. Floor division on
  a secret was missed and every constant-time XOR was reported as a
  division. Keyed on the operator symbol dis prints, which cannot silently
  renumber.
- The instruction regex required a byte offset that 3.13 no longer prints,
  so all but the first instruction of each source line was skipped. On 3.13
  `key_coef // (2 * gamma2)` came back PASSED. Every column ahead of the
  opcode is now optional, and the source line carries forward.

PHP: `_parse_opcache_output` delegated to the VLD parser on the claim the
formats were "similar enough"; they share nothing structural. `analyze()`
also called the VLD parser unconditionally, so the function was dead either
way. VLD is an unbundled PECL build, so this is the path most users hit: a
file with intdiv, mt_rand and base64_encode reported zero findings. Adds a
real parser, plus intdiv/fdiv, which perform division through a call and so
emit no DIV opcode.

JavaScript and TypeScript: V8 prints `67 E> 0x… @ 14 : 3e 03 04  Div a0,
[4]`, not `offset : Mnemonic`, so every file parsed as zero instructions.
`--no-lazy` covers functions a module exports without calling. Bytecode
blocks are filtered to names the file declares, since V8 dumps node's
internals identically. Source positions are byte offsets, now converted to
lines except for transpiled TypeScript, where they index generated output.

Go: `go build` links the runtime in, so 23 of 24 findings came from the
allocator and garbage collector while the caller's code was silent. Objdump
headers name the source file, so the filter is exact. Go also writes arm64
in Plan 9 syntax, where a 32-bit divide is SDIVW — absent from the table, so
`int32` division, the shape of every polynomial coefficient divide, read as
clean.

Rust: compiled as a bin crate, so any library file failed E0601. Crypto
lives in libraries.

Swift: target triples were hardcoded to Apple platforms, so swiftc rejected
them on Linux and no Swift file could be analyzed there.

Java: the weak-RNG pattern matched only `new Random(`, missing the
fully-qualified form that needs no import.

All six source-level scanners treated comments as code, so `/** a / b */`
counted as a division; comment bodies are now blanked while preserving
offsets. Finally, the text report labelled every scripting-language error
`[WARN]` while the summary counted it as an error: analyzer.py runs as
`__main__` while script_analyzers imports it as `analyzer`, giving two
Severity enums, so identity comparison failed. Severity is compared by
value now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Record triage verdicts for every fixture and assert the pairs hold

expectations.json gives each fixture's cases a verdict and the reasoning
behind it: 53 cases over 13 languages, covering arithmetic everywhere,
conditional branches on the native backends, weak RNG wherever a backend
detects it, and PHP's encoding functions.

TestTriageMatrix asserts the mechanical half — that the analyzer still
reports both members of every true/false-positive pair, since the premise of
the skill's triage step is that the tool cannot tell them apart. Four of its
checks need no toolchain, so the matrix stays guarded on machines that
cannot compile most of these languages: every supported extension has a
fixture, every fixture pairs a true with a false positive, every line-based
locator resolves exactly once, and no fixture leaks its verdict into a
comment. The run itself fails rather than skipping when no fixture could be
exercised, and names the languages it skipped.

Two cases record measured behaviour rather than the intent: Rust claims no
branch true positive, because rustc emits no flagged conditional branch in
the tag comparison at O0 on arm64 and asserting one would encode a fixture
artefact; and Go's branch false positive is its stack-growth check, which
appears in every Go function and depends on nothing the caller passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Trim the constant-time-analysis skill and state its real coverage

SKILL.md goes from 219 to 165 lines by removing what was duplicated or
inert: a decision tree that restated the trigger bullets below it, five
near-identical invocation blocks, and JVM/.NET setup copied from
references/vm-compiled.md. It gains the frontmatter it was missing —
allowed-tools, which it never declared and so inherited everything, and an
explicit effort level.

The additions come from evaluating the skill rather than from review. Runs
at three effort levels all recommended fixing a secret-dependent divide by
handing the compiler a constant divisor; measuring that fix showed it still
emits a real divide on gcc riscv64 at every optimization level, on gcc arm64
and x86_64 at Os and Oz, and on clang arm64 at O0 and Oz. The sweep section
now says so, with the table.

Coverage is not uniform across backends, and a clean report means different
things depending on the language, so there is a table for that too. Triage
guidance now also covers the weak-RNG and encoding findings, where no
operand is secret and the question is what the result is used for — seeding
a nonce or jittering a retry delay.

"When NOT to Use" names the constant-time-testing skill and draws the line
between them: that one measures a running binary, this one reads compiler
output and never executes the code.

Both READMEs listed nine languages when thirteen are supported, and the
skill's file tree was missing four reference files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix warning-level detectors that no source could ever match

Extending the triage matrix to the comparison, lookup and encoding families
found that several of those detectors were unreachable. The tables listed
them, so the coverage looked complete, but no input could trigger them.

- Ruby built every warning pattern as `\.<name>`, which is right for method
  calls like `.start_with?(` but turns the module-qualified keys into
  `\.base64\.encode64`. Five entries — both base64 functions, json.parse and
  both marshal functions — could not match any Ruby source.
- JavaScript did the same to its non-method entries, so `btoa(`, `atob(`,
  `decodeURIComponent(` and `JSON.parse(` never matched.
- The JS keyed-property mnemonics were V8 9.x spellings. Node 22 emits
  `GetKeyedProperty`, so secret-indexed array access — the whole point of
  that family — was undetectable. The named-property entry is removed rather
  than updated: a named access has a constant property name and cannot be
  indexed by a secret, so flagging it reports every `Math.trunc` call.
- Java, Kotlin and C# selected patterns with if/elif chains covering three or
  four keys and skipping the rest, leaving the base64 and convert entries
  unreachable. They now share `qualified_call_pattern`, which distinguishes
  the two shapes the keys use: `string.equals` is a method on any receiver,
  while `arrays.equals` and `base64.getencoder` are members of a named type
  and must match only the qualified form. Case-insensitivity is embedded in
  the pattern because the keys are lowercase while the source spells them
  `Base64.getEncoder` and `.Equals`.

Ruby's warning mnemonics also kept the dots from their keys, unlike every
other severity and backend, so they read `BASE64.ENCODE64`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Cover the comparison, lookup and encoding families in the triage matrix

The matrix covered two of the four families the analyzer implements. It now
covers all four, taking it from 53 cases to 99.

Each of the eight bytecode and scripting fixtures gains three pairs: an
early-exit comparison of a secret tag against the same comparison on a public
header, a table lookup indexed by a key byte against one indexed by a fixed
offset into a public header, and a base64 encode of a decrypted session key
against one of the public algorithm identifier. PHP already had the encoding
pair. Native fixtures are unchanged — their instruction tables carry no
comparison or load entries, and C's tag comparison is already covered as a
branch.

All three families are warning severity, so their cases run with
`include_warnings`. That is worth noting on its own: the invocation the skill
documents does not pass `--warnings`, so a reviewer following it sees none of
this — including early-exit comparison, which is the most common timing bug
in practice.

Every case was measured against the real toolchain before being written down.
Python's fixture avoids a `list[bytes]` annotation in a signature because the
subscript in the annotation is itself reported at module scope.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Refuse an architecture the compiler cannot target

Every compiler looked its architecture up with `.get()` and simply omitted the
target flag when the lookup missed, while `analyze_source` still labelled the
report with the architecture that was requested and applied that
architecture's instruction table. So `--arch riscv64` on Swift, whose Linux map
holds two entries, compiled for the host and reported `architecture: riscv64`
with no findings; `--arch mips` on gcc reported `passed: true` the same way.

"PASSED for riscv64" is what a reviewer writes down, so producing host output
under the requested label is worse than failing. All five compilers now refuse
and name what they do support.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Stop the Go symbol filter readmitting the runtime by basename

The filter compares the objdump block's source path against the file under
analysis, but when the realpath comparison failed it fell back to comparing
basenames — and Go's runtime ships map.go, slice.go, string.go, time.go and
select.go. Analyzing a user file named map.go therefore matched every
`TEXT runtime.…(SB) /usr/lib/go/src/runtime/map.go` block, reported
runtime.makeBucketArray among the findings, and parsed 14 functions instead of
5. That is the behaviour the filter exists to prevent, and the existing test
missed it only because its fixture is named triage_go.go.

objdump prints an absolute path, so the exact comparison is sufficient there;
the basename fallback now applies only when the printed path is relative.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Make comment blanking string-aware

Blanking comments before the source-level scans stopped doc comments counting
as code, but the regexes knew nothing about string literals, so a comment
marker inside one blanked live code:

- `const u = "http://example.com"; const k = a / b;` lost the division, and
  `String u = "http://x"; Random r = new Random();` reported clean — for Java,
  Kotlin and C# the source scan is the only detector for `new Random()`.
- Ruby's `puts "tag #{a / b}"` lost everything from the interpolation on.
- Worst, a lone `const marker = "/*";` plus any later `*/` anywhere in the file
  blanked every line in between, with no diagnostic.

String and template literals are now matched first and preserved. This is still
a regex rather than a lexer — unterminated literals, JavaScript regex literals,
Ruby heredocs and `%w[]` are not modelled — and the docstring says so; the
failure mode is a blanked or unblanked span, never a crash.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Test the empty-parse guards and close smaller robustness gaps

The test named for PHP's "nothing parsed" refusal asserted only that the parser
returned empty lists, which the pre-fix code did too, so deleting the raise in
`analyze()` left the suite green. It now asserts the RuntimeError, and the Go
equivalent — the symbol filter keeping nothing — is covered as well. Both were
confirmed to fail with their guard removed.

Four gaps that each degrade quietly:

- The OPcache offset group was `\d{4}` exactly, so an op_array past 9999
  opcodes stopped parsing mid-function. `functions` is non-empty by then, so
  the empty-parse guard cannot fire and the report reads as a partial pass.
- The V8 name filter treated "could not read the compiled file" as "filtering
  disabled", listing hundreds of node-internal functions with nothing to
  distinguish that from a genuinely noisy file. Unreadable input is now an
  explicit None with a note on stderr.
- `last_line_num` was not reset per code object, so the first instructions of a
  function inherited the previous function's line when dis omits one.
- The triage matrix printed its skip list to stdout, where CI summaries do not
  show it, so an image that lost a toolchain would keep passing while the claim
  that all backends are exercised quietly stopped being true. Skips are now
  emitted per language and appear in the count.

Two test-quality fixes: the matrix completeness check compared set sizes, so
renaming one language while dropping another stayed green — it compares sets
now, through an explicit display-name to `detect_language()` mapping. And
`_have` uses `shutil.which` rather than running `--version`, because javap
exits non-zero for it, which would have skipped Java and Kotlin everywhere.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Pass --warnings in the documented invocation

The Quick Start ran the analyzer with no flags, which reports only
error-severity findings: division, modulo and weak RNG. Four families are
warning severity and stayed silent — secret-dependent branches, early-exit
comparison, secret-indexed table lookups, and variable-time encoding. A
reviewer following the skill exactly never saw the early-exit comparison of an
authentication tag, which is the most common timing bug in practice and what
Lucky Thirteen was.

`--warnings` is now in the documented command, the directory sweep and the flag
table, with the two consequences of turning it on stated alongside. Warnings do
not affect the PASSED line, so `Result: PASSED` beside `Warnings: 6` is normal
and is not a clean result. And comparison and lookup findings need their own
triage question: for a lookup it is the index that must be secret, not the
contents. Confirmed comparison findings have a per-language constant-time
primitive, so those are tabulated rather than left as "write it branch-free";
a secret-indexed lookup gets the honest answer that no drop-in exists.

Also names the plugin that owns constant-time-testing, since a reader cannot
assume it is installed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix the fixture path and the sdist include list

Two defects that both make something unreachable:

The pointer to the triage fixtures was written as a bare relative path, while
every other path in SKILL.md is `{baseDir}`-prefixed. The skill runs against the
user's repository, so a model following it looked for
`ct_analyzer/tests/triage_samples/` inside the project being audited. The
validator resolves markdown links rather than inline code, so this passed 359
reference checks while being unusable at runtime.

The sdist include list named three globs covering C, Go and Rust, so a packaged
install shipped none of the Python, Ruby, PHP, JS, TS, Java, C#, Kotlin or Swift
samples, nor the triage fixtures and their manifest, and the tests that read
them could not run from an sdist. A built sdist now carries all 27.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Test that cross-compilation really targets the requested architecture

Nothing covered cross-compilation. The pre-existing cross-architecture test
asks for arm64, which is the host on an aarch64 machine, so it passes there
without ever crossing — and a silent fall back to the host is exactly the
failure `reject_arch` was added to prevent.

These assert instruction names unique to the target: x86 `DIVL`/`DIVQ`/`IDIVL`
and RISC-V `DIVU`/`DIVW`/`REMU` must appear, and arm64 `SDIV` must not. Dropping
clang's `--target` makes both fail, which a report-shape assertion would not
have caught. Go gets the same treatment for amd64, where GOARCH cross-builds
because CGO is disabled.

Only the paths that work are covered: on this host clang crosses to both
targets and Go to amd64, while gcc, rustc and Go/riscv64 cannot cross here at
all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Drive an explicit compiler with its own flags, and name it in the report

Cross-compiling with gcc was impossible through the analyzer. `--arch x86_64`
passed gcc's native ISA switch `-m64`, which the gcc on an arm64 host rejects,
and the escape hatch that should have worked — handing over the cross binary —
was broken: `get_compiler` treated every unrecognized `--compiler` value as
clang, so `--compiler x86_64-linux-gnu-gcc` was driven with `--target=` and
failed on "unrecognized command-line option".

An explicit compiler is now dispatched on which compiler it actually is: by
filename first, falling back to its `--version` banner, so `cc` is recognized
as whatever it points at. `--compiler x86_64-linux-gnu-gcc --arch x86_64` and
`--compiler riscv64-linux-gnu-gcc --arch riscv64` both work and emit that
target's division instructions.

Nothing is substituted on the caller's behalf. The analyzer could look for
`<triple>-gcc` and use it automatically, but a Debian cross gcc is frequently a
different major version than the host's, so its codegen is not the host gcc's;
silently swapping binaries would put one compiler's output under another's name
— the same mislabelling this commit removes. Instead, when gcc is asked for an
architecture it cannot build, the error names the binary to pass, and the report
records the binary that ran rather than the family, so `compiler:` is no longer
"gcc" when a cross build produced the assembly.

SKILL.md now states how each toolchain crosses, since "sweep more than one
--arch" was not actionable for gcc users, and advises comparing against the
toolchain that builds the product rather than a packaged cross build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Detect software division, and fix the armhf flags that hid it

Installing the remaining cross toolchains turned up two problems that are not
about missing dependencies.

The arm flags were self-contradictory: `-march=armv7-a -mfloat-abi=hard` is
rejected with "selected architecture lacks an FPU", because armv7-a alone
specifies no FPU. Adding `-mfpu=vfpv3-d16`, Debian's armhf baseline, makes the
build work.

With it building, arm reported no findings at all on a fixture with three
divisions. armv7-a has no hardware divider, so gcc emits `bl __aeabi_idiv`
rather than a division instruction, and the analyzer matches mnemonics: a
branch-with-link is not in any table, so the file read as PASSED. Software
division loops over its operands, making it more operand-dependent than the
instruction it replaces, so this was the worst kind of false negative. Calls to
the libgcc and Arm EABI division routines are now reported by call target, which
also covers __divti3 — used on x86_64 for __int128 division, which crypto code
does perform.

Verified against every cross target now installed: x86_64 DIVQ/IDIVL, i386
DIVL/IDIVL, arm AEABI_IDIV, riscv64 DIVU/DIVW/REMU, ppc64le DIVDU/DIVW, s390x
DLGR/DSGFR, arm64 SDIV/UDIV. The ppc64le and s390x instruction tables had never
been exercised before.

Also stops the new "pass a cross build" hint from telling the caller to do what
they just did: it is suppressed when the compiler already is a cross build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Cross-compile a freestanding probe, not a fixture that needs a cross libc

CI failed on `test_clang_targets_riscv64` with "bits/libc-header-start.h file
not found". The test compiled the C triage fixture, which includes <stdint.h>,
so cross-compiling it needs the target's C library headers as well as the
compiler. The x86_64 case passed only because the runner is x86_64, making that
target native; my dev box passed riscv64 only because installing
gcc-riscv64-linux-gnu had pulled in libc6-dev-riscv64-cross. So the test was
asserting an environment property, not the analyzer's behaviour.

Nothing about checking that a division instruction reaches the assembly needs a
libc. The cross tests now use a freestanding sample with no includes, verified
to compile for x86_64, riscv64 and s390x with `-nostdinc` — that is, with no
headers reachable at all — while still yielding that target's division
mnemonics.

This also corrects the claim in SKILL.md that clang needs nothing installed to
cross-compile: it needs no second compiler, but a source including libc headers
still needs the target's headers. When that is what failed, the error now names
the package that supplies them, which is the analyzer's own advice to sweep
`--arch` being made actionable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Scott Arciszewski <147527775+tob-scott-a@users.noreply.github.com>
Co-authored-by: kz-tob <kara.zaffarano@trailofbits.com>
2026-08-03 14:12:27 -04:00

63 lines
2.1 KiB
TOML
Raw Permalink Blame History

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "ct-analyzer"
version = "0.1.0"
description = "Constant-time assembly analyzer for detecting timing side-channel vulnerabilities"
readme = "README.md"
requires-python = ">=3.10"
authors = [
{name = "Scott Arciszewski", email = "opensource@trailofbits.com"}
]
keywords = [
"cryptography",
"security",
"timing-attack",
"side-channel",
"constant-time",
"static-analysis",
]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Topic :: Security :: Cryptography",
"Topic :: Software Development :: Quality Assurance",
]
[project.scripts]
ct-analyzer = "ct_analyzer.analyzer:main"
[project.urls]
Homepage = "https://github.com/trailofbits/skills"
Documentation = "https://github.com/trailofbits/skills/tree/main/plugins/constant-time-analysis#readme"
Repository = "https://github.com/trailofbits/skills.git"
[tool.ty.environment]
# The tests run as scripts and reach the modules via `sys.path.insert`, importing
# `analyzer` rather than `ct_analyzer.analyzer`. Without this, ty reports every one
# of those imports as unresolved, which buries the diagnostics that matter.
extra-paths = ["ct_analyzer"]
[tool.hatch.build.targets.wheel]
packages = ["ct_analyzer"]
[tool.hatch.build.targets.sdist]
# Every fixture the test suite reads, not just the three C/Go/Rust globs that
# were listed: a packaged install was missing the Python, Ruby, PHP, JS, TS,
# Java, C#, Kotlin and Swift samples, so the tests that use them could not run
# from an sdist. `triage_samples` also needs expectations.json, which is the
# manifest driving the triage matrix.
include = [
"ct_analyzer/**/*.py",
"ct_analyzer/tests/test_samples/*",
"ct_analyzer/tests/triage_samples/*",
]
Reference in New Issue View Git Blame Copy Permalink