mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
feat(pdf): sandbox optional content classification (#724)
* feat(pdf): sandbox optional content classification * fix(pdf): bind advisory to complete input * fix(pdf): close classifier protocol edge cases * fix(pdf): reject overflowing confidence values * fix(pdf): keep worker adapter outcomes closed * fix(pdf): close worker and publication races
This commit is contained in:
Edward Cheng-I Wu
committed by
GitHub
GitHub
parent
96580b2de7
commit
ca0746aed4
@@ -6,6 +6,8 @@ All notable changes to this project will be documented in this file.
|
||||
|
||||
### Added
|
||||
|
||||
- **Opt-in, process-isolated PDF text/OCR advisory (#512 follow-up; safe replacement for the concept proposed in external PR #623).** Default calls keep the original sidecar shape, tool version, and structural `PASS / FAIL / UNAVAILABLE` decision; only a caller that selects `--classify-content` starts a fixed child worker over the exact already-hashed bytes and receives an all-or-nothing extension whose verdict scope is explicitly `STRUCTURE_ONLY`. The parent never imports the optional native classifier, uses no shell, writes the exact input off the timeout-control loop, enforces one five-second execution deadline plus a shared 0.2-second teardown grace, and caps only stdout/stderr. Each loop iteration polls and then immediately observes the clock, accepting an exit only when that observation is strictly before the deadline; a late-returning poll cannot turn a timeout into success. POSIX cleanup kills the isolated worker group before joining pipe helpers; the portable Windows path terminates only the direct worker. Launch/helper, exit/signal, pipe, and protocol failures become closed reason codes; non-finite confidence and out-of-range OCR pages are rejected; only `TEXT_AVAILABLE / OCR_RECOMMENDED / unavailable` advisory states are emitted. The stdout-only legacy CLI performs no alias precheck, so malformed input paths remain exit-0 structural `UNAVAILABLE` data. When a file write is requested, conservative NFC/casefold keys, resolved targets, and existing-inode checks reject literal, `..`, case/canonical-equivalent, symlink, and hard-link aliases before the worker or either write; `samefile` errors fail closed except for an absent leaf, while an unresolvable input remains the structural preflight's responsibility. Raw upstream types and exception text cannot enter the prompt-facing sidecar; an explicitly requested, exclusive POSIX-only mode-0600 local diagnostic retains at most 512 bytes of untrusted operator detail, with its resolved parent dirfd/inode bound before the worker so a later parent-symlink retarget cannot redirect raw detail. Its created fd is inode-bound: any pre-success fchmod/write/file-fsync/close/parent-fsync failure removes the final leaf only if a fresh no-follow lookup still names that exact created inode, parent-fsyncs best-effort, and preserves the primary error. Thus a partial/malformed diagnostic does not permanently consume the exclusive path and retry succeeds, while an attacker replacement leaf is never deleted. Non-POSIX platforms or platforms without `fchmod` reject that option before path creation. POSIX sidecar output binds the resolved parent dirfd and inode before the worker starts, creates a fixed-length random-named private `0700` staging directory through that dirfd, and uses only anchored dirfd-relative operations thereafter. The complete fixed-name payload is file-fsynced, its open inode is checked against the staging entry, atomically installed with dirfd-relative `os.replace`, checked again at the target, and parent-fsynced; an observed check-to-replace substitution is rejected and its installed attacker entry removed, retargeting a parent symlink cannot redirect publication, final-component links are replaced rather than followed, and a legal 255-byte destination basename still fits. Cleanup attempts close/unlink/rmdir independently, preserves any primary publication error over secondary close failures, and removes unpublished staging. The parent and private staging directory remain caller-controlled prerequisites: Python exposes no compare-and-rename primitive, so a same-UID actor that keeps racing after the instantaneous installed-inode postcondition is outside the isolation claim. Non-POSIX `--output` fails closed because Python does not expose the required anchored dirfd publication there; classifier-to-stdout remains available. The optional dependency has a separate pin and deterministic absent state rather than being installed unconditionally. Three closed Draft 2020-12 schemas, synthetic PDFs, temporary fake modules, and fake workers cover present/absent, late exit/timeout, non-zero/signal, malformed/invalid, oversize, page-bound, helper-startup, deadline/teardown, diagnostic failure/retry and attacker-leaf retention, diagnostic-parent and parent/final/staging alias races, close failures, long basenames, privacy, and permission paths without downloading the package, using a live PDF, invoking a model, or wiring an automatic OCR/anchor gate. This proves the isolation/protocol slice only, not classifier accuracy or downstream efficacy.
|
||||
|
||||
- **Offline role-topology utility design and synthetic seed (#582).** Two independent task classes now freeze their own matched role-count arms: decomposable reviewer evidence review uses 1/2/5/7-seat topologies, while sequential writing/revision uses 1/2/5/8-seat chains and a separate requirement-based evaluator that forbids accept/reject labels. A closed study plan fixes per-call token caps, empty tool/web/network allowance, three paired replicates, zero retries, first-partial stop, no imputation, small-seed uncertainty disclosure, arm/expert blinding, and a human-verified `unique verified value` rule that credits only first-originated, evidence-anchored units retained in the final output. Four repository-owned synthetic fixtures, closed plan/seed/materialized-manifest schemas, and a suite-specific validator/dry-run/materializer produce 186 self-contained neutral prompt templates with complete hash-bound role contracts, dependency placeholders, conservative input-budget receipts, and dispatch/measurement permanently false. No subject, expert, adjudicator, model, API, tool, dispatch, or measurement ran; reviewer interpretation still depends on #653 and #582 remains open.
|
||||
|
||||
- **Offline candidate-ledger substrate (#655 Track A).** Three closed Draft 2020-12 contracts now bind the complete high-impact consentable-plan projection, consistent known-reference or explicit-unknown provider retention disclosure, one visible initial attempt per planned query/index pair, hash- and time-bound retry authorizations, monotonic retained attempt/hit evidence, and explicit relevance success/failure evidence bound to the exact claim, candidate, assessor contract, and canonical prompt bytes. A pure local finalizer preserves failures and every raw terminal state, applies one NFKC visible-semantic-text boundary, treats only strict `10.<4-9 digits>/<suffix>` values as DOI identity, blocks no-DOI bridges between distinct DOI components, and applies the frozen filter/dedup/canonical/top-40 rules. CLI persistence fails before path creation for `session_only` and writes once only for hash-bound `explicit_local_export`, using an exact hash-bound absolute output path, exclusive/no-follow creation, mode 0600, file/directory fsync, and truthful persistence/export/path metadata; exact replay needs no network, model, discovery adapter, stance classification, rendering, evidence-row changes, pipeline wiring, or dispatch. Synthetic fixtures, direct Unicode/DOI/persistence mutation regressions, an exact module-use-allowlisted no-capability integration guard, resolver byte pins, pytest-manifest registration, and spec-consistency wiring cover this substrate. The live probe and Track B remain unimplemented and unmeasured; #655 stays open.
|
||||
|
||||
@@ -80,6 +80,103 @@ anchor downstream, so the conservative bucket is the honest one.
|
||||
Exit code 0 whenever a verdict was produced (the verdict is data, not an error); 2 on usage
|
||||
errors only — so orchestration can always consume the JSON without exit-code branching.
|
||||
|
||||
The closed Draft 2020-12 sidecar schema is
|
||||
`shared/contracts/pdf/pdf_read_preflight.schema.json`. It accepts this unchanged legacy
|
||||
shape and the all-or-nothing opt-in extension below.
|
||||
|
||||
### Optional content advisory — isolated and diagnostic-only (2026-08-13 follow-up)
|
||||
|
||||
`--classify-content` is an explicit opt-in diagnostic consumer. It is not used by the
|
||||
default Stage-1 invocation and never changes the structural verdict. The parent sends the
|
||||
exact bytes already read and hashed above to the fixed
|
||||
`scripts/pdf_content_classifier_worker.py` child over stdin. Only that child imports the
|
||||
optional native `pdf_inspector` package. The exact input write runs outside the timeout
|
||||
control loop and is accepted only when complete. The parent uses `shell=False`, one
|
||||
five-second execution deadline from child startup, one shared 0.2-second teardown grace,
|
||||
and concurrent 8,192-byte stdout / 4,096-byte stderr caps; there is no claimed stdin cap.
|
||||
Every loop iteration polls and then immediately observes the monotonic clock. The poll
|
||||
result is accepted only if that observation is strictly before the deadline. An exit
|
||||
whose poll returns at or after the boundary is therefore `WORKER_TIMEOUT`.
|
||||
POSIX cleanup kills the isolated worker group before reader/writer joins, including when
|
||||
the leader has exited while descendants retain pipe handles. The portable Windows path
|
||||
terminates only the direct worker. Timeout, helper-startup, non-zero/signal exit, cap
|
||||
breach, pipe failure, malformed JSON, and invalid output all yield closed outcomes.
|
||||
Non-`PASS` structural inputs do not start the child.
|
||||
|
||||
Calls without this flag preserve the original sidecar field set and
|
||||
`tool: pdf_read_preflight/1.0.0`. Opted-in sidecars change the tool version to 1.1.0 and
|
||||
add all three extension fields together: `verdict_scope: STRUCTURE_ONLY`, the closed
|
||||
`content_advisory`, and the closed `content_classification` object. A partial extension is
|
||||
schema-invalid.
|
||||
|
||||
The parent applies a hand-written closed validator equivalent to
|
||||
`shared/contracts/pdf/pdf_content_classifier_worker.schema.json`, without adding a
|
||||
runtime `jsonschema` dependency. It also enforces stricter runtime invariants: exact
|
||||
keys, finite confidence in `[0,1]`, sorted unique integer OCR pages, and every page index
|
||||
below the structural `reader_page_count`. Open upstream types collapse to
|
||||
`TEXT_AVAILABLE` only for the exact positive `text_based` + empty-page combination and
|
||||
otherwise to `OCR_RECOMMENDED`; raw upstream type and exception strings never enter the
|
||||
sidecar.
|
||||
|
||||
An opted-in scanned result is therefore represented honestly as structural
|
||||
`verdict: PASS` with `verdict_scope: STRUCTURE_ONLY` and
|
||||
`content_advisory: OCR_RECOMMENDED`, never as a content pass. A missing optional
|
||||
dependency is deterministic `CONTENT_UNAVAILABLE / DEPENDENCY_ABSENT` and leaves the
|
||||
structural verdict unchanged.
|
||||
|
||||
On POSIX, `--classifier-diagnostics <path>` requires the opt-in flag and creates an
|
||||
exclusive, non-overwriting mode-`0600` local JSON file. Platforms without POSIX
|
||||
`fchmod` reject this option before creating the path; classification without a local
|
||||
diagnostic remains available. The artifact may contain at most 512 bytes of explicitly
|
||||
untrusted worker detail plus byte counts; neither its path nor detail appears in the
|
||||
prompt-facing sidecar. Contract:
|
||||
`shared/contracts/pdf/pdf_content_classifier_diagnostic.schema.json`. Full frozen design
|
||||
and residual-risk boundary:
|
||||
`docs/design/2026-08-13-512-pdf-content-classification-sandbox-spec.md`.
|
||||
|
||||
The stdout-only legacy invocation performs no alias precheck: an unreadable input or
|
||||
symlink loop remains an exit-0 structural `UNAVAILABLE` verdict. When `--output` or
|
||||
`--classifier-diagnostics` is present, every requested write target must resolve safely
|
||||
before parsing or worker launch. Conservative NFC/casefold keys cover literal, `..`, case-only,
|
||||
and canonically equivalent spellings; resolved keys cover symlinks; `samefile` covers
|
||||
existing hard links. `samefile` errors fail closed except `ENOENT`, which may mean an
|
||||
uncreated target. Failure to resolve the input itself is left to structural preflight and
|
||||
does not block a separately safe write target.
|
||||
|
||||
The diagnostic remains exclusive-create/no-follow. Its resolved parent directory is
|
||||
opened and inode-bound before worker launch, and the final file is created relative to
|
||||
that dirfd; a later parent-symlink retarget cannot redirect raw detail. The created fd's
|
||||
inode is recorded before fchmod or writing. Any failure before complete publication,
|
||||
including fchmod, partial write, file fsync, close, or parent fsync, performs a fresh
|
||||
no-follow lookup and unlinks only when the leaf still names that exact created inode;
|
||||
parent fsync during cleanup is best-effort and never replaces the primary error. The
|
||||
exclusive path is therefore retryable after a partial diagnostic, while a symlink or
|
||||
hard-link attacker replacement is retained rather than deleted.
|
||||
|
||||
POSIX sidecar output uses a different publication contract. Before structural parsing or worker launch, the CLI opens the
|
||||
resolved parent directory, records its inode, and creates through that dirfd a
|
||||
fixed-length random-named private `0700` staging directory. All subsequent staging and
|
||||
publication operations are relative to these anchored dirfds, so retargeting the parent
|
||||
symlink cannot redirect output. Complete bytes go to the fixed staging filename
|
||||
`payload`, independent of the destination name (so a legal 255-byte basename works).
|
||||
The open payload is fsynced and its inode is checked against the no-follow staging entry;
|
||||
a substituted symlink or hard link is rejected. Dirfd-relative `os.replace` atomically
|
||||
replaces the target entry, the installed inode is rechecked, and the parent is fsynced.
|
||||
Thus final-component links are replaced rather than followed and cannot overwrite the
|
||||
source PDF or diagnostic. Cleanup attempts file close, unlink, directory close/rmdir,
|
||||
and parent close independently, preserves a primary error over secondary close errors,
|
||||
and removes unpublished staging. Publication failures are usage errors. Non-POSIX
|
||||
`--output` fails closed because Python does not expose the required anchored publication
|
||||
contract there; stdout classification remains available.
|
||||
|
||||
This publication contract assumes the output parent is caller-controlled. The private
|
||||
random staging directory is `0700`, and the worker group is terminated before
|
||||
publication, but Python exposes no atomic compare-inode-and-rename primitive. The
|
||||
pre-replace open-inode check plus the instantaneous post-install inode check reject both
|
||||
tested swap timings; if the latter observes an attacker inode, that installed entry is
|
||||
removed before failure. A same-UID actor that continues changing entries after the final
|
||||
postcondition remains outside this process-isolation claim.
|
||||
|
||||
### Layer 2 — prompt rules
|
||||
|
||||
- **Three emitters** (`synthesis_agent`, `draft_writer_agent`, `report_compiler_agent`): a
|
||||
@@ -138,3 +235,13 @@ enumeration exercises recursion), lying root `/Count` (FAIL), truncated tail (UN
|
||||
FAIL, never PASS), encrypted marker (UNAVAILABLE), page-tree cycle (UNAVAILABLE via guard),
|
||||
non-PDF bytes (UNAVAILABLE), missing file (UNAVAILABLE), pypdf absent (monkeypatched →
|
||||
UNAVAILABLE with `pypdf-not-installed` warning), sidecar shape + hash stability, exit codes.
|
||||
|
||||
The 2026-08-13 follow-up adds no live PDF or package download. Temporary fake modules and
|
||||
workers cover dependency present/absent, text/scanned results, exception isolation,
|
||||
timeout, helper-startup failure, one-deadline/one-grace teardown, leader-exit inherited
|
||||
pipes, non-zero/signal exits, malformed/invalid/oversize output, finite confidence, page
|
||||
bounds, clock-before-poll late-exit rejection, stdout-only malformed-input compatibility,
|
||||
conservative pre-run path-alias rejection, output/diagnostic parent-retarget and final-entry races,
|
||||
staging symlink/hard-link swaps, close-failure cleanup, 255-byte basenames, diagnostic
|
||||
partial-write/fsync/close cleanup and attacker-leaf retention, privacy/mode, and all three closed schemas. The test file is also registered in
|
||||
`scripts/_ci_pytest_manifest.toml`.
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
# #512 follow-up — opt-in PDF content advisory with process isolation
|
||||
|
||||
**Date:** 2026-08-13 · **Provenance:** safe maintainer-owned replacement for the
|
||||
useful diagnostic idea in external PR #623 · **Status:** frozen minimal slice
|
||||
|
||||
## Decision
|
||||
|
||||
The structural contract delivered by #512 remains authoritative and unchanged:
|
||||
`PASS / FAIL / UNAVAILABLE` says whether the local PDF page structure can license a
|
||||
page locator. It does not say whether a page contains usable extracted text.
|
||||
|
||||
This slice adds one explicit diagnostic consumer without placing a new native parser
|
||||
inside the structural preflight process:
|
||||
|
||||
```text
|
||||
operator --classify-content
|
||||
-> pdf_read_preflight.py (existing structural parse + exact SHA-256)
|
||||
-> fixed child worker over the same already-hashed bytes
|
||||
-> closed validation in the parent
|
||||
-> content_advisory in the sidecar
|
||||
```
|
||||
|
||||
Default Stage-1 and library calls preserve the original sidecar shape and tool version
|
||||
and do not run the classifier. The classifier is invoked
|
||||
only when the operator selects `--classify-content` and the structural verdict is
|
||||
`PASS`. This is an observable, diagnostic-only use path, not an OCR router, anchor gate,
|
||||
or agent-prompt change.
|
||||
|
||||
## Frozen scope
|
||||
|
||||
- `scripts/pdf_read_preflight.py`: parent process, closed consumer, timeout/caps,
|
||||
sidecar projection, and optional local diagnostic writer.
|
||||
- `scripts/pdf_content_classifier_worker.py`: the only module that imports or calls
|
||||
`pdf_inspector`.
|
||||
- `shared/contracts/pdf/`: closed Draft 2020-12 schemas for the prompt-facing sidecar,
|
||||
worker stdout, and local-only diagnostic.
|
||||
- `requirements-pdf-content-classifier.txt`: isolated optional dependency pin; the
|
||||
package is not added to the default developer/CI environment.
|
||||
- synthetic and fake-worker tests in `scripts/test_pdf_read_preflight.py`, registered
|
||||
in the unified pytest manifest.
|
||||
|
||||
No pipeline, agent definition, prompt, claim-audit finalizer, OCR tool, model, network,
|
||||
or live/private PDF is added. No claim is made that scanned-page detection improves an
|
||||
academic output or closes a measured reliability gap.
|
||||
|
||||
## Trust and process boundaries
|
||||
|
||||
### Exact-byte binding
|
||||
|
||||
The existing structural preflight reads and hashes the PDF once. The parent sends those
|
||||
same bytes to the child on stdin. The worker does not reopen a path, so a file replacement
|
||||
between structural validation and classification cannot bind a result from different
|
||||
bytes to the structural SHA-256.
|
||||
|
||||
### Native parser containment
|
||||
|
||||
The parent never imports `pdf_inspector`. It starts the fixed repository worker with an
|
||||
argv list and `shell=False`. On POSIX the child is the process-group leader of a new
|
||||
session; the parent kills that group before reader/writer joins, including when the
|
||||
leader has exited while an ordinary descendant retains inherited pipe handles. Every
|
||||
terminal path repeats best-effort cleanup. On Windows the portable stdlib path terminates
|
||||
and reaps only the direct worker and makes no process-tree containment claim. The parent
|
||||
applies:
|
||||
|
||||
- one 5-second execution deadline beginning immediately after child startup;
|
||||
- one explicitly separate, shared 0.2-second teardown grace (not one grace per wait or
|
||||
helper);
|
||||
- 8,192-byte stdout limit;
|
||||
- 4,096-byte stderr limit;
|
||||
- distinct closed outcomes for launch failure, timeout, non-zero exit, signal exit,
|
||||
stdout/stderr limit, helper-startup/pipe failure, malformed JSON, and invalid closed
|
||||
output.
|
||||
|
||||
The loop polls and then immediately observes the monotonic clock. Only a poll result
|
||||
whose following observation is strictly before the execution deadline may be processed
|
||||
as a worker exit. A poll that returns at or after the boundary is `WORKER_TIMEOUT`.
|
||||
|
||||
Readers drain both pipes concurrently and retain no more than each limit plus one byte.
|
||||
Those are the only byte caps. The exact PDF input is written on a separate thread, so a
|
||||
child that never reads stdin cannot block the parent's timeout loop, and a classified
|
||||
result is accepted only after the complete input write. Helper construction failures are
|
||||
closed `WORKER_IO_ERROR` outcomes inside the same immediate post-`Popen` cleanup region.
|
||||
A native abort or segmentation fault terminates the child, not the structural preflight
|
||||
process.
|
||||
|
||||
This is process isolation, not a general OS sandbox. A POSIX descendant that deliberately
|
||||
detaches into another session or process group, and every host-wide resource failure,
|
||||
remain outside the containment claim; the optional third-party parser is not trusted.
|
||||
|
||||
### Worker stdout contract
|
||||
|
||||
`pdf_content_classifier_worker/1` is closed and permits only:
|
||||
|
||||
| `status` | `reason` | value fields |
|
||||
|---|---|---|
|
||||
| `CLASSIFIED` | `CLASSIFIED` | `TEXT_AVAILABLE` or `OCR_RECOMMENDED`; finite confidence in `[0,1]`; at most 50,000 unique non-negative page indexes |
|
||||
| `UNAVAILABLE` | `DEPENDENCY_ABSENT`, `CLASSIFIER_ERROR`, or `INVALID_CLASSIFIER_RESULT` | all value fields `null` |
|
||||
|
||||
The worker recognizes only the positive upstream combination `pdf_type ==
|
||||
"text_based"` with an empty OCR-page list as `TEXT_AVAILABLE`. Every other non-empty
|
||||
upstream type is reduced to `OCR_RECOMMENDED`; the open vendor enum is never emitted.
|
||||
The parent applies a hand-written validator equivalent to the closed worker schema,
|
||||
requires sorted unique integer pages, rejects JSON `NaN`/infinities and unknown fields,
|
||||
and binds every page index to
|
||||
`0 <= page < reader_page_count` from the structural parse.
|
||||
|
||||
### Sidecar semantics
|
||||
|
||||
The existing `schema: pdf_read_preflight/1` remains compatible. Calls without the
|
||||
opt-in keep the original field set and `tool: pdf_read_preflight/1.0.0`; opted-in
|
||||
sidecars use tool 1.1.0 and gain the following all-or-nothing extension:
|
||||
|
||||
- `verdict_scope: STRUCTURE_ONLY` — makes the old verdict's scope explicit;
|
||||
- `content_advisory` — one of `TEXT_AVAILABLE`,
|
||||
`OCR_RECOMMENDED`, `CONTENT_UNAVAILABLE`, or `STRUCTURAL_UNAVAILABLE`;
|
||||
- `content_classification` — a closed object carrying request state, closed reason,
|
||||
classification, confidence, and bounded page indexes.
|
||||
|
||||
An image-only PDF may truthfully have:
|
||||
|
||||
```json
|
||||
{
|
||||
"verdict": "PASS",
|
||||
"verdict_scope": "STRUCTURE_ONLY",
|
||||
"content_advisory": "OCR_RECOMMENDED"
|
||||
}
|
||||
```
|
||||
|
||||
That means the page tree is structurally coherent while text usability was not
|
||||
established. It is not a content `PASS`. Dependency absence or any worker failure yields
|
||||
`CONTENT_UNAVAILABLE` and never changes the structural verdict. A non-`PASS` structural
|
||||
result yields `STRUCTURAL_UNAVAILABLE` without starting the optional child.
|
||||
|
||||
### Operator diagnostics
|
||||
|
||||
The prompt-facing sidecar contains only closed reason codes. The upstream type and raw
|
||||
exception are absent. On POSIX, with the separate, explicit
|
||||
`--classifier-diagnostics <local-path>` option, the operator may create one exclusive,
|
||||
non-overwriting `0600` JSON file containing:
|
||||
|
||||
- the closed reason code;
|
||||
- observed stdout/stderr byte counts;
|
||||
- at most 512 bytes of clearly named `untrusted_detail`.
|
||||
|
||||
The diagnostic path and detail never appear in the sidecar. The file is local,
|
||||
unencrypted, untrusted operator evidence and must not be copied into an agent prompt.
|
||||
Platforms without POSIX `fchmod` reject the diagnostics option before path creation;
|
||||
the ordinary classifier subprocess remains available under its narrower Windows
|
||||
direct-worker containment claim.
|
||||
|
||||
The stdout-only legacy CLI performs no alias precheck, so an unreadable input or symlink
|
||||
loop remains an exit-0 structural `UNAVAILABLE` verdict. With either write option, every
|
||||
write target must resolve safely before structural parsing or worker launch.
|
||||
NFC/casefold canonical keys conservatively reject literal, `..`, case-only, and Unicode
|
||||
canonical-equivalent aliases even when their leaves do not exist; resolved keys reject
|
||||
symlink aliases; existing-inode comparison rejects hard links. Existing-inode errors
|
||||
fail closed except `ENOENT`, while failure to resolve the input itself stays a structural
|
||||
preflight concern and does not suppress an otherwise safe output of that verdict.
|
||||
|
||||
The local diagnostic remains exclusive-create/no-follow. Its resolved parent directory
|
||||
is opened and inode-bound before the worker, and final creation is relative to that
|
||||
dirfd, so a parent-symlink retarget cannot redirect raw diagnostic detail. The created
|
||||
fd is inode-bound before fchmod/write. On any pre-success fchmod, partial-write,
|
||||
file-fsync, close, or parent-fsync failure, cleanup performs a fresh no-follow lookup and
|
||||
unlinks the leaf only if it still identifies the created inode; it then best-effort
|
||||
parent-fsyncs without replacing the primary error. A malformed partial diagnostic cannot
|
||||
permanently consume the exclusive destination and immediate retry works. A symlink or
|
||||
hard-link attacker replacement has a different inode and is not removed.
|
||||
|
||||
For ordinary sidecar output on
|
||||
POSIX, the resolved parent directory is opened and inode-bound before the worker starts.
|
||||
A fixed-length random-named private `0700` staging directory is created and opened
|
||||
through that parent dirfd; every later operation is relative to the anchored parent or
|
||||
staging dirfd, so a parent-symlink retarget cannot redirect publication. Complete bytes
|
||||
use the fixed staging leaf `payload`, allowing a legal 255-byte destination basename.
|
||||
After file fsync, the open payload inode must still match the no-follow staging entry;
|
||||
symlink or hard-link replacement is rejected. Dirfd-relative `os.replace` installs the
|
||||
payload, its installed inode is rechecked, and the parent is fsynced. Final-component
|
||||
links are replaced rather than followed and cannot truncate the input or diagnostic.
|
||||
Cleanup guards close, unlink, staging-dir close/rmdir, and parent close independently;
|
||||
secondary cleanup failures never replace a primary publication error, and unpublished
|
||||
staging is removed. Any output `OSError` becomes a CLI usage error. Non-POSIX
|
||||
`--output` fails closed because the needed anchored dirfd publication surface is absent;
|
||||
classification to stdout retains its narrower direct-worker Windows contract.
|
||||
|
||||
The output parent must be caller-controlled. Python's standard library exposes no
|
||||
atomic compare-inode-and-rename operation. The private random staging directory is
|
||||
`0700`, the worker group is terminated before publication, and identity is checked both
|
||||
immediately before replace and immediately after install. A substitution observed after
|
||||
install is rejected and the observed attacker entry is removed. Those instantaneous
|
||||
postconditions cover the tested pre-check and exact check-to-replace swaps; a same-UID
|
||||
actor that continues racing after the final check is outside this process-isolation
|
||||
claim.
|
||||
|
||||
## Optional dependency model
|
||||
|
||||
`pdf-inspector` is not installed by `requirements-dev.txt`. Operators who deliberately
|
||||
select this diagnostic may install the isolated pin in
|
||||
`requirements-pdf-content-classifier.txt`. Only a top-level `ModuleNotFoundError` naming
|
||||
`pdf_inspector` emits the deterministic closed `DEPENDENCY_ABSENT` state. Internal or
|
||||
transitive import failures emit `CLASSIFIER_ERROR`, with bounded detail available only in
|
||||
the explicit local diagnostic; structural output is unaffected.
|
||||
|
||||
The hermetic suite does not download or execute the real optional package. It inserts a
|
||||
temporary fake `pdf_inspector` module into the child environment to cover the present,
|
||||
absent, scanned, text, malformed-result, and exception adapters. Therefore this slice
|
||||
proves the process and contract boundary, not the current third-party package's empirical
|
||||
classification accuracy.
|
||||
|
||||
## Test matrix
|
||||
|
||||
- Existing #512 synthetic structural corpus remains green.
|
||||
- Default path and structural non-`PASS` path prove the child is not started.
|
||||
- Actual repository worker with temporary fake module: dependency absent, text,
|
||||
scanned/open upstream type, classifier exception, and exact input bytes.
|
||||
- Fake workers: timeout, non-zero exit, signal, malformed JSON, unknown key/enum,
|
||||
non-finite/out-of-range confidence, invalid/duplicate/out-of-range pages, stdout flood,
|
||||
stderr flood, all three helper-startup failures, and a direct descendant that inherits
|
||||
pipes and must be terminated before joins.
|
||||
- True top-level dependency absence is distinct from internal and transitive import
|
||||
failures; the latter are local-diagnostic-only classifier errors.
|
||||
- Schema mutations reject the legacy/extension tool versions on the wrong shape.
|
||||
- Prompt-facing sidecar excludes raw worker detail; local diagnostic is bounded,
|
||||
exclusive, mode `0600`, and schema-valid.
|
||||
- A shared small teardown grace cannot accumulate across sequential waits; conservative
|
||||
case/canonical folding, literal/`..`, symlink, hard-link, `samefile` error, and
|
||||
malformed-input compatibility cases pin the precheck boundary.
|
||||
- A clock-before-poll adversarial sequence rejects an otherwise successful exit first
|
||||
observable at the deadline.
|
||||
- Post-check output symlink/hard-link races against both input and diagnostic replace the
|
||||
hostile final entry without following it; injected replace failure cleans staging.
|
||||
- Parent-symlink retargeting stays on the pre-worker bound directory; staging symlink and
|
||||
hard-link swaps are rejected by open-inode identity; a secondary close failure neither
|
||||
masks the primary error nor leaves staging; a 255-byte basename publishes.
|
||||
- Diagnostic-parent retargeting likewise stays on its pre-worker bound directory.
|
||||
- Diagnostic partial-write, file-fsync, and close failures remove only the created inode
|
||||
and permit retry; symlink/hard-link attacker replacements survive cleanup unchanged.
|
||||
- All three Draft 2020-12 schemas validate; the parent contains no
|
||||
`pdf_inspector` import.
|
||||
|
||||
## Residual risk and follow-on gate
|
||||
|
||||
This slice deliberately stops at an operator-visible diagnostic. Automatic OCR routing,
|
||||
changing anchor eligibility, or sending the advisory into writer/auditor context would
|
||||
change citation-integrity behavior and agent definitions. That requires a separate
|
||||
issue-first design, explicit consumer semantics, prompt-injection review for every field,
|
||||
and its own evidence. This slice is not that authorization.
|
||||
@@ -0,0 +1,5 @@
|
||||
# Optional, opt-in dependency for scripts/pdf_content_classifier_worker.py.
|
||||
# Kept out of requirements-dev.txt so the dependency-absent path remains a
|
||||
# supported deterministic state and the native parser is never imported by the
|
||||
# structural preflight process.
|
||||
pdf-inspector==0.2.6
|
||||
@@ -543,3 +543,7 @@ path = "scripts/test_build_claim_standing_candidate_ledger.py"
|
||||
[[pytest]]
|
||||
id = "655-claim-standing-candidate-ledger-integration"
|
||||
path = "scripts/test_check_claim_standing_candidate_ledger_integration.py"
|
||||
|
||||
[[pytest]]
|
||||
id = "512-pdf-read-preflight-and-content-sandbox"
|
||||
path = "scripts/test_pdf_read_preflight.py"
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
"""Isolated optional PDF content-classifier worker.
|
||||
|
||||
This process is the only place that imports or invokes ``pdf_inspector``. It
|
||||
receives the exact bytes already hashed by ``pdf_read_preflight.py`` on stdin
|
||||
and emits one small, closed JSON object on stdout. Any native panic, abort, or
|
||||
segmentation fault is therefore contained to this child and interpreted by the
|
||||
parent as an unavailable advisory signal.
|
||||
|
||||
The worker deliberately never emits the upstream classifier's free-form
|
||||
``pdf_type`` or exception text on stdout. Bounded exception detail is written
|
||||
only to stderr, which the parent discards unless the operator explicitly asks
|
||||
for a separate local diagnostic file.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
SCHEMA = "pdf_content_classifier_worker/1"
|
||||
MAX_PAGE_ENTRIES = 50_000
|
||||
MAX_OPERATOR_DETAIL_BYTES = 512
|
||||
|
||||
|
||||
def _emit(payload: dict[str, Any]) -> None:
|
||||
raw = json.dumps(
|
||||
payload,
|
||||
ensure_ascii=False,
|
||||
allow_nan=False,
|
||||
separators=(",", ":"),
|
||||
sort_keys=True,
|
||||
).encode("utf-8")
|
||||
sys.stdout.buffer.write(raw + b"\n")
|
||||
sys.stdout.buffer.flush()
|
||||
|
||||
|
||||
def _unavailable(reason: str) -> dict[str, Any]:
|
||||
return {
|
||||
"schema": SCHEMA,
|
||||
"status": "UNAVAILABLE",
|
||||
"reason": reason,
|
||||
"classification": None,
|
||||
"confidence": None,
|
||||
"pages_needing_ocr": None,
|
||||
}
|
||||
|
||||
|
||||
def _operator_detail(exc: BaseException) -> None:
|
||||
# This stream is never copied into the prompt-facing sidecar. Bound bytes
|
||||
# before writing so even a hostile exception string cannot flood the pipe.
|
||||
# Rendering the diagnostic must never replace the closed stdout result with
|
||||
# another crash: third-party exceptions can have a broken __str__, and
|
||||
# stderr can disappear while handling the original failure.
|
||||
try:
|
||||
try:
|
||||
detail = str(exc)
|
||||
except BaseException:
|
||||
detail = "<unprintable exception>"
|
||||
text = f"{type(exc).__name__}: {detail}".encode(
|
||||
"utf-8",
|
||||
errors="replace",
|
||||
)[:MAX_OPERATOR_DETAIL_BYTES]
|
||||
except BaseException:
|
||||
text = b"unprintable classifier exception"
|
||||
try:
|
||||
sys.stderr.buffer.write(text)
|
||||
sys.stderr.buffer.flush()
|
||||
except BaseException:
|
||||
pass
|
||||
|
||||
|
||||
def _normalize_result(classified: Any) -> dict[str, Any]:
|
||||
try:
|
||||
raw_pdf_type = classified.pdf_type
|
||||
raw_confidence = classified.confidence
|
||||
raw_pages = classified.pages_needing_ocr
|
||||
except BaseException as exc:
|
||||
_operator_detail(exc)
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
|
||||
if type(raw_pdf_type) is not str or not raw_pdf_type:
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
if type(raw_confidence) not in (int, float):
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
try:
|
||||
confidence = float(raw_confidence)
|
||||
except BaseException as exc:
|
||||
_operator_detail(exc)
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
if not math.isfinite(confidence) or not 0.0 <= confidence <= 1.0:
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
|
||||
try:
|
||||
iterator = iter(raw_pages)
|
||||
except BaseException as exc:
|
||||
_operator_detail(exc)
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
|
||||
pages: list[int] = []
|
||||
seen: set[int] = set()
|
||||
try:
|
||||
for page in iterator:
|
||||
if len(pages) >= MAX_PAGE_ENTRIES:
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
if type(page) is not int or page < 0:
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
if page in seen:
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
seen.add(page)
|
||||
pages.append(page)
|
||||
except BaseException as exc:
|
||||
_operator_detail(exc)
|
||||
return _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
|
||||
pages.sort()
|
||||
# Do not expose an open upstream enum. The one positively recognized
|
||||
# state is text_based with no OCR pages; every other non-empty upstream
|
||||
# type is conservatively reduced to the closed OCR_RECOMMENDED advisory.
|
||||
classification = (
|
||||
"TEXT_AVAILABLE"
|
||||
if raw_pdf_type == "text_based" and not pages
|
||||
else "OCR_RECOMMENDED"
|
||||
)
|
||||
return {
|
||||
"schema": SCHEMA,
|
||||
"status": "CLASSIFIED",
|
||||
"reason": "CLASSIFIED",
|
||||
"classification": classification,
|
||||
"confidence": confidence,
|
||||
"pages_needing_ocr": pages,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
pdf_inspector_spec = importlib.util.find_spec("pdf_inspector")
|
||||
except BaseException as exc:
|
||||
_operator_detail(exc)
|
||||
_emit(_unavailable("CLASSIFIER_ERROR"))
|
||||
return 0
|
||||
if pdf_inspector_spec is None:
|
||||
_emit(_unavailable("DEPENDENCY_ABSENT"))
|
||||
return 0
|
||||
|
||||
try:
|
||||
pdf_inspector = importlib.import_module("pdf_inspector")
|
||||
except BaseException as exc:
|
||||
_operator_detail(exc)
|
||||
_emit(_unavailable("CLASSIFIER_ERROR"))
|
||||
return 0
|
||||
|
||||
try:
|
||||
data = sys.stdin.buffer.read()
|
||||
classified = pdf_inspector.classify_pdf_bytes(data)
|
||||
except BaseException as exc:
|
||||
_operator_detail(exc)
|
||||
_emit(_unavailable("CLASSIFIER_ERROR"))
|
||||
return 0
|
||||
|
||||
try:
|
||||
normalized = _normalize_result(classified)
|
||||
except BaseException as exc:
|
||||
# Final adapter backstop: a future normalization branch must not let a
|
||||
# malformed third-party object escape the closed stdout contract.
|
||||
_operator_detail(exc)
|
||||
normalized = _unavailable("INVALID_CLASSIFIER_RESULT")
|
||||
_emit(normalized)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+1104
-21
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -3,6 +3,37 @@
|
||||
Schema files for cross-skill contracts: reviewer sprint contracts, Material Passport
|
||||
ports, and (v3.6.7+) cross-model audit artifact pipelines.
|
||||
|
||||
## PDF read-integrity and optional content advisory (#512 follow-up)
|
||||
|
||||
- `pdf/pdf_read_preflight.schema.json` accepts the unchanged legacy structural sidecar
|
||||
or the all-or-nothing opt-in content extension. In that extension, `verdict` is
|
||||
explicitly `verdict_scope: STRUCTURE_ONLY`; `OCR_RECOMMENDED` never rewrites that
|
||||
structural value into a content claim. The schema binds the legacy shape to tool
|
||||
version 1.0.0 and the extension shape to 1.1.0.
|
||||
- `pdf/pdf_content_classifier_worker.schema.json` closes the stdout of the fixed
|
||||
isolated worker to two classifications, three unavailable reasons, finite bounded
|
||||
confidence, and bounded non-negative page indexes. Runtime additionally binds every
|
||||
page to the structural page count.
|
||||
- `pdf/pdf_content_classifier_diagnostic.schema.json` is the separate POSIX-only,
|
||||
local mode-`0600` operator artifact. Platforms without `fchmod` reject its CLI
|
||||
option before path creation. Its explicitly untrusted detail is capped and never
|
||||
copied into or referenced by the sidecar. File-writing CLI invocations use
|
||||
conservative NFC/casefold keys, path resolution, and existing-inode checks to reject
|
||||
aliases before worker launch; the stdout-only legacy path adds no such precondition.
|
||||
POSIX sidecar output pre-binds the parent dirfd/inode, then uses a private `0700`
|
||||
fixed-name staging directory and anchored dirfd-relative publication. Open-inode
|
||||
checks reject staging entry swaps; atomic final-entry replacement does not follow
|
||||
post-check links. Non-POSIX output fails closed; stdout classification remains
|
||||
available. The diagnostic parent is likewise dirfd-bound before the worker. These are
|
||||
instantaneous inode postconditions, not a general same-UID sandbox; callers control
|
||||
the output parent. A failed diagnostic unlinks only the no-follow leaf still matching
|
||||
its created fd inode, preserving the primary error and any attacker replacement while
|
||||
keeping its exclusive path retryable.
|
||||
|
||||
Runtime: `scripts/pdf_read_preflight.py` and
|
||||
`scripts/pdf_content_classifier_worker.py`. Frozen opt-in scope and residual risk:
|
||||
`docs/design/2026-08-13-512-pdf-content-classification-sandbox-spec.md`.
|
||||
|
||||
## Claim-standing candidate ledger (#655 Track A)
|
||||
|
||||
- `claim_standing/query_plan.schema.json` (`claim-standing-query-plan/1.0`)
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://github.com/Imbad0202/academic-research-skills/shared/contracts/pdf/pdf_content_classifier_diagnostic.schema.json",
|
||||
"title": "POSIX local-only PDF classifier operator diagnostic",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema",
|
||||
"reason",
|
||||
"untrusted_detail",
|
||||
"stdout_bytes_observed",
|
||||
"stderr_bytes_observed"
|
||||
],
|
||||
"properties": {
|
||||
"schema": {"const": "pdf_content_classifier_diagnostic/1"},
|
||||
"reason": {
|
||||
"enum": [
|
||||
"NOT_REQUESTED",
|
||||
"STRUCTURAL_NOT_PASS",
|
||||
"CLASSIFIED",
|
||||
"DEPENDENCY_ABSENT",
|
||||
"CLASSIFIER_ERROR",
|
||||
"INVALID_CLASSIFIER_RESULT",
|
||||
"WORKER_LAUNCH_ERROR",
|
||||
"WORKER_TIMEOUT",
|
||||
"WORKER_NONZERO_EXIT",
|
||||
"WORKER_SIGNAL",
|
||||
"WORKER_STDOUT_LIMIT",
|
||||
"WORKER_STDERR_LIMIT",
|
||||
"WORKER_IO_ERROR",
|
||||
"WORKER_MALFORMED_OUTPUT",
|
||||
"WORKER_INVALID_OUTPUT"
|
||||
]
|
||||
},
|
||||
"untrusted_detail": {"type": "string", "maxLength": 512},
|
||||
"stdout_bytes_observed": {"type": "integer", "minimum": 0},
|
||||
"stderr_bytes_observed": {"type": "integer", "minimum": 0}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://github.com/Imbad0202/academic-research-skills/shared/contracts/pdf/pdf_content_classifier_worker.schema.json",
|
||||
"title": "Isolated PDF content-classifier worker result",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema",
|
||||
"status",
|
||||
"reason",
|
||||
"classification",
|
||||
"confidence",
|
||||
"pages_needing_ocr"
|
||||
],
|
||||
"properties": {
|
||||
"schema": {"const": "pdf_content_classifier_worker/1"},
|
||||
"status": {"enum": ["CLASSIFIED", "UNAVAILABLE"]},
|
||||
"reason": {
|
||||
"enum": [
|
||||
"CLASSIFIED",
|
||||
"DEPENDENCY_ABSENT",
|
||||
"CLASSIFIER_ERROR",
|
||||
"INVALID_CLASSIFIER_RESULT"
|
||||
]
|
||||
},
|
||||
"classification": {
|
||||
"enum": ["TEXT_AVAILABLE", "OCR_RECOMMENDED", null]
|
||||
},
|
||||
"confidence": {
|
||||
"type": ["number", "null"],
|
||||
"minimum": 0.0,
|
||||
"maximum": 1.0
|
||||
},
|
||||
"pages_needing_ocr": {
|
||||
"oneOf": [
|
||||
{
|
||||
"type": "array",
|
||||
"maxItems": 50000,
|
||||
"uniqueItems": true,
|
||||
"items": {"type": "integer", "minimum": 0}
|
||||
},
|
||||
{"type": "null"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"oneOf": [
|
||||
{
|
||||
"properties": {
|
||||
"status": {"const": "CLASSIFIED"},
|
||||
"reason": {"const": "CLASSIFIED"},
|
||||
"classification": {"enum": ["TEXT_AVAILABLE", "OCR_RECOMMENDED"]},
|
||||
"confidence": {"type": "number", "minimum": 0.0, "maximum": 1.0},
|
||||
"pages_needing_ocr": {
|
||||
"type": "array",
|
||||
"maxItems": 50000,
|
||||
"uniqueItems": true,
|
||||
"items": {"type": "integer", "minimum": 0}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"properties": {
|
||||
"status": {"const": "UNAVAILABLE"},
|
||||
"reason": {
|
||||
"enum": [
|
||||
"DEPENDENCY_ABSENT",
|
||||
"CLASSIFIER_ERROR",
|
||||
"INVALID_CLASSIFIER_RESULT"
|
||||
]
|
||||
},
|
||||
"classification": {"type": "null"},
|
||||
"confidence": {"type": "null"},
|
||||
"pages_needing_ocr": {"type": "null"}
|
||||
}
|
||||
}
|
||||
],
|
||||
"allOf": [
|
||||
{
|
||||
"if": {
|
||||
"properties": {"classification": {"const": "TEXT_AVAILABLE"}},
|
||||
"required": ["classification"]
|
||||
},
|
||||
"then": {
|
||||
"properties": {"pages_needing_ocr": {"maxItems": 0}}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://github.com/Imbad0202/academic-research-skills/shared/contracts/pdf/pdf_read_preflight.schema.json",
|
||||
"title": "PDF read-integrity sidecar with opt-in content extension",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema",
|
||||
"verdict",
|
||||
"file",
|
||||
"sha256",
|
||||
"declared_page_count",
|
||||
"enumerated_page_count",
|
||||
"reader_page_count",
|
||||
"warnings",
|
||||
"generated_at",
|
||||
"tool"
|
||||
],
|
||||
"properties": {
|
||||
"schema": {"const": "pdf_read_preflight/1"},
|
||||
"verdict": {"enum": ["PASS", "FAIL", "UNAVAILABLE"]},
|
||||
"verdict_scope": {"const": "STRUCTURE_ONLY"},
|
||||
"file": {"type": "string"},
|
||||
"sha256": {
|
||||
"oneOf": [
|
||||
{"type": "string", "pattern": "^[0-9a-f]{64}$"},
|
||||
{"type": "null"}
|
||||
]
|
||||
},
|
||||
"declared_page_count": {"type": ["integer", "null"]},
|
||||
"enumerated_page_count": {"type": ["integer", "null"]},
|
||||
"reader_page_count": {"type": ["integer", "null"]},
|
||||
"warnings": {"type": "array", "items": {"type": "string"}},
|
||||
"content_advisory": {
|
||||
"enum": [
|
||||
"TEXT_AVAILABLE",
|
||||
"OCR_RECOMMENDED",
|
||||
"CONTENT_UNAVAILABLE",
|
||||
"STRUCTURAL_UNAVAILABLE"
|
||||
]
|
||||
},
|
||||
"content_classification": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema",
|
||||
"requested",
|
||||
"status",
|
||||
"reason",
|
||||
"classification",
|
||||
"confidence",
|
||||
"pages_needing_ocr"
|
||||
],
|
||||
"properties": {
|
||||
"schema": {"const": "pdf_content_classification/1"},
|
||||
"requested": {"const": true},
|
||||
"status": {"enum": ["CLASSIFIED", "UNAVAILABLE"]},
|
||||
"reason": {
|
||||
"enum": [
|
||||
"STRUCTURAL_NOT_PASS",
|
||||
"CLASSIFIED",
|
||||
"DEPENDENCY_ABSENT",
|
||||
"CLASSIFIER_ERROR",
|
||||
"INVALID_CLASSIFIER_RESULT",
|
||||
"WORKER_LAUNCH_ERROR",
|
||||
"WORKER_TIMEOUT",
|
||||
"WORKER_NONZERO_EXIT",
|
||||
"WORKER_SIGNAL",
|
||||
"WORKER_STDOUT_LIMIT",
|
||||
"WORKER_STDERR_LIMIT",
|
||||
"WORKER_IO_ERROR",
|
||||
"WORKER_MALFORMED_OUTPUT",
|
||||
"WORKER_INVALID_OUTPUT"
|
||||
]
|
||||
},
|
||||
"classification": {
|
||||
"enum": ["TEXT_AVAILABLE", "OCR_RECOMMENDED", null]
|
||||
},
|
||||
"confidence": {
|
||||
"type": ["number", "null"],
|
||||
"minimum": 0.0,
|
||||
"maximum": 1.0
|
||||
},
|
||||
"pages_needing_ocr": {
|
||||
"oneOf": [
|
||||
{
|
||||
"type": "array",
|
||||
"maxItems": 50000,
|
||||
"uniqueItems": true,
|
||||
"items": {"type": "integer", "minimum": 0}
|
||||
},
|
||||
{"type": "null"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"oneOf": [
|
||||
{
|
||||
"properties": {
|
||||
"status": {"const": "CLASSIFIED"},
|
||||
"reason": {"const": "CLASSIFIED"},
|
||||
"classification": {"enum": ["TEXT_AVAILABLE", "OCR_RECOMMENDED"]},
|
||||
"confidence": {"type": "number", "minimum": 0.0, "maximum": 1.0},
|
||||
"pages_needing_ocr": {
|
||||
"type": "array",
|
||||
"maxItems": 50000,
|
||||
"uniqueItems": true,
|
||||
"items": {"type": "integer", "minimum": 0}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"properties": {
|
||||
"status": {"const": "UNAVAILABLE"},
|
||||
"reason": {
|
||||
"enum": [
|
||||
"STRUCTURAL_NOT_PASS",
|
||||
"DEPENDENCY_ABSENT",
|
||||
"CLASSIFIER_ERROR",
|
||||
"INVALID_CLASSIFIER_RESULT",
|
||||
"WORKER_LAUNCH_ERROR",
|
||||
"WORKER_TIMEOUT",
|
||||
"WORKER_NONZERO_EXIT",
|
||||
"WORKER_SIGNAL",
|
||||
"WORKER_STDOUT_LIMIT",
|
||||
"WORKER_STDERR_LIMIT",
|
||||
"WORKER_IO_ERROR",
|
||||
"WORKER_MALFORMED_OUTPUT",
|
||||
"WORKER_INVALID_OUTPUT"
|
||||
]
|
||||
},
|
||||
"classification": {"type": "null"},
|
||||
"confidence": {"type": "null"},
|
||||
"pages_needing_ocr": {"type": "null"}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"generated_at": {"type": "string", "format": "date-time"},
|
||||
"tool": {"type": "string", "pattern": "^pdf_read_preflight/[0-9]+\\.[0-9]+\\.[0-9]+$"}
|
||||
},
|
||||
"oneOf": [
|
||||
{
|
||||
"properties": {
|
||||
"tool": {"const": "pdf_read_preflight/1.0.0"}
|
||||
},
|
||||
"not": {
|
||||
"anyOf": [
|
||||
{"required": ["verdict_scope"]},
|
||||
{"required": ["content_advisory"]},
|
||||
{"required": ["content_classification"]}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"required": ["verdict_scope", "content_advisory", "content_classification"],
|
||||
"properties": {
|
||||
"tool": {"const": "pdf_read_preflight/1.1.0"}
|
||||
},
|
||||
"oneOf": [
|
||||
{
|
||||
"properties": {
|
||||
"verdict": {"enum": ["FAIL", "UNAVAILABLE"]},
|
||||
"content_advisory": {"const": "STRUCTURAL_UNAVAILABLE"},
|
||||
"content_classification": {
|
||||
"properties": {
|
||||
"status": {"const": "UNAVAILABLE"},
|
||||
"reason": {"const": "STRUCTURAL_NOT_PASS"}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"properties": {
|
||||
"verdict": {"const": "PASS"},
|
||||
"content_advisory": {"const": "CONTENT_UNAVAILABLE"},
|
||||
"content_classification": {
|
||||
"properties": {
|
||||
"status": {"const": "UNAVAILABLE"},
|
||||
"reason": {
|
||||
"enum": [
|
||||
"DEPENDENCY_ABSENT",
|
||||
"CLASSIFIER_ERROR",
|
||||
"INVALID_CLASSIFIER_RESULT",
|
||||
"WORKER_LAUNCH_ERROR",
|
||||
"WORKER_TIMEOUT",
|
||||
"WORKER_NONZERO_EXIT",
|
||||
"WORKER_SIGNAL",
|
||||
"WORKER_STDOUT_LIMIT",
|
||||
"WORKER_STDERR_LIMIT",
|
||||
"WORKER_IO_ERROR",
|
||||
"WORKER_MALFORMED_OUTPUT",
|
||||
"WORKER_INVALID_OUTPUT"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"properties": {
|
||||
"verdict": {"const": "PASS"},
|
||||
"content_advisory": {"const": "TEXT_AVAILABLE"},
|
||||
"content_classification": {
|
||||
"properties": {
|
||||
"status": {"const": "CLASSIFIED"},
|
||||
"classification": {"const": "TEXT_AVAILABLE"},
|
||||
"pages_needing_ocr": {"maxItems": 0}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"properties": {
|
||||
"verdict": {"const": "PASS"},
|
||||
"content_advisory": {"const": "OCR_RECOMMENDED"},
|
||||
"content_classification": {
|
||||
"properties": {
|
||||
"status": {"const": "CLASSIFIED"},
|
||||
"classification": {"const": "OCR_RECOMMENDED"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user