feat(daymade-docs): add pdf-to-html skill

Convert a PDF into a single self-contained, readable HTML file that
preserves images, charts and reading order, with optional parallel
translation into another language. Distilled from a real
PDF-to-other-language HTML session.

Bundles three scripts (structured extraction with decorative-image
detection, data-driven HTML build with font-size heading inference and
base64-inlined images, adaptive headless-Chrome visual verification) and
two references (Dynamic-Workflow parallel translation with fidelity
rules; failure-cases / do-not-attempt). Registered under daymade-docs
(1.1.0 -> 1.2.0).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
daymade
2026-06-07 01:31:40 +08:00
parent 6102d7edb7
commit 80a856edfe
7 changed files with 873 additions and 2 deletions
+3 -2
View File
@@ -135,7 +135,7 @@
"description": "Documentation suite plugin that exposes document conversion, Mermaid diagram generation, PDF/PPT creation, and documentation cleanup skills under one shared namespace",
"source": "./daymade-docs",
"strict": false,
"version": "1.1.0",
"version": "1.2.0",
"category": "suite",
"keywords": [
"suite",
@@ -151,7 +151,8 @@
"./mermaid-tools",
"./pdf-creator",
"./ppt-creator",
"./docs-cleaner"
"./docs-cleaner",
"./pdf-to-html"
]
},
{
+150
View File
@@ -0,0 +1,150 @@
---
name: pdf-to-html
description: Converts a PDF into one self-contained, readable HTML file that preserves images, tables, charts and reading order — optionally translating it into another language while keeping every figure. Uses structured extraction (PyMuPDF), font-size-driven layout, compressed base64-inlined images (a single portable file), and mandatory headless-Chrome visual verification. Use whenever someone wants to READ a PDF as a web page or clean document, turn a PDF into HTML, or translate a PDF into another language while keeping its images/tables/charts intact — e.g. "PDF 转 HTML", "把这个 PDF 转成中文网页版", "make this report readable", "translate this PDF but don't lose the charts", "I just want to read this PDF on my phone". Distinct from doc-to-markdown (plain Markdown text) and pdf-creator (Markdown→PDF) — this one produces a styled, image-faithful HTML reading experience.
---
# PDF to HTML
Turn a PDF into a single, self-contained, readable HTML file — images, tables,
charts and reading order preserved — and optionally translate it, keeping every
figure in place.
The pipeline is **extract → look → (translate) → build → verify**. The middle
"look" and final "verify" steps are where faithfulness actually comes from: a PDF
is a layout, not just a text stream, so you read the rendered pages before
building and the rendered HTML before delivering.
This skill runs **inline** (no `context: fork`): translation orchestrates a
Dynamic Workflow, and a subagent cannot spawn one.
## When to use / not use
- **Use** when the goal is to *read* a PDF as HTML/web page, to convert a PDF to
a styled HTML document, or to translate a PDF into another language while
keeping its figures and tables.
- **doc-to-markdown** instead if they want plain Markdown text (no styling, figures optional).
- **pdf-creator** instead for the reverse direction (Markdown → PDF).
## What it does NOT do
- **Scanned/image-only PDFs** (no text layer): OCR first (e.g. `ocrmypdf`), then use this.
- **Complex multi-column tables**: cell *text* is preserved and readable, but column
alignment can flatten into a text flow — PyMuPDF reads a table as text blocks, not a
grid, so the grid lines are gone. Tables that are *images* in the PDF survive as
images. If the table's grid structure is essential, use **doc-to-markdown** (pandoc
rebuilds real tables) or convert that page separately.
- **Pixel-perfect facsimile**: output is a clean *re-flow* that keeps images and
reading order, not a 1:1 copy of the original page layout.
- **Rewriting**: it translates and re-lays-out; it does not summarize, add a TL;DR,
or editorialize. Faithfulness is the point (see Fidelity below).
## Dependencies
`uv` (runs Python with inline deps), Google Chrome or Chromium (visual
verification). Python packages come via `uv run --with`: PyMuPDF, Pillow, numpy.
Nothing to pre-install beyond Chrome and uv.
## Workflow
Copy this checklist and tick as you go:
```
- [ ] 1. Extract structure + render pages (extract_pdf.py)
- [ ] 2. Read pages/*.png — SEE the layout, find content vs decorative images
- [ ] 3. (only if translating) run the translation workflow
- [ ] 4. Build the single-file HTML (build_html.py)
- [ ] 5. Verify visually (verify_render.py → Read every segment)
- [ ] 6. Deliver the .html
```
### 1. Extract
```bash
uv run --with pymupdf python scripts/extract_pdf.py input.pdf
```
Writes `input-build/` with `structure.json` (text blocks with font sizes + image
blocks flagged `decorative`), `images/`, and `pages/` (one PNG per page).
### 2. Look before you build
Read `input-build/pages/*.png`. This is not optional: you need to see the real
layout, confirm which images are content vs decoration, and spot tables/charts.
For a long PDF, read every page; for a short one it's quick. This is also where
you understand the document well enough to translate it well.
### 3. Translate (optional)
Only if the user asked for another language. Read
[references/translation_workflow.md](references/translation_workflow.md) and
follow it: a Dynamic Workflow translates pages in parallel, captions data charts,
and reconciles terminology. It produces two overlay files (`units.json`,
`caps.json`) that step 4 consumes. **Do not** hand-translate inline for anything
longer than a page — the workflow keeps terminology consistent and is far faster.
### 4. Build
```bash
# original-language HTML
uv run --with Pillow python scripts/build_html.py input-build/structure.json --out output.html
# translated HTML (overlays from step 3)
uv run --with Pillow python scripts/build_html.py input-build/structure.json --out output.html \
--translation input-build/units.json --captions input-build/caps.json --lang zh-CN
```
`build_html.py` is **data-driven**: it infers heading levels from font size (most
common size = body; larger steps up to h3/h2/h1), drops decorative images, and
inlines content images as compressed base64 → one portable file. It is not
hand-tuned to any document. If a particular PDF has an unusual structure (e.g.
multi-column, sidebars, a figure the size heuristic misreads), read the script and
adjust — it's short and meant to be edited per document.
### 5. Verify visually (mandatory)
```bash
uv run --with Pillow --with numpy python scripts/verify_render.py output.html
```
Then **Read every `seg-*.png`** and check: fonts render (no tofu boxes), no
clipped tables/figures, headings/lists look right, all expected images present.
Text being correct does not mean the render is correct (failure_cases #7). Fix and
re-verify until it's clean.
A quick structural cross-check is fine too, but count occurrences correctly:
`grep -o '<figure>' output.html | wc -l`**not** `grep -c` (failure_cases #1).
### 6. Deliver
Hand over the single `.html`. It's self-contained (images inlined), so it opens
with a double-click and nothing can go missing.
## Scripts
| Script | Run with | Purpose |
|--------|----------|---------|
| `scripts/extract_pdf.py` | `uv run --with pymupdf` | PDF → structure.json + images/ + page renders |
| `scripts/build_html.py` | `uv run --with Pillow` | structure.json (+ optional translation/captions) → single-file HTML |
| `scripts/verify_render.py` | `uv run --with Pillow --with numpy` | headless-Chrome render → readable PNG segments |
## Fidelity (read before translating)
The deliverable looks authoritative, so wrong content is worse than ugly content.
The non-negotiable rules — and the specific ways this has gone wrong before — are
in [references/failure_cases.md](references/failure_cases.md). The one that bites
hardest: **never give a real person an inferred translated name, and copy every
number/proper-noun verbatim** (failure_cases #6). Read that file before any
translation run; skim it before any run.
## Next Step
After producing the HTML, suggest the natural follow-up:
```
Conversion complete: output.html (single self-contained file).
Options:
A) Make a PDF of it — run /daymade-docs:pdf-creator if you want a print/share copy (Recommended if they need to send it)
B) Extract the text as Markdown instead — run /daymade-docs:doc-to-markdown (if they wanted editable text, not a reading page)
C) No thanks — the HTML is what I wanted
```
@@ -0,0 +1,78 @@
# Failure Cases — Do NOT Attempt
Real traps from building this pipeline. Each one cost a wrong turn; reading them
saves you the same detour. Skim before you start, re-read #6 before translating.
## Contents
- Verification traps (#1, #7)
- Chrome rendering limit (#2)
- Workflow / agent traps (#3, #4, #5)
- Fidelity rule — the important one (#6)
- Image classification (#8)
---
## 1. `grep -c` counts lines, not matches
When you sanity-check the built HTML ("are there 3 `<li>`? 4 `<figure>`?"),
`grep -c '<li>' file` counts **lines that contain a match**. Minified HTML often
puts several `<li>` on one line, so `grep -c` reports `1` when there are really 3
— and you "discover" a structure bug that isn't there.
**Do:** `grep -o '<li>' file | wc -l` to count occurrences.
## 2. Chrome headless screenshot caps around 16384px
A 2x device-scale-factor screenshot of a long page **silently truncates** once
physical height passes ~16384px — no error, the bottom just vanishes. You verify
the top, declare success, and miss that the last sections never rendered.
**Do:** probe real height at 1x first, then pick a scale that keeps the whole
page under the cap. `verify_render.py` already does this; remember it for any
manual screenshot.
## 3. Dynamic Workflow return value is wrapped
A workflow's task-output file is `{summary, agentCount, logs, result}` — the value
your script returned lives under **`result`**. `json.load(f)["units"]` throws
`KeyError`; read `json.load(f)["result"]["units"]`.
## 4. Translated text may arrive HTML-entity escaped
A translation agent sometimes emits `>` as `&gt;`. If you then `html.escape()` it
again you get `&amp;gt;`, which renders as the literal `&gt;`. **Do:**
`html.unescape()` each translated string once before merging it in.
## 5. Agent socket failure → resume, don't restart
In a multi-agent workflow an individual agent can die on a transient socket close
("The socket connection was closed unexpectedly"). Don't re-run the whole batch —
**resume** the workflow: completed agents return from cache, only the failed ones
re-run. (Don't pre-conclude it's a "blip" either — but for a one-off transient,
resume is the cheap correct move.)
## 6. FIDELITY: never invent a name, number, or fact (the important one)
When translating, do **not** give a real person a translated name you inferred.
Real case: a romaji name with no given Han/Kanji form was "helpfully" rendered
into characters by sound — that assigns a real human an identity the source never
stated, and it was probably wrong too. **Keep the original spelling.**
Same rule for everything factual: copy numbers, percentages, years, and proper
nouns (people, companies, institutions, products) verbatim — never infer them.
For any data chart, have the agent Read the original image and match values
**pixel-by-pixel** before writing a caption; don't write numbers "from memory".
And translate faithfully: a translation is not a rewrite. Do not add a TL;DR the
author didn't write, a "why this matters to *you*" localization aside, or a
one-line conclusion the source doesn't state. Those are the reviewer's favorite
overreach; refuse them.
This is where a faithful-looking deliverable most easily goes wrong and where it
most damages trust.
## 7. Text-correct is not render-correct
Confirming the text is right (grep found the words, python read the string) is
**not** enough. Fonts fall back to tofu boxes, tables overflow their column, a
translated heading wraps into an ugly orphan — none of it shows up until you LOOK
at the rendered page. Visual verification is a required step, not an optional one.
## 8. Decorative images vs content images
A PDF carries lots of non-content rasters: footer logos repeated every page,
hairline rules, bullet glyphs. They're tiny (often < 3 KB) or appear at the same
bbox on every page. Inlining them pollutes the reading flow. Classify by byte
size + repeated-bbox-across-pages (`extract_pdf.py` flags `decorative`), and drop
them by default. Keep them only if a document genuinely uses a small image as
content.
@@ -0,0 +1,152 @@
# Translation Workflow (optional)
Read this only when the user wants the HTML in a different language than the PDF.
Translation runs as a Dynamic Workflow so pages translate in parallel and a final
pass keeps terminology consistent. It must run in the **main context** (this skill
is inline) — a subagent cannot spawn the workflow's agents.
## Contents
- When to translate
- Step A: prepare translation units
- Step B: run the workflow (parallel translate → caption charts → reconcile)
- Glossary discipline
- Fidelity rules
- Chart handling — ask the user
- Text-overlay convention
- Step C: merge back and build
## When to translate
Only when the user asks ("translate to X", "中文版", "make an English version").
Otherwise build directly from the original text — don't translate unprompted.
## Step A: prepare translation units
Extract the text to translate, each with a stable id that **matches the key
`build_html.py` expects** (`p{page}_t{Nth-text-block-on-that-page}`, page numbers
skipped). This is what lets the translation merge back onto the right block.
```python
import json
struct = json.load(open("build/structure.json"))
units = []
for pg in struct["pages"]:
p, ti = pg["page"], 0
for b in pg["blocks"]:
if b["type"] != "text":
continue
t = b["text"].strip()
if t.isdigit() and len(t) <= 4: # page number — skip, same rule as build_html.py
continue
units.append({"id": f"p{p}_t{ti}", "page": p, "src": b["text"]})
ti += 1
json.dump(units, open("build/units_src.json", "w"), ensure_ascii=False, indent=1)
print(len(units), "units")
```
## Step B: run the workflow
Before launching, read the rendered `pages/*.png` and decide the **register**
(who reads this, how formal) and a **glossary** — these go into every agent prompt
so the whole document sounds like one translator. Skeleton (adapt the bracketed
parts; keep the structure):
```javascript
export const meta = {
name: 'translate-pdf-units',
description: 'Translate extracted PDF units in parallel, caption charts, reconcile terminology',
phases: [{ title: 'Translate' }, { title: 'Captions' }, { title: 'Reconcile' }],
}
const BG = `[1-2 sentences: what this document is, who the reader is, target language + register].`
const GLOSSARY = `[key term -> target-language definition; list names/orgs/products to keep verbatim].`
const CONV = `Output convention: blank line = paragraph break; lines starting "- " = list items; ` +
`a numbered sub-heading on its own line gets prefixed "## ". Keep ALL numbers, percentages, ` +
`years and proper nouns verbatim. Never invent a translated name for a real person.`
const U = { type:'object', properties:{ units:{ type:'array', items:{ type:'object',
properties:{ id:{type:'string'}, tr:{type:'string'} }, required:['id','tr'] } } }, required:['units'] }
const C = { type:'object', properties:{ chart:{type:'string'}, title:{type:'string'}, caption:{type:'string'} },
required:['chart','title','caption'] }
phase('Translate') // one agent per page, in parallel
const pages = [/* 1, 2, ... N */]
const translated = await parallel(pages.map(p => () =>
agent(`${BG}\n\nRead /ABS/build/units_src.json. Translate every unit whose page==${p}.\n` +
`${CONV}\n\n${GLOSSARY}\n\nReturn units:[{id,tr}] with ids exactly as in the file; omit none.`,
{ label:`tr p${p}`, phase:'Translate', schema:U })))
phase('Captions') // one agent per data chart
const charts = [/* { file:'img-p5-1.png' }, ... only real data charts */]
const caps = await parallel(charts.map(c => () =>
agent(`Read /ABS/build/images/${c.file}. It is a data chart labeled in the source language. ` +
`Output a target-language title and a 2-4 sentence reading of the REAL data (axes, what rises/` +
`falls, key values, highest/lowest). State only values actually visible — invent nothing. chart="${c.file}".`,
{ label:`cap ${c.file}`, phase:'Captions', schema:C })))
phase('Reconcile') // one pass to unify terminology
const all = translated.filter(Boolean).flatMap(t => t.units || [])
const fixed = await agent(`${BG}\n\nUnify terminology per the glossary, smooth cross-page seams, ` +
`change no meaning, add or drop nothing, keep the markdown convention. Return all ${all.length} units, ` +
`ids unchanged.\n${GLOSSARY}\n\n${JSON.stringify(all)}`, { label:'reconcile', schema:U })
return { units: (fixed && fixed.units) || all, captions: caps.filter(Boolean) }
```
Notes: per-page agents each Read the same units file and filter by page — simple
and robust for a short document. If a page agent dies on a socket close, **resume**
the workflow (failure_cases #5), don't re-run all of them.
## Glossary discipline
Fix the glossary before translating and pass it to every agent. Without it,
recurring terms drift across pages (the same word translated three ways). The
reconcile pass enforces it globally.
## Fidelity rules
A translation is faithful, not a rewrite. Copy numbers/percentages/years and
proper nouns verbatim. **Never give a real person an inferred translated name**
(failure_cases #6). For charts, match the original image pixel-by-pixel. Do not
add a TL;DR, a localization "why this matters to you" aside, or a conclusion the
source didn't write — that's overreach a reviewer will praise and the author never
asked for.
## Chart handling — ask the user
A data chart's internal labels are in the source language. Three options; the
default is the safest. Use AskUserQuestion:
- **Keep original image + target-language caption** (default — zero data risk; the
caption explains the chart in the reader's language).
- **Keep original + unify a heading bar / frame** (the `--captions` path already
draws a heading bar; reduces the "pasted from elsewhere" look).
- **Redraw as a native target-language chart** (best integration, but you must read
every value off the original correctly — only do this once the data is verified,
and re-draw line charts from real endpoints/trend, not guessed points).
## Text-overlay convention
Translated text uses light markdown: blank line = paragraph, `- ` = list item,
`## ` = sub-heading. `build_html.py`'s renderer understands these, and they also
fix the common case where a PDF splits "...end of section. Next-heading" into one
block across a page break — mark the heading with `## ` and it lays out correctly.
## Step C: merge back and build
Turn the workflow result into the two overlay files `build_html.py` consumes.
Mind two traps: the workflow output is wrapped in `result` (failure_cases #3), and
each string must be `html.unescape`d once (failure_cases #4).
```python
import json, html
res = json.load(open("workflow-output.json"))["result"] # <-- ["result"], not top level
units = {u["id"]: html.unescape(u["tr"]) for u in res["units"]}
caps = {c["chart"]: {"title": html.unescape(c["title"]),
"caption": html.unescape(c["caption"])} for c in res.get("captions", [])}
json.dump(units, open("build/units.json", "w"), ensure_ascii=False, indent=1)
json.dump(caps, open("build/caps.json", "w"), ensure_ascii=False, indent=1)
# Verify no unit was dropped before building.
src_ids = {u["id"] for u in json.load(open("build/units_src.json"))}
missing = src_ids - set(units)
print("missing translations:", missing or "none")
```
Then build with the overlays and the right `lang`:
```bash
uv run --with Pillow python build_html.py build/structure.json --out output.html \
--translation build/units.json --captions build/caps.json --lang zh-CN --title "..."
```
@@ -0,0 +1,224 @@
#!/usr/bin/env python3
"""Rebuild structure.json into one self-contained, readable HTML file.
Data-driven, not template-per-document: heading levels are inferred from font
size (the most common size is body text; larger sizes step up to h3/h2/h1), so
this works on an arbitrary PDF without hand-coding its sections. Images are
compressed and inlined as base64 -> a single portable .html you can double-click.
Optional overlays (both produced by the translation workflow):
--translation units.json {"p1_t0": "translated text", ...}
key = p{page}_t{Nth-text-block-on-that-page}.
A block with a translation renders its translation;
others keep the original text.
--captions caps.json {"img-p5-1.png": {"title": "...", "caption": "..."}}
attaches a heading bar + caption under that figure
(used to explain a chart whose insides stay original).
Text overlay convention (so the renderer can lay out translated prose well):
blank line = paragraph break · "- " line = list item · "## " line = sub-heading.
Original (untranslated) text has none of these, so it just flows as paragraphs.
Usage:
uv run --with Pillow python build_html.py build/structure.json --out out.html
uv run --with Pillow python build_html.py build/structure.json --out out.html \\
--translation build/units.json --captions build/caps.json --title "..." --lang zh-CN
"""
import os
import io
import re
import sys
import json
import html
import base64
import argparse
from collections import Counter
from PIL import Image
# Inline a content image up to this width. Bigger than any reading viewport, small
# enough to keep the single file manageable. Charts/line art stay PNG (crisp text);
# wide photos go JPEG (much smaller). Threshold below splits the two.
MAX_IMG_WIDTH = 1400
PHOTO_WIDTH_THRESHOLD = 1000 # rendered width above which we prefer JPEG
JPEG_QUALITY = 82
def load_json(path):
if not path:
return {}
if not os.path.isfile(path):
sys.exit(f"error: no such file: {path}")
with open(path) as f:
return json.load(f)
def data_uri(img_path):
"""Compress + base64 a content image. JPEG for wide photos, PNG otherwise."""
im = Image.open(img_path)
if im.width > MAX_IMG_WIDTH:
h = round(im.height * MAX_IMG_WIDTH / im.width)
im = im.resize((MAX_IMG_WIDTH, h), Image.LANCZOS)
buf = io.BytesIO()
if im.width >= PHOTO_WIDTH_THRESHOLD and im.mode in ("RGB", "RGBA", "P"):
im.convert("RGB").save(buf, "JPEG", quality=JPEG_QUALITY)
mime = "jpeg"
else:
im.save(buf, "PNG")
mime = "png"
return f"data:image/{mime};base64," + base64.b64encode(buf.getvalue()).decode()
def is_page_number(text):
"""A standalone short numeric block is a page number — drop it."""
return text.strip().isdigit() and len(text.strip()) <= 4
def md(text):
"""Render the lightweight text-overlay convention to HTML.
Safe on original (non-translated) text too: with no '## ', '- ' or blank
lines it simply becomes paragraphs.
"""
out = []
for blk in re.split(r"\n\s*\n", (text or "").strip()):
lines = [l for l in blk.split("\n") if l.strip()]
if not lines:
continue
if all(l.strip().startswith("- ") for l in lines):
items = "".join(f"<li>{html.escape(l.strip()[2:].strip())}</li>" for l in lines)
out.append(f"<ul>{items}</ul>")
elif lines[0].strip().startswith("## "):
out.append(f"<h3>{html.escape(lines[0].strip()[3:].strip())}</h3>")
rest = " ".join(l.strip() for l in lines[1:])
if rest:
out.append(f"<p>{html.escape(rest)}</p>")
else:
out.append(f"<p>{html.escape(' '.join(l.strip() for l in lines))}</p>")
return "\n".join(out)
def build(structure_path, out_path, translation, captions, title, lang, drop_decorative):
struct = load_json(structure_path)
tr = load_json(translation)
caps = load_json(captions)
img_dir = os.path.join(os.path.dirname(structure_path), "images")
pages = struct["pages"]
page_width = struct.get("meta", {}).get("page_width", 612)
# Body text size = the most common max-size among real text blocks.
sizes = [round(b["size"]) for pg in pages for b in pg["blocks"]
if b["type"] == "text" and not is_page_number(b["text"]) and b["size"]]
body_size = Counter(sizes).most_common(1)[0][0] if sizes else 11
# Heading levels come from the document's ACTUAL distinct sizes above body,
# not a fixed multiplier — otherwise a 44pt title and a 16pt sub-heading both
# land in h1. Largest distinct size -> h1, next -> h2, next and smaller -> h3.
big_sizes = sorted({s for s in sizes if s > body_size}, reverse=True)
tier = {s: ("h1", "h2", "h3")[min(i, 2)] for i, s in enumerate(big_sizes)}
def tag_of(size):
return tier.get(round(size), "p")
parts = []
for pg in pages:
p = pg["page"]
ti = 0
for b in pg["blocks"]:
if b["type"] == "text":
raw = b["text"]
if is_page_number(raw):
continue
key = f"p{p}_t{ti}"
ti += 1
content = tr.get(key, raw)
tag = tag_of(b["size"])
if tag == "p":
parts.append(md(content)) # paragraphs / lists / sub-heads
else:
parts.append(f"<{tag}>{html.escape(content.replace(chr(10), ' ').strip())}</{tag}>")
elif b["type"] == "image":
if drop_decorative and b.get("decorative"):
continue
src = os.path.join(img_dir, b["file"])
if not os.path.isfile(src):
continue
# Display size from how big the image is ON THE PAGE (its bbox),
# so a small inline icon doesn't blow up to full column width.
ratio = (b["bbox"][2] - b["bbox"][0]) / page_width if page_width else 1
szcls = "wide" if ratio >= 0.5 else ("mid" if ratio >= 0.25 else "small")
cap = caps.get(b["file"])
if cap:
parts.append(
f'<figure class="cap">'
f'<div class="cap-head">{html.escape(cap.get("title", ""))}</div>'
f'<img src="{data_uri(src)}" alt="{html.escape(cap.get("title",""))}">'
f'<figcaption>{html.escape(cap.get("caption", ""))}</figcaption></figure>')
else:
parts.append(f'<figure class="{szcls}"><img src="{data_uri(src)}" alt=""></figure>')
doc_title = title or struct.get("meta", {}).get("title") or "Document"
body_html = "\n".join(parts)
page_html = HTML_TEMPLATE.format(lang=html.escape(lang),
title=html.escape(doc_title),
body=body_html)
with open(out_path, "w") as f:
f.write(page_html)
kb = os.path.getsize(out_path) // 1024
print(f"wrote {out_path} ({kb} KB) — body size={body_size}pt, "
f"{len(parts)} blocks, {len(tr)} translated, {len(caps)} captioned")
print("NEXT: verify visually — render with verify_render.py and Read the segments.")
# Neutral, light, professional reading layout. Accent is a calm slate-blue, not a
# brand color, so it suits an arbitrary document. 760px column + 1.85 line-height
# reads well for both Latin and CJK; responsive break at 680px stacks figures.
HTML_TEMPLATE = """<!DOCTYPE html>
<html lang="{lang}">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>{title}</title>
<style>
:root {{ --ink:#1d1d1f; --muted:#6b6b70; --line:#e4e4e8; --soft:#f6f6f8; --accent:#3b5b8c; }}
* {{ box-sizing:border-box; }}
body {{ margin:0; background:#fff; color:var(--ink);
font-family:-apple-system,BlinkMacSystemFont,"PingFang SC","Hiragino Sans GB",
"Microsoft YaHei","Segoe UI",Roboto,sans-serif;
font-size:17px; line-height:1.85; -webkit-font-smoothing:antialiased; }}
.wrap {{ max-width:760px; margin:0 auto; padding:48px 24px 96px; }}
p {{ margin:0 0 1.1em; }}
h1 {{ font-size:34px; font-weight:700; line-height:1.3; margin:8px 0 22px; }}
h2 {{ font-size:24px; font-weight:700; line-height:1.35; margin:42px 0 16px; }}
h3 {{ font-size:19px; font-weight:700; margin:30px 0 10px; }}
ul {{ margin:0 0 1.2em; padding-left:1.3em; }}
li {{ margin:0 0 .55em; }}
figure {{ margin:30px 0; }}
figure img {{ width:100%; border:1px solid var(--line); border-radius:10px; display:block; }}
figure.small {{ text-align:center; }}
figure.small img {{ width:auto; max-width:140px; display:inline-block; }}
figure.mid img {{ max-width:62%; margin:0 auto; }}
figure.cap {{ border:1px solid var(--line); border-radius:12px; overflow:hidden; }}
figure.cap img {{ border:none; border-radius:0; padding:14px 14px 6px; }}
.cap-head {{ background:var(--accent); color:#fff; font-weight:700; font-size:15px; padding:11px 18px; }}
figcaption {{ font-size:13.5px; color:var(--muted); padding:6px 18px 16px; line-height:1.7; }}
@media (max-width:680px) {{ .wrap {{ padding:28px 18px 60px; }} h1 {{ font-size:27px; }} body {{ font-size:16px; }} }}
</style>
</head>
<body><div class="wrap">
{body}
</div></body>
</html>"""
if __name__ == "__main__":
ap = argparse.ArgumentParser(description="Build single-file HTML from structure.json.")
ap.add_argument("structure", help="path to structure.json from extract_pdf.py")
ap.add_argument("--out", required=True, help="output .html path")
ap.add_argument("--translation", default=None, help="optional units.json overlay")
ap.add_argument("--captions", default=None, help="optional figure-caption json")
ap.add_argument("--title", default=None, help="override document title")
ap.add_argument("--lang", default="en", help="html lang attribute (e.g. zh-CN)")
ap.add_argument("--keep-decorative", action="store_true",
help="keep images flagged decorative (default: drop them)")
args = ap.parse_args()
build(args.structure, args.out, args.translation, args.captions,
args.title, args.lang, drop_decorative=not args.keep_decorative)
@@ -0,0 +1,142 @@
#!/usr/bin/env python3
"""Extract a PDF's structure so it can be rebuilt as faithful, readable HTML.
The point of a separate extraction step is a *verifiable intermediate output*:
structure.json is the plan. Inspect it (and the rendered page PNGs) before
building, instead of going PDF -> HTML in one opaque jump.
Outputs (under --outdir, default "<pdf-stem>-build/"):
structure.json per-page blocks in reading order. Text blocks carry their
bbox + max font size (font size is what build_html.py uses to
infer heading levels). Image blocks carry bbox, pixel size,
byte size, and a `decorative` flag.
images/ every embedded raster at original resolution.
pages/ one rendered PNG per page — so Claude can SEE the layout and
read figures, not just the text stream. Text-correct is not
layout-correct; always look at these.
Reading order: PyMuPDF's get_text("dict") already returns blocks in reading
order, so block order is preserved as-is — this is what lets an image sit in the
right place between paragraphs.
Usage:
uv run --with pymupdf python extract_pdf.py input.pdf
uv run --with pymupdf python extract_pdf.py input.pdf --outdir build --dpi 150
"""
import sys
import os
import json
import argparse
import fitz # PyMuPDF
# A raster under this many bytes is almost always a rule / spacer / bullet glyph,
# not content worth inlining. Real figures in Office/Google-Docs exports are tens
# of KB and up; decorative separators are well under 3 KB. Marked, not deleted —
# build_html.py decides whether to drop it, and you can override per document.
DECORATIVE_MAX_BYTES = 3000
def extract(pdf_path, outdir, dpi):
if not os.path.isfile(pdf_path):
sys.exit(f"error: no such file: {pdf_path}")
try:
doc = fitz.open(pdf_path)
except Exception as e:
sys.exit(f"error: cannot open PDF ({e}). If it's a scanned image PDF, "
f"OCR it first (e.g. ocrmypdf) — this skill needs real text.")
os.makedirs(f"{outdir}/images", exist_ok=True)
os.makedirs(f"{outdir}/pages", exist_ok=True)
npages = len(doc)
bbox_counts = {} # bucketed image bbox -> how many pages it appears on
raw_pages = []
for pno in range(npages):
page = doc.load_page(pno)
# Render the page so Claude can look at the real layout. dpi 120 is a
# readable default; raise for tiny print.
page.get_pixmap(dpi=dpi).save(f"{outdir}/pages/page-{pno+1:02d}.png")
blocks = []
nimg = 0
for b in page.get_text("dict")["blocks"]:
if b["type"] == 0: # text
text, sizes = "", []
for line in b["lines"]:
for span in line["spans"]:
text += span["text"]
sizes.append(round(span["size"], 1))
text += "\n"
text = text.strip()
if text:
blocks.append({
"type": "text",
"bbox": [round(x) for x in b["bbox"]],
"text": text,
"size": max(sizes) if sizes else 0,
})
elif b["type"] == 1: # image
nimg += 1
ext = b.get("ext", "png")
fn = f"img-p{pno+1}-{nimg}.{ext}"
data = b["image"]
with open(f"{outdir}/images/{fn}", "wb") as f:
f.write(data)
# Bucket the bbox so near-identical positions across pages collapse
# to one key — that is how we detect repeating headers/footers.
key = tuple(round(x / 5) * 5 for x in b["bbox"])
bbox_counts[key] = bbox_counts.get(key, 0) + 1
blocks.append({
"type": "image",
"bbox": [round(x) for x in b["bbox"]],
"file": fn,
"w": b.get("width"),
"h": b.get("height"),
"bytes": len(data),
"_bbox_key": list(key),
})
raw_pages.append({"page": pno + 1, "blocks": blocks})
# Mark decorative images: tiny byte size, OR the same bbox repeating on more
# than half the pages (a running header/footer logo). max(2, ...) so short
# documents don't false-positive a 2-page coincidence.
repeat_threshold = max(2, npages // 2)
for pg in raw_pages:
for blk in pg["blocks"]:
if blk["type"] == "image":
repeated = bbox_counts.get(tuple(blk["_bbox_key"]), 0) > repeat_threshold
blk["decorative"] = bool(blk["bytes"] < DECORATIVE_MAX_BYTES or repeated)
del blk["_bbox_key"]
meta = {
"source": os.path.basename(pdf_path),
"pages": npages,
"page_width": round(doc[0].rect.width) if npages else 612,
"title": doc.metadata.get("title") or "",
"render_dpi": dpi,
}
out = {"meta": meta, "pages": raw_pages}
with open(f"{outdir}/structure.json", "w") as f:
json.dump(out, f, ensure_ascii=False, indent=1)
# Console summary so a successful run is self-evident (and easy to sanity-check).
print(f"source: {meta['source']} pages: {npages} title: {meta['title'] or '(none)'}")
for pg in raw_pages:
ntext = sum(1 for b in pg["blocks"] if b["type"] == "text")
imgs = [b for b in pg["blocks"] if b["type"] == "image"]
content_imgs = sum(1 for b in imgs if not b["decorative"])
print(f" page {pg['page']:>2}: {ntext} text blocks, "
f"{content_imgs} content image(s), {len(imgs)-content_imgs} decorative")
print(f"\nwrote {outdir}/structure.json + images/ + pages/")
print("NEXT: Read the pages/*.png to see the real layout before building.")
if __name__ == "__main__":
ap = argparse.ArgumentParser(description="Extract PDF structure for HTML rebuild.")
ap.add_argument("pdf")
ap.add_argument("--outdir", default=None, help="default: <pdf-stem>-build/")
ap.add_argument("--dpi", type=int, default=120, help="page render DPI (default 120)")
args = ap.parse_args()
outdir = args.outdir or os.path.splitext(os.path.basename(args.pdf))[0] + "-build"
extract(args.pdf, outdir, args.dpi)
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""Render the built HTML with headless Chrome and slice it into readable segments.
Why this exists: text-correct is not render-correct. Fonts can fall back, tables
can overflow, a translated heading can wrap badly none of which show up unless
you LOOK. After running this, Read each seg-*.png and check the layout.
Two real gotchas this script handles for you:
1. Chrome's headless screenshot caps height around 16384 physical px. A 2x shot
of a long page silently truncates. So we first probe the real content height
at 1x, then pick the largest device-scale-factor that keeps the full page
under the cap (crisp when it fits, still complete when it doesn't).
2. A full-page shot is one tall image; thumbnailed, the text is unreadable. So
we slice into ~2600px-tall segments each one is legible when Read.
Usage:
uv run --with Pillow --with numpy python verify_render.py out.html
uv run --with Pillow --with numpy python verify_render.py out.html --outdir shots --width 840 --scale 2
"""
import os
import sys
import shutil
import argparse
import subprocess
from PIL import Image
import numpy as np
# Stay comfortably under Chrome's ~16384px headless screenshot ceiling.
MAX_PHYSICAL = 15000
SEGMENT_PHYSICAL = 2600 # tall enough to be efficient, short enough to read
def find_chrome():
candidates = [
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
"/Applications/Chromium.app/Contents/MacOS/Chromium",
]
for c in candidates:
if os.path.isfile(c):
return c
for name in ("google-chrome", "chromium", "chromium-browser", "chrome"):
p = shutil.which(name)
if p:
return p
sys.exit("error: Chrome/Chromium not found — it's required for visual "
"verification. Install Google Chrome, or pass a different verifier.")
def shoot(chrome, html_path, out_png, width, height, scale):
subprocess.run([
chrome, "--headless", "--disable-gpu", "--no-sandbox", "--hide-scrollbars",
"--no-proxy-server", # local file:// must not route via a proxy
f"--force-device-scale-factor={scale}",
"--virtual-time-budget=10000", # let base64 images + fonts settle
f"--window-size={width},{height}",
f"--screenshot={out_png}", f"file://{html_path}",
], check=False, capture_output=True)
if not os.path.isfile(out_png):
sys.exit(f"error: Chrome produced no screenshot ({out_png}). "
f"Check the HTML path and that Chrome runs headless on this machine.")
def content_height(png):
"""Bottom of the actual content (trim the blank tail below the page)."""
a = np.array(Image.open(png).convert("L"))
nonwhite = np.where((a < 250).any(axis=1))[0]
return int(nonwhite.max()) + 1 if len(nonwhite) else a.shape[0]
def verify(html_path, outdir, width, desired_scale):
if not os.path.isfile(html_path):
sys.exit(f"error: no such file: {html_path}")
chrome = find_chrome()
os.makedirs(outdir, exist_ok=True)
# 1) Probe true content height at 1x (1 CSS px == 1 device px here).
probe = os.path.join(outdir, "_probe.png")
shoot(chrome, html_path, probe, width, 16000, 1)
css_height = content_height(probe)
# 2) Largest scale that keeps the whole page under the physical cap.
# Largest scale that keeps the whole page under the cap. Don't round up —
# that can nudge scale*height back over the cap and force an unwanted 1x.
scale = max(1.0, min(desired_scale, MAX_PHYSICAL / max(css_height, 1)))
if scale * css_height <= MAX_PHYSICAL:
final = os.path.join(outdir, "_full.png")
shoot(chrome, html_path, final, width, css_height + 40, scale)
note = f"scale {scale}x"
else:
# Page taller than the cap even at 1x — keep the complete 1x probe, trimmed.
final = probe
scale = 1.0
note = "scale 1x (page exceeds cap; rendered complete but not magnified)"
# 3) Slice into readable segments.
im = Image.open(final)
full = im.crop((0, 0, im.width, min(im.height, round((css_height + 40) * scale))))
n = (full.height + SEGMENT_PHYSICAL - 1) // SEGMENT_PHYSICAL
paths = []
for i in range(n):
top, bot = i * SEGMENT_PHYSICAL, min(full.height, (i + 1) * SEGMENT_PHYSICAL)
seg = os.path.join(outdir, f"seg-{i+1:02d}.png")
full.crop((0, top, full.width, bot)).save(seg)
paths.append(seg)
if os.path.exists(probe) and final != probe:
os.remove(probe)
print(f"rendered {html_path} at {note} -> {n} segment(s) in {outdir}/")
for p in paths:
print(f" {p}")
print("\nNEXT: Read every seg-*.png and check: fonts render (no tofu boxes), "
"tables/figures aren't clipped, headings/lists look right, images present.")
if __name__ == "__main__":
ap = argparse.ArgumentParser(description="Headless-render HTML and slice into readable PNGs.")
ap.add_argument("html")
ap.add_argument("--outdir", default="render-check", help="where to write segments")
ap.add_argument("--width", type=int, default=840, help="viewport CSS width (default 840)")
ap.add_argument("--scale", type=float, default=2.0, help="desired device scale (default 2)")
args = ap.parse_args()
verify(args.html, args.outdir, args.width, args.scale)