mirror of
https://github.com/Astro-Han/karpathy-llm-wiki.git
synced 2026-09-14 19:09:17 +08:00
feat: add grounding invariant with mechanical evidence checker
Every load-bearing fact (number, date, direct quote) must exist verbatim in the raw files an article links. Compile establishes the invariant (locate before write, exact literals, derived values show components); lint verifies it with scripts/check_evidence.py, which greps candidate literals against linked raws and reports misses plus unreferenced raw files. Lint is now three tiers: safe auto-fixes, mechanical reports (the script), judgment reports. Also moves adding See Also cross-references out of auto-fix, where judgment never belonged.
This commit is contained in:
Yuhan Lei
@@ -2,3 +2,4 @@
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
__pycache__/
|
||||
|
||||
@@ -36,6 +36,10 @@ Triggers only on the first Ingest. Check whether `raw/` and `wiki/` exist. Creat
|
||||
|
||||
If Query or Lint cannot find the wiki structure, tell the user: "Run an ingest first to initialize the wiki." Do not auto-create.
|
||||
|
||||
## The Grounding Invariant
|
||||
|
||||
Every load-bearing fact in wiki/ — numbers, dates, direct quotes — exists verbatim in the raw/ files linked by that article's Raw field. Compile *establishes* this invariant (locate before you write); lint *verifies* it (`scripts/check_evidence.py` greps each fact in the linked raws). Because raw/ is immutable and log.md is append-only, the invariant holds permanently once verified: lint only needs to re-check articles touched since the last lint entry.
|
||||
|
||||
---
|
||||
|
||||
## Ingest
|
||||
@@ -78,6 +82,8 @@ Determine where the new content belongs:
|
||||
|
||||
These are not mutually exclusive. A single source may warrant merging into one article while also creating a separate article for a distinct concept it introduces. In all cases, check for factual conflicts: if the new source contradicts existing content, annotate the disagreement with source attribution. When merging, note the conflict within the merged article. When the conflicting content lives in separate articles, note it in both and cross-link them.
|
||||
|
||||
**Source fidelity.** Every number, date, and direct quote must be located in the raw file (grep or read) *before* it is written; write the value exactly as found — if the source says 42K, write 42K, not 42,000. Derived values (sums, deltas, counts you computed) must show their components so each component is findable in raw. If you cannot locate a value, do not write its exact form; drop it or state it without precision.
|
||||
|
||||
See `references/article-template.md` for article format. Key points:
|
||||
- Sources field: author, organization, or publication name + date, semicolon-separated.
|
||||
- Raw field: markdown links to raw/ files, semicolon-separated.
|
||||
@@ -149,9 +155,9 @@ When the user explicitly asks to archive or save the answer to the wiki:
|
||||
|
||||
## Lint
|
||||
|
||||
Quality checks on the wiki. Two categories with different authority levels.
|
||||
Quality checks on the wiki. Three categories with different authority levels.
|
||||
|
||||
### Deterministic Checks (auto-fix)
|
||||
### Safe Fixes (auto-fix)
|
||||
|
||||
Fix these automatically:
|
||||
|
||||
@@ -170,16 +176,24 @@ Fix these automatically:
|
||||
- Zero or multiple matches → report to the user.
|
||||
|
||||
**See Also** — within each topic directory:
|
||||
- Add obviously missing cross-references between related articles.
|
||||
- Remove links to deleted files.
|
||||
|
||||
### Heuristic Checks (report only)
|
||||
### Mechanical Reports (no fixes)
|
||||
|
||||
Run these mechanically with `scripts/check_evidence.py <wiki-root>` (optionally followed by article paths to limit scope). Report findings; never auto-fix facts.
|
||||
|
||||
**Source fidelity** — default scope: articles touched since the last lint entry in log.md; all articles when the user asks for a full sweep. Reported suspects are candidates, not verdicts: derived values and product names may appear. Judge each against the raw context and report only real mismatches.
|
||||
|
||||
**Unreferenced raw files** — files logged with a No material disposition are excluded; everything else is a genuine backlog reminder.
|
||||
|
||||
### Judgment Reports (no fixes)
|
||||
|
||||
These rely on your judgment. Report findings without auto-fixing:
|
||||
|
||||
- Factual contradictions across articles
|
||||
- Outdated claims superseded by newer sources
|
||||
- Missing conflict annotations where sources disagree
|
||||
- Obviously missing cross-references between related articles (suggest them; do not add silently)
|
||||
- Orphan pages with no inbound links from other wiki articles
|
||||
- Missing cross-topic references
|
||||
- Concepts frequently mentioned but lacking a dedicated page
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Mechanical evidence check for a Karpathy-style LLM wiki.
|
||||
|
||||
Report-only; never modifies files. Two sweeps:
|
||||
|
||||
1. Fidelity — extract candidate literals (numbers, ISO dates, direct
|
||||
quotes) from each wiki article and verify that each candidate appears
|
||||
verbatim in the raw files linked by that article's Raw field. Misses
|
||||
are listed as suspects. Derived values, product names, and deliberate
|
||||
paraphrases will show up as suspects; judging them is the reader's
|
||||
job, not this script's.
|
||||
2. Inventory — list raw files that no article's Raw field references,
|
||||
excluding files whose ingest was logged as "no material".
|
||||
|
||||
Usage: check_evidence.py [wiki-root] [article.md ...]
|
||||
Defaults: wiki-root is the current directory; all wiki/**/*.md articles
|
||||
except index.md and log.md are checked.
|
||||
"""
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
NUMBER_RE = re.compile(r"\d[\d,]*(?:\.\d+)?\s?[KMB%]")
|
||||
PLAIN_NUMBER_RE = re.compile(r"\d[\d,]{1,}")
|
||||
DATE_RE = re.compile(r"\d{4}-\d{2}(?:-\d{2})?")
|
||||
QUOTE_RES = [re.compile(r'"([^"\n]{15,})"'), re.compile(r"“([^”\n]{15,})”")]
|
||||
METADATA_RE = re.compile(r"^>\s*(Sources?|Raw|Collected|Published|Updated):")
|
||||
LINK_RE = re.compile(r"\[([^\]]*)\]\(([^)]*)\)")
|
||||
INLINE_CODE_RE = re.compile(r"`[^`\n]*`")
|
||||
FENCED_CODE_RE = re.compile(r"```.*?```", re.DOTALL)
|
||||
RAW_LINK_RE = re.compile(r"\(([^)]+\.md)[^)]*\)")
|
||||
NO_MATERIAL_RE = re.compile(r"no material:\s*(\S+)")
|
||||
WS_RE = re.compile(r"\s+")
|
||||
|
||||
SKIP_FILES = {"index.md", "log.md"}
|
||||
|
||||
|
||||
def normalize(text: str) -> str:
|
||||
return WS_RE.sub(" ", text).strip()
|
||||
|
||||
|
||||
def strip_noise(text: str) -> str:
|
||||
text = FENCED_CODE_RE.sub(" ", text)
|
||||
text = INLINE_CODE_RE.sub(" ", text)
|
||||
text = LINK_RE.sub(r"\1", text)
|
||||
return text
|
||||
|
||||
|
||||
def extract_candidates(text: str) -> list[str]:
|
||||
candidates = []
|
||||
for line in text.splitlines():
|
||||
if METADATA_RE.match(line.strip()):
|
||||
continue
|
||||
line = strip_noise(line)
|
||||
candidates.extend(m.group(0) for m in DATE_RE.finditer(line))
|
||||
candidates.extend(m.group(0) for m in NUMBER_RE.finditer(line))
|
||||
candidates.extend(
|
||||
m.group(0) for m in PLAIN_NUMBER_RE.finditer(line) if "," in m.group(0)
|
||||
)
|
||||
for quote_re in QUOTE_RES:
|
||||
candidates.extend(m.group(1) for m in quote_re.finditer(line))
|
||||
seen = set()
|
||||
unique = []
|
||||
for cand in candidates:
|
||||
cand = cand.strip(".,;:()[]")
|
||||
if cand and cand not in seen:
|
||||
seen.add(cand)
|
||||
unique.append(cand)
|
||||
return unique
|
||||
|
||||
|
||||
def raw_links_of(article_text: str) -> list[str]:
|
||||
links = []
|
||||
for line in article_text.splitlines():
|
||||
if re.match(r"^>\s*Raw:", line.strip()):
|
||||
links.extend(RAW_LINK_RE.findall(line))
|
||||
return links
|
||||
|
||||
|
||||
def iter_articles(wiki_dir: Path):
|
||||
for path in sorted(wiki_dir.rglob("*.md")):
|
||||
if path.name not in SKIP_FILES:
|
||||
yield path
|
||||
|
||||
|
||||
def check_article(article: Path) -> list[str]:
|
||||
text = article.read_text(encoding="utf-8")
|
||||
raws = []
|
||||
for link in raw_links_of(text):
|
||||
target = (article.parent / link).resolve()
|
||||
if target.is_file():
|
||||
raws.append(normalize(target.read_text(encoding="utf-8")))
|
||||
if not raws:
|
||||
return []
|
||||
misses = []
|
||||
for cand in extract_candidates(text):
|
||||
needle = normalize(cand)
|
||||
if not any(needle in raw for raw in raws):
|
||||
misses.append(cand)
|
||||
return misses
|
||||
|
||||
|
||||
def no_material_names(log_file: Path) -> set[str]:
|
||||
if not log_file.is_file():
|
||||
return set()
|
||||
names = set()
|
||||
for line in log_file.read_text(encoding="utf-8").splitlines():
|
||||
m = NO_MATERIAL_RE.search(line)
|
||||
if m:
|
||||
names.add(Path(m.group(1).rstrip("`.,;")).name)
|
||||
return names
|
||||
|
||||
|
||||
def referenced_raws(root: Path) -> set[Path]:
|
||||
referenced = set()
|
||||
for article in iter_articles(root / "wiki"):
|
||||
for link in raw_links_of(article.read_text(encoding="utf-8")):
|
||||
target = (article.parent / link).resolve()
|
||||
referenced.add(target)
|
||||
return referenced
|
||||
|
||||
|
||||
def unreferenced_raws(root: Path) -> list[str]:
|
||||
raw_dir = root / "raw"
|
||||
if not raw_dir.is_dir():
|
||||
return []
|
||||
referenced = referenced_raws(root)
|
||||
disposed = no_material_names(root / "wiki" / "log.md")
|
||||
missing = []
|
||||
for path in sorted(raw_dir.rglob("*.md")):
|
||||
if path.resolve() not in referenced and path.name not in disposed:
|
||||
missing.append(path.relative_to(root).as_posix())
|
||||
return missing
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
root = Path(argv[1]).resolve() if len(argv) > 1 else Path.cwd()
|
||||
wiki_dir = root / "wiki"
|
||||
if not wiki_dir.is_dir():
|
||||
print(f"no wiki/ directory under {root}")
|
||||
return 1
|
||||
|
||||
if len(argv) > 2:
|
||||
articles = [Path(a) for a in argv[2:]]
|
||||
else:
|
||||
articles = list(iter_articles(wiki_dir))
|
||||
|
||||
print("# Evidence check\n")
|
||||
print("## Fidelity suspects")
|
||||
suspect_count = 0
|
||||
for article in articles:
|
||||
misses = check_article(article)
|
||||
if misses:
|
||||
label = article
|
||||
try:
|
||||
label = article.resolve().relative_to(root)
|
||||
except ValueError:
|
||||
pass
|
||||
print(f"\n{label}")
|
||||
for miss in misses:
|
||||
print(f"- {miss}")
|
||||
suspect_count += 1
|
||||
if suspect_count == 0:
|
||||
print("\n(none)")
|
||||
|
||||
print("\n## Unreferenced raw files")
|
||||
orphans = unreferenced_raws(root)
|
||||
for path in orphans:
|
||||
print(f"- {path}")
|
||||
if not orphans:
|
||||
print("(none)")
|
||||
|
||||
print(f"\n## Summary\n{suspect_count} fidelity suspect(s), {len(orphans)} unreferenced raw file(s)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv))
|
||||
@@ -0,0 +1,119 @@
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT = Path(__file__).resolve().parent.parent / "scripts" / "check_evidence.py"
|
||||
|
||||
RAW_CONTENT = """# Ghostty Update
|
||||
|
||||
> Source: https://example.com/ghostty
|
||||
> Collected: 2026-04-17
|
||||
> Published: 2026-04-16
|
||||
|
||||
Ghostty reached 42K stars on GitHub in April 2026.
|
||||
The maintainer said "the terminal should feel invisible to users" in the interview.
|
||||
Daily active users reached 10,000 by March.
|
||||
"""
|
||||
|
||||
ARTICLE_CONTENT = """# Ghostty
|
||||
|
||||
> Sources: Example, 2026-04-16
|
||||
> Raw: [ghostty](../../raw/ai-research/2026-04-17-ghostty.md)
|
||||
|
||||
## Overview
|
||||
|
||||
Ghostty has 42K stars and reached 10,000 daily active users.
|
||||
The maintainer said "the terminal should feel invisible to users".
|
||||
|
||||
## Growth
|
||||
|
||||
Forks grew to 3,020 last week.
|
||||
Install with `--limit 500` after downloading.
|
||||
|
||||
```
|
||||
ignore this 8888 number
|
||||
```
|
||||
"""
|
||||
|
||||
SECOND_RAW = """# Unrelated Notes
|
||||
|
||||
> Source: https://example.com/notes
|
||||
> Collected: 2026-05-01
|
||||
> Published: Unknown
|
||||
|
||||
Nothing here is compiled anywhere.
|
||||
"""
|
||||
|
||||
|
||||
def make_wiki(root: Path, log: str = ""):
|
||||
(root / "raw" / "ai-research").mkdir(parents=True)
|
||||
(root / "raw" / "ai-research" / "2026-04-17-ghostty.md").write_text(RAW_CONTENT)
|
||||
(root / "raw" / "misc").mkdir(parents=True)
|
||||
(root / "raw" / "misc" / "notes.md").write_text(SECOND_RAW)
|
||||
(root / "wiki" / "ai-research").mkdir(parents=True)
|
||||
(root / "wiki" / "ai-research" / "ghostty.md").write_text(ARTICLE_CONTENT)
|
||||
(root / "wiki" / "index.md").write_text("# Knowledge Base Index\n")
|
||||
(root / "wiki" / "log.md").write_text(log or "# Wiki Log\n")
|
||||
|
||||
|
||||
def run_checker(root: Path, *args: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(SCRIPT), str(root), *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
|
||||
class FidelityCheckTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.tmp.name)
|
||||
make_wiki(self.root)
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_flags_value_absent_from_raw(self):
|
||||
result = run_checker(self.root)
|
||||
self.assertIn("3,020", result.stdout)
|
||||
|
||||
def test_passes_values_and_quotes_present_in_raw(self):
|
||||
result = run_checker(self.root)
|
||||
self.assertNotIn("42K", result.stdout.replace("3,020", ""))
|
||||
self.assertNotIn("10,000", result.stdout)
|
||||
self.assertNotIn("invisible to users", result.stdout)
|
||||
|
||||
def test_ignores_numbers_in_code(self):
|
||||
result = run_checker(self.root)
|
||||
self.assertNotIn("500", result.stdout)
|
||||
self.assertNotIn("8888", result.stdout)
|
||||
|
||||
|
||||
class RawInventoryTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.tmp.name)
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_reports_raw_file_never_compiled(self):
|
||||
make_wiki(self.root)
|
||||
result = run_checker(self.root)
|
||||
self.assertIn("raw/misc/notes.md", result.stdout)
|
||||
|
||||
def test_no_material_disposition_suppresses_report(self):
|
||||
log = (
|
||||
"# Wiki Log\n\n"
|
||||
"## [2026-05-01] ingest | no material: notes.md\n"
|
||||
"- Disposition: No material\n"
|
||||
)
|
||||
make_wiki(self.root, log=log)
|
||||
result = run_checker(self.root)
|
||||
self.assertNotIn("notes.md", result.stdout)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user