feat(server): run a vault as a Dockerized HTTP + MCP memory service (#185)

Everything else here is local: skills on your machine, a vault on your disk.
This puts a vault behind a URL so an agent elsewhere can search and write it.

One container, one vault, one API key. The vault stays plain markdown on a
mounted volume, so Obsidian still opens it. The container does no LLM work --
search and packing delegate to the existing graphrag/context_pack modules, and
the caller's agent does the thinking.

- obsidian_wiki/server.py: four vault functions, five REST routes and four MCP
  tools as transport over them. Auth is one bearer key; the process refuses to
  start without WIKI_API_KEY unless WIKI_ALLOW_ANONYMOUS=1. Every path is
  resolved inside the vault or refused.
- Dockerfile, docker-compose.yml, .dockerignore, and a ghcr push on v* tags.
- New [server] extra; the core wheel stays dependency-free.
- docs/deployment.md, plus the WIKI_* vars in configuration.md and .env.example.

Requires mcp>=2.0: 2.0 renamed FastMCP to MCPServer, and session_manager only
exists once streamable_http_app() has been called -- a mounted sub-app's own
lifespan is ignored, so it is wired into the FastAPI lifespan by hand.
This commit is contained in:
Arnav
2026-08-21 14:37:37 -07:00
committed by GitHub
parent 9ae541c849
commit 52c9f2bae2
13 changed files with 506 additions and 0 deletions
+6
View File
@@ -0,0 +1,6 @@
.git
.venv
vault/
assets/
.pytest_cache
**/__pycache__
+9
View File
@@ -207,3 +207,12 @@ CODE_UNDERSTANDING_CODEGRAPH_BIN=
# wiki-ingest will promote them to proper wiki pages when you run it.
# Promoted files are moved into _raw/_archived/ after processing, not deleted.
OBSIDIAN_RAW_DIR=_raw
# --- Memory server (optional) ---
#
# Only used by the Dockerized HTTP + MCP server (`python -m obsidian_wiki.server`),
# which puts this vault behind a URL so remote agents can use it as memory.
# Ignored entirely when you use the skills locally. See docs/deployment.md.
# The server refuses to start without a key unless WIKI_ALLOW_ANONYMOUS=1.
WIKI_API_KEY=
WIKI_PORT=8080
+33
View File
@@ -0,0 +1,33 @@
name: docker
# Build and push the memory-server image alongside the PyPI release.
on:
push:
tags: ["v*"]
workflow_dispatch:
jobs:
publish:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/metadata-action@v5
id: meta
with:
images: ghcr.io/${{ github.repository }}
- uses: docker/build-push-action@v6
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# .git is not in the build context, so hatch-vcs needs the tag passed in.
build-args: VERSION=${{ steps.meta.outputs.version }}
+27
View File
@@ -0,0 +1,27 @@
# Memory service: one container, one vault on a mounted volume.
# Build: docker build -t obsidian-wiki .
# Run: docker run -p 8080:8080 -e WIKI_API_KEY=... -v wiki-data:/vault obsidian-wiki
FROM python:3.12-slim
# git: obsidian_wiki/sync.py shells out to it for vault backup.
RUN apt-get update && apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY . .
# .git is excluded from the build context, so hatch-vcs cannot read the tag.
# CI passes the real one: docker build --build-arg VERSION=2026.8.1
ARG VERSION=0.0.0
ENV HATCH_VCS_PRETEND_VERSION=$VERSION SETUPTOOLS_SCM_PRETEND_VERSION=$VERSION
RUN pip install --no-cache-dir '.[server]'
RUN useradd --create-home wiki && mkdir -p /vault && chown wiki:wiki /vault
USER wiki
ENV OBSIDIAN_VAULT_PATH=/vault WIKI_PORT=8080
VOLUME /vault
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \
CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health').status==200 else 1)"
CMD ["python", "-m", "obsidian_wiki.server"]
+1
View File
@@ -170,6 +170,7 @@ Full data, per-run logs and the scaling measurements are in
| **[Architecture](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/architecture.md)** | The four ingest stages, vault structure, what we added to Karpathy's pattern |
| **[Session Brain](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/session-brain.md)** | Topic graph over your agent session history |
| **[Browser Extension](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/browser-extension.md)** | Capture pages into the vault, and fill web forms from it |
| **[Deployment](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/deployment.md)** | Run a vault as a Dockerized memory service agents reach over HTTP/MCP |
| **[Contributing](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/contributing.md)** | Adding skills, keeping the READMEs in sync |
## Contributing
+1
View File
@@ -156,6 +156,7 @@ obsidian-wiki sessions-query "the auth bug with the weird retry loop"
| **[Architecture](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/architecture.md)** | 四個匯入階段、vault 結構、我們在 Karpathy 模式上加了什麼 |
| **[Session Brain](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/session-brain.md)** | 建立在 agent session 歷史之上的主題圖譜 |
| **[Browser Extension](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/browser-extension.md)** | 將網頁擷取進 vault並用 vault 內容填寫網頁表單 |
| **[Deployment](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/deployment.md)** | 以 Docker 將 vault 部署成記憶服務,讓 agent 透過 HTTP/MCP 存取 |
| **[Contributing](https://github.com/Ar9av/obsidian-wiki/blob/main/docs/contributing.md)** | 新增 skill、維持兩份 README 同步 |
## 參與貢獻
+12
View File
@@ -0,0 +1,12 @@
services:
wiki:
build: .
ports: ["8080:8080"]
environment:
WIKI_API_KEY: ${WIKI_API_KEY:?set WIKI_API_KEY in .env}
volumes:
- wiki-data:/vault
restart: unless-stopped
volumes:
wiki-data:
+1
View File
@@ -12,6 +12,7 @@ Everything beyond the [README](../README.md) landing page.
| [Architecture](architecture.md) | The four ingest stages, vault structure, what we added to Karpathy's pattern |
| [Session Brain](session-brain.md) | Topic graph over your agent session history |
| [Browser Extension](browser-extension.md) | Capture pages into the vault, and fill web forms from it |
| [Deployment](deployment.md) | Run a vault as a Dockerized memory service agents reach over HTTP/MCP |
| [Contributing](contributing.md) | Adding skills, keeping the two READMEs in sync |
New here? Read [Installation](installation.md), then [Skills Reference](skills.md).
+11
View File
@@ -251,3 +251,14 @@ Pages can carry a `visibility/` tag marking their intended reach. This is **enti
**Filtered mode** is opt-in, triggered by phrases like "public only", "user-facing answer", "no internal content", or "as a user would see it" in a query. Default mode shows everything.
`visibility/` tags are **system tags** — they don't count toward the 5-tag limit and are listed separately from domain/type tags in the taxonomy.
## Memory server (optional)
Only read by `obsidian_wiki/server.py`, the Dockerized HTTP + MCP front end. Irrelevant to local
skill use. Full guide: [Deployment](deployment.md).
| Variable | What it does | Default |
|---|---|---|
| `WIKI_API_KEY` | Bearer token required on every `/v1/*` and `/mcp` request | *(none — the server refuses to start without it)* |
| `WIKI_ALLOW_ANONYMOUS` | `1` disables auth entirely. Local development only | *(unset)* |
| `WIKI_PORT` | Port the server listens on | `8080` |
+98
View File
@@ -0,0 +1,98 @@
# Deployment — run your vault as a memory service
Everything else in this project is local: skills on your machine, a vault on your disk. This is the
one piece that puts a vault behind a URL, so an agent somewhere else — another machine, CI, a hosted
product — can search it and write to it.
**One container, one vault, one API key.** The vault stays plain markdown on a mounted volume, so
you can still open it in Obsidian. The container does no LLM work: it searches, reads, writes, and
packs. The calling agent does the thinking.
## Run it
```bash
echo "WIKI_API_KEY=$(openssl rand -hex 24)" > .env
docker compose up --build
```
The vault lives in the named volume `wiki-data`. To use a vault you already have, swap the volume
for a bind mount:
```yaml
volumes:
- /path/to/your/vault:/vault
```
Locally, without Docker:
```bash
pip install 'obsidian-wiki[server]'
WIKI_API_KEY=dev OBSIDIAN_VAULT_PATH=~/vault python -m obsidian_wiki.server
```
## Configuration
| Variable | What it does | Default |
|---|---|---|
| `OBSIDIAN_VAULT_PATH` | Vault the service serves | `/vault` |
| `WIKI_API_KEY` | Bearer token for every `/v1/*` and `/mcp` request | *(none — required)* |
| `WIKI_ALLOW_ANONYMOUS` | `1` disables auth entirely. Local development only | *(unset)* |
| `WIKI_PORT` | Port to listen on | `8080` |
The process **refuses to start** without `WIKI_API_KEY` unless `WIKI_ALLOW_ANONYMOUS=1`. There is no
default key.
## Connect an agent (MCP)
```bash
claude mcp add --transport http wiki-memory http://localhost:8080/mcp/ \
--header "Authorization: Bearer $WIKI_API_KEY"
```
Four tools: `memory_search`, `memory_read`, `memory_write`, `memory_context_pack`.
## REST
Every route below `/v1` needs `Authorization: Bearer <key>`. `/health` does not.
| Method | Path | Notes |
|---|---|---|
| `GET` | `/health` | Liveness; also reports whether the vault directory exists |
| `GET` | `/v1/search?q=&limit=` | Ranked pages with summaries, from the same GraphRAG index the `wiki-query` skill uses |
| `GET` | `/v1/pages/{path}` | One page as markdown, by vault-relative path |
| `POST` | `/v1/pages` | `{title, category, content, tags, sources, summary, upsert}` |
| `POST` | `/v1/context-pack` | `{topic, budget, recent, public_only, metadata_only}` |
```bash
curl -X POST localhost:8080/v1/pages -H "Authorization: Bearer $WIKI_API_KEY" \
-H 'content-type: application/json' \
-d '{"title":"Merkle Tree","category":"concepts","summary":"Hash tree for cheap diffing.","content":"Each node hashes its children."}'
# -> {"path":"concepts/merkle-tree.md", ...}
```
Writes land at `<category>/<slug-of-title>.md` with the six required frontmatter keys plus `summary`,
and append a line to `log.md`. `created:` is preserved across updates. Use `category: "_raw"` for a
rough capture you intend to promote with `wiki-ingest` later — the same role `_raw/` plays for the
`wiki-capture` skill.
`POST /v1/pages` writes exactly what you send. It does not distil, dedupe, or cross-link — those are
the agent's job, via the `wiki-capture` and `cross-linker` skills.
## Backups
The vault is a directory, so back it up like one:
```bash
docker run --rm -v wiki-data:/vault -v ~/.aws:/root/.aws:ro amazon/aws-cli \
s3 sync /vault s3://your-bucket/vault
```
Or point the vault at a git remote and use `obsidian-wiki sync` — see
[Configuration](configuration.md).
## What this is not
Single-tenant: one container serves one vault behind one key. No per-user isolation, no quotas, no
rate limiting, no billing. To serve several people, run a container per vault and put a reverse proxy
in front. Terminate TLS at that proxy — the container speaks plain HTTP, and the API key is only as
private as the connection carrying it.
+219
View File
@@ -0,0 +1,219 @@
"""HTTP + MCP front end for a vault, so remote agents can use it as memory.
Single tenant: one process, one vault, one API key. The container does no LLM
work — search, packing and frontmatter parsing all delegate to the existing
`graphrag` / `context_pack` modules, and the caller's agent does the thinking.
Needs the optional extra: ``pip install 'obsidian-wiki[server]'``.
Run it with ``python -m obsidian_wiki.server``.
"""
from __future__ import annotations
import hmac
import os
import re
from contextlib import asynccontextmanager
from datetime import date
from pathlib import Path
from typing import Any
from fastapi import Depends, FastAPI, HTTPException, Request
from mcp.server.mcpserver import MCPServer
from pydantic import BaseModel, Field
from obsidian_wiki.context_pack import ContextError, build_context_pack
from obsidian_wiki.graphrag import query as graph_query
VAULT = Path(os.environ.get("OBSIDIAN_VAULT_PATH", "/vault")).expanduser()
API_KEY = os.environ.get("WIKI_API_KEY", "")
ANONYMOUS = os.environ.get("WIKI_ALLOW_ANONYMOUS") == "1"
if not API_KEY and not ANONYMOUS:
raise RuntimeError(
"refusing to start without WIKI_API_KEY. "
"Set it, or set WIKI_ALLOW_ANONYMOUS=1 for local development."
)
# --- vault operations -------------------------------------------------------
# Every route and every MCP tool goes through these four functions, so the
# path check below is the single trust boundary for the whole service.
def _resolve(rel: str) -> Path:
"""Resolve a caller-supplied path inside the vault, or refuse."""
root = VAULT.resolve()
target = (root / rel).resolve()
if target != root and root not in target.parents:
raise HTTPException(400, f"path escapes the vault: {rel}")
return target
def _slug(title: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-") or "untitled"
def search(q: str, limit: int = 8) -> dict[str, Any]:
return graph_query(VAULT, q, top_n=limit)
def read_page(path: str) -> dict[str, Any]:
target = _resolve(path)
if not target.is_file():
raise HTTPException(404, f"no such page: {path}")
return {"path": path, "markdown": target.read_text(encoding="utf-8")}
def write_page(
title: str,
category: str,
content: str,
*,
tags: list[str] | None = None,
sources: list[str] | None = None,
summary: str = "",
upsert: bool = True,
) -> dict[str, Any]:
rel = f"{_slug(category)}/{_slug(title)}.md"
target = _resolve(rel)
if target.exists() and not upsert:
raise HTTPException(409, f"page already exists: {rel}")
today = date.today().isoformat()
created = today
if target.exists():
# Preserve the original created: date across updates.
match = re.search(r"^created:\s*(\S+)", target.read_text(encoding="utf-8"), re.MULTILINE)
created = match.group(1) if match else today
front = "\n".join(
[
"---",
f"title: {title}",
f"category: {_slug(category)}",
"tags: [" + ", ".join(tags or []) + "]",
"sources: [" + ", ".join(sources or []) + "]",
f"summary: {summary}" if summary else "summary:",
f"created: {created}",
f"updated: {today}",
"---",
"",
"",
]
)
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(front + content.rstrip() + "\n", encoding="utf-8")
log = VAULT / "log.md"
log.parent.mkdir(parents=True, exist_ok=True)
with log.open("a", encoding="utf-8") as handle:
handle.write(f"- {today} — wrote [[{title}]] via API ({rel})\n")
return {"path": rel, "created": created, "updated": today}
def context_pack(
topic: str,
*,
budget: int = 8000,
recent: bool = False,
public_only: bool = False,
metadata_only: bool = False,
) -> dict[str, Any]:
try:
return build_context_pack(
VAULT, topic, budget=budget, recent=recent,
public_only=public_only, metadata_only=metadata_only,
)
except ContextError as exc:
raise HTTPException(400, str(exc)) from exc
# --- HTTP -------------------------------------------------------------------
def require_key(request: Request) -> None:
if ANONYMOUS:
return
header = request.headers.get("authorization", "")
token = header[7:] if header.lower().startswith("bearer ") else ""
if not hmac.compare_digest(token, API_KEY):
raise HTTPException(401, "missing or invalid API key")
class PageWrite(BaseModel):
title: str
category: str = "concepts"
content: str
tags: list[str] = Field(default_factory=list)
sources: list[str] = Field(default_factory=list)
summary: str = ""
upsert: bool = True
class PackRequest(BaseModel):
topic: str = ""
budget: int = 8000
recent: bool = False
public_only: bool = False
metadata_only: bool = False
mcp = MCPServer("obsidian-wiki")
mcp.tool(name="memory_search", description="Search the wiki. Returns ranked pages with summaries.")(search)
mcp.tool(name="memory_read", description="Read one wiki page as markdown, by vault-relative path.")(read_page)
mcp.tool(name="memory_write", description="Write a wiki page. Use category '_raw' for a rough capture.")(write_page)
mcp.tool(name="memory_context_pack", description="Compile a token-bounded context pack on a topic.")(context_pack)
# Must be built before `mcp.session_manager` exists. Mounted at /mcp below, so
# its own path is "/". Stateless: no server-side session state to lose on restart.
_mcp_app = mcp.streamable_http_app(streamable_http_path="/", stateless_http=True)
@asynccontextmanager
async def lifespan(_: FastAPI):
# Without running the session manager here, /mcp accepts the first request
# and then hangs.
async with mcp.session_manager.run():
yield
app = FastAPI(title="obsidian-wiki memory", lifespan=lifespan)
app.mount("/mcp", _mcp_app)
@app.get("/health")
def health() -> dict[str, Any]:
return {"ok": VAULT.is_dir(), "vault": str(VAULT)}
@app.get("/v1/search", dependencies=[Depends(require_key)])
def http_search(q: str, limit: int = 8) -> dict[str, Any]:
return search(q, limit)
@app.get("/v1/pages/{path:path}", dependencies=[Depends(require_key)])
def http_read(path: str) -> dict[str, Any]:
return read_page(path)
@app.post("/v1/pages", dependencies=[Depends(require_key)])
def http_write(body: PageWrite) -> dict[str, Any]:
return write_page(
body.title, body.category, body.content,
tags=body.tags, sources=body.sources, summary=body.summary, upsert=body.upsert,
)
@app.post("/v1/context-pack", dependencies=[Depends(require_key)])
def http_pack(body: PackRequest) -> dict[str, Any]:
return context_pack(
body.topic, budget=body.budget, recent=body.recent,
public_only=body.public_only, metadata_only=body.metadata_only,
)
def main() -> None:
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("WIKI_PORT", "8080")))
if __name__ == "__main__":
main()
+2
View File
@@ -38,6 +38,8 @@ dependencies = []
ast = ["tree-sitter>=0.21", "tree-sitter-languages>=1.10"]
# Higher-fidelity community detection (Leiden algorithm; greedy label propagation is the default fallback).
graph = ["leidenalg>=0.10", "igraph>=0.11"]
# HTTP + MCP memory server (obsidian_wiki/server.py). The core wheel stays dependency-free.
server = ["fastapi>=0.115", "uvicorn[standard]>=0.30", "mcp>=2.0"]
[project.urls]
Homepage = "https://github.com/Ar9av/obsidian-wiki"
+86
View File
@@ -0,0 +1,86 @@
"""Server checks. Skipped unless the optional [server] extra is installed."""
from __future__ import annotations
import importlib
import sys
import pytest
pytest.importorskip("fastapi")
pytest.importorskip("mcp")
from fastapi.testclient import TestClient # noqa: E402
KEY = "test-key"
@pytest.fixture
def client(tmp_path, monkeypatch):
monkeypatch.setenv("OBSIDIAN_VAULT_PATH", str(tmp_path))
monkeypatch.setenv("WIKI_API_KEY", KEY)
monkeypatch.delenv("WIKI_ALLOW_ANONYMOUS", raising=False)
sys.modules.pop("obsidian_wiki.server", None)
server = importlib.import_module("obsidian_wiki.server")
with TestClient(server.app) as c:
c.headers["authorization"] = f"Bearer {KEY}"
yield c
def test_health_needs_no_key(client):
client.headers.pop("authorization")
assert client.get("/health").json()["ok"] is True
def test_missing_and_wrong_key_are_rejected(client):
client.headers.pop("authorization")
assert client.get("/v1/search", params={"q": "x"}).status_code == 401
client.headers["authorization"] = "Bearer nope"
assert client.get("/v1/search", params={"q": "x"}).status_code == 401
def test_write_then_search_and_read_round_trips(client, tmp_path):
written = client.post("/v1/pages", json={
"title": "Vector Clocks",
"category": "concepts",
"summary": "Ordering events without a global clock.",
"tags": ["distributed-systems"],
"content": "Vector clocks track causality across replicas.",
}).json()
assert written["path"] == "concepts/vector-clocks.md"
on_disk = (tmp_path / written["path"]).read_text()
assert "title: Vector Clocks" in on_disk and "updated:" in on_disk
assert "vector-clocks.md" in (tmp_path / "log.md").read_text()
hits = client.get("/v1/search", params={"q": "vector clocks"}).json()
assert any("vector-clocks" in c["page"] for c in hits["candidates"])
assert "causality" in client.get(f"/v1/pages/{written['path']}").json()["markdown"]
def test_created_date_survives_an_update(client, tmp_path):
body = {"title": "Raft", "category": "concepts", "content": "one"}
first = client.post("/v1/pages", json=body).json()
body["content"] = "two"
assert client.post("/v1/pages", json=body).json()["created"] == first["created"]
assert (tmp_path / first["path"]).read_text().endswith("two\n")
def test_upsert_false_conflicts(client):
body = {"title": "Paxos", "category": "concepts", "content": "x", "upsert": False}
assert client.post("/v1/pages", json=body).status_code == 200
assert client.post("/v1/pages", json=body).status_code == 409
@pytest.mark.parametrize("path", ["../../etc/passwd", "concepts/../../escape.md", "/etc/passwd"])
def test_path_traversal_is_refused(client, path):
# A leading slash is absorbed by the route, so absolute paths land as relative
# ones inside the vault — a 404, never a read outside it.
assert client.get(f"/v1/pages/{path}").status_code in (400, 404)
def test_write_cannot_escape_the_vault(client, tmp_path):
resp = client.post("/v1/pages", json={
"title": "escape", "category": "../../..", "content": "x",
})
# The category is slugified before it becomes a directory, so dots never survive.
assert ".." not in resp.json()["path"]
assert not (tmp_path.parent / "escape.md").exists()