docs: document the index.yml configuration schema

index.yml — qmd's declarative collection config — had no README documentation at all, despite being the single source of truth for every collection's path, glob, ignore rules, contexts, and update command. This documents it in full, with the per-collection `update` hook the maintainer publicly called under-documented as the centerpiece, and overhauls example-index.yml so the repo's starter template actually teaches the schema. - README: new "Configuring index.yml" section — annotated YAML example plus a key-reference table for global_context, editor_uri, models.*, and per-collection path/pattern/ignore/update/includeByDefault/context, with file-location rules (XDG_CONFIG_HOME, QMD_CONFIG_DIR, named {name}.yml, project-local .qmd/index.yml). - README: "Automatic update commands" subsection — execution via bash -c in the collection's directory, run-then-reindex order, non-zero-exit abort, and the qmd collection update-cmd shortcut, with a sample qmd update trace. - README: ignore documented as YAML-only and additive with the un-overridable built-in exclusions; XDG_CONFIG_HOME/QMD_CONFIG_DIR added to the env-var table; Model Configuration cross-references the models:/QMD_EMBED_MODEL override path. - example-index.yml: rewritten into a fully-commented starter template where each collection demonstrates a distinct feature; models: left commented so it can't drift from the live defaults. Every key verified against src/collections.ts and its consumers (src/cli/qmd.ts, src/store.ts). example-index.yml parses as valid YAML. Companion to #715.
2026-09-14 20:27:06 +08:00 · 2026-06-08 21:48:36 -07:00
parent 636602409c
commit 8fbffc838c
3 changed files with 200 additions and 10 deletions
@@ -4,6 +4,30 @@

 ### Documentation

+- README: added a "Configuring `index.yml`" section documenting the full config
+  schema — `global_context`, `editor_uri`, the `models.embed`/`rerank`/`generate`
+  overrides, and per-collection `path`/`pattern`/`ignore`/`update`/
+  `includeByDefault`/`context` — with file-location rules (`XDG_CONFIG_HOME`,
+  `QMD_CONFIG_DIR`, named `{name}.yml`, project-local `.qmd/index.yml`). Every key
+  is verified against `src/collections.ts` and its consumers. Documents behavior
+  that previously existed only in code, where the absence of docs led contributors
+  to repeatedly re-submit already-shipped model-resolution fixes (#502, #559, #564)
+  and to request config that already works (#645, #678). Added
+  `XDG_CONFIG_HOME`/`QMD_CONFIG_DIR` to the environment-variable table and noted
+  the `index.yml` `models:` / `QMD_EMBED_MODEL` override path in the Model
+  Configuration section.
+- README: expanded the per-collection `update` field into an "Automatic update
+  commands" subsection — the feature the maintainer publicly called under-documented
+  — covering execution via `bash -c` in the collection's directory, the run-then-
+  reindex order, the non-zero-exit abort behavior, and the `qmd collection
+  update-cmd` set/clear shortcut. The `ignore` key now states it is YAML-only (no
+  CLI command) and additive with the un-overridable built-in exclusions.
+- `example-index.yml`: overhauled from three near-identical collections into a
+  fully-commented starter template where each collection demonstrates a distinct
+  feature (hierarchical context, auto-`update`, `ignore` patterns, non-markdown
+  globs, `includeByDefault: false`, and an all-fields example), plus commented
+  `editor_uri`/`models` stubs. README now links to it. Model URIs are intentionally
+  left as placeholders so the template can't drift from the defaults.
 - README: documented collection filtering (`-c` semantics), the `collection
  show`/`include`/`exclude`/`update-cmd` subcommands, the `--intent`/`--no-rerank`/
  `-C`/`--full-path` search flags, the `--format <kind>` output selector (with the
@@ -648,6 +648,101 @@ qmd context list
 qmd context rm qmd://notes/old
 ```

+### Configuring `index.yml`
+
+The `collection` and `context` commands above all read and write a single YAML
+config file — you can also edit it directly. Everything QMD knows about your
+collections (paths, masks, exclusions, per-collection update hooks, contexts, and
+optional model overrides) lives here. A fully-commented starter template ships as
+[`example-index.yml`](example-index.yml) in this repo.
+
+**Location:** `~/.config/qmd/index.yml` by default. The directory honors
+`XDG_CONFIG_HOME` (→ `$XDG_CONFIG_HOME/qmd/index.yml`) and `QMD_CONFIG_DIR`. A
+named index uses `{name}.yml` — `qmd --index work …` reads/writes `work.yml`.
+A **project-local** index created with `qmd init` lives at `.qmd/index.yml`
+(`.qmd/index.yaml` is also accepted) alongside a project-local `index.sqlite`,
+so config and index stay inside the project instead of `~/.config` / `~/.cache`.
+
+```yaml
+# ~/.config/qmd/index.yml
+
+# Context applied to every collection (system-message style). Optional.
+global_context: "Knowledge base for my projects"
+
+# Terminal hyperlink template for search results. Optional.
+# Overridden by the QMD_EDITOR_URI env var. See "Editor Links" below.
+editor_uri: "vscode://file{path}:{line}:{col}"
+
+# Override the default GGUF models per role. Optional — omit to use the
+# built-in defaults. `qmd init` writes this block pre-filled with the
+# resolved defaults. See "Model Configuration" for the default URIs.
+models:
+  embed: "hf:ggml-org/embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf"
+  rerank: "hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf"
+  generate: "hf:tobil/qmd-query-expansion-1.7B-gguf/qmd-query-expansion-1.7B-q4_k_m.gguf"
+
+# One entry per collection. The key is the collection name.
+collections:
+  notes:
+    path: /Users/me/notes        # absolute path to index (required)
+    pattern: "**/*.md"           # glob mask (default: **/*.md)
+    ignore:                      # glob patterns to exclude from indexing
+      - "Archive/**"
+      - "**/drafts/**"
+    update: "git pull --rebase"  # bash command run before each `qmd update`
+    includeByDefault: true       # include in unscoped queries (default: true)
+    context:                     # path prefix → description; longest match wins
+      "/": "Personal notes and ideas"
+      "/work": "Work-related notes"
+```
+
+| Key | Scope | Purpose |
+|-----|-------|---------|
+| `global_context` | top-level | Context prepended for every collection. Set via `qmd context add /`. |
+| `editor_uri` (alias `editor_uri_template`) | top-level | Hyperlink template for clickable result paths; `QMD_EDITOR_URI` overrides. |
+| `models.embed` / `.rerank` / `.generate` | top-level | HuggingFace GGUF URIs (`hf:<user>/<repo>/<file>`) overriding the built-in defaults per role. |
+| `collections.<name>.path` | per-collection | Absolute directory to index. |
+| `collections.<name>.pattern` | per-collection | Glob mask. Set via `qmd collection add --mask`. Default `**/*.md`. |
+| `collections.<name>.ignore` | per-collection | Glob patterns excluded from indexing — useful to stop nested collections double-indexing. **YAML-only — no CLI command sets this.** Additive with QMD's built-in exclusions (`node_modules`, `.git`, `.cache`, `vendor`, `dist`, `build`), which you cannot un-ignore. |
+| `collections.<name>.update` | per-collection | Bash command run before `qmd update` re-indexes this collection. Set via `qmd collection update-cmd`. |
+| `collections.<name>.includeByDefault` | per-collection | Whether unscoped queries search it. Toggle with `qmd collection include`/`exclude`. Default `true`. |
+| `collections.<name>.context` | per-collection | Path-prefix → description map; the most specific (longest) matching prefix wins. Set via `qmd context add`. |
+
+> **Note:** Editing `index.yml` changes which directories and models QMD *uses*,
+> but does not re-index on its own. Run `qmd update` after changing `path`,
+> `pattern`, or `ignore`, and `qmd embed` after changing `models.embed`.
+
+#### Automatic update commands
+
+A collection's `update` field is QMD's built-in refresh hook: when you run
+`qmd update`, each collection's `update` command runs **first**, then the
+collection is re-indexed. This keeps a collection in sync with an upstream source
+(a git remote, a sync script) without wrapping `qmd` yourself.
+
+```yaml
+collections:
+  wiki:
+    path: ~/reference/wiki
+    update: "git pull --ff-only"
+```
+
+    $ qmd update
+    [1/3] wiki (**/*.md)
+        Running update command: git pull --ff-only
+        Already up to date.
+    Collection: ~/reference/wiki (**/*.md)
+    Indexed: 0 new, 2 updated, 340 unchanged, 0 removed
+
+The command runs via `bash -c` in the collection's own directory (its `path`), not
+your current working directory. If it exits non-zero, `qmd update` prints the
+failure and **aborts the entire run** — collections after the failing one are not
+re-indexed. Set or clear it from the CLI instead of editing YAML by hand:
+
+```sh
+qmd collection update-cmd wiki 'git pull --ff-only'   # set
+qmd collection update-cmd wiki                         # clear
+```
+
 ### Search Commands

 ```
@@ -967,6 +1062,8 @@ llm_cache       -- Cached LLM responses (query expansion, rerank scores)
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `XDG_CACHE_HOME` | `~/.cache` | Cache directory location |
+| `XDG_CONFIG_HOME` | `~/.config` | Config directory location (where `index.yml` lives) |
+| `QMD_CONFIG_DIR` | unset | Override the config directory outright (takes precedence over `XDG_CONFIG_HOME`) |
 | `QMD_LLAMA_GPU` | `auto` | Force llama.cpp GPU backend (`metal`, `vulkan`, `cuda`) or disable GPU with `false` |
 | `QMD_FORCE_CPU` | unset | Set to `1`/`true` to force CPU mode before any CUDA/Vulkan/Metal probing. Equivalent CLI flag: `--no-gpu`. |
 | `QMD_EMBED_PARALLELISM` | automatic | Override embedding/reranking context parallelism (1-8). Windows CUDA defaults to `1` because parallel CUDA contexts can crash with `ggml-cuda.cu:98`; use Vulkan or raise this only if your driver is stable. |
@@ -1086,7 +1183,7 @@ Query ──► LLM Expansion ──► [Original, Variant 1, Variant 2]

 ## Model Configuration

-Models are configured in `src/llm.ts` as HuggingFace URIs:
+The default models are defined in `src/llm.ts` as HuggingFace URIs:

 ```typescript
 const DEFAULT_EMBED_MODEL = "hf:ggml-org/embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf";
@@ -1094,6 +1191,10 @@ const DEFAULT_RERANK_MODEL = "hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-re
 const DEFAULT_GENERATE_MODEL = "hf:tobil/qmd-query-expansion-1.7B-gguf/qmd-query-expansion-1.7B-q4_k_m.gguf";
 ```

+Override them per-role without touching source via the `models:` block in
+`index.yml` (see [Configuring `index.yml`](#configuring-indexyml)) or the
+`QMD_EMBED_MODEL` env var. Re-run `qmd embed` after changing the embedding model.
+
 ### EmbeddingGemma Prompt Format

 ```
@@ -1,33 +1,98 @@
 # QMD Collections Configuration
 # Location: ~/.config/qmd/index.yml
 #
-# This file defines all collections and their contexts.
-# You can manually edit this file - changes take effect immediately.
+# This file is the single source of truth for your collections. Every
+# `qmd collection ...` and `qmd context ...` command reads and writes it, and
+# you can also edit it directly — changes take effect on the next command.
+# (Editing does NOT re-index; run `qmd update` after changing a path, pattern,
+# or ignore list.)
+#
+# Copy this file to ~/.config/qmd/index.yml and customize it.

-# Global context applied to all collections
-# Use this for universal search instructions or patterns
+# ─── Global context ──────────────────────────────────────────────────────────
+# Prepended to results across ALL collections. Use it for conventions that span
+# your whole knowledge base. Set via: qmd context add / "..."
 global_context: "If you see a relevant [[WikiWord]], you can search for that WikiWord to get more context."

-# Collection definitions
+# ─── Terminal hyperlinks (optional) ──────────────────────────────────────────
+# Template that turns result paths into clickable editor links.
+# Overridden by the QMD_EDITOR_URI env var.
+# editor_uri: "vscode://file{path}:{line}:{col}"
+
+# ─── Model overrides (optional) ──────────────────────────────────────────────
+# Omit to use the built-in GGUF defaults. `qmd init` writes this block
+# pre-filled with the resolved defaults. See "Model Configuration" in the README
+# for the current default URIs (kept there so this template can't go stale).
+# models:
+#   embed: "hf:<user>/<repo>/<file>.gguf"
+#   rerank: "hf:<user>/<repo>/<file>.gguf"
+#   generate: "hf:<user>/<repo>/<file>.gguf"
+
+# ─── Collections ─────────────────────────────────────────────────────────────
 collections:
-  # Meeting notes
-  Meetings:
+
+  # ── Basic: just a path and a pattern ───────────────────────────────────────
+  meetings:
    path: ~/Documents/Meetings
    pattern: "**/*.md"
    context:
      "/": "Meeting notes and summaries"

-  # Daily journal entries
+  # ── Hierarchical context: path-prefix → description ────────────────────────
+  # Context is matched by path prefix; the most specific (longest) match wins.
  journals:
    path: ~/Documents/Notes
    pattern: "**/*.md"
    context:
+      "/": "Personal notes vault"
      "/journal/2024": "Daily notes from 2024"
      "/journal/2025": "Daily notes from 2025"
-      "/": "Notes vault"
+      "/recipes": "Cooking recipes and meal planning"

+  # ── Auto-update: run a command before re-indexing ──────────────────────────
+  # On `qmd update`, this runs via `bash -c` in the collection's directory
+  # BEFORE re-indexing. A non-zero exit aborts the whole `qmd update` run.
+  # Set via CLI: qmd collection update-cmd wiki 'git pull --ff-only'
+  wiki:
+    path: ~/reference/wiki
+    pattern: "**/*.md"
+    update: "git pull --ff-only"
+    context:
+      "/": "External wiki — automatically pulled on every qmd update"
+
+  # ── Ignore patterns: exclude files from indexing ───────────────────────────
+  # YAML-only — there is no CLI command for this. Additive with the built-in
+  # exclusions (node_modules, .git, .cache, vendor, dist, build), which you
+  # cannot un-ignore. Non-markdown patterns work too.
+  project-docs:
+    path: ~/work/project
+    pattern: "**/*.{md,mdx,txt}"
+    ignore:
+      - "drafts/**"           # skip draft documents
+      - "*.tmp"               # skip temp files
+      - "archive/2023/**"     # skip old archive
+    context:
+      "/": "Team project documentation"
+      "/api": "API reference and endpoint docs"
+      "/guides": "How-to guides and tutorials"
+
+  # ── Exclude from default searches ──────────────────────────────────────────
+  # includeByDefault: false → skipped unless you name it: qmd query -c vendor "..."
+  # Set via CLI: qmd collection exclude vendor
+  vendor:
+    path: ~/work/vendor
+    pattern: "**/*.md"
+    includeByDefault: false
+    context:
+      "/": "Third-party vendor docs (excluded from default searches; use -c vendor)"
+
+  # ── Everything together ────────────────────────────────────────────────────
  codex:
    path: ~/Documents/Codex
    pattern: "**/*.md"
+    ignore:
+      - "Sessions/**"
+    update: "git stash && git pull --rebase --ff-only && git stash pop"
+    includeByDefault: true    # true is the default; shown here for clarity
    context:
      "/": "Thematic collections of important concepts and discussions"