feat(concurrency): opt-in parallelism for I/O-bound MCP tools

Adds a `concurrency: 1-8` parameter to ctx_batch_execute and
ctx_fetch_and_index, plus a shared `runPool` primitive, observability
extractor, and Parallel I/O guidance across all 14 adapter routing
docs.

What ships
- src/concurrency/runPool.ts (new): generic in-flight-capped worker
  pool returning Promise.allSettled-style results. Single primitive
  used by both batch tools — no copy-pasted worker logic.
- ctx_batch_execute: serial branch unchanged (shared timeout budget,
  cascading skip). Parallel branch routed through runPool. Description
  hardened with PARALLELIZE I/O ✅/❌ guidance and NON-NEGOTIABLE
  THINK IN CODE clause.
- ctx_fetch_and_index: accepts both legacy `{url, source}` (single,
  exact backward-compat wording) and new `{requests: [{url, source}]}`
  (batch). Workers fetch in parallel via runPool; FTS5 writes drain
  serially through indexFetched to avoid SQLite WAL contention.
  Per-URL preview capped at 384 chars in batch mode (~3KB total) so
  context-savings hold under 8-URL fan-outs. composeFetchCacheKey
  wiring preserved across the refactor — same-label-different-URL
  collisions stay fixed (commit 1f1243e regression test enforced).
- effectiveConcurrency = min(N, os.cpus().length) when capByCpuCount
  set. Response surfaces capped count in caveman style.
- mcp_tool_call extractor (src/session/extract.ts) persists tool_input
  for mcp__* events with UTF-8-aware truncation at 2KB. Unlocks
  getMcpToolUsage() analytics — median/max concurrency per batch tool
  visible in ctx_stats.
- 14 adapter routing docs updated with the same Parallel I/O
  paragraph adapted to each host's tool-call prefix style. GitHub
  rate-limit caveat included consistently.

Hardening from 2-round architectural review
- Worker try/catch + Promise.allSettled isolation: one job throw no
  longer strands siblings or leaves undefined output slots.
- Timeout sentinel routes through formatCommandOutput: __CM_FS__
  markers stripped + bytes counted on partial-stdout-on-timeout.
- trackIndexed moved after FTS5 write succeeds (no over-count on
  failed indexes).
- UTF-8-aware truncate (Buffer.byteLength + continuation-byte
  walk-back): multi-byte payloads (CJK, 4-byte symbols) honor the
  byte budget without landing mid-codepoint.
- cpuCountForCap helper deleted: was CommonJS require in an ESM
  file, silently always returning 1. Replaced with top-level
  `cpus` import from node:os.

Tests (per CONTRIBUTING.md no-new-test-files rule, all under
existing files)
- 7 runPool unit tests: order, throw isolation, in-flight cap,
  job-count clamp, os.cpus cap, onSettled callback ordering.
- 13 ctx_fetch_and_index batch source-level tests: schema accepts
  both shapes, serial-write contract holds, backward-compat wording
  preserved, batch preview cap enforced, caveman header formatting,
  composeFetchCacheKey wiring across the refactor.
- 3 P0 hardening tests: throw-isolation, timeout marker stripping,
  5-cmd × 100ms at concurrency=5 < 200ms (CI-checked timing
  regression replacing the deleted bench).
- 4 mcp_tool_call extractor tests including UTF-8 multibyte
  regression.
- 3 getMcpToolUsage analytics tests.

Verification
- 138/138 server.test.ts pass; 309/309 across server + extract +
  analytics on cw/ctx-analytics.
- On next: 318/326 pass. 8 pre-existing unrelated failures
  (ctx_index projectRoot resolution from #365, ctx_execute_file env
  cascade, getSessionDir pre-detection) untouched.
- Typecheck clean.

Co-Authored-By: Sebastian Breguel <sebastianbreguel@gmail.com>
This commit is contained in:
Mert Koseoglu
2026-05-02 21:43:37 +03:00
parent 6ada7536b1
commit b392c2fe2f
21 changed files with 1204 additions and 565 deletions
+11
View File
@@ -40,6 +40,17 @@ Use `mcp__context-mode__ctx_execute(language: "shell", code: "grep ...")` in san
4. **WEB**: `mcp__context-mode__ctx_fetch_and_index(url, source)` then `mcp__context-mode__ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `mcp__context-mode__ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `mcp__context-mode__ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `mcp__context-mode__ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+11
View File
@@ -40,6 +40,17 @@ Use `ctx_execute(language: "shell", code: "grep ...")` in sandbox.
4. **WEB**: `ctx_fetch_and_index(url, source)` then `ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Subagent routing
Routing block auto-injected into subagent prompts. Bash-type subagents upgraded to general-purpose. No manual instruction needed.
+11
View File
@@ -41,6 +41,17 @@ Use `ctx_execute(language: "shell", code: "grep ...")` in sandbox.
4. **WEB**: `ctx_fetch_and_index(url, source)` then `ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+11
View File
@@ -20,6 +20,17 @@ Analyze/count/filter/compare/search/parse/transform data: **write code** via `ct
4. **WEB**: `ctx_fetch_and_index(url)` then `ctx_search(queries)` — never dump raw HTML.
5. **INDEX**: `ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Forbidden Actions
- DO NOT use Bash for >20 lines output — use `ctx_execute` or `ctx_batch_execute`.
+11
View File
@@ -40,6 +40,17 @@ Use `mcp__context-mode__ctx_execute(language: "shell", code: "grep ...")` in san
4. **WEB**: `mcp__context-mode__ctx_fetch_and_index(url, source)` then `mcp__context-mode__ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `mcp__context-mode__ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `mcp__context-mode__ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `mcp__context-mode__ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
@@ -40,6 +40,9 @@ Use `ctx_execute(language: "shell", code: "grep ...")` in sandbox.
4. **WEB**: `ctx_fetch_and_index(url, source)` then `ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `ctx_index(content, source)` — store in FTS5 for later search.
### Parallel I/O batches
Pass `concurrency: 4-8` to `ctx_batch_execute` and `ctx_fetch_and_index` for network/API batches. Keep `concurrency: 1` for CPU-bound work (test, build, lint). GitHub gh: cap at 4.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+11
View File
@@ -40,6 +40,17 @@ Use `context-mode_ctx_execute(language: "shell", code: "grep ...")` in sandbox.
4. **WEB**: `context-mode_ctx_fetch_and_index(url, source)` then `context-mode_ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `context-mode_ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `context-mode_ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `context-mode_ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+11
View File
@@ -40,6 +40,17 @@ Use `@context-mode/ctx_execute(language: "shell", code: "grep ...")` in sandbox.
4. **WEB**: `@context-mode/ctx_fetch_and_index(url, source)` then `@context-mode/ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `@context-mode/ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `@context-mode/ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `@context-mode/ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+11
View File
@@ -40,6 +40,17 @@ Use `context-mode__ctx_execute(language: "shell", code: "grep ...")` in sandbox.
4. **WEB**: `context-mode__ctx_fetch_and_index(url, source)` then `context-mode__ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `context-mode__ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `context-mode__ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `context-mode__ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+11
View File
@@ -40,6 +40,17 @@ Use `context-mode_ctx_execute(language: "shell", code: "grep ...")` in sandbox.
4. **WEB**: `context-mode_ctx_fetch_and_index(url, source)` then `context-mode_ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `context-mode_ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `context-mode_ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `context-mode_ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+11
View File
@@ -41,6 +41,17 @@ Use `ctx_execute(language: "shell", code: "grep ...")` in sandbox.
4. **WEB**: `ctx_fetch_and_index(url, source)` then `ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+11
View File
@@ -40,6 +40,17 @@ Use `mcp__context-mode__ctx_execute(language: "shell", code: "grep ...")` in san
4. **WEB**: `mcp__context-mode__ctx_fetch_and_index(url, source)` then `mcp__context-mode__ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `mcp__context-mode__ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `mcp__context-mode__ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `mcp__context-mode__ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Subagent routing
Routing block auto-injected into subagent prompts. Bash-type subagents upgraded to general-purpose. No manual instruction needed.
@@ -40,6 +40,9 @@ Use `ctx_execute(language: "shell", code: "grep ...")` in sandbox.
4. **WEB**: `ctx_fetch_and_index(url, source)` then `ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `ctx_index(content, source)` — store in FTS5 for later search.
### Parallel I/O batches
Pass `concurrency: 4-8` to `ctx_batch_execute` and `ctx_fetch_and_index` for network/API batches. Keep `concurrency: 1` for CPU-bound work (test, build, lint). GitHub gh: cap at 4.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+11
View File
@@ -40,6 +40,17 @@ Use `mcp:context-mode:ctx_execute(language: "shell", code: "grep ...")` in sandb
4. **WEB**: `mcp:context-mode:ctx_fetch_and_index(url, source)` then `mcp:context-mode:ctx_search(queries)` — raw HTML never enters context.
5. **INDEX**: `mcp:context-mode:ctx_index(content, source)` — store in FTS5 for later search.
## Parallel I/O batches
For multi-URL fetches or multi-API calls, **always** include `concurrency: N` (1-8):
- `mcp:context-mode:ctx_batch_execute(commands: [3+ network commands], concurrency: 5)` — gh, curl, dig, docker inspect, multi-region cloud queries
- `mcp:context-mode:ctx_fetch_and_index(requests: [{url, source}, ...], concurrency: 5)` — multi-URL batch fetch
**Use concurrency 4-8** for I/O-bound work (network calls, API queries). **Keep concurrency 1** for CPU-bound (npm test, build, lint) or commands sharing state (ports, lock files, same-repo writes).
GitHub API rate-limit: cap at 4 for `gh` calls.
## Output
Terse like caveman. Technical substance exact. Only fluff die.
+81
View File
@@ -0,0 +1,81 @@
/**
* Generic in-flight-capped worker pool.
*
* Used by:
* - runBatchCommands (ctx_batch_execute parallel branch)
* - runBatchFetch (ctx_fetch_and_index batch path)
*
* Returns Promise.allSettled-style results so one job's throw cannot
* strand siblings. Caller maps fulfilled/rejected per index. Output
* order is preserved by input index (not completion order).
*
* Designed to be the SINGLE concurrency primitive for the project —
* all "run N independent operations with at most M in flight" needs
* route here. Avoids the worker-pool copy-paste flagged in the
* concurrency PRD architectural review (finding G).
*/
import { cpus } from "node:os";
export interface PoolJob<T> {
run(): Promise<T>;
}
export interface RunPoolOptions {
/** Hard concurrency cap (1-N). Auto-clamped to job count. */
concurrency: number;
/** Optional: also clamp by `os.cpus().length` (memory-pressure safety). Default false. */
capByCpuCount?: boolean;
/** Optional: per-settled callback (e.g. for progress reporting / metrics). */
onSettled?: (idx: number, result: PromiseSettledResult<unknown>) => void;
}
export interface RunPoolResult<T> {
/** Per-index settled result, ordered by input index. */
settled: PromiseSettledResult<T>[];
/** Concurrency actually used after all caps applied. */
effectiveConcurrency: number;
/** True when effectiveConcurrency < requested concurrency. */
capped: boolean;
}
export async function runPool<T>(
jobs: PoolJob<T>[],
opts: RunPoolOptions,
): Promise<RunPoolResult<T>> {
const { concurrency, capByCpuCount = false, onSettled } = opts;
if (jobs.length === 0) {
return { settled: [], effectiveConcurrency: 0, capped: false };
}
const requested = Math.max(1, concurrency);
const cpuCap = capByCpuCount ? Math.max(1, cpus().length) : requested;
const effectiveConcurrency = Math.min(requested, cpuCap, jobs.length);
const capped = effectiveConcurrency < requested;
const settled: PromiseSettledResult<T>[] = new Array(jobs.length);
let nextIdx = 0;
async function worker(): Promise<void> {
while (true) {
const idx = nextIdx++;
if (idx >= jobs.length) return;
try {
const value = await jobs[idx].run();
settled[idx] = { status: "fulfilled", value };
} catch (err) {
settled[idx] = { status: "rejected", reason: err };
}
onSettled?.(idx, settled[idx]);
}
}
const workers: Promise<void>[] = [];
for (let w = 0; w < effectiveConcurrency; w++) workers.push(worker());
// allSettled defends against any promise rejection escaping a worker
// (the worker already swallows its own errors, but this is belt-and-braces).
await Promise.allSettled(workers);
return { settled, effectiveConcurrency, capped };
}
+354 -272
View File
@@ -5,12 +5,13 @@ import { createRequire } from "node:module";
import { createHash } from "node:crypto";
import { existsSync, unlinkSync, readdirSync, readFileSync, writeFileSync, rmSync, mkdirSync, cpSync, statSync, symlinkSync, lstatSync } from "node:fs";
import { execSync, type ChildProcess } from "node:child_process";
import { join, dirname, resolve, sep, isAbsolute } from "node:path";
import { join, dirname, resolve, sep } from "node:path";
import { fileURLToPath } from "node:url";
import { homedir, tmpdir } from "node:os";
import { homedir, tmpdir, cpus } from "node:os";
import { request as httpsRequest } from "node:https";
import { z } from "zod";
import { PolyglotExecutor } from "./executor.js";
import { runPool, type PoolJob } from "./concurrency/runPool.js";
import { ContentStore, cleanupStaleDBs, cleanupStaleContentDBs, type SearchResult, type IndexResult } from "./store.js";
import { composeFetchCacheKey } from "./fetch-cache.js";
import {
@@ -31,9 +32,8 @@ import { startLifecycleGuard } from "./lifecycle.js";
import { getWorktreeSuffix, SessionDB } from "./session/db.js";
import { searchAllSources } from "./search/unified.js";
import { buildNodeCommand, type HookAdapter } from "./adapters/types.js";
import { detectPlatform, getSessionDirSegments } from "./adapters/detect.js";
import { loadDatabase } from "./db-base.js";
import { AnalyticsEngine, formatReport, getLifetimeStats } from "./session/analytics.js";
import { AnalyticsEngine, formatReport } from "./session/analytics.js";
const __pkg_dir = dirname(fileURLToPath(import.meta.url));
const VERSION: string = (() => {
for (const rel of ["../package.json", "./package.json"]) {
@@ -69,13 +69,9 @@ server.server.setRequestHandler(ListPromptsRequestSchema, async () => ({ prompts
server.server.setRequestHandler(ListResourcesRequestSchema, async () => ({ resources: [] }));
server.server.setRequestHandler(ListResourceTemplatesRequestSchema, async () => ({ resourceTemplates: [] }));
// Pass `getProjectDir` (env cascade) lazily so the executor stays in sync
// with the same resolver used by ctx_index / resolveProjectPath. Capturing
// a snapshot at construction would diverge when only CONTEXT_MODE_PROJECT_DIR
// (or another non-CLAUDE_PROJECT_DIR var) is set — see PR #365.
const executor = new PolyglotExecutor({
runtimes,
projectRoot: () => getProjectDir(),
projectRoot: process.env.CLAUDE_PROJECT_DIR,
});
// ─────────────────────────────────────────────────────────
@@ -129,28 +125,10 @@ let _insightChild: ChildProcess | null = null;
/**
* Get the platform-specific sessions directory from the detected adapter.
*
* Pre-detection path (race window before MCP `initialize` completes):
* call `detectPlatform()` (sync, env-var-based) and look up segments via
* `getSessionDirSegments()` (sync map, no adapter instantiation). This keeps
* non-Claude platforms from spilling sessions into `~/.claude/`.
*
* Last-resort `.claude` fallback only fires if the segments map returns null
* (e.g., "unknown" PlatformId) or if anything throws.
* Falls back to ~/.claude/context-mode/sessions/ before adapter detection.
*/
function getSessionDir(): string {
if (_detectedAdapter) return _detectedAdapter.getSessionDir();
try {
const signal = detectPlatform();
const segments = getSessionDirSegments(signal.platform);
if (segments) {
const dir = join(homedir(), ...segments, "context-mode", "sessions");
mkdirSync(dir, { recursive: true });
return dir;
}
} catch { /* fall through to default */ }
const dir = join(homedir(), ".claude", "context-mode", "sessions");
mkdirSync(dir, { recursive: true });
return dir;
@@ -173,15 +151,10 @@ function getProjectDir(): string {
|| process.env.VSCODE_CWD
|| process.env.OPENCODE_PROJECT_DIR
|| process.env.PI_PROJECT_DIR
|| process.env.IDEA_INITIAL_DIRECTORY
|| process.env.CONTEXT_MODE_PROJECT_DIR
|| process.cwd();
}
function resolveProjectPath(filePath: string): string {
return isAbsolute(filePath) ? filePath : resolve(getProjectDir(), filePath);
}
/**
* Consistent project dir hashing across all DB paths.
* Normalizes Windows backslashes before hashing so the same project
@@ -374,46 +347,9 @@ function trackResponse(toolName: string, response: ToolResult): ToolResult {
sessionStats.calls[toolName] = (sessionStats.calls[toolName] || 0) + 1;
sessionStats.bytesReturned[toolName] =
(sessionStats.bytesReturned[toolName] || 0) + bytes;
// Persist to SessionDB so counters survive process restart, --continue, upgrade.
// Best-effort: never throws, never blocks. Deferred via setImmediate so the
// SQLite open/select/update/close (~1-5ms even after worktree-suffix cache)
// does not extend the response path on any of macOS / Linux / Windows.
setImmediate(() => persistToolCallCounter(toolName, bytes));
return response;
}
/**
* Increment the per-session, per-tool counter in SessionDB so ctx_stats
* keeps showing the right numbers after the server restarts mid-session
* (e.g. on `npm update -g context-mode` or `claude --continue`).
*
* The session_id used is whatever session_meta currently holds as the
* most recent session — populated by the SessionStart hook.
*/
function persistToolCallCounter(toolName: string, bytes: number): void {
try {
const dbHash = hashProjectDir();
const worktreeSuffix = getWorktreeSuffix();
const sessionDbPath = join(
getSessionDir(),
`${dbHash}${worktreeSuffix}.db`,
);
if (!existsSync(sessionDbPath)) return;
const sdb = new SessionDB({ dbPath: sessionDbPath });
try {
const sid = sdb.getLatestSessionId();
if (!sid) return;
sdb.incrementToolCall(sid, toolName, bytes);
} finally {
sdb.close();
}
} catch {
// best-effort: counter must never throw
}
}
function trackIndexed(bytes: number): void {
sessionStats.bytesIndexed += bytes;
}
@@ -488,11 +424,7 @@ function checkFilePathDenyPolicy(
toolName: string,
): ToolResult | null {
try {
// Use the canonical getProjectDir() helper so deny-policy enforcement
// works on every supported adapter. The previous shortcut skipped the
// full env cascade and either failed open on non-Claude hosts or
// matched against an unrelated repo's deny rules.
const projectDir = getProjectDir();
const projectDir = process.env.CLAUDE_PROJECT_DIR ?? process.cwd();
const denyGlobs = readToolDenyPatterns("Read", projectDir);
const result = evaluateFilePath(
filePath,
@@ -758,34 +690,40 @@ export async function runBatchCommands(
return { outputs, timedOut };
}
// Parallel path — per-command timeout, in-flight cap, order preserved by index.
const outputs: string[] = new Array(commands.length);
let timedOut = false;
const cap = Math.min(concurrency, commands.length);
let nextIdx = 0;
async function worker(): Promise<void> {
while (true) {
const idx = nextIdx++;
if (idx >= commands.length) return;
const cmd = commands[idx];
// Parallel path — delegated to the shared runPool primitive.
// Each job returns { output, timedOut }; runPool handles in-flight cap,
// throw isolation (Promise.allSettled semantics), and order preservation.
const jobs: PoolJob<{ output: string; timedOut: boolean }>[] = commands.map((cmd) => ({
run: async () => {
const result = await executor.execute({
language: "shell",
code: `${nodeOptsPrefix}${cmd.command} 2>&1`,
timeout,
});
if (result.timedOut) {
timedOut = true;
outputs[idx] = `# ${cmd.label}\n\n(timed out after ${timeout}ms)\n`;
} else {
outputs[idx] = formatCommandOutput(cmd.label, result.stdout, onFsBytes);
}
// Always route partial stdout through formatCommandOutput so __CM_FS__
// markers are stripped + counted, even when the command timed out.
const formatted = formatCommandOutput(cmd.label, result.stdout, onFsBytes);
const output = result.timedOut
? formatted.replace(/\n$/, "") + `\n(timed out after ${timeout}ms)\n`
: formatted;
return { output, timedOut: !!result.timedOut };
},
}));
const { settled } = await runPool(jobs, { concurrency });
const outputs: string[] = new Array(commands.length);
let timedOut = false;
for (let i = 0; i < settled.length; i++) {
const r = settled[i];
if (r.status === "fulfilled") {
outputs[i] = r.value.output;
if (r.value.timedOut) timedOut = true;
} else {
// Isolated executor throw (spawn EAGAIN, ENOMEM, EMFILE, …) — siblings keep running.
const message = r.reason instanceof Error ? r.reason.message : String(r.reason);
outputs[i] = `# ${commands[i].label}\n\n(executor error: ${message})\n`;
}
}
const workers: Promise<void>[] = [];
for (let w = 0; w < cap; w++) workers.push(worker());
await Promise.all(workers);
return { outputs, timedOut };
}
@@ -1322,18 +1260,16 @@ server.registerTool(
}
try {
const resolvedPath = path ? resolveProjectPath(path) : undefined;
// Track the raw bytes being indexed (content or file)
if (content) trackIndexed(Buffer.byteLength(content));
else if (resolvedPath) {
else if (path) {
try {
const fs = await import("fs");
trackIndexed(fs.readFileSync(resolvedPath).byteLength);
trackIndexed(fs.readFileSync(path).byteLength);
} catch { /* ignore — file read errors handled by store */ }
}
const store = getStore();
const result = store.index({ content, path: resolvedPath, source: source ?? resolvedPath });
const result = store.index({ content, path, source });
return trackResponse("ctx_index", {
content: [
@@ -1507,25 +1443,19 @@ server.registerTool(
let totalSize = 0;
const sections: string[] = [];
// Open SessionDB once before the loop (Blocker 4: avoid open/close per query).
// The DB filename must match what session-snapshot/session-extract write to,
// which is `${hash}${getWorktreeSuffix()}.db` — bug #4 was the missing suffix.
// Open SessionDB once before the loop (Blocker 4: avoid open/close per query)
let timelineDB: InstanceType<typeof SessionDB> | null = null;
if (sort === "timeline") {
try {
const sessionsDir = getSessionDir();
const dbFile = join(sessionsDir, `${hashProjectDir()}${getWorktreeSuffix()}.db`);
const dbFile = join(sessionsDir, `${hashProjectDir()}.db`);
if (existsSync(dbFile)) {
timelineDB = new SessionDB({ dbPath: dbFile });
}
} catch { /* SessionDB unavailable — search ContentStore + auto-memory only */ }
}
// Adapter-aware config dir. Falls back to CLAUDE_CONFIG_DIR / ~/.claude
// only when no platform adapter has been detected (e.g. raw `npx context-mode dev`).
const configDir = _detectedAdapter?.getConfigDir()
|| process.env.CLAUDE_CONFIG_DIR
|| join(homedir(), ".claude");
const configDir = process.env.CLAUDE_CONFIG_DIR || join(homedir(), ".claude");
try {
for (const q of queryList) {
@@ -1546,7 +1476,6 @@ server.registerTool(
sessionDB: timelineDB,
projectDir: getProjectDir(),
configDir,
adapter: _detectedAdapter ?? undefined,
});
} else {
results = store.searchWithFallback(q, effectiveLimit, source, contentType);
@@ -1695,23 +1624,166 @@ main();
`;
}
// ─────────────────────────────────────────────────────────
// fetch_and_index helpers — split into parallel-safe fetch and serial-only index
// ─────────────────────────────────────────────────────────
const FETCH_TTL_MS = 24 * 60 * 60 * 1000; // 24 hours
const FETCH_PREVIEW_LIMIT = 3072;
type FetchOneResult =
| { kind: "cached"; label: string; chunkCount: number; estimatedBytes: number; ageStr: string }
| { kind: "fetched"; url: string; source?: string; markdown: string; header: string }
| { kind: "fetch_error"; url: string; error: string; reason: "exit" | "read" | "empty" | "throw" };
/**
* Pure fetch step — TTL cache check + subprocess fetch. SAFE TO RUN IN PARALLEL.
* Performs zero SQLite writes (only reads source meta). Caller must funnel
* fetched results through `indexFetched` serially to avoid FTS5 WAL contention.
*/
async function fetchOneUrl(url: string, source: string | undefined, force: boolean | undefined): Promise<FetchOneResult> {
if (!force) {
const store = getStore();
// Cache key composes (source, url) so two distinct URLs sharing the same
// `source` label do not collide — they each get their own cache slot
// (commit 1f1243e regression test enforced).
const cacheKey = composeFetchCacheKey(source, url);
const meta = store.getSourceMeta(cacheKey);
if (meta) {
const indexedAt = new Date(meta.indexedAt + "Z"); // SQLite datetime is UTC without Z
const ageMs = Date.now() - indexedAt.getTime();
if (ageMs < FETCH_TTL_MS) {
const ageHours = Math.floor(ageMs / (60 * 60 * 1000));
const ageMin = Math.floor(ageMs / (60 * 1000));
const ageStr = ageHours > 0 ? `${ageHours}h ago` : ageMin > 0 ? `${ageMin}m ago` : "just now";
const estimatedBytes = meta.chunkCount * 1600; // ~1.6KB/chunk avg
return { kind: "cached", label: meta.label, chunkCount: meta.chunkCount, estimatedBytes, ageStr };
}
// Stale — fall through to re-fetch silently
}
}
const outputPath = join(tmpdir(), `ctx-fetch-${Date.now()}-${Math.random().toString(36).slice(2)}.dat`);
try {
const fetchCode = buildFetchCode(url, outputPath);
const result = await executor.execute({
language: "javascript",
code: fetchCode,
timeout: 30_000,
});
if (result.exitCode !== 0) {
return { kind: "fetch_error", url, error: result.stderr || result.stdout || "unknown error", reason: "exit" };
}
const header = (result.stdout || "").trim();
let markdown: string;
try {
markdown = readFileSync(outputPath, "utf-8").trim();
} catch {
return { kind: "fetch_error", url, error: "could not read subprocess output", reason: "read" };
}
if (markdown.length === 0) {
return { kind: "fetch_error", url, error: "empty content", reason: "empty" };
}
return { kind: "fetched", url, source, markdown, header };
} catch (err: unknown) {
return {
kind: "fetch_error",
url,
error: err instanceof Error ? err.message : String(err),
reason: "throw",
};
} finally {
try { rmSync(outputPath); } catch { /* already gone */ }
}
}
interface IndexedFetchResult {
label: string;
totalChunks: number;
totalBytes: number;
preview: string;
}
/**
* Serial-only indexing step — single FTS5 write per call. Caller loops over
* fetched results and calls this one-at-a-time to avoid SQLite WAL contention
* (PRD finding E).
*/
function indexFetched(f: { url: string; source?: string; markdown: string; header: string }): IndexedFetchResult {
const store = getStore();
// Storage label composed via composeFetchCacheKey so two URLs sharing a
// `source` label do not overwrite each other (commit 1f1243e). ctx_search()
// still finds both via LIKE-mode source filter on the `source` substring.
const storageLabel = composeFetchCacheKey(f.source, f.url);
let indexed: IndexResult;
if (f.header === "__CM_CT__:json") {
indexed = store.indexJSON(f.markdown, storageLabel);
} else if (f.header === "__CM_CT__:text") {
indexed = store.indexPlainText(f.markdown, storageLabel);
} else {
indexed = store.index({ content: f.markdown, source: storageLabel });
}
// Track AFTER the FTS5 write succeeds — failed indexes shouldn't inflate the counter.
trackIndexed(Buffer.byteLength(f.markdown));
const preview = f.markdown.length > FETCH_PREVIEW_LIMIT
? f.markdown.slice(0, FETCH_PREVIEW_LIMIT) + "\n\n…[truncated — use ctx_search() for full content]"
: f.markdown;
return {
label: indexed.label,
totalChunks: indexed.totalChunks,
totalBytes: Buffer.byteLength(f.markdown),
preview,
};
}
server.registerTool(
"ctx_fetch_and_index",
{
title: "Fetch & Index URL",
title: "Fetch & Index URL(s)",
description:
"Fetches URL content, converts HTML to markdown, indexes into searchable knowledge base, " +
"and returns a ~3KB preview. Full content stays in sandbox — use ctx_search() for deeper lookups.\n\n" +
"Better than WebFetch: preview is immediate, full content is searchable, raw HTML never enters context.\n\n" +
"Content-type aware: HTML is converted to markdown, JSON is chunked by key paths, plain text is indexed directly.\n\n" +
"PARALLELIZE I/O: For multi-URL research (library evaluation, migration scans, doc comparisons), pass `requests: [{url, source}, ...]` with `concurrency: 4-8` — speeds up by 3-5x on real workloads.\n" +
" ✅ Use concurrency: 4-8 for: library docs sweep, multi-changelog scan, competitive pricing pages, multi-region docs, GitHub raw file pulls.\n" +
" ❌ Single URL → use the legacy {url, source} shape (concurrency irrelevant).\n" +
" Example: requests: [{url: 'https://react.dev/...', source: 'react'}, {url: 'https://vuejs.org/...', source: 'vue'}], concurrency: 5.\n" +
" Indexing is serial regardless of concurrency — fetches race, FTS5 writes don't (avoids SQLite WAL contention).\n\n" +
"When reporting results — terse like caveman. Technical substance exact. Only fluff die. Pattern: [thing] [action] [reason]. [next step].",
inputSchema: z.object({
url: z.string().describe("The URL to fetch and index"),
url: z.string().optional().describe("Single URL to fetch and index (legacy single-shape)"),
source: z
.string()
.optional()
.describe(
"Label for the indexed content (e.g., 'React useEffect docs', 'Supabase Auth API')",
"Label for the indexed content when using single `url` (e.g., 'React useEffect docs', 'Supabase Auth API'). For batch, put source in each requests entry.",
),
requests: z
.array(
z.object({
url: z.string().describe("URL to fetch"),
source: z.string().optional().describe("Label for this URL's indexed content"),
}),
)
.min(1)
.optional()
.describe(
"Batch shape: array of {url, source?} entries. Use with concurrency>1 for parallel fetch. " +
"Each request indexed under its own source label. Output preserves input order.",
),
concurrency: z
.coerce.number()
.int()
.min(1)
.max(8)
.optional()
.default(1)
.describe(
"Max URLs to fetch in parallel (1-8, default: 1). " +
"Use 4-8 for I/O-bound multi-URL batches (library docs, changelogs, pricing pages). " +
"Capped by os.cpus().length on small machines (response notes when capped). " +
"Indexing is always serial regardless — only fetches race.",
),
force: z
.boolean()
@@ -1719,141 +1791,166 @@ server.registerTool(
.describe("Skip cache and re-fetch even if content was recently indexed"),
}),
},
async ({ url, source, force }) => {
// TTL cache: if source was indexed within 24h, return cached hint.
// Cache key composes (source, url) so two distinct URLs sharing the same
// `source` label do not collide — they each get their own cache slot.
if (!force) {
const store = getStore();
const cacheKey = composeFetchCacheKey(source, url);
const meta = store.getSourceMeta(cacheKey);
if (meta) {
const indexedAt = new Date(meta.indexedAt + "Z"); // SQLite datetime is UTC without Z
const ageMs = Date.now() - indexedAt.getTime();
const TTL_MS = 24 * 60 * 60 * 1000; // 24 hours
if (ageMs < TTL_MS) {
const ageHours = Math.floor(ageMs / (60 * 60 * 1000));
const ageMin = Math.floor(ageMs / (60 * 1000));
const ageStr = ageHours > 0 ? `${ageHours}h ago` : ageMin > 0 ? `${ageMin}m ago` : "just now";
// Track cache savings — estimate ~1.6KB per chunk (average indexed content size)
const estimatedBytes = meta.chunkCount * 1600;
sessionStats.cacheHits++;
sessionStats.cacheBytesSaved += estimatedBytes;
return trackResponse("ctx_fetch_and_index", {
content: [{
type: "text" as const,
text: `Cached: **${meta.label}** — ${meta.chunkCount} sections, indexed ${ageStr} (fresh, TTL: 24h).\nTo refresh: call ctx_fetch_and_index again with \`force: true\`.\n\nYou MUST call ctx_search() to answer questions about this content — this cached response contains no content.\nUse: ctx_search(queries: [...], source: "${meta.label}")`,
}],
});
}
// Stale (>24h) — fall through to re-fetch silently
}
}
// Generate a unique temp file path for the subprocess to write fetched content.
// This bypasses the executor's 100KB stdout truncation — content goes file→handler directly.
const outputPath = join(tmpdir(), `ctx-fetch-${Date.now()}-${Math.random().toString(36).slice(2)}.dat`);
try {
const fetchCode = buildFetchCode(url, outputPath);
const result = await executor.execute({
language: "javascript",
code: fetchCode,
timeout: 30_000,
});
if (result.exitCode !== 0) {
return trackResponse("ctx_fetch_and_index", {
content: [
{
type: "text" as const,
text: `Failed to fetch ${url}: ${result.stderr || result.stdout}`,
},
],
isError: true,
});
}
// Parse content-type marker from stdout (content is in the temp file)
const store = getStore();
const header = (result.stdout || "").trim();
// Read full content from temp file
let markdown: string;
try {
markdown = readFileSync(outputPath, "utf-8").trim();
} catch {
return trackResponse("ctx_fetch_and_index", {
content: [
{
type: "text" as const,
text: `Fetched ${url} but could not read subprocess output`,
},
],
isError: true,
});
}
if (markdown.length === 0) {
return trackResponse("ctx_fetch_and_index", {
content: [
{
type: "text" as const,
text: `Fetched ${url} but got empty content`,
},
],
isError: true,
});
}
trackIndexed(Buffer.byteLength(markdown));
// Route to the appropriate indexing strategy based on Content-Type.
// Storage label includes URL via composeFetchCacheKey so two URLs sharing
// a `source` label do not overwrite each other; ctx_search() still finds
// both via LIKE-mode source filter on the `source` substring.
const storageLabel = composeFetchCacheKey(source, url);
let indexed: IndexResult;
if (header === "__CM_CT__:json") {
indexed = store.indexJSON(markdown, storageLabel);
} else if (header === "__CM_CT__:text") {
indexed = store.indexPlainText(markdown, storageLabel);
} else {
// HTML (default) — content is already converted to markdown
indexed = store.index({ content: markdown, source: storageLabel });
}
// Build preview — first ~3KB of markdown for immediate use
const PREVIEW_LIMIT = 3072;
const preview = markdown.length > PREVIEW_LIMIT
? markdown.slice(0, PREVIEW_LIMIT) + "\n\n…[truncated — use ctx_search() for full content]"
: markdown;
const totalKB = (Buffer.byteLength(markdown) / 1024).toFixed(1);
const text = [
`Fetched and indexed **${indexed.totalChunks} sections** (${totalKB}KB) from: ${indexed.label}`,
`Full content indexed in sandbox — use ctx_search(queries: [...], source: "${indexed.label}") for specific lookups.`,
"",
"---",
"",
preview,
].join("\n");
async ({ url, source, requests, concurrency, force }) => {
// Normalize input: legacy {url} or new {requests: [...]}.
// requests wins when both are provided (explicit batch intent).
const batch: { url: string; source?: string }[] = requests
? requests
: url
? [{ url, source }]
: [];
if (batch.length === 0) {
return trackResponse("ctx_fetch_and_index", {
content: [{ type: "text" as const, text }],
});
} catch (err: unknown) {
const message = err instanceof Error ? err.message : String(err);
return trackResponse("ctx_fetch_and_index", {
content: [
{ type: "text" as const, text: `Fetch error: ${message}` },
],
content: [{
type: "text" as const,
text: "ctx_fetch_and_index requires either `url` (single) or `requests: [{url, source?}, ...]` (batch).",
}],
isError: true,
});
} finally {
// Clean up temp file
try { rmSync(outputPath); } catch { /* already gone */ }
}
const isLegacySingle = !requests && batch.length === 1;
const requestedConcurrency = concurrency ?? 1;
// Parallel fetch via shared runPool primitive. capByCpuCount only for batch
// — single-URL doesn't need the cap (only one job, executor is one subprocess).
const jobs: PoolJob<FetchOneResult>[] = batch.map((req) => ({
run: () => fetchOneUrl(req.url, req.source, force),
}));
const { settled, effectiveConcurrency, capped } = await runPool(jobs, {
concurrency: requestedConcurrency,
capByCpuCount: !isLegacySingle && requestedConcurrency > 1,
});
// Serial index drain — workers race on fetch, but store.index* runs one at a time.
type Finalized =
| { kind: "cached"; label: string; chunkCount: number; ageStr: string }
| { kind: "fetched"; indexed: IndexedFetchResult }
| { kind: "fetch_error"; url: string; error: string; reason: "exit" | "read" | "empty" | "throw" }
| { kind: "job_error"; url: string; error: string };
const finalized: Finalized[] = [];
for (let i = 0; i < settled.length; i++) {
const r = settled[i];
if (r.status === "rejected") {
const message = r.reason instanceof Error ? r.reason.message : String(r.reason);
finalized.push({ kind: "job_error", url: batch[i].url, error: message });
continue;
}
const v = r.value;
if (v.kind === "cached") {
sessionStats.cacheHits++;
sessionStats.cacheBytesSaved += v.estimatedBytes;
finalized.push({ kind: "cached", label: v.label, chunkCount: v.chunkCount, ageStr: v.ageStr });
} else if (v.kind === "fetch_error") {
finalized.push({ kind: "fetch_error", url: v.url, error: v.error, reason: v.reason });
} else {
// Serial FTS5 write here — no parallel store.index calls.
finalized.push({ kind: "fetched", indexed: indexFetched(v) });
}
}
// Backward-compat single-URL response shape — preserve the EXACT original wording.
if (isLegacySingle) {
const r = finalized[0];
if (r.kind === "cached") {
return trackResponse("ctx_fetch_and_index", {
content: [{
type: "text" as const,
text: `Cached: **${r.label}** — ${r.chunkCount} sections, indexed ${r.ageStr} (fresh, TTL: 24h).\nTo refresh: call ctx_fetch_and_index again with \`force: true\`.\n\nYou MUST call ctx_search() to answer questions about this content — this cached response contains no content.\nUse: ctx_search(queries: [...], source: "${r.label}")`,
}],
});
}
if (r.kind === "fetched") {
const totalKB = (r.indexed.totalBytes / 1024).toFixed(1);
const text = [
`Fetched and indexed **${r.indexed.totalChunks} sections** (${totalKB}KB) from: ${r.indexed.label}`,
`Full content indexed in sandbox — use ctx_search(queries: [...], source: "${r.indexed.label}") for specific lookups.`,
"",
"---",
"",
r.indexed.preview,
].join("\n");
return trackResponse("ctx_fetch_and_index", {
content: [{ type: "text" as const, text }],
});
}
// fetch_error — preserve original error wording per reason
if (r.kind === "fetch_error") {
const text =
r.reason === "empty" ? `Fetched ${r.url} but got empty content`
: r.reason === "read" ? `Fetched ${r.url} but could not read subprocess output`
: r.reason === "exit" ? `Failed to fetch ${r.url}: ${r.error}`
: /* throw */ `Fetch error: ${r.error}`;
return trackResponse("ctx_fetch_and_index", {
content: [{ type: "text" as const, text }],
isError: true,
});
}
// job_error
return trackResponse("ctx_fetch_and_index", {
content: [{ type: "text" as const, text: `Fetch error: ${r.error}` }],
isError: true,
});
}
// Batch response — aggregated summary; isError only when EVERY URL failed.
// Per-URL preview capped tightly so a 8-URL batch doesn't undo the
// context-savings the tool exists to deliver (PRD review finding G1).
const FETCH_BATCH_PREVIEW_LIMIT = 384; // ~3KB total for 8-URL batches
const lines: string[] = [];
let totalSections = 0;
let totalBytes = 0;
let cachedCount = 0;
let fetchedCount = 0;
let errorCount = 0;
const snippets: string[] = [];
for (const r of finalized) {
if (r.kind === "cached") {
cachedCount++;
lines.push(`- [cache] ${r.label} — ${r.chunkCount} sections (${r.ageStr})`);
} else if (r.kind === "fetched") {
fetchedCount++;
totalSections += r.indexed.totalChunks;
totalBytes += r.indexed.totalBytes;
const kb = (r.indexed.totalBytes / 1024).toFixed(1);
lines.push(`- [new] ${r.indexed.label} — ${r.indexed.totalChunks} sections (${kb}KB)`);
const snippet = r.indexed.preview.length > FETCH_BATCH_PREVIEW_LIMIT
? r.indexed.preview.slice(0, FETCH_BATCH_PREVIEW_LIMIT).trimEnd() + "…"
: r.indexed.preview;
snippets.push(`### ${r.indexed.label}\n\n${snippet}`);
} else {
errorCount++;
lines.push(`- [err] ${r.url}: ${r.error}`);
}
}
const totalKB = (totalBytes / 1024).toFixed(1);
const cappedNote = capped
? ` cap=${effectiveConcurrency}/${cpus().length}cpu`
: "";
// Caveman style — terse status line: counts + sections + size.
// Singular forms used at count=1 to avoid grammar drift ("1 errors" → "1 error").
const fmt = (n: number, sing: string, plur: string) => `${n} ${n === 1 ? sing : plur}`;
const headerLine =
`fetched ${batch.length} c=${effectiveConcurrency}${cappedNote}. ` +
`ok=${fetchedCount} cache=${cachedCount} err=${errorCount}. ` +
`${fmt(totalSections, "section", "sections")} ${totalKB}KB.`;
const text = [
headerLine,
"",
...lines,
"",
`ctx_search(queries: [...], source: "<label>") for full content.`,
...(snippets.length > 0 ? ["", "---", "", ...snippets] : []),
].join("\n");
return trackResponse("ctx_fetch_and_index", {
content: [{ type: "text" as const, text }],
isError: errorCount === batch.length, // only mark error if every URL failed
});
},
);
@@ -2059,12 +2156,6 @@ server.registerTool(
async () => {
// ONE call, ONE source — AnalyticsEngine.queryAll()
let text: string;
// Lifetime stats (across all SessionDBs + auto-memory) — best-effort.
let lifetime;
try {
lifetime = getLifetimeStats({ sessionsDir: getSessionDir() });
} catch { /* ignore — formatReport tolerates undefined */ }
try {
const dbHash = hashProjectDir();
const worktreeSuffix = getWorktreeSuffix();
@@ -2079,7 +2170,9 @@ server.registerTool(
try {
const engine = new AnalyticsEngine(sdb);
const report = engine.queryAll(sessionStats);
text = formatReport(report, VERSION, _latestVersion, { lifetime });
// MCP usage is read-only and cheap; only available when DB exists.
const mcpUsage = engine.getMcpToolUsage();
text = formatReport(report, VERSION, _latestVersion, mcpUsage);
} finally {
sdb.close();
}
@@ -2087,13 +2180,13 @@ server.registerTool(
// No session DB — build a minimal report from runtime stats only
const engine = new AnalyticsEngine(createMinimalDb());
const report = engine.queryAll(sessionStats);
text = formatReport(report, VERSION, _latestVersion, { lifetime });
text = formatReport(report, VERSION, _latestVersion);
}
} catch {
// Session DB not available or incompatible — build minimal report from runtime stats
const engine = new AnalyticsEngine(createMinimalDb());
const report = engine.queryAll(sessionStats);
text = formatReport(report, VERSION, _latestVersion, { lifetime });
text = formatReport(report, VERSION, _latestVersion);
}
return trackResponse("ctx_stats", {
@@ -2407,23 +2500,15 @@ server.registerTool(
"First run installs dependencies (~30s). Subsequent runs open instantly.",
inputSchema: z.object({
port: z.coerce.number().optional().describe("Port to serve on (default: 4747)"),
sessionDir: z.string().optional().describe("Override INSIGHT_SESSION_DIR: directory containing context-mode session .db files"),
contentDir: z.string().optional().describe("Override INSIGHT_CONTENT_DIR: directory containing context-mode content/index .db files"),
insightSessionDir: z.string().optional().describe("Alias for sessionDir / INSIGHT_SESSION_DIR"),
insightContentDir: z.string().optional().describe("Alias for contentDir / INSIGHT_CONTENT_DIR"),
}),
},
async ({ port: userPort, sessionDir, contentDir: inputContentDir, insightSessionDir, insightContentDir }) => {
async ({ port: userPort }) => {
const port = userPort || 4747;
const userSessionDir = sessionDir || insightSessionDir;
const userContentDir = inputContentDir || insightContentDir;
// __pkg_dir is build/ for tsc, plugin root for bundle — resolve to plugin root
const pluginRoot = existsSync(resolve(__pkg_dir, "package.json")) ? __pkg_dir : dirname(__pkg_dir);
const insightSource = resolve(pluginRoot, "insight");
// Use adapter-aware path by default, with explicit overrides for hosts whose
// MCP adapter/session DB lives outside the detected default path.
const sessDir = userSessionDir ? resolve(userSessionDir) : getSessionDir();
const contentDir = userContentDir ? resolve(userContentDir) : join(dirname(sessDir), "content");
// Use adapter-aware path: derive from sessions dir (works across all 12 adapters)
const sessDir = getSessionDir();
const cacheDir = join(dirname(sessDir), "insight-cache");
// Verify source exists
@@ -2502,10 +2587,9 @@ server.registerTool(
// Port is free, proceed with spawn
}
if (portOccupied && (sourceUpdated || userSessionDir || userContentDir)) {
// Source or data-dir configuration changed while a server is already running —
// kill it so fresh code/env runs.
steps.push(sourceUpdated ? "Killing stale dashboard server (source updated)..." : "Killing existing dashboard server (data dir override)...");
if (portOccupied && sourceUpdated) {
// Source was updated but stale server is running on port — kill it so fresh code runs
steps.push("Killing stale dashboard server (source updated)...");
try {
if (process.platform === "win32") {
execSync(`for /f "tokens=5" %a in ('netstat -ano ^| findstr :${port}') do taskkill /F /PID %a`, { stdio: "pipe" });
@@ -2545,8 +2629,8 @@ server.registerTool(
env: {
...process.env,
PORT: String(port),
INSIGHT_SESSION_DIR: sessDir,
INSIGHT_CONTENT_DIR: contentDir,
INSIGHT_SESSION_DIR: getSessionDir(),
INSIGHT_CONTENT_DIR: join(dirname(getSessionDir()), "content"),
INSIGHT_PARENT_PID: String(process.pid),
},
detached: true,
@@ -2590,8 +2674,6 @@ server.registerTool(
else execSync(`xdg-open "${url}" 2>/dev/null || sensible-browser "${url}" 2>/dev/null`, { stdio: "pipe" });
} catch { /* browser open is best-effort */ }
if (userSessionDir) steps.push(`Session dir: ${sessDir}`);
if (userContentDir) steps.push(`Content dir: ${contentDir}`);
steps.push(`Dashboard running at ${url}`);
return trackResponse("ctx_insight", {
+111 -265
View File
@@ -9,11 +9,6 @@
* const report = engine.queryAll(runtimeStats);
*/
import { existsSync, readdirSync, statSync } from "node:fs";
import { join } from "node:path";
import { homedir } from "node:os";
import { loadDatabase as loadDatabaseImpl } from "../db-base.js";
function semverNewer(a: string, b: string): boolean {
const pa = a.split(".").map(Number);
const pb = b.split(".").map(Number);
@@ -67,6 +62,14 @@ export interface SandboxIO {
outputBytes: number;
}
/** MCP tool usage row — concurrency stats for batch-style tools. */
export interface McpToolUsageRow {
tool_name: string;
calls: number;
median_concurrency: number | null;
max_concurrency: number | null;
}
// ─────────────────────────────────────────────────────────
// Runtime stats — passed in from server.ts (can't come from DB)
// ─────────────────────────────────────────────────────────
@@ -243,6 +246,80 @@ export class AnalyticsEngine {
return { inputBytes, outputBytes };
}
/**
* MCP tool usage — call counts and concurrency stats per MCP tool.
*
* Reads `mcp_tool_call` events, parses the JSON payload, and aggregates:
* - call count per tool_name
* - median + max of `params.concurrency` (only for tools that take it,
* e.g. ctx_batch_execute, ctx_fetch_and_index). Returns null when the
* tool doesn't carry a concurrency param so callers can render N/A.
*
* Best-effort: malformed rows or truncated payloads are skipped silently.
*/
getMcpToolUsage(): McpToolUsageRow[] {
let rows: Array<{ data: string }>;
try {
rows = this.db.prepare(
"SELECT data FROM session_events WHERE category = 'mcp_tool_call'",
).all() as Array<{ data: string }>;
} catch {
return [];
}
// toolName -> { calls, concurrencies }
const agg = new Map<string, { calls: number; concurrencies: number[] }>();
for (const row of rows) {
let parsed: { tool_name?: unknown; params?: unknown; truncated?: unknown };
try {
parsed = JSON.parse(row.data);
} catch {
continue;
}
const toolName = typeof parsed.tool_name === "string" ? parsed.tool_name : null;
if (!toolName) continue;
const bucket = agg.get(toolName) ?? { calls: 0, concurrencies: [] };
bucket.calls += 1;
// Skip concurrency extraction when the row was truncated — the params
// blob is a substring of JSON that may not parse cleanly.
if (parsed.truncated !== true && parsed.params && typeof parsed.params === "object") {
const c = (parsed.params as Record<string, unknown>).concurrency;
if (typeof c === "number" && Number.isFinite(c) && c > 0) {
bucket.concurrencies.push(c);
}
}
agg.set(toolName, bucket);
}
const out: McpToolUsageRow[] = [];
for (const [tool_name, b] of agg) {
let median: number | null = null;
let max: number | null = null;
if (b.concurrencies.length > 0) {
const sorted = [...b.concurrencies].sort((a, c) => a - c);
const mid = Math.floor(sorted.length / 2);
median = sorted.length % 2 === 0
? (sorted[mid - 1] + sorted[mid]) / 2
: sorted[mid];
max = sorted[sorted.length - 1];
}
out.push({
tool_name,
calls: b.calls,
median_concurrency: median,
max_concurrency: max,
});
}
// Stable sort: most-called first, then alphabetical
out.sort((a, c) => c.calls - a.calls || a.tool_name.localeCompare(c.tool_name));
return out;
}
// ═══════════════════════════════════════════════════════
// queryAll — single unified report from ONE source
// ═══════════════════════════════════════════════════════
@@ -260,36 +337,11 @@ export class AnalyticsEngine {
).get() as { session_id: string } | undefined;
const sid = latestSession?.session_id ?? "";
// ── Hydrate runtime stats from persistent tool_calls table ──
// Bug #1 + #2: counters survive process restart, --continue, upgrade.
// The persistent values include the in-memory ones (we write to DB on
// every trackResponse), so REPLACE rather than ADD to avoid double-count.
const mergedCalls: Record<string, number> = { ...runtimeStats.calls };
const mergedBytes: Record<string, number> = { ...runtimeStats.bytesReturned };
if (sid) {
try {
const persistedRows = this.db.prepare(
"SELECT tool, calls, bytes_returned FROM tool_calls WHERE session_id = ?",
).all(sid) as Array<{ tool: string; calls: number; bytes_returned: number }>;
for (const row of persistedRows) {
// Take MAX so a fresh process (in-memory = 0) inherits prior totals,
// but a long-running process with more in-memory than DB still wins.
mergedCalls[row.tool] = Math.max(mergedCalls[row.tool] || 0, row.calls);
mergedBytes[row.tool] = Math.max(
mergedBytes[row.tool] || 0,
row.bytes_returned,
);
}
} catch {
// tool_calls table missing on older DBs — fall back to runtime only
}
}
// ── Runtime savings ──
const totalBytesReturned = Object.values(mergedBytes).reduce(
const totalBytesReturned = Object.values(runtimeStats.bytesReturned).reduce(
(sum, b) => sum + b, 0,
);
const totalCalls = Object.values(mergedCalls).reduce(
const totalCalls = Object.values(runtimeStats.calls).reduce(
(sum, c) => sum + c, 0,
);
const keptOut = runtimeStats.bytesIndexed + runtimeStats.bytesSandboxed;
@@ -300,14 +352,14 @@ export class AnalyticsEngine {
: 0;
const toolNames = new Set([
...Object.keys(mergedCalls),
...Object.keys(mergedBytes),
...Object.keys(runtimeStats.calls),
...Object.keys(runtimeStats.bytesReturned),
]);
const byTool = Array.from(toolNames).sort().map((tool) => ({
tool,
calls: mergedCalls[tool] || 0,
context_kb: Math.round((mergedBytes[tool] || 0) / 1024 * 10) / 10,
tokens: Math.round((mergedBytes[tool] || 0) / 4),
calls: runtimeStats.calls[tool] || 0,
context_kb: Math.round((runtimeStats.bytesReturned[tool] || 0) / 1024 * 10) / 10,
tokens: Math.round((runtimeStats.bytesReturned[tool] || 0) / 4),
}));
const uptimeMs = Date.now() - runtimeStats.sessionStart;
@@ -428,126 +480,6 @@ export class AnalyticsEngine {
}
}
// ─────────────────────────────────────────────────────────
// Lifetime stats (Bug #3 + #4)
// ─────────────────────────────────────────────────────────
/** Aggregated stats spanning every SessionDB + auto-memory under the user's profile. */
export interface LifetimeStats {
totalEvents: number;
totalSessions: number;
autoMemoryCount: number;
autoMemoryProjects: number;
/** Per-prefix breakdown of auto-memory files (user/feedback/project/...). */
autoMemoryByPrefix: Record<string, number>;
}
/** Extract leading prefix from auto-memory filename: `feedback_push.md` → `feedback`. */
function autoMemoryPrefix(filename: string): string {
const base = filename.replace(/\.md$/i, "");
const m = base.match(/^([a-z]+)/i);
return m ? m[1].toLowerCase() : "other";
}
/**
* Aggregate lifetime stats from all SessionDB files in `sessionsDir` and
* all auto-memory markdown files under `memoryRoot/<project>/memory/`.
*
* Best-effort: silently ignores missing/unreadable files so ctx_stats
* can never be broken by a corrupt sidecar.
*/
export function getLifetimeStats(opts?: {
sessionsDir?: string;
memoryRoot?: string;
/** Override for tests — defaults to db-base loadDatabase(). */
loadDatabase?: () => unknown;
}): LifetimeStats {
const sessionsDir = opts?.sessionsDir
?? join(homedir(), ".claude", "context-mode", "sessions");
const memoryRoot = opts?.memoryRoot
?? join(homedir(), ".claude", "projects");
let totalEvents = 0;
let totalSessions = 0;
// ── SessionDB aggregation ──
if (existsSync(sessionsDir)) {
let dbFiles: string[] = [];
try {
dbFiles = readdirSync(sessionsDir).filter((f) => f.endsWith(".db"));
} catch { /* unreadable */ }
if (dbFiles.length > 0) {
// Lazy-load better-sqlite3 / bun-sqlite via the same path the runtime uses.
let DatabaseCtor: ReturnType<typeof loadDatabaseImpl> | null = null;
try {
DatabaseCtor = opts?.loadDatabase
? (opts.loadDatabase() as ReturnType<typeof loadDatabaseImpl>)
: loadDatabaseImpl();
} catch { /* sqlite unavailable */ }
if (DatabaseCtor) {
for (const file of dbFiles) {
const dbPath = join(sessionsDir, file);
try {
const sdb = new DatabaseCtor(dbPath, { readonly: true });
try {
const ev = sdb.prepare("SELECT COUNT(*) AS cnt FROM session_events").get() as { cnt: number } | undefined;
const ss = sdb.prepare("SELECT COUNT(*) AS cnt FROM session_meta").get() as { cnt: number } | undefined;
totalEvents += ev?.cnt ?? 0;
totalSessions += ss?.cnt ?? 0;
} finally {
sdb.close();
}
} catch {
// missing tables / corrupt file — skip
}
}
}
}
}
// ── Auto-memory file scan ──
let autoMemoryCount = 0;
let autoMemoryProjects = 0;
const autoMemoryByPrefix: Record<string, number> = {};
if (existsSync(memoryRoot)) {
let projectDirs: string[] = [];
try {
projectDirs = readdirSync(memoryRoot).filter((entry) => {
try {
return statSync(join(memoryRoot, entry)).isDirectory();
} catch { return false; }
});
} catch { /* unreadable */ }
for (const proj of projectDirs) {
const memDir = join(memoryRoot, proj, "memory");
if (!existsSync(memDir)) continue;
let mdFiles: string[] = [];
try {
mdFiles = readdirSync(memDir).filter((f) => f.endsWith(".md"));
} catch { continue; }
if (mdFiles.length === 0) continue;
autoMemoryProjects++;
autoMemoryCount += mdFiles.length;
for (const f of mdFiles) {
const prefix = autoMemoryPrefix(f);
autoMemoryByPrefix[prefix] = (autoMemoryByPrefix[prefix] ?? 0) + 1;
}
}
}
return {
totalEvents,
totalSessions,
autoMemoryCount,
autoMemoryProjects,
autoMemoryByPrefix,
};
}
// ─────────────────────────────────────────────────────────
// formatReport — renders FullReport as sales-grade savings dashboard
// ─────────────────────────────────────────────────────────
@@ -576,19 +508,6 @@ function fmtNum(n: number): string {
return String(n);
}
// ─────────────────────────────────────────────────────────
// Pricing (Bug #6) — Anthropic Opus input rate
// ─────────────────────────────────────────────────────────
/** Opus 4 input price: $15 per 1M tokens. */
export const OPUS_INPUT_PRICE_PER_TOKEN = 15 / 1_000_000;
/** Convert a token count to a USD string at the Opus input rate. */
export function tokensToUsd(tokens: number): string {
const safe = Number.isFinite(tokens) && tokens > 0 ? tokens : 0;
return `$${(safe * OPUS_INPUT_PRICE_PER_TOKEN).toFixed(2)}`;
}
/**
* Build a proportional bar using █ chars, scaled to a fixed width.
* Returns e.g. "████████████████████████████████████████" for full width.
@@ -601,82 +520,19 @@ function dataBar(bytes: number, maxBytes: number, width: number = 40): string {
/**
* Render project memory section with category bars.
*
* Shows persistent event data, and \u2014 when supplied \u2014 lifetime totals
* across every project's SessionDB so users see the cumulative value
* (Bug #3).
*
* Caps the category list at `topN` and prints "N more categories" with the
* actual remaining count (Bug #5 \u2014 was hardcoded "9 more").
* Shows persistent event data across all sessions.
*/
function renderProjectMemory(
pm: FullReport["projectMemory"],
opts?: { lifetime?: LifetimeStats; topN?: number },
): string[] {
if (pm.total_events === 0 && (opts?.lifetime?.totalEvents ?? 0) === 0) return [];
function renderProjectMemory(pm: FullReport["projectMemory"]): string[] {
if (pm.total_events === 0) return [];
const out: string[] = [];
const topN = opts?.topN ?? 2;
out.push("");
out.push(`Persistent memory \u2713 preserved across compact, restart & upgrade`);
// Lifetime line (Bug #3) \u2014 collapses to project-only when lifetime missing.
const lifeEvents = opts?.lifetime?.totalEvents ?? pm.total_events;
const lifeSessions = opts?.lifetime?.totalSessions ?? pm.session_count;
const sessionLabel = lifeSessions === 1 ? "1 session" : `${fmtNum(lifeSessions)} sessions`;
// Estimate lifetime savings: ~4 bytes/token (~1KB/event) \u2192 at Opus rates.
const lifetimeTokens = lifeEvents * 256; // ~1KB per event / 4 bytes per token
out.push(` ${fmtNum(lifeEvents)} events \u00b7 ${sessionLabel} \u00b7 ~${tokensToUsd(lifetimeTokens)} saved lifetime`);
const sessionLabel = pm.session_count === 1 ? "1 session" : `${pm.session_count} sessions`;
out.push(`${fmtNum(pm.total_events)} events remembered across ${sessionLabel} \u2014 searchable after compact & restart`);
out.push("");
const cats = pm.by_category;
const visible = cats.slice(0, topN);
const maxCount = visible.length > 0 ? visible[0].count : 1;
for (const cat of visible) {
const maxCount = pm.by_category.length > 0 ? pm.by_category[0].count : 1;
for (const cat of pm.by_category) {
out.push(` ${cat.label.padEnd(18)} ${String(cat.count).padStart(5)} ${dataBar(cat.count, maxCount, 30)}`);
}
// Bug #5: real overflow count, not hardcoded.
const remaining = Math.max(0, cats.length - topN);
if (remaining > 0) {
out.push(` ... ${remaining} more categor${remaining === 1 ? "y" : "ies"}`);
}
return out;
}
/**
* Render the auto-memory section (Bug #4) \u2014 files Claude Code captured
* under ~/.claude/projects/<project>/memory/ across the user's machine.
*/
function renderAutoMemory(lifetime: LifetimeStats | undefined): string[] {
if (!lifetime || lifetime.autoMemoryCount === 0) return [];
const out: string[] = [];
out.push("");
out.push(
`Auto-memory \u2713 ${lifetime.autoMemoryCount} preference${lifetime.autoMemoryCount === 1 ? "" : "s"} learned across ${lifetime.autoMemoryProjects} project${lifetime.autoMemoryProjects === 1 ? "" : "s"}`,
);
const entries = Object.entries(lifetime.autoMemoryByPrefix)
.sort((a, b) => b[1] - a[1])
.slice(0, 6);
for (const [prefix, count] of entries) {
out.push(` ${prefix.padEnd(12)} ${String(count).padStart(2)}`);
}
return out;
}
/** Render the closing "Bottom line" footer (Bug #8). */
function renderBottomLine(sessionTokensSaved: number, lifetime: LifetimeStats | undefined): string[] {
const out: string[] = [];
const sessionUsd = tokensToUsd(sessionTokensSaved);
// Lifetime estimate: ~1KB/event \u00f7 4 bytes/token = 256 tokens/event.
const lifetimeTokens = (lifetime?.totalEvents ?? 0) * 256;
const lifetimeUsd = tokensToUsd(lifetimeTokens);
out.push("");
out.push("\u2500".repeat(65));
out.push("Your AI talks less, remembers more, costs less.");
out.push(`${sessionUsd} this session, ${lifetimeUsd} lifetime, and counting.`);
out.push("\u2500".repeat(65));
return out;
}
@@ -694,11 +550,10 @@ export function formatReport(
report: FullReport,
version?: string,
latestVersion?: string | null,
opts?: { lifetime?: LifetimeStats },
mcpUsage?: McpToolUsageRow[],
): string {
const lines: string[] = [];
const duration = formatDuration(report.session.uptime_min);
const lifetime = opts?.lifetime;
// ── Compute real savings ──
const totalKeptOut =
@@ -708,9 +563,6 @@ export function formatReport(
const grandTotal = totalKeptOut + totalReturned;
const savingsPct = grandTotal > 0 ? (totalKeptOut / grandTotal) * 100 : 0;
const tokensSaved = Math.round(totalKeptOut / 4);
const ratioMultiplier = totalReturned > 0
? Math.max(1, Math.round(grandTotal / Math.max(totalReturned, 1)))
: 0;
// ── Fresh session: no savings yet ──
if (totalKeptOut === 0) {
@@ -724,9 +576,7 @@ export function formatReport(
}
// Project memory
lines.push(...renderProjectMemory(report.projectMemory, { lifetime }));
lines.push(...renderAutoMemory(lifetime));
lines.push(...renderBottomLine(0, lifetime));
lines.push(...renderProjectMemory(report.projectMemory));
// Footer
lines.push("");
@@ -741,10 +591,7 @@ export function formatReport(
// ── Active session: visual savings dashboard ──
// Line 1: Hero metric — the screenshottable number
// Bug #6: include Opus pricing on the hero line for credibility.
lines.push(
`${fmtNum(tokensSaved)} tokens saved · ${savingsPct.toFixed(1)}% reduction · ${duration} · ~${tokensToUsd(tokensSaved)} saved (Opus)`,
);
lines.push(`${fmtNum(tokensSaved)} tokens saved · ${savingsPct.toFixed(1)}% reduction · ${duration}`);
lines.push("");
// Lines 2-3: Before/After comparison bars — the visual proof
@@ -753,12 +600,7 @@ export function formatReport(
lines.push("");
// Value statement — the line people share
// Bug #7: replace meaningless "3.0x" ratio with "3× longer sessions".
if (ratioMultiplier >= 2) {
lines.push(`${kb(totalKeptOut)} kept out of your conversation — ${ratioMultiplier}× longer sessions before compact.`);
} else {
lines.push(`${kb(totalKeptOut)} kept out of your conversation. Never entered context.`);
}
lines.push(`${kb(totalKeptOut)} kept out of your conversation. Never entered context.`);
lines.push("");
// Compact stats row
@@ -790,14 +632,18 @@ export function formatReport(
}
}
// ── Project memory — persistent across sessions (Bug #3 + #5) ──
lines.push(...renderProjectMemory(report.projectMemory, { lifetime }));
// ── MCP concurrency usage (only when batch tools recorded a concurrency) ──
if (mcpUsage && mcpUsage.length > 0) {
const concurrent = mcpUsage.filter((u) => u.median_concurrency != null);
for (const u of concurrent) {
lines.push(
`MCP concurrency usage: ${u.tool_name} median=${u.median_concurrency} max=${u.max_concurrency} (${u.calls} calls)`,
);
}
}
// ── Auto-memory — Claude Code's preference learnings (Bug #4) ──
lines.push(...renderAutoMemory(lifetime));
// ── Bottom line — business value framing (Bug #8) ──
lines.push(...renderBottomLine(tokensSaved, lifetime));
// ── Project memory — persistent across sessions ──
lines.push(...renderProjectMemory(report.projectMemory));
// ── Footer ──
lines.push("");
+62
View File
@@ -509,6 +509,67 @@ function extractMcp(input: HookInput): SessionEvent[] {
}];
}
/**
* Category 27: mcp_tool_call
* Records the raw MCP call shape (tool_name + tool_input) so analytics
* can compute usage patterns like batch concurrency.
*
* Distinct from `extractMcp` (category "mcp"), which captures the textual
* call+response for FTS5 search. This emits a structured JSON payload
* keyed by tool_name + params, capped to ~2KB to keep SQLite rows small.
*
* Priority 4 (informational) — should not crowd out high-signal events
* during FIFO eviction.
*/
const MCP_PARAMS_BUDGET_BYTES = 2048;
/**
* UTF-8-aware string truncation. Returns the longest prefix of `s` whose
* UTF-8 byte length is <= `maxBytes`, never landing mid-multibyte-codepoint.
*
* Naive `s.slice(0, N)` operates on UTF-16 code units, so a 2KB cap could
* either over-shoot (multi-byte codepoints occupy fewer code units than
* bytes — e.g. a chunk of CJK / emoji-heavy JSON would silently exceed
* the byte budget) or land mid surrogate pair (corrupt JSON downstream).
*/
function truncateToBytes(s: string, maxBytes: number): { value: string; truncated: boolean } {
if (Buffer.byteLength(s, "utf8") <= maxBytes) return { value: s, truncated: false };
const buf = Buffer.from(s, "utf8");
// Walk back from maxBytes until the byte starts a fresh codepoint:
// 0xxxxxxx → ASCII (start)
// 11xxxxxx → start of multi-byte
// 10xxxxxx → continuation; keep walking
let cut = maxBytes;
while (cut > 0 && (buf[cut] & 0xc0) === 0x80) cut--;
return { value: buf.subarray(0, cut).toString("utf8"), truncated: true };
}
function extractMcpToolCall(input: HookInput): SessionEvent[] {
const { tool_name, tool_input } = input;
if (!tool_name.startsWith("mcp__")) return [];
// Serialize params, then truncate the *string* (not the object) so the
// shape stays diagnosable even when the payload is huge.
let paramsStr: string;
try {
paramsStr = JSON.stringify(tool_input ?? {});
} catch {
paramsStr = "{}";
}
const { value: cappedStr, truncated } = truncateToBytes(paramsStr, MCP_PARAMS_BUDGET_BYTES);
const payload = truncated
? `{"tool_name":${JSON.stringify(tool_name)},"params_raw":${JSON.stringify(cappedStr)},"truncated":true}`
: `{"tool_name":${JSON.stringify(tool_name)},"params":${cappedStr}}`;
return [{
type: "mcp_tool_call",
category: "mcp_tool_call",
data: safeString(payload),
priority: 4,
}];
}
/**
* Category 6 (tool-based): decision
* AskUserQuestion tool — tracks questions posed to user and their answers.
@@ -899,6 +960,7 @@ export function extractEvents(input: HookInput): SessionEvent[] {
events.push(...extractSkill(input));
events.push(...extractSubagent(input));
events.push(...extractMcp(input));
events.push(...extractMcpToolCall(input));
events.push(...extractDecision(input));
events.push(...extractConstraint(input));
events.push(...extractWorktree(input));
+63
View File
@@ -254,4 +254,67 @@ describe("Analytics Metrics", () => {
expect(report.continuity.compact_count).toBe(0);
});
});
// ─── MCP tool usage ────────────────────────────────────
describe("getMcpToolUsage", () => {
it("returns median+max concurrency for batch tools", () => {
// Insert mcp_tool_call rows with varied concurrency values for the same
// tool and one row for a tool without a concurrency param.
const concurrencies = [4, 8, 6, 8]; // median = (6+8)/2 = 7, max = 8
for (const c of concurrencies) {
insertEvent(db, {
session_id: SESSION_ID,
type: "mcp_tool_call",
category: "mcp_tool_call",
priority: 4,
data: JSON.stringify({
tool_name: "mcp__context-mode__ctx_batch_execute",
params: { commands: [], concurrency: c },
}),
});
}
// Tool without a concurrency param — should report nulls
insertEvent(db, {
session_id: SESSION_ID,
type: "mcp_tool_call",
category: "mcp_tool_call",
priority: 4,
data: JSON.stringify({
tool_name: "mcp__context-mode__ctx_search",
params: { queries: ["foo"] },
}),
});
// Truncated row — must be counted as a call but skipped for concurrency
insertEvent(db, {
session_id: SESSION_ID,
type: "mcp_tool_call",
category: "mcp_tool_call",
priority: 4,
data: JSON.stringify({
tool_name: "mcp__context-mode__ctx_batch_execute",
params_raw: '{"commands":[{"label":"x"',
truncated: true,
}),
});
const usage = engine.getMcpToolUsage();
const batch = usage.find((u) => u.tool_name === "mcp__context-mode__ctx_batch_execute");
expect(batch).toBeDefined();
expect(batch!.calls).toBe(5); // 4 normal + 1 truncated
expect(batch!.median_concurrency).toBe(7);
expect(batch!.max_concurrency).toBe(8);
const search = usage.find((u) => u.tool_name === "mcp__context-mode__ctx_search");
expect(search).toBeDefined();
expect(search!.calls).toBe(1);
expect(search!.median_concurrency).toBeNull();
expect(search!.max_concurrency).toBeNull();
});
it("returns empty array when no mcp_tool_call events exist", () => {
expect(engine.getMcpToolUsage()).toEqual([]);
});
});
});
+310 -15
View File
@@ -2311,6 +2311,300 @@ describe("runBatchCommands edge cases", () => {
});
});
// ═══════════════════════════════════════════════════════════════════════════
// runBatchCommands hardening — P0 fixes per PRD-concurrency-architectural §0
// ═══════════════════════════════════════════════════════════════════════════
describe("runBatchCommands P0 hardening", () => {
test("finding A: executor throw is isolated, siblings complete", async () => {
// One worker's executor.execute() throws (e.g. spawn EAGAIN under load).
// Without try/catch, Promise.all would reject and strand sibling outputs
// as `undefined`, surfacing as the literal "undefined" after .join("\n").
const exec = mkMockExecutor((code) => {
if (code.includes("boom")) throw new Error("spawn EAGAIN");
return { stdout: code.includes("a") ? "alpha" : code.includes("b") ? "beta" : "gamma" };
});
const cmds: BatchCommand[] = [
{ label: "A", command: "echo a" },
{ label: "BOOM", command: "echo boom" },
{ label: "B", command: "echo b" },
{ label: "C", command: "echo c" },
];
const { outputs } = await runBatchCommands(
cmds,
{ timeout: 5000, concurrency: 4, nodeOptsPrefix: NOOP_PREFIX },
exec,
);
expect(outputs).toHaveLength(4);
expect(outputs[0]).toContain("# A");
expect(outputs[0]).toContain("alpha");
expect(outputs[1]).toContain("# BOOM");
expect(outputs[1]).toContain("(executor error: spawn EAGAIN)");
expect(outputs[2]).toContain("beta");
expect(outputs[3]).toContain("gamma");
// Critically: no `undefined` slots
expect(outputs.every((o) => typeof o === "string" && o.length > 0)).toBe(true);
});
test("finding B: timed-out parallel command still strips __CM_FS__ markers + counts bytes", async () => {
// Real subprocess timeouts often return partial stdout *with* the marker.
// Pre-fix the parallel branch wrote the (timed out) sentinel directly,
// bypassing formatCommandOutput → markers leaked into context, bytes uncounted.
const exec = mkMockExecutor(() => ({
stdout: "partial line 1\n__CM_FS__:512\npartial line 2\n",
timedOut: true,
}));
let totalBytes = 0;
const { outputs, timedOut } = await runBatchCommands(
[{ label: "SLOW", command: "x" }],
{ timeout: 100, concurrency: 2, nodeOptsPrefix: NOOP_PREFIX, onFsBytes: (b) => { totalBytes += b; } },
exec,
);
expect(timedOut).toBe(true);
expect(totalBytes).toBe(512); // marker counted
expect(outputs[0]).not.toContain("__CM_FS__"); // marker stripped
expect(outputs[0]).toContain("partial line 1");
expect(outputs[0]).toContain("partial line 2");
expect(outputs[0]).toContain("(timed out after 100ms)"); // sentinel still appended
});
test("finding D: timing-regression — 5 cmds × 100ms at concurrency=5 finishes in <200ms", async () => {
// Replaces the deleted bench (CONTRIBUTING.md L275 forbids new test files).
// Asserts ≥3× speedup over serial. CI-checked.
const exec = mkMockExecutor(async () => {
await new Promise((r) => setTimeout(r, 100));
return { stdout: "ok" };
});
const cmds: BatchCommand[] = Array.from({ length: 5 }, (_, i) => ({
label: `C${i}`,
command: "x",
}));
const start = Date.now();
const { outputs, timedOut } = await runBatchCommands(
cmds,
{ timeout: 5000, concurrency: 5, nodeOptsPrefix: NOOP_PREFIX },
exec,
);
const elapsed = Date.now() - start;
expect(timedOut).toBe(false);
expect(outputs).toHaveLength(5);
// Serial would be ~500ms (5×100). Parallel should be ~100ms + overhead.
// Threshold 200ms gives generous CI room while still catching a regression to serial.
expect(elapsed).toBeLessThan(200);
});
});
// ═══════════════════════════════════════════════════════════════════════════
// runPool — shared concurrency primitive (PRD finding G)
// ═══════════════════════════════════════════════════════════════════════════
import { runPool, type PoolJob } from "../../src/concurrency/runPool.js";
describe("runPool primitive", () => {
test("empty jobs returns empty settled array", async () => {
const { settled, effectiveConcurrency, capped } = await runPool([], { concurrency: 4 });
expect(settled).toHaveLength(0);
expect(effectiveConcurrency).toBe(0);
expect(capped).toBe(false);
});
test("happy path: order preserved, all fulfilled", async () => {
const jobs: PoolJob<number>[] = [10, 20, 30, 40].map((v, i) => ({
run: async () => {
await new Promise((r) => setTimeout(r, (4 - i) * 10)); // reverse-order delay
return v;
},
}));
const { settled, effectiveConcurrency } = await runPool(jobs, { concurrency: 4 });
expect(effectiveConcurrency).toBe(4);
expect(settled.map((s) => s.status === "fulfilled" ? s.value : null)).toEqual([10, 20, 30, 40]);
});
test("throw isolation: one job rejects, siblings still fulfill", async () => {
const jobs: PoolJob<string>[] = [
{ run: async () => "a" },
{ run: async () => { throw new Error("boom"); } },
{ run: async () => "c" },
];
const { settled } = await runPool(jobs, { concurrency: 3 });
expect(settled[0]).toEqual({ status: "fulfilled", value: "a" });
expect(settled[1].status).toBe("rejected");
expect((settled[1] as { reason: Error }).reason.message).toBe("boom");
expect(settled[2]).toEqual({ status: "fulfilled", value: "c" });
});
test("in-flight cap: never exceeds concurrency", async () => {
let inFlight = 0;
let maxInFlight = 0;
const jobs: PoolJob<void>[] = Array.from({ length: 10 }, () => ({
run: async () => {
inFlight++;
maxInFlight = Math.max(maxInFlight, inFlight);
await new Promise((r) => setTimeout(r, 20));
inFlight--;
},
}));
const { effectiveConcurrency, capped } = await runPool(jobs, { concurrency: 3 });
expect(maxInFlight).toBeLessThanOrEqual(3);
expect(maxInFlight).toBeGreaterThanOrEqual(2); // proves at least some parallelism
expect(effectiveConcurrency).toBe(3);
expect(capped).toBe(false);
});
test("auto-clamp to job count when concurrency > jobs.length", async () => {
const jobs: PoolJob<number>[] = [{ run: async () => 1 }, { run: async () => 2 }];
const { effectiveConcurrency, capped } = await runPool(jobs, { concurrency: 8 });
expect(effectiveConcurrency).toBe(2);
expect(capped).toBe(true);
});
test("capByCpuCount caps by os.cpus().length", async () => {
// We can't predict the test runner's cpu count, so just assert the bounds.
const jobs: PoolJob<number>[] = Array.from({ length: 32 }, (_, i) => ({ run: async () => i }));
const { effectiveConcurrency, capped } = await runPool(jobs, { concurrency: 32, capByCpuCount: true });
const cores = require("node:os").cpus().length;
expect(effectiveConcurrency).toBeLessThanOrEqual(cores);
expect(effectiveConcurrency).toBeLessThanOrEqual(32);
expect(capped).toBe(effectiveConcurrency < 32);
});
test("onSettled callback fires per job in completion order", async () => {
const events: number[] = [];
const jobs: PoolJob<number>[] = [
{ run: async () => { await new Promise((r) => setTimeout(r, 30)); return 0; } },
{ run: async () => { await new Promise((r) => setTimeout(r, 10)); return 1; } },
{ run: async () => { await new Promise((r) => setTimeout(r, 20)); return 2; } },
];
await runPool(jobs, { concurrency: 3, onSettled: (idx) => { events.push(idx); } });
// Job 1 (10ms) completes first, then 2 (20ms), then 0 (30ms)
expect(events).toEqual([1, 2, 0]);
});
});
// ═══════════════════════════════════════════════════════════════════════════
// ctx_fetch_and_index batch path — schema + handler-level checks
// (Full subprocess fetch tested in tests/mcp-integration.ts;
// these tests verify schema acceptance + serial-index contract via source-level read.)
// ═══════════════════════════════════════════════════════════════════════════
describe("ctx_fetch_and_index batch refactor", () => {
const fetchHandlerSrc = readFileSync(
resolve(__dirname, "../../src/server.ts"),
"utf-8",
);
test("schema accepts both legacy {url} and batch {requests}", () => {
expect(fetchHandlerSrc).toContain('url: z.string().optional()');
expect(fetchHandlerSrc).toContain('requests: z');
// Zod array of {url, source?}
expect(fetchHandlerSrc).toContain("z.object({\n url: z.string()");
expect(fetchHandlerSrc).toContain('source: z.string().optional()');
});
test("handler exposes concurrency 1-8 with default 1", () => {
// Find the fetch_and_index registerTool block, then assert concurrency schema near it.
// Stop anchor: the next registerTool call (ctx_batch_execute).
const fetchBlockMatch = fetchHandlerSrc.match(/registerTool\(\s*"ctx_fetch_and_index"[\s\S]+?registerTool\(\s*"ctx_batch_execute"/);
expect(fetchBlockMatch).not.toBeNull();
const block = fetchBlockMatch![0];
expect(block).toContain("concurrency: z");
expect(block).toMatch(/\.min\(1\)\s*\n?\s*\.max\(8\)/);
expect(block).toContain(".default(1)");
});
test("PARALLELIZE I/O guidance + locked requests:[] schema in description", () => {
expect(fetchHandlerSrc).toContain("PARALLELIZE I/O");
expect(fetchHandlerSrc).toContain("requests: [{url, source}");
expect(fetchHandlerSrc).toContain("3-5x");
expect(fetchHandlerSrc).toContain("✅");
expect(fetchHandlerSrc).toContain("❌");
});
test("serial-write contract: index drain is a for-loop calling indexFetched serially", () => {
// The handler must NOT spawn parallel store.index calls. The drain is a
// for-loop over `settled` calling indexFetched serially. Anti-pattern check.
expect(fetchHandlerSrc).toContain("Serial index drain");
expect(fetchHandlerSrc).toContain("indexFetched(v)");
// No `await Promise.all(... indexFetched ...)` pattern anywhere
expect(fetchHandlerSrc).not.toMatch(/Promise\.all\([^)]*indexFetched/);
});
test("backward compat: legacy single-URL response wording preserved", () => {
// Original handler returned "Cached: **${label}**" / "Fetched and indexed **N sections**"
// The refactor must keep these EXACT strings for the legacy path so
// tests/mcp-integration.ts and any user-side scripts grepping the response don't break.
expect(fetchHandlerSrc).toContain("Cached: **${r.label}**");
expect(fetchHandlerSrc).toContain("Fetched and indexed **${r.indexed.totalChunks} sections**");
// The source escapes backticks inside a template literal — match the escaped form.
expect(fetchHandlerSrc).toContain("To refresh: call ctx_fetch_and_index again with");
expect(fetchHandlerSrc).toContain("force: true");
});
test("isLegacySingle gate prevents batch response wrapping for single-URL calls", () => {
expect(fetchHandlerSrc).toContain("const isLegacySingle = !requests && batch.length === 1");
expect(fetchHandlerSrc).toContain("if (isLegacySingle)");
});
test("capped-concurrency note appears only when capped", () => {
expect(fetchHandlerSrc).toMatch(/cappedNote\s*=\s*capped\s*\?/);
// Caveman style — `cap=N/Mcpu` instead of "capped from N to M; M cores available".
expect(fetchHandlerSrc).toContain("cap=${effectiveConcurrency}/${cpus().length}cpu");
});
test("batch isError only when ALL URLs fail (errorCount === batch.length)", () => {
expect(fetchHandlerSrc).toContain("isError: errorCount === batch.length");
});
test("batch preview is capped to prevent context flooding (review F2)", () => {
// Per-URL preview in batch mode capped tightly so an 8-URL batch doesn't
// dump ~24KB of context (8 × 3072 char single-URL preview cap).
expect(fetchHandlerSrc).toContain("FETCH_BATCH_PREVIEW_LIMIT");
// Cap value must be ≤500 chars (8 URLs × 500 = ~4KB max snippets total)
const limitMatch = fetchHandlerSrc.match(/FETCH_BATCH_PREVIEW_LIMIT\s*=\s*(\d+)/);
expect(limitMatch).not.toBeNull();
expect(parseInt(limitMatch![1])).toBeLessThanOrEqual(500);
// Must actually be applied to per-URL previews in the batch loop
expect(fetchHandlerSrc).toMatch(/preview\.length\s*>\s*FETCH_BATCH_PREVIEW_LIMIT/);
});
test("batch header uses singular form for count=1 (review F5 plural fix)", () => {
// Per CLAUDE.md "Terse like caveman" + grammar correctness:
// "1 errors" → "1 error" via the fmt() helper.
expect(fetchHandlerSrc).toContain('const fmt = (n: number, sing: string, plur: string)');
expect(fetchHandlerSrc).toContain('n === 1 ? sing : plur');
});
test("batch header uses caveman style (review F5 terse format)", () => {
// Old: "Batch fetched N URLs at concurrency=X (capped from Y to X; Z cores available): a fetched, b cached, c errors. d new sections (eKB total)."
// New: "fetched N c=X cap=X/Zcpu. ok=a cache=b err=c. d sections eKB."
expect(fetchHandlerSrc).toContain("`fetched ${batch.length} c=${effectiveConcurrency}");
expect(fetchHandlerSrc).toContain("ok=${fetchedCount} cache=${cachedCount} err=${errorCount}");
expect(fetchHandlerSrc).not.toContain("Batch fetched"); // old verbose wording gone
});
test("fetchOneUrl is parallel-safe (no SQLite writes)", () => {
// Verify by inspecting the helper source: it calls store.getSourceMeta (read)
// but never store.index/indexJSON/indexPlainText (writes).
const fetchOneSrc = fetchHandlerSrc.match(/async function fetchOneUrl\([\s\S]+?^}/m);
expect(fetchOneSrc).not.toBeNull();
const block = fetchOneSrc![0];
expect(block).toContain("store.getSourceMeta"); // read OK
expect(block).not.toContain("store.index"); // no writes
expect(block).not.toContain("store.indexJSON");
expect(block).not.toContain("store.indexPlainText");
});
test("indexFetched is serial-only (single FTS5 write per call)", () => {
const indexFetchedSrc = fetchHandlerSrc.match(/function indexFetched\([\s\S]+?^}/m);
expect(indexFetchedSrc).not.toBeNull();
const block = indexFetchedSrc![0];
// Has exactly one of: store.index / store.indexJSON / store.indexPlainText per branch
expect(block).toContain("store.indexJSON");
expect(block).toContain("store.indexPlainText");
expect(block).toContain("store.index");
});
});
// ═══════════════════════════════════════════════════════════════════════════
// ctx_doctor resource cleanup regression (#247)
// ═══════════════════════════════════════════════════════════════════════════
@@ -2528,24 +2822,25 @@ describe("ctx_fetch_and_index cache key includes URL (Fix 6/10)", () => {
resolve(__dirname, "../../src/server.ts"),
"utf-8",
);
// Locate the ctx_fetch_and_index handler block
const block = serverSrc.match(
/registerTool\(\s*"ctx_fetch_and_index"[\s\S]*?^\);/m,
);
expect(block, "ctx_fetch_and_index handler not found").not.toBeNull();
const body = block![0];
// The cache lookup may live in the handler block OR in an extracted helper
// (post-refactor: `fetchOneUrl` is the parallel-safe fetcher invoked by both
// single-URL and batch paths). Either location must use composeFetchCacheKey,
// not the bare label/url variable.
// Cache lookup must call getSourceMeta with a key composed from label+url,
// not the bare label/url. The fix uses composeFetchCacheKey().
const lookupCall = body.match(
/getSourceMeta\(\s*([^)]+)\s*\)/,
);
// composeFetchCacheKey must be imported and referenced
expect(serverSrc).toContain('from "./fetch-cache.js"');
expect(serverSrc).toContain("composeFetchCacheKey");
// Find ANY getSourceMeta call across the file
const lookupCall = serverSrc.match(/getSourceMeta\(\s*([^)]+)\s*\)/);
expect(lookupCall, "getSourceMeta call missing").not.toBeNull();
const arg = lookupCall![1];
const arg = lookupCall![1].trim();
// Must NOT be the bare `label` variable (that was the bug).
expect(arg.trim()).not.toBe("label");
// Must reference both the label and the url, ideally via composeFetchCacheKey.
expect(body).toContain("composeFetchCacheKey");
expect(arg).not.toBe("label");
// Argument must be a key derived from composition (`cacheKey`, `storageLabel`,
// or a direct `composeFetchCacheKey(...)` call). Reject any single-token
// identifier that doesn't carry the composition contract.
expect(arg).toMatch(/cacheKey|storageLabel|composeFetchCacheKey/);
});
test("ContentStore: per-(label,url) keys do not collide on getSourceMeta", () => {
+85 -13
View File
@@ -1444,14 +1444,15 @@ describe("MCP Events", () => {
};
const events = extractEvents(input);
assert.equal(events.length, 1);
assert.equal(events[0].type, "mcp");
assert.equal(events[0].category, "mcp");
assert.ok(events[0].data.includes("jira_get"), "data should include tool short name");
assert.ok(events[0].data.includes("CVX-5909"), "data should include first string arg");
const mcpEvents = events.filter(e => e.category === "mcp");
assert.equal(mcpEvents.length, 1);
assert.equal(mcpEvents[0].type, "mcp");
assert.equal(mcpEvents[0].category, "mcp");
assert.ok(mcpEvents[0].data.includes("jira_get"), "data should include tool short name");
assert.ok(mcpEvents[0].data.includes("CVX-5909"), "data should include first string arg");
// Response body is now searchable, not just the call shape
assert.ok(
events[0].data.includes("MQTT reconnect storm"),
mcpEvents[0].data.includes("MQTT reconnect storm"),
"data should include tool_response body so FTS5 can index it",
);
});
@@ -1468,9 +1469,10 @@ describe("MCP Events", () => {
};
const events = extractEvents(input);
assert.equal(events.length, 1);
const mcpEvents = events.filter(e => e.category === "mcp");
assert.equal(mcpEvents.length, 1);
assert.ok(
events[0].data.includes(bigResponse),
mcpEvents[0].data.includes(bigResponse),
"large tool_response must be preserved in full",
);
});
@@ -1484,9 +1486,10 @@ describe("MCP Events", () => {
};
const events = extractEvents(input);
assert.equal(events.length, 1);
assert.equal(events[0].type, "mcp");
assert.equal(events[0].data, "ctx_stats", "no \\nresponse: suffix when tool_response absent");
const mcpEvents = events.filter(e => e.category === "mcp");
assert.equal(mcpEvents.length, 1);
assert.equal(mcpEvents[0].type, "mcp");
assert.equal(mcpEvents[0].data, "ctx_stats", "no \\nresponse: suffix when tool_response absent");
});
test("gracefully handles empty tool_response", () => {
@@ -1497,8 +1500,77 @@ describe("MCP Events", () => {
};
const events = extractEvents(input);
assert.equal(events.length, 1);
assert.equal(events[0].data, "ctx_stats", "empty tool_response should not add suffix");
const mcpEvents = events.filter(e => e.category === "mcp");
assert.equal(mcpEvents.length, 1);
assert.equal(mcpEvents[0].data, "ctx_stats", "empty tool_response should not add suffix");
});
test("emits mcp_tool_call category for mcp__* events with truncated params", () => {
// Small payload — no truncation, params parsed verbatim
const small = extractEvents({
tool_name: "mcp__context-mode__ctx_batch_execute",
tool_input: { commands: [{ label: "x", command: "ls" }], concurrency: 6 },
});
const smallCall = small.find(e => e.category === "mcp_tool_call");
assert.ok(smallCall, "mcp_tool_call event should be emitted");
assert.equal(smallCall!.type, "mcp_tool_call");
assert.equal(smallCall!.priority, 4);
const smallPayload = JSON.parse(smallCall!.data);
assert.equal(smallPayload.tool_name, "mcp__context-mode__ctx_batch_execute");
assert.equal(smallPayload.params.concurrency, 6);
assert.equal(smallPayload.truncated, undefined);
// Large payload — params JSON exceeds 2KB, must be truncated with sentinel
const bigCommands = Array.from({ length: 200 }, (_, i) => ({
label: `cmd-${i}`,
command: "echo " + "x".repeat(50),
}));
const big = extractEvents({
tool_name: "mcp__context-mode__ctx_batch_execute",
tool_input: { commands: bigCommands, concurrency: 8 },
});
const bigCall = big.find(e => e.category === "mcp_tool_call");
assert.ok(bigCall, "mcp_tool_call event should be emitted for large payload");
// 2KB params budget + JSON-escape overhead + wrapper (~300 bytes max).
assert.ok(bigCall!.data.length <= 2500, "data should be capped near 2KB params budget");
const bigPayload = JSON.parse(bigCall!.data);
assert.equal(bigPayload.truncated, true, "truncation sentinel must be set");
assert.equal(typeof bigPayload.params_raw, "string", "raw substring preserved");
assert.equal(bigPayload.tool_name, "mcp__context-mode__ctx_batch_execute");
});
test("UTF-8-aware truncation: never lands mid-codepoint (review F3)", () => {
// Naive `string.slice(0, N)` operates on UTF-16 code units. With multi-byte
// characters, that can either over-shoot the byte budget OR slice mid
// surrogate pair, producing an unpaired surrogate that becomes U+FFFD
// after a SQLite TEXT round-trip.
//
// Repro shape: enough multi-byte characters to push the JSON payload past
// 2KB. We use 3-byte (CJK) and 4-byte (math symbol) characters so any
// mid-codepoint slice is observable.
const cjkChunk = "中文测试".repeat(200); // 4 × 3 bytes × 200 = 2400 bytes raw
const symbolChunk = "𝕏".repeat(100); // 1 × 4 bytes × 100 = 400 bytes raw
const big = extractEvents({
tool_name: "mcp__context-mode__ctx_batch_execute",
tool_input: { mixed: cjkChunk + symbolChunk, concurrency: 4 },
});
const call = big.find(e => e.category === "mcp_tool_call");
assert.ok(call, "mcp_tool_call event should be emitted for multibyte payload");
const payload = JSON.parse(call!.data);
assert.equal(payload.truncated, true, "multibyte payload should trip truncation");
// Critical invariant 1 — params_raw must round-trip through Buffer.from cleanly.
// If the slice landed mid-codepoint, JSON.parse would have already thrown above
// (invalid JSON token) OR the string would contain U+FFFD replacement chars.
assert.equal(typeof payload.params_raw, "string");
assert.ok(!payload.params_raw.includes("<22>"), "no replacement chars (mid-codepoint slice)");
// Critical invariant 2 — the truncated raw must satisfy the BYTE budget,
// not the UTF-16 code-unit budget. Since CJK = 3 bytes/char and the raw
// payload includes JSON quoting overhead, a UTF-16-only cap would let
// ~6KB of bytes through; UTF-8-aware cap holds it ≤ 2KB.
const rawBytes = Buffer.byteLength(payload.params_raw, "utf8");
assert.ok(rawBytes <= 2048, `raw bytes (${rawBytes}) exceed 2KB budget`);
});
});