mirror of
https://github.com/jackwener/OpenCLI.git
synced 2026-09-14 18:25:42 +08:00
fix(browser): preserve structured network captures (#2406)
* fix(browser): preserve structured network captures * test(browser): cover direct CDP request capture * fix(browser): redact credential-shaped request values * fix(browser): redact bare CSRF request fields --------- Co-authored-by: OpenCLI-sol <opencli-sol@users.noreply.github.com>
This commit is contained in:
@@ -57,7 +57,7 @@ opencli browser network
|
||||
- `shape` — response body 的路径→类型映射(不含原 body,省 token)
|
||||
- `status / url / method / ct / size`
|
||||
|
||||
静态资源 / 埋点 / 追踪默认已过滤。默认会保留 JSON / XML / plain text / `text/javascript` 这类 API 响应;如果你确定浏览器 DevTools 里有目标请求但这里缺失,用 `--all` 查一遍是否被 content-type 或 URL 噪音过滤挡掉。
|
||||
静态资源 / 埋点 / 追踪默认已过滤。默认会保留 JSON / XML / plain text / `text/javascript`,也会识别 `text/x-component` 与明确的 `/rsc-action/` React Server Component 流。如果你确定浏览器 DevTools 里有目标请求但这里缺失,用 `--all` 查一遍是否被其他 content-type 或 URL 噪音过滤挡掉。capture queue 是破坏性读取;Core 会先缓存本批原始条目再做展示过滤,所以紧接着的空 `--all` 仍可复用该 session 的 raw cache,而不是永久丢掉被隐藏的条目。
|
||||
|
||||
如果是冷启动,先看 `opencli browser analyze <url>` 里的 `api_candidates`:
|
||||
|
||||
@@ -102,11 +102,13 @@ opencli browser network --detail <key>
|
||||
|
||||
capture 会持久化到 `~/.opencli/cache/browser-network/<session>.json`(默认 TTL 24h),所以 `--detail` 即使跨多条其他命令也还在。
|
||||
|
||||
`--detail` 还会在 capture provider 支持时返回 `request`:method 仍在顶层;headers 中 cookie、Authorization、CSRF/XSRF、token/key/secret/session 等值会替换为 `<redacted>`;可安全识别的 JSON object / URL-encoded form 会保留结构,位置数组、opaque 或截断 body 只保留 kind、shape、full size、truncated/omitted 状态。不要因为 body 被安全省略就拿 URL 单独 replay——这说明请求合同仍不完整。
|
||||
|
||||
这也意味着私有页面的 response 可能落在本地 cache。侦察结束要删除相关 session capture 并释放 browser session;不要依赖 24h TTL 代替清理。
|
||||
|
||||
### 关键 request headers
|
||||
|
||||
`browser network` 当前只抓响应(body + status + ct),抓不到请求头。要看请求头就在 DevTools Network 面板里点这条 request,或用 `browser eval` 手动 `fetch(url)` 复现一次观察浏览器发出去的头:
|
||||
先用 `browser network --detail <key>` 看脱敏后的 request headers / body shape;不要打印或复制 credential 原值。旧 capture provider 若没有返回 `request`,再去 DevTools Network 面板核字段名,或用页面自然动作重新 capture,不能用 `browser eval` 猜造一份缺 header/body 的 URL-only 请求:
|
||||
|
||||
| 看到 | 含义 | 对应策略 |
|
||||
|------|------|---------|
|
||||
|
||||
@@ -34,6 +34,8 @@ Never paste credentials or response bodies into the ledger. Store structural fac
|
||||
|
||||
Dynamic evidence proves that a request occurred. Static scanning expands recall to lazy pagination, detail, search, and routes that this session did not trigger. Neither alone proves a production contract.
|
||||
|
||||
Do not equate “structured” with JSON. React Server Components (`text/x-component`), streamed HTML fragments, protobuf-like payloads, and positional arrays may carry the authoritative data. Preserve their content type, request context, truncation state, and structural shape even when the default network view would normally hide them.
|
||||
|
||||
jsluice is optional and stays outside adapter runtime. Feed it script text through stdin, keep source locations, and treat `EXPR` as unknown. Do not persist suspected secret values. A candidate becomes useful only after dynamic occurrence or a safe replay verifies its shape and semantics.
|
||||
|
||||
## 4. Attribute requests with causal diffs
|
||||
@@ -74,6 +76,8 @@ A read contract must prove all of these:
|
||||
5. **Auth boundary**: cookies/CSRF/origin/runtime requirements are explicit and do not leak secrets.
|
||||
6. **Failure semantics**: auth, HTTP, malformed/truncated body, repeated cursor/page, timeout, and partial data fail typed.
|
||||
|
||||
Replay the complete request contract, not a URL-shaped fragment. A captured URL returning 4xx/5xx does not reject the underlying endpoint when headers, body, cookies, runtime action identifiers, or page-owned signing were omitted. Record the missing context and use `INTERCEPT` until it can be reproduced safely; never guess absent request fields from a bundle string.
|
||||
|
||||
A direct API-backed write contract additionally must prove:
|
||||
|
||||
1. target identity is deterministically bound in the request;
|
||||
@@ -90,13 +94,17 @@ A direct API-backed write contract additionally must prove:
|
||||
Browser capture queues may be destructive drains. Before relying on them:
|
||||
|
||||
- install capture before the action and drain stale entries;
|
||||
- cache the raw selected capture before applying display-only MIME, static-resource, or shape filters;
|
||||
- allow in-flight responses to settle;
|
||||
- treat bodyless or truncated relevant entries as possible data loss;
|
||||
- inspect non-JSON structured streams with request method, safely redacted headers, body shape, and size/truncation metadata;
|
||||
- merge all relevant completed responses in the action window;
|
||||
- identify pages/cursors by content, not arrival order alone;
|
||||
- deduplicate by stable entity ID;
|
||||
- reject repeated pages/cursors and page-cap exhaustion rather than return accumulated partial rows.
|
||||
|
||||
Never copy authorization, cookies, CSRF/XSRF values, API keys, session identifiers, or token-bearing request bodies into output, ledgers, fixtures, or site memory. Redact keyed values; if a positional or opaque body cannot be sanitized confidently, preserve only its kind, shape, full size, and truncation/omission state.
|
||||
|
||||
A cached page may render without a fresh request. A DOM fallback is valid only when it is strictly scoped to the target container, preserves the public columns, and can distinguish empty state from structure drift. Do not silently switch to a weaker page-wide selector.
|
||||
|
||||
## 8. Choose strategy per command
|
||||
|
||||
+41
-2
@@ -3,10 +3,12 @@ import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
const { MockWebSocket } = vi.hoisted(() => {
|
||||
class MockWebSocket {
|
||||
static OPEN = 1;
|
||||
static lastInstance: MockWebSocket | undefined;
|
||||
readyState = 1;
|
||||
private handlers = new Map<string, Array<(...args: unknown[]) => void>>();
|
||||
|
||||
constructor(_url: string) {
|
||||
MockWebSocket.lastInstance = this;
|
||||
queueMicrotask(() => this.emit('open'));
|
||||
}
|
||||
|
||||
@@ -22,7 +24,7 @@ const { MockWebSocket } = vi.hoisted(() => {
|
||||
this.readyState = 3;
|
||||
}
|
||||
|
||||
private emit(event: string, ...args: unknown[]): void {
|
||||
emit(event: string, ...args: unknown[]): void {
|
||||
for (const handler of this.handlers.get(event) ?? []) {
|
||||
handler(...args);
|
||||
}
|
||||
@@ -36,7 +38,7 @@ vi.mock('ws', () => ({
|
||||
WebSocket: MockWebSocket,
|
||||
}));
|
||||
|
||||
import { CDPBridge } from './cdp.js';
|
||||
import { CDPBridge, CDP_REQUEST_BODY_CAPTURE_LIMIT } from './cdp.js';
|
||||
|
||||
describe('CDPBridge cookies', () => {
|
||||
beforeEach(() => {
|
||||
@@ -96,4 +98,41 @@ describe('CDPBridge cookies', () => {
|
||||
['Page.getLayoutMetrics', {}],
|
||||
]);
|
||||
});
|
||||
|
||||
it('captures request headers and bounded post data on direct CDP pages', async () => {
|
||||
vi.stubEnv('OPENCLI_CDP_ENDPOINT', 'ws://127.0.0.1:9222/devtools/page/1');
|
||||
|
||||
const bridge = new CDPBridge();
|
||||
const fullBody = 'x'.repeat(CDP_REQUEST_BODY_CAPTURE_LIMIT + 5);
|
||||
vi.spyOn(bridge, 'send').mockImplementation(async (method: string) => {
|
||||
if (method === 'Network.getRequestPostData') return { postData: fullBody };
|
||||
return {};
|
||||
});
|
||||
|
||||
const page = await bridge.connect();
|
||||
await page.startNetworkCapture?.();
|
||||
MockWebSocket.lastInstance?.emit('message', Buffer.from(JSON.stringify({
|
||||
method: 'Network.requestWillBeSent',
|
||||
params: {
|
||||
requestId: 'request-1',
|
||||
request: {
|
||||
method: 'POST',
|
||||
url: 'https://example.test/rsc-action/actions/pagination',
|
||||
headers: { Authorization: 'Bearer secret', 'Content-Type': 'application/json' },
|
||||
hasPostData: true,
|
||||
},
|
||||
},
|
||||
})));
|
||||
|
||||
const entries = await page.readNetworkCapture?.() as Array<Record<string, unknown>>;
|
||||
expect(entries).toHaveLength(1);
|
||||
expect(entries[0]).toMatchObject({
|
||||
method: 'POST',
|
||||
requestHeaders: { Authorization: 'Bearer secret', 'Content-Type': 'application/json' },
|
||||
requestBodyKind: 'string',
|
||||
requestBodyFullSize: fullBody.length,
|
||||
requestBodyTruncated: true,
|
||||
});
|
||||
expect(String(entries[0].requestBodyPreview)).toHaveLength(CDP_REQUEST_BODY_CAPTURE_LIMIT);
|
||||
});
|
||||
});
|
||||
|
||||
+44
-1
@@ -46,6 +46,7 @@ const CDP_SEND_TIMEOUT = 30_000;
|
||||
// surface `responseBodyFullSize` + `responseBodyTruncated` so downstream layers
|
||||
// can tell the agent what happened instead of lying about the payload.
|
||||
export const CDP_RESPONSE_BODY_CAPTURE_LIMIT = 8 * 1024 * 1024;
|
||||
export const CDP_REQUEST_BODY_CAPTURE_LIMIT = 1 * 1024 * 1024;
|
||||
|
||||
export class CDPBridge implements IBrowserFactory {
|
||||
private _ws: WebSocket | null = null;
|
||||
@@ -191,6 +192,11 @@ class CDPPage extends CDPBasePage {
|
||||
private _networkCapturePattern = '';
|
||||
private _networkEntries: Array<{
|
||||
url: string; method: string; responseStatus?: number;
|
||||
requestHeaders?: Record<string, string>;
|
||||
requestBodyKind?: string;
|
||||
requestBodyPreview?: string;
|
||||
requestBodyFullSize?: number;
|
||||
requestBodyTruncated?: boolean;
|
||||
responseContentType?: string;
|
||||
responsePreview?: string;
|
||||
responseBodyFullSize?: number;
|
||||
@@ -313,14 +319,51 @@ class CDPPage extends CDPBasePage {
|
||||
|
||||
// Step 1: Record request method/url on requestWillBeSent
|
||||
this.bridge.on('Network.requestWillBeSent', (params: unknown) => {
|
||||
const p = params as { requestId: string; request: { method: string; url: string }; timestamp: number };
|
||||
const p = params as {
|
||||
requestId: string;
|
||||
request: {
|
||||
method: string;
|
||||
url: string;
|
||||
headers?: Record<string, unknown>;
|
||||
postData?: string;
|
||||
hasPostData?: boolean;
|
||||
};
|
||||
timestamp: number;
|
||||
};
|
||||
if (!this._networkCapturePattern || p.request.url.includes(this._networkCapturePattern)) {
|
||||
const rawBody = typeof p.request.postData === 'string' ? p.request.postData : '';
|
||||
const bodyTruncated = rawBody.length > CDP_REQUEST_BODY_CAPTURE_LIMIT;
|
||||
const idx = this._networkEntries.push({
|
||||
url: p.request.url,
|
||||
method: p.request.method,
|
||||
requestHeaders: Object.fromEntries(
|
||||
Object.entries(p.request.headers ?? {}).map(([name, value]) => [name, String(value)]),
|
||||
),
|
||||
requestBodyKind: p.request.hasPostData ? 'string' : 'empty',
|
||||
requestBodyPreview: bodyTruncated ? rawBody.slice(0, CDP_REQUEST_BODY_CAPTURE_LIMIT) : rawBody,
|
||||
requestBodyFullSize: rawBody.length,
|
||||
requestBodyTruncated: bodyTruncated,
|
||||
timestamp: Date.now(),
|
||||
}) - 1;
|
||||
this._pendingRequests.set(p.requestId, idx);
|
||||
|
||||
if (p.request.hasPostData && p.request.postData === undefined) {
|
||||
const requestBodyFetch = this.bridge.send('Network.getRequestPostData', { requestId: p.requestId }).then((result: unknown) => {
|
||||
const postData = (result as { postData?: string } | undefined)?.postData;
|
||||
if (typeof postData !== 'string') return;
|
||||
const truncated = postData.length > CDP_REQUEST_BODY_CAPTURE_LIMIT;
|
||||
this._networkEntries[idx].requestBodyPreview = truncated
|
||||
? postData.slice(0, CDP_REQUEST_BODY_CAPTURE_LIMIT)
|
||||
: postData;
|
||||
this._networkEntries[idx].requestBodyFullSize = postData.length;
|
||||
this._networkEntries[idx].requestBodyTruncated = truncated;
|
||||
}).catch(() => {
|
||||
// Some request types do not expose post data.
|
||||
}).finally(() => {
|
||||
this._pendingBodyFetches.delete(requestBodyFetch);
|
||||
});
|
||||
this._pendingBodyFetches.add(requestBodyFetch);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import type { SafeNetworkRequest } from './network-request.js';
|
||||
|
||||
export const DEFAULT_TTL_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
@@ -32,6 +33,8 @@ export interface CachedNetworkEntry {
|
||||
body_truncated?: boolean;
|
||||
body_full_size?: number;
|
||||
timestamp?: number;
|
||||
/** Sanitized request context; credential values and opaque bodies are omitted. */
|
||||
request?: SafeNetworkRequest;
|
||||
}
|
||||
|
||||
export interface NetworkCacheFile {
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { sanitizeCapturedRequest, sanitizeCapturedUrl } from './network-request.js';
|
||||
|
||||
describe('network request sanitization', () => {
|
||||
it('redacts sensitive headers and nested JSON fields while preserving shape', () => {
|
||||
const request = sanitizeCapturedRequest({
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
Authorization: 'Bearer live-secret',
|
||||
Cookie: 'sid=live-cookie',
|
||||
'X-CSRF': 'bare-header-csrf',
|
||||
'X-Trace-Id': 'trace-1',
|
||||
'X-Runtime-Id': 'a'.repeat(48),
|
||||
},
|
||||
bodyKind: 'string',
|
||||
bodyPreview: JSON.stringify({
|
||||
query: 'timeline',
|
||||
variables: { cursor: 'next', csrfToken: 'live-csrf', csrf: 'bare-body-csrf' },
|
||||
}),
|
||||
bodyFullSize: 91,
|
||||
});
|
||||
|
||||
expect(request).toMatchObject({
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
Authorization: '<redacted>',
|
||||
Cookie: '<redacted>',
|
||||
'X-CSRF': '<redacted>',
|
||||
'X-Trace-Id': 'trace-1',
|
||||
'X-Runtime-Id': '<redacted>',
|
||||
},
|
||||
body_kind: 'json',
|
||||
body: {
|
||||
query: 'timeline',
|
||||
variables: { cursor: 'next', csrfToken: '<redacted>', csrf: '<redacted>' },
|
||||
},
|
||||
body_full_size: 91,
|
||||
redacted: true,
|
||||
});
|
||||
expect(request?.body_shape?.['$.variables.csrfToken']).toBe('string');
|
||||
expect(JSON.stringify(request)).not.toContain('live-secret');
|
||||
expect(JSON.stringify(request)).not.toContain('live-cookie');
|
||||
expect(JSON.stringify(request)).not.toContain('live-csrf');
|
||||
expect(JSON.stringify(request)).not.toContain('bare-header-csrf');
|
||||
expect(JSON.stringify(request)).not.toContain('bare-body-csrf');
|
||||
});
|
||||
|
||||
it('redacts form credentials and preserves repeated safe fields', () => {
|
||||
const request = sanitizeCapturedRequest({
|
||||
headers: { 'content-type': 'application/x-www-form-urlencoded' },
|
||||
bodyKind: 'string',
|
||||
bodyPreview: 'q=opencli&tag=one&tag=two&access_token=secret&xsrf=bare-form-xsrf',
|
||||
});
|
||||
|
||||
expect(request?.body_kind).toBe('form');
|
||||
expect(request?.body).toEqual({
|
||||
q: 'opencli',
|
||||
tag: ['one', 'two'],
|
||||
access_token: '<redacted>',
|
||||
xsrf: '<redacted>',
|
||||
});
|
||||
expect(request?.redacted).toBe(true);
|
||||
});
|
||||
|
||||
it('omits positional and truncated bodies but keeps their shape and size', () => {
|
||||
const positional = sanitizeCapturedRequest({
|
||||
bodyKind: 'string',
|
||||
bodyPreview: JSON.stringify(['opaque-runtime-token', { cursor: 'next' }]),
|
||||
});
|
||||
expect(positional?.body_kind).toBe('json');
|
||||
expect(positional?.body_omitted).toBe(true);
|
||||
expect(positional?.body_shape?.['$']).toBe('array(2)');
|
||||
expect(positional).not.toHaveProperty('body');
|
||||
|
||||
const truncated = sanitizeCapturedRequest({
|
||||
bodyKind: 'string',
|
||||
bodyPreview: '{"partial":',
|
||||
bodyFullSize: 50_000,
|
||||
bodyTruncated: true,
|
||||
});
|
||||
expect(truncated).toMatchObject({
|
||||
body_kind: 'opaque',
|
||||
body_full_size: 50_000,
|
||||
body_truncated: true,
|
||||
body_omitted: true,
|
||||
});
|
||||
});
|
||||
|
||||
it('redacts credential-shaped URL query parameters', () => {
|
||||
const url = sanitizeCapturedUrl('https://api.example.test/rsc-action?page=2&csrf_token=secret&xsrf=bare-query-xsrf');
|
||||
expect(url).toContain('page=2');
|
||||
expect(url).toContain('csrf_token=%3Credacted%3E');
|
||||
expect(url).toContain('xsrf=%3Credacted%3E');
|
||||
expect(url).not.toContain('secret');
|
||||
expect(url).not.toContain('bare-query-xsrf');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,217 @@
|
||||
import { inferShape, type Shape } from './shape.js';
|
||||
|
||||
const REDACTED = '<redacted>';
|
||||
|
||||
export interface SafeNetworkRequest {
|
||||
headers?: Record<string, string>;
|
||||
body_kind?: 'empty' | 'json' | 'form' | 'opaque';
|
||||
body?: unknown;
|
||||
body_shape?: Shape;
|
||||
body_full_size?: number;
|
||||
body_truncated?: boolean;
|
||||
body_omitted?: boolean;
|
||||
redacted?: boolean;
|
||||
}
|
||||
|
||||
export interface CapturedRequestMetadata {
|
||||
headers?: unknown;
|
||||
bodyKind?: unknown;
|
||||
bodyPreview?: unknown;
|
||||
bodyFullSize?: unknown;
|
||||
bodyTruncated?: unknown;
|
||||
}
|
||||
|
||||
function normalizedName(name: string): string {
|
||||
return name.toLowerCase().replace(/[^a-z0-9]/g, '');
|
||||
}
|
||||
|
||||
function isSensitiveName(name: string): boolean {
|
||||
const normalized = normalizedName(name);
|
||||
return normalized === 'authorization'
|
||||
|| normalized === 'proxyauthorization'
|
||||
|| normalized === 'cookie'
|
||||
|| normalized === 'setcookie'
|
||||
|| normalized === 'sapisid'
|
||||
|| normalized === 'sid'
|
||||
|| normalized === 'liat'
|
||||
|| normalized === 'jsessionid'
|
||||
|| normalized === 'password'
|
||||
|| normalized === 'passwd'
|
||||
|| normalized === 'apikey'
|
||||
|| normalized === 'auth'
|
||||
|| normalized === 'authentication'
|
||||
|| normalized === 'credential'
|
||||
|| normalized === 'credentials'
|
||||
|| normalized === 'signature'
|
||||
|| normalized === 'sig'
|
||||
|| normalized === 'csrf'
|
||||
|| normalized === 'xcsrf'
|
||||
|| normalized === 'xsrf'
|
||||
|| normalized === 'xxsrf'
|
||||
|| normalized === 'clientsecret'
|
||||
|| normalized.endsWith('authorization')
|
||||
|| normalized.endsWith('cookie')
|
||||
|| normalized.endsWith('apikey')
|
||||
|| normalized.endsWith('password')
|
||||
|| normalized.endsWith('passwd')
|
||||
|| normalized.endsWith('token')
|
||||
|| normalized.endsWith('secret')
|
||||
|| normalized.endsWith('sessionid');
|
||||
}
|
||||
|
||||
function isCredentialLikeValue(value: string): boolean {
|
||||
const trimmed = value.trim();
|
||||
return /^(?:bearer|basic)\s+\S+/i.test(trimmed)
|
||||
|| /^eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(trimmed)
|
||||
|| /^[a-f0-9]{32,}$/i.test(trimmed)
|
||||
|| /^[A-Za-z0-9_-]{48,}$/.test(trimmed);
|
||||
}
|
||||
|
||||
function sanitizeHeaders(raw: unknown): { headers?: Record<string, string>; redacted: boolean } {
|
||||
if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return { redacted: false };
|
||||
const headers: Record<string, string> = {};
|
||||
let redacted = false;
|
||||
for (const [name, value] of Object.entries(raw as Record<string, unknown>)) {
|
||||
const stringValue = String(value);
|
||||
if (isSensitiveName(name) || isCredentialLikeValue(stringValue)) {
|
||||
headers[name] = REDACTED;
|
||||
redacted = true;
|
||||
} else {
|
||||
headers[name] = stringValue;
|
||||
}
|
||||
}
|
||||
return { ...(Object.keys(headers).length > 0 ? { headers } : {}), redacted };
|
||||
}
|
||||
|
||||
function redactObject(value: unknown): { value: unknown; redacted: boolean } {
|
||||
if (Array.isArray(value)) {
|
||||
let redacted = false;
|
||||
const next = value.map((item) => {
|
||||
const result = redactObject(item);
|
||||
redacted ||= result.redacted;
|
||||
return result.value;
|
||||
});
|
||||
return { value: next, redacted };
|
||||
}
|
||||
if (typeof value === 'string' && isCredentialLikeValue(value)) {
|
||||
return { value: REDACTED, redacted: true };
|
||||
}
|
||||
if (!value || typeof value !== 'object') return { value, redacted: false };
|
||||
|
||||
let redacted = false;
|
||||
const next: Record<string, unknown> = {};
|
||||
for (const [key, item] of Object.entries(value as Record<string, unknown>)) {
|
||||
if (isSensitiveName(key)) {
|
||||
next[key] = REDACTED;
|
||||
redacted = true;
|
||||
continue;
|
||||
}
|
||||
const result = redactObject(item);
|
||||
next[key] = result.value;
|
||||
redacted ||= result.redacted;
|
||||
}
|
||||
return { value: next, redacted };
|
||||
}
|
||||
|
||||
function formBody(raw: string): { body: Record<string, string | string[]>; redacted: boolean } {
|
||||
const values = new URLSearchParams(raw);
|
||||
const body: Record<string, string | string[]> = {};
|
||||
let redacted = false;
|
||||
for (const [key, value] of values.entries()) {
|
||||
const safeValue = isSensitiveName(key) || isCredentialLikeValue(value) ? REDACTED : value;
|
||||
redacted ||= safeValue === REDACTED;
|
||||
const previous = body[key];
|
||||
if (previous === undefined) body[key] = safeValue;
|
||||
else if (Array.isArray(previous)) previous.push(safeValue);
|
||||
else body[key] = [previous, safeValue];
|
||||
}
|
||||
return { body, redacted };
|
||||
}
|
||||
|
||||
function headerValue(headers: Record<string, string> | undefined, name: string): string {
|
||||
if (!headers) return '';
|
||||
const match = Object.entries(headers).find(([key]) => key.toLowerCase() === name);
|
||||
return match?.[1] ?? '';
|
||||
}
|
||||
|
||||
/**
|
||||
* Preserve request structure without turning `browser network` into a secret
|
||||
* dumper. Known credential fields are redacted. JSON objects and URL-encoded
|
||||
* forms remain inspectable; positional JSON and opaque bodies expose shape /
|
||||
* size only because their secret-bearing slots cannot be identified safely.
|
||||
*/
|
||||
export function sanitizeCapturedRequest(raw: CapturedRequestMetadata): SafeNetworkRequest | undefined {
|
||||
const sanitizedHeaders = sanitizeHeaders(raw.headers);
|
||||
const preview = typeof raw.bodyPreview === 'string' ? raw.bodyPreview : undefined;
|
||||
const fullSize = typeof raw.bodyFullSize === 'number' && Number.isFinite(raw.bodyFullSize)
|
||||
? raw.bodyFullSize
|
||||
: preview?.length;
|
||||
const truncated = raw.bodyTruncated === true;
|
||||
const rawKind = typeof raw.bodyKind === 'string' ? raw.bodyKind : undefined;
|
||||
|
||||
if (!sanitizedHeaders.headers && preview === undefined && rawKind === undefined) return undefined;
|
||||
|
||||
const request: SafeNetworkRequest = {
|
||||
...(sanitizedHeaders.headers ? { headers: sanitizedHeaders.headers } : {}),
|
||||
...(typeof fullSize === 'number' ? { body_full_size: fullSize } : {}),
|
||||
...(truncated ? { body_truncated: true } : {}),
|
||||
...(sanitizedHeaders.redacted ? { redacted: true } : {}),
|
||||
};
|
||||
|
||||
if (rawKind === 'empty' || preview === '') {
|
||||
request.body_kind = 'empty';
|
||||
return request;
|
||||
}
|
||||
if (preview === undefined) return request;
|
||||
|
||||
const contentType = headerValue(sanitizedHeaders.headers, 'content-type').toLowerCase();
|
||||
if (!truncated) {
|
||||
try {
|
||||
const parsed = JSON.parse(preview) as unknown;
|
||||
request.body_kind = 'json';
|
||||
request.body_shape = inferShape(parsed);
|
||||
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
|
||||
const sanitized = redactObject(parsed);
|
||||
request.body = sanitized.value;
|
||||
if (sanitized.redacted) request.redacted = true;
|
||||
} else {
|
||||
request.body_omitted = true;
|
||||
}
|
||||
return request;
|
||||
} catch {
|
||||
// Non-JSON request; inspect content type below.
|
||||
}
|
||||
|
||||
if (contentType.includes('application/x-www-form-urlencoded')) {
|
||||
const sanitized = formBody(preview);
|
||||
request.body_kind = 'form';
|
||||
request.body = sanitized.body;
|
||||
request.body_shape = inferShape(sanitized.body);
|
||||
if (sanitized.redacted) request.redacted = true;
|
||||
return request;
|
||||
}
|
||||
}
|
||||
|
||||
request.body_kind = 'opaque';
|
||||
request.body_shape = inferShape(preview);
|
||||
request.body_omitted = true;
|
||||
return request;
|
||||
}
|
||||
|
||||
/** Redact credential-shaped query parameters while preserving route identity. */
|
||||
export function sanitizeCapturedUrl(rawUrl: string): string {
|
||||
try {
|
||||
const url = new URL(rawUrl);
|
||||
let changed = false;
|
||||
for (const key of [...url.searchParams.keys()]) {
|
||||
const value = url.searchParams.get(key) ?? '';
|
||||
if (isSensitiveName(key) || isCredentialLikeValue(value)) {
|
||||
url.searchParams.set(key, REDACTED);
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
return changed ? url.toString() : rawUrl;
|
||||
} catch {
|
||||
return rawUrl;
|
||||
}
|
||||
}
|
||||
+106
@@ -1996,6 +1996,112 @@ describe('browser network command', () => {
|
||||
expect(out.entries[0].shape['$.messages']).toBe('array(1)');
|
||||
});
|
||||
|
||||
it('treats React Server Component responses as API-like traffic', async () => {
|
||||
browserState.page!.readNetworkCapture = vi.fn().mockResolvedValue([
|
||||
{
|
||||
url: 'https://www.linkedin.com/flagship-web/rsc-action/actions/pagination',
|
||||
method: 'POST',
|
||||
responseStatus: 200,
|
||||
responseContentType: 'text/x-component',
|
||||
responsePreview: '1:{"posts":[{"id":"p1"}]}',
|
||||
},
|
||||
{
|
||||
url: 'https://www.linkedin.com/flagship-web/rsc-action/actions/detail',
|
||||
method: 'POST',
|
||||
responseStatus: 200,
|
||||
responseContentType: 'text/html',
|
||||
responsePreview: '<rsc-stream>',
|
||||
},
|
||||
]);
|
||||
const program = createProgram('', '');
|
||||
|
||||
await program.parseAsync(['node', 'opencli', 'browser', '--session', 'test', 'network']);
|
||||
|
||||
const out = lastJsonLog();
|
||||
expect(out.count).toBe(2);
|
||||
expect(out.filtered_out).toBe(0);
|
||||
expect(out.entries.map((entry: any) => entry.key)).toEqual([
|
||||
'POST www.linkedin.com/flagship-web/rsc-action/actions/pagination',
|
||||
'POST www.linkedin.com/flagship-web/rsc-action/actions/detail',
|
||||
]);
|
||||
});
|
||||
|
||||
it('caches the raw drained batch before display filtering so a later --all can recover it', async () => {
|
||||
browserState.page!.readNetworkCapture = vi.fn()
|
||||
.mockResolvedValueOnce([
|
||||
{
|
||||
url: 'https://example.com/page-fragment',
|
||||
method: 'GET',
|
||||
responseStatus: 200,
|
||||
responseContentType: 'text/html',
|
||||
responsePreview: '<main>hidden from default output</main>',
|
||||
},
|
||||
])
|
||||
.mockResolvedValueOnce([]);
|
||||
const program = createProgram('', '');
|
||||
|
||||
await program.parseAsync(['node', 'opencli', 'browser', '--session', 'test', 'network']);
|
||||
expect(lastJsonLog()).toMatchObject({ count: 0, filtered_out: 1 });
|
||||
|
||||
consoleLogSpy.mockClear();
|
||||
await program.parseAsync(['node', 'opencli', 'browser', '--session', 'test', 'network', '--all']);
|
||||
|
||||
const out = lastJsonLog();
|
||||
expect(out.count).toBe(1);
|
||||
expect(out.cache_reused).toBe(true);
|
||||
expect(out.entries[0].key).toBe('GET example.com/page-fragment');
|
||||
});
|
||||
|
||||
it('--detail exposes sanitized request context without credential values', async () => {
|
||||
browserState.page!.readNetworkCapture = vi.fn().mockResolvedValue([
|
||||
{
|
||||
url: 'https://www.linkedin.com/flagship-web/rsc-action/actions/pagination?csrf_token=url-secret',
|
||||
method: 'POST',
|
||||
requestHeaders: {
|
||||
'Content-Type': 'application/json',
|
||||
Cookie: 'li_at=cookie-secret',
|
||||
'X-CSRF-Token': 'header-secret',
|
||||
'X-Trace-Id': 'trace-1',
|
||||
},
|
||||
requestBodyKind: 'string',
|
||||
requestBodyPreview: JSON.stringify({ variables: { cursor: 'next', accessToken: 'body-secret' } }),
|
||||
requestBodyFullSize: 123,
|
||||
requestBodyTruncated: false,
|
||||
responseStatus: 200,
|
||||
responseContentType: 'text/x-component',
|
||||
responsePreview: '1:{"posts":[]}',
|
||||
},
|
||||
]);
|
||||
const program = createProgram('', '');
|
||||
|
||||
await program.parseAsync(['node', 'opencli', 'browser', '--session', 'test', 'network']);
|
||||
consoleLogSpy.mockClear();
|
||||
await program.parseAsync([
|
||||
'node', 'opencli', 'browser', '--session', 'test', 'network',
|
||||
'--detail', 'POST www.linkedin.com/flagship-web/rsc-action/actions/pagination',
|
||||
]);
|
||||
|
||||
const out = lastJsonLog();
|
||||
expect(out.url).toContain('csrf_token=%3Credacted%3E');
|
||||
expect(out.request).toMatchObject({
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
Cookie: '<redacted>',
|
||||
'X-CSRF-Token': '<redacted>',
|
||||
'X-Trace-Id': 'trace-1',
|
||||
},
|
||||
body_kind: 'json',
|
||||
body: { variables: { cursor: 'next', accessToken: '<redacted>' } },
|
||||
body_full_size: 123,
|
||||
redacted: true,
|
||||
});
|
||||
expect(out.request.body_shape['$.variables.accessToken']).toBe('string');
|
||||
expect(JSON.stringify(out)).not.toContain('url-secret');
|
||||
expect(JSON.stringify(out)).not.toContain('cookie-secret');
|
||||
expect(JSON.stringify(out)).not.toContain('header-secret');
|
||||
expect(JSON.stringify(out)).not.toContain('body-secret');
|
||||
});
|
||||
|
||||
it('--raw emits full bodies inline for every entry', async () => {
|
||||
const program = createProgram('', '');
|
||||
|
||||
|
||||
+66
-28
@@ -27,6 +27,7 @@ import { buildFindJs, buildSemanticFindJs, isFindError, type FindResult, type Fi
|
||||
import { inferShape } from './browser/shape.js';
|
||||
import { assignKeys } from './browser/network-key.js';
|
||||
import { DEFAULT_TTL_MS, findEntry, loadNetworkCache, saveNetworkCache, type CachedNetworkEntry } from './browser/network-cache.js';
|
||||
import { sanitizeCapturedRequest, sanitizeCapturedUrl, type SafeNetworkRequest } from './browser/network-request.js';
|
||||
import { NETWORK_INTERCEPTOR_JS } from './browser/network-interceptor.js';
|
||||
import { parseFilter, shapeMatchesFilter } from './browser/shape-filter.js';
|
||||
import { buildHtmlTreeJs, type HtmlTreeResult } from './browser/html-tree.js';
|
||||
@@ -60,6 +61,8 @@ type BrowserNetworkItem = {
|
||||
bodyTruncated?: boolean;
|
||||
/** Epoch milliseconds when the request was observed. */
|
||||
timestamp?: number;
|
||||
/** Sanitized request context captured by CDP. */
|
||||
request?: SafeNetworkRequest;
|
||||
};
|
||||
|
||||
function parseDurationMs(raw: unknown, flagName: string): number | null | { error: string } {
|
||||
@@ -199,8 +202,15 @@ async function captureNetworkItems(page: import('./types.js').IPage): Promise<Br
|
||||
? (e.responseBodyFullSize as number)
|
||||
: (preview ? preview.length : 0);
|
||||
const truncated = e.responseBodyTruncated === true;
|
||||
const request = sanitizeCapturedRequest({
|
||||
headers: e.requestHeaders,
|
||||
bodyKind: e.requestBodyKind,
|
||||
bodyPreview: e.requestBodyPreview,
|
||||
bodyFullSize: e.requestBodyFullSize,
|
||||
bodyTruncated: e.requestBodyTruncated,
|
||||
});
|
||||
return {
|
||||
url: (e.url as string) || '',
|
||||
url: sanitizeCapturedUrl((e.url as string) || ''),
|
||||
method: (e.method as string) || 'GET',
|
||||
status: (e.responseStatus as number) || 0,
|
||||
size: fullSize,
|
||||
@@ -209,6 +219,7 @@ async function captureNetworkItems(page: import('./types.js').IPage): Promise<Br
|
||||
bodyFullSize: fullSize,
|
||||
bodyTruncated: truncated,
|
||||
timestamp: timestampFromRaw(e.timestamp),
|
||||
...(request ? { request } : {}),
|
||||
};
|
||||
});
|
||||
}
|
||||
@@ -216,7 +227,11 @@ async function captureNetworkItems(page: import('./types.js').IPage): Promise<Br
|
||||
const raw = await page.evaluate(`(function(){ var out = window.__opencli_net || []; window.__opencli_net = []; return JSON.stringify(out); })()`) as string;
|
||||
try {
|
||||
const parsed = JSON.parse(raw) as BrowserNetworkItem[];
|
||||
return parsed.map((item) => ({ ...item, timestamp: timestampFromRaw(item.timestamp) }));
|
||||
return parsed.map((item) => ({
|
||||
...item,
|
||||
url: sanitizeCapturedUrl(item.url),
|
||||
timestamp: timestampFromRaw(item.timestamp),
|
||||
}));
|
||||
} catch {
|
||||
if (process.env.OPENCLI_VERBOSE) log.warn(`[network] Failed to parse interceptor buffer: ${typeof raw === 'string' ? raw.slice(0, 200) : String(raw)}`);
|
||||
return [];
|
||||
@@ -224,11 +239,11 @@ async function captureNetworkItems(page: import('./types.js').IPage): Promise<Br
|
||||
}
|
||||
|
||||
/** Drop static-resource / telemetry noise so agents see only API-shaped traffic. */
|
||||
function filterNetworkItems(items: BrowserNetworkItem[]): BrowserNetworkItem[] {
|
||||
function filterNetworkItems<T extends Pick<BrowserNetworkItem, 'url' | 'ct'>>(items: T[]): T[] {
|
||||
return items.filter((r) => {
|
||||
const ct = r.ct?.toLowerCase() ?? '';
|
||||
return (
|
||||
(ct.includes('json') || ct.includes('xml') || ct.includes('text/plain') || ct.includes('javascript')) &&
|
||||
(ct.includes('json') || ct.includes('xml') || ct.includes('text/plain') || ct.includes('javascript') || ct.includes('text/x-component') || /\/rsc-action(?:\/|\?|$)/i.test(r.url)) &&
|
||||
!/\.(js|css|png|jpg|gif|svg|woff|ico|map)(\?|$)/i.test(r.url) &&
|
||||
!/analytics|tracking|telemetry|beacon|pixel|gtag|fbevents/i.test(r.url)
|
||||
);
|
||||
@@ -2684,6 +2699,7 @@ Examples:
|
||||
...(typeof entry.timestamp === 'number' ? { timestamp: toIsoTimestamp(entry.timestamp) } : {}),
|
||||
shape: inferShape(entry.body),
|
||||
body: outputBody,
|
||||
...(entry.request ? { request: entry.request } : {}),
|
||||
};
|
||||
if (captureTruncated || transportTruncated) {
|
||||
detailEnvelope.body_truncated = true;
|
||||
@@ -2734,33 +2750,54 @@ Examples:
|
||||
return;
|
||||
}
|
||||
|
||||
let items = opts.all ? rawItems : filterNetworkItems(rawItems);
|
||||
items = filterByTimeWindow(items, { sinceMs, untilMs });
|
||||
if (opts.failed) items = items.filter((item) => item.status === 0 || item.status >= 400);
|
||||
const filteredOut = rawItems.length - items.length;
|
||||
let selectedRaw = filterByTimeWindow(rawItems, { sinceMs, untilMs });
|
||||
if (opts.failed) selectedRaw = selectedRaw.filter((item) => item.status === 0 || item.status >= 400);
|
||||
|
||||
const keyed = assignKeys(items);
|
||||
const cacheEntries: CachedNetworkEntry[] = keyed.map((it) => ({
|
||||
key: it.key,
|
||||
url: it.url,
|
||||
method: it.method,
|
||||
status: it.status,
|
||||
size: it.size,
|
||||
ct: it.ct,
|
||||
body: it.body,
|
||||
...(typeof it.timestamp === 'number' ? { timestamp: it.timestamp } : {}),
|
||||
...(it.bodyTruncated ? { body_truncated: true } : {}),
|
||||
...(it.bodyTruncated && typeof it.bodyFullSize === 'number'
|
||||
? { body_full_size: it.bodyFullSize }
|
||||
: {}),
|
||||
}));
|
||||
// The live CDP buffer is a destructive drain. Assign keys and persist
|
||||
// the selected raw batch before applying display-only MIME/static/shape
|
||||
// filters, otherwise a default call permanently destroys RSC and other
|
||||
// non-standard responses before a later --all / --detail can inspect it.
|
||||
const keyed = assignKeys(selectedRaw);
|
||||
let cacheEntries: CachedNetworkEntry[] = keyed.map((it) => ({
|
||||
key: it.key,
|
||||
url: it.url,
|
||||
method: it.method,
|
||||
status: it.status,
|
||||
size: it.size,
|
||||
ct: it.ct,
|
||||
body: it.body,
|
||||
...(typeof it.timestamp === 'number' ? { timestamp: it.timestamp } : {}),
|
||||
...(it.bodyTruncated ? { body_truncated: true } : {}),
|
||||
...(it.bodyTruncated && typeof it.bodyFullSize === 'number'
|
||||
? { body_full_size: it.bodyFullSize }
|
||||
: {}),
|
||||
...(it.request ? { request: it.request } : {}),
|
||||
}));
|
||||
|
||||
// A repeated call commonly sees an empty live batch because the first
|
||||
// call drained it. Reuse the still-fresh raw cache instead of replacing
|
||||
// it with an empty file, so `network` followed by `network --all` works.
|
||||
let reusedCache = false;
|
||||
if (rawItems.length === 0 && opts.all) {
|
||||
const cached = loadNetworkCache(session, { ttlMs });
|
||||
if (cached.status === 'ok' && cached.file) {
|
||||
cacheEntries = filterByTimeWindow(cached.file.entries, { sinceMs, untilMs });
|
||||
if (opts.failed) cacheEntries = cacheEntries.filter((item) => item.status === 0 || item.status >= 400);
|
||||
reusedCache = true;
|
||||
}
|
||||
}
|
||||
|
||||
const displayEntries = opts.all ? cacheEntries : filterNetworkItems(cacheEntries);
|
||||
const filteredOut = (reusedCache ? cacheEntries.length : rawItems.length) - displayEntries.length;
|
||||
// Soft failure: the caller already has the data, so surface a warning
|
||||
// via the output envelope rather than erroring out the whole command.
|
||||
let cacheWarning: string | null = null;
|
||||
try {
|
||||
saveNetworkCache(session, cacheEntries);
|
||||
} catch (err) {
|
||||
cacheWarning = `Could not persist capture cache: ${(err as Error).message}. --detail lookups may miss this capture.`;
|
||||
if (!reusedCache && rawItems.length > 0) {
|
||||
try {
|
||||
saveNetworkCache(session, cacheEntries);
|
||||
} catch (err) {
|
||||
cacheWarning = `Could not persist capture cache: ${(err as Error).message}. --detail lookups may miss this capture.`;
|
||||
}
|
||||
}
|
||||
|
||||
// Pair each cache entry with its shape up front so --filter can read
|
||||
@@ -2768,7 +2805,7 @@ Examples:
|
||||
// body. Cache persistence above stored the unfiltered set on purpose:
|
||||
// later `--detail <key>` lookups must still see requests that the
|
||||
// current --filter narrowed out.
|
||||
const shaped = cacheEntries.map((e) => ({ entry: e, shape: inferShape(e.body) }));
|
||||
const shaped = displayEntries.map((e) => ({ entry: e, shape: inferShape(e.body) }));
|
||||
const visible = filterFields
|
||||
? shaped.filter((s) => shapeMatchesFilter(s.shape, filterFields))
|
||||
: shaped;
|
||||
@@ -2780,6 +2817,7 @@ Examples:
|
||||
count: visible.length,
|
||||
filtered_out: filteredOut,
|
||||
};
|
||||
if (reusedCache) envelope.cache_reused = true;
|
||||
if (filterFields) {
|
||||
envelope.filter = filterFields;
|
||||
envelope.filter_dropped = filterDropped;
|
||||
|
||||
Reference in New Issue
Block a user