Files
trailofbits__skills/plugins/gh-cli/hooks/test_helper.bash
T

131 lines
4.1 KiB
Bash
Raw Normal View History

#!/usr/bin/env bash
# Test helper functions for gh-cli BATS tests
# shellcheck disable=SC2154 # status/output are BATS-provided variables
# shellcheck disable=SC2016 # Single quotes for jq filter are intentional
# Paths to the hook scripts under test
FETCH_HOOK="${BATS_TEST_DIRNAME}/intercept-github-fetch.sh"
CURL_HOOK="${BATS_TEST_DIRNAME}/intercept-github-curl.sh"
# Run the WebFetch hook with a URL
# Usage: run_fetch_hook "https://github.com/owner/repo"
run_fetch_hook() {
local url="$1"
run bash -c 'jq -n --arg url "$1" '"'"'{"tool_input":{"url":$url,"prompt":"test"}}'"'"' | "$2"' _ "$url" "$FETCH_HOOK"
}
gh-cli: intercept GitHub fetches from MCP fetch tools (#274) * gh-cli: intercept GitHub fetches from MCP fetch tools The fetch hook only matched WebFetch, so a GitHub URL fetched through an MCP fetch tool bypassed it entirely. Match `mcp__.*[Ff]etch` as well, which covers Exa's `web_fetch_exa` and equivalents from other MCP servers. A matcher alone is not enough: WebFetch passes a single `url`, while MCP fetch tools pass a `urls` array and batch several pages into one call. Read both shapes. A tool call is atomic, so one GitHub URL anywhere in a batch denies the whole call, and each offending URL is labeled with its own suggestion. Single-URL calls keep the message they had. Split the URL classification into suggest_api, suggest_raw, and suggest_github_com behind a suggest_for_url dispatcher so it can run per URL in a loop, and collapse eight verbatim copies of the clone hint into one helper. Behavior is unchanged; blob and tree merge into one branch because they emitted identical text. The plugin README's interception table now also lists the pull, issues, releases, and gist patterns the hook already handled but never documented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * gh-cli: address review on the MCP fetch interceptor Test the matcher. Nothing exercised the regex that decides whether the hook runs at all, so a matcher firing on nothing would still pass every other suite here. matcher.bats reads it out of hooks.json and checks it against real tool names, and refuses to run against an empty matcher, which would make every match assertion pass vacuously. Widen the matcher to fetch, scrape, crawl, and extract. Firecrawl's scrape and Tavily's extract retrieve a URL like any fetch tool but carry no "fetch" in the name, so they bypassed the hook while the README promised "any MCP fetch tool". The README now names what matches. Check a string-valued `urls`. A server declaring `urls: string | string[]` sent a bare string, `arrays` dropped it, and the fetch went out unauthenticated. Listing the field twice keeps it under both shapes. `prompt` is still not scanned, so a prompt mentioning a GitHub URL does not false-deny. Give the closing note its own line when several URLs are denied; it previously trailed only the last entry. Guard assert_suggestion_starts_with against an empty reason and an empty prefix, either of which let it pass while inspecting nothing. Document the api.github.com contents, releases, and actions rows. The generic `gh api` row was actively wrong for /contents/, pointing readers at the anti-pattern the shim exists to block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 10:38:13 -04:00
# Run the fetch hook with a `urls` array, as MCP fetch tools (e.g. Exa's
# web_fetch_exa) pass them.
# Usage: run_fetch_hook_urls "https://example.com" "https://github.com/owner/repo"
run_fetch_hook_urls() {
local hook="$FETCH_HOOK"
run bash -c 'hook="$1"; shift; jq -n '"'"'{"tool_input":{"urls":$ARGS.positional}}'"'"' --args "$@" | "$hook"' _ "$hook" "$@"
}
# Run the Bash hook with a command
# Usage: run_curl_hook "curl https://api.github.com/..."
run_curl_hook() {
local cmd="$1"
run bash -c 'jq -n --arg cmd "$1" '"'"'{"tool_input":{"command":$cmd}}'"'"' | "$2"' _ "$cmd" "$CURL_HOOK"
}
# Run hook without gh available (for testing early exit)
better ci validations, fix marketplace error (#85) * better ci validations, fix marketplace error * fix lints * fix lint script * fix lint script2 * fix ruff errors * fix abs path regex * fix shellcheck lint * fix: resolve code review findings for PR #85 - Remove dead CHANGED_FILES code path (unreachable in CI) - Add version and description consistency checks between plugin.json and marketplace.json - Eliminate duplicate plugin.json reads via parse_plugin_json() - Fix bats test discovery for filenames with spaces (print0/xargs -0) - Sync marketplace.json with plugin.json for 5 pre-existing mismatches (second-opinion version, ask-questions/burpsuite/fix-review/insecure-defaults descriptions) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: gh-cli bats tests fail when gh is in /usr/bin The _no_gh test helpers used PATH=/usr/bin:/bin to exclude gh, but on Ubuntu CI runners gh is installed at /usr/bin/gh. Fix by creating a temp directory with symlinks to only jq and bash. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: improve plugin descriptions for better skill triggering - ask-questions-if-underspecified: restore trigger context ("asking questions") while keeping invocation constraint - burpsuite-project-parser: drop filler "directly from the command line", use outcome-oriented "for security analysis" - insecure-defaults: restore specific scenarios (hardcoded credentials, fallback secrets, weak auth defaults) for better matching Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Dan Guido <dan@trailofbits.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 16:28:33 +01:00
# Create a minimal PATH containing jq but not gh.
# On Ubuntu CI runners, both live in /usr/bin, so we can't just restrict PATH.
# Instead, create a temp dir with only a jq symlink.
_make_path_without_gh() {
local tmpdir
tmpdir="$(mktemp -d)"
ln -s "$(command -v jq)" "${tmpdir}/jq"
ln -s "$(command -v bash)" "${tmpdir}/bash"
echo "$tmpdir"
}
run_fetch_hook_no_gh() {
local url="$1"
better ci validations, fix marketplace error (#85) * better ci validations, fix marketplace error * fix lints * fix lint script * fix lint script2 * fix ruff errors * fix abs path regex * fix shellcheck lint * fix: resolve code review findings for PR #85 - Remove dead CHANGED_FILES code path (unreachable in CI) - Add version and description consistency checks between plugin.json and marketplace.json - Eliminate duplicate plugin.json reads via parse_plugin_json() - Fix bats test discovery for filenames with spaces (print0/xargs -0) - Sync marketplace.json with plugin.json for 5 pre-existing mismatches (second-opinion version, ask-questions/burpsuite/fix-review/insecure-defaults descriptions) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: gh-cli bats tests fail when gh is in /usr/bin The _no_gh test helpers used PATH=/usr/bin:/bin to exclude gh, but on Ubuntu CI runners gh is installed at /usr/bin/gh. Fix by creating a temp directory with symlinks to only jq and bash. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: improve plugin descriptions for better skill triggering - ask-questions-if-underspecified: restore trigger context ("asking questions") while keeping invocation constraint - burpsuite-project-parser: drop filler "directly from the command line", use outcome-oriented "for security analysis" - insecure-defaults: restore specific scenarios (hardcoded credentials, fallback secrets, weak auth defaults) for better matching Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Dan Guido <dan@trailofbits.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 16:28:33 +01:00
local safe_path
safe_path="$(_make_path_without_gh)"
run env PATH="$safe_path" bash -c 'jq -n --arg url "$1" '"'"'{"tool_input":{"url":$url,"prompt":"test"}}'"'"' 2>/dev/null | "$2"' _ "$url" "$FETCH_HOOK"
rm -rf "$safe_path"
}
run_curl_hook_no_gh() {
local cmd="$1"
better ci validations, fix marketplace error (#85) * better ci validations, fix marketplace error * fix lints * fix lint script * fix lint script2 * fix ruff errors * fix abs path regex * fix shellcheck lint * fix: resolve code review findings for PR #85 - Remove dead CHANGED_FILES code path (unreachable in CI) - Add version and description consistency checks between plugin.json and marketplace.json - Eliminate duplicate plugin.json reads via parse_plugin_json() - Fix bats test discovery for filenames with spaces (print0/xargs -0) - Sync marketplace.json with plugin.json for 5 pre-existing mismatches (second-opinion version, ask-questions/burpsuite/fix-review/insecure-defaults descriptions) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: gh-cli bats tests fail when gh is in /usr/bin The _no_gh test helpers used PATH=/usr/bin:/bin to exclude gh, but on Ubuntu CI runners gh is installed at /usr/bin/gh. Fix by creating a temp directory with symlinks to only jq and bash. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: improve plugin descriptions for better skill triggering - ask-questions-if-underspecified: restore trigger context ("asking questions") while keeping invocation constraint - burpsuite-project-parser: drop filler "directly from the command line", use outcome-oriented "for security analysis" - insecure-defaults: restore specific scenarios (hardcoded credentials, fallback secrets, weak auth defaults) for better matching Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Dan Guido <dan@trailofbits.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 16:28:33 +01:00
local safe_path
safe_path="$(_make_path_without_gh)"
run env PATH="$safe_path" bash -c 'jq -n --arg cmd "$1" '"'"'{"tool_input":{"command":$cmd}}'"'"' 2>/dev/null | "$2"' _ "$cmd" "$CURL_HOOK"
rm -rf "$safe_path"
}
# Assert the hook allowed the action (exit 0, no output)
assert_allow() {
if [[ $status -ne 0 ]]; then
echo "Expected exit 0 (allow), got exit $status"
echo "Output: $output"
return 1
fi
if [[ -n "$output" ]]; then
echo "Expected no output (allow), got:"
echo "$output"
return 1
fi
}
# Assert the hook denied the action (JSON output with deny decision)
assert_deny() {
if [[ $status -ne 0 ]]; then
echo "Expected exit 0 with deny JSON, got exit $status"
echo "Output: $output"
return 1
fi
if [[ -z "$output" ]]; then
echo "Expected deny JSON output, got empty"
return 1
fi
if ! echo "$output" | jq -e '.hookSpecificOutput.permissionDecision == "deny"' >/dev/null 2>&1; then
echo "Expected permissionDecision: deny"
echo "Output: $output"
return 1
fi
}
# Assert the suggestion contains expected text
# Usage: assert_suggestion_contains "gh repo view"
assert_suggestion_contains() {
local expected="$1"
local reason
reason=$(echo "$output" | jq -r '.hookSpecificOutput.permissionDecisionReason // empty' 2>/dev/null)
if [[ -z "$reason" ]]; then
echo "No permissionDecisionReason found in output"
echo "Output: $output"
return 1
fi
if [[ "$reason" != *"$expected"* ]]; then
echo "Expected suggestion to contain: $expected"
echo "Got: $reason"
return 1
fi
}
gh-cli: intercept GitHub fetches from MCP fetch tools (#274) * gh-cli: intercept GitHub fetches from MCP fetch tools The fetch hook only matched WebFetch, so a GitHub URL fetched through an MCP fetch tool bypassed it entirely. Match `mcp__.*[Ff]etch` as well, which covers Exa's `web_fetch_exa` and equivalents from other MCP servers. A matcher alone is not enough: WebFetch passes a single `url`, while MCP fetch tools pass a `urls` array and batch several pages into one call. Read both shapes. A tool call is atomic, so one GitHub URL anywhere in a batch denies the whole call, and each offending URL is labeled with its own suggestion. Single-URL calls keep the message they had. Split the URL classification into suggest_api, suggest_raw, and suggest_github_com behind a suggest_for_url dispatcher so it can run per URL in a loop, and collapse eight verbatim copies of the clone hint into one helper. Behavior is unchanged; blob and tree merge into one branch because they emitted identical text. The plugin README's interception table now also lists the pull, issues, releases, and gist patterns the hook already handled but never documented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * gh-cli: address review on the MCP fetch interceptor Test the matcher. Nothing exercised the regex that decides whether the hook runs at all, so a matcher firing on nothing would still pass every other suite here. matcher.bats reads it out of hooks.json and checks it against real tool names, and refuses to run against an empty matcher, which would make every match assertion pass vacuously. Widen the matcher to fetch, scrape, crawl, and extract. Firecrawl's scrape and Tavily's extract retrieve a URL like any fetch tool but carry no "fetch" in the name, so they bypassed the hook while the README promised "any MCP fetch tool". The README now names what matches. Check a string-valued `urls`. A server declaring `urls: string | string[]` sent a bare string, `arrays` dropped it, and the fetch went out unauthenticated. Listing the field twice keeps it under both shapes. `prompt` is still not scanned, so a prompt mentioning a GitHub URL does not false-deny. Give the closing note its own line when several URLs are denied; it previously trailed only the last entry. Guard assert_suggestion_starts_with against an empty reason and an empty prefix, either of which let it pass while inspecting nothing. Document the api.github.com contents, releases, and actions rows. The generic `gh api` row was actively wrong for /contents/, pointing readers at the anti-pattern the shim exists to block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 10:38:13 -04:00
# Assert the suggestion begins with expected text
# Usage: assert_suggestion_starts_with "Use "
assert_suggestion_starts_with() {
local expected="$1"
local reason
if [[ -z "$expected" ]]; then
echo "assert_suggestion_starts_with called with an empty prefix — it would pass vacuously"
return 1
fi
reason=$(echo "$output" | jq -r '.hookSpecificOutput.permissionDecisionReason // empty' 2>/dev/null)
if [[ -z "$reason" ]]; then
echo "No permissionDecisionReason found in output"
echo "Output: $output"
return 1
fi
if [[ "$reason" != "$expected"* ]]; then
echo "Expected suggestion to start with: $expected"
echo "Got: $reason"
return 1
fi
}