mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
feat(showcase): bin/railway Ruby tooling for Railway ops (#5082)
## Summary Adds `showcase/bin/railway` — a single-file Ruby tool (stdlib only, no Bundler/Gemfile) that exposes 9 subcommands for Showcase Railway operations: | Subcommand | Purpose | |---|---| | `snapshot` | Capture an env's services + config into a YAML snapshot. | | `restore` | Restore an env to a snapshot (force-redeploy each service). | | `rollback` | Roll a single service back one deploy (`--to DEPLOYMENT_ID` for specific). | | `rollback-commit` | Restore an env to the snapshot committed at a given git SHA. | | `promote` | Promote staging digests to production with prechecks. | | `pin` | Pin a service to a specific image digest. | | `env-diff` | Diff two envs; exits 1 on drift. | | `resolve-digest` | Resolve an image tag to its `sha256:` digest via GHCR. | | `lint-prod` | CI gate: warn if any prod service is not digest-pinned. Supports `--exit-zero` (advisory mode) and `--format json` (machine-readable output). | Spec: https://www.notion.so/36d3aa38185281df97e4cfe11dad7d47 (companion to the main rollback spec) ## Design highlights - **Stdlib only** — `net/http`, `json`, `yaml`, `optparse`. No gem deps, no Gemfile, no Bundler. - **Auth** — reads `RAILWAY_TOKEN` env var first, falls back to `~/.railway/config.json`. Never invokes `railway login`/`railway logout`/`op`. - **GraphQL** — direct calls to `https://backboard.railway.app/graphql/v2` using `serviceInstanceDeployV2` for force-redeploy. - **GHCR digest resolution** — uses OCI Distribution Spec manifest HEAD + `Docker-Content-Digest` header. Anonymous token for public packages. - **Snapshot YAML schema** — versioned (`version: 1`). Records env-var KEYS only, never values, so snapshots are safe to commit. - **Production protection** — every mutating subcommand requires `--yes` AND a typed `production` confirmation phrase on stdin. `--non-interactive` skips the prompt but still requires `--yes`. No way to mutate prod without explicit acknowledgement. - **Uniform exit codes** — 0 clean, 1 drift/findings/refused, 2 error. ## Promote precheck classes Per the spec: - **MOVE**: image digests; startCommand (only with `--include-startcommand`); auto-update-disable. - **VERIFY-REFUSE**: service-set parity, critical env-key parity (RAILWAY_TOKEN, GHCR_TOKEN, SHARED_SECRET, OPS_TRIGGER_TOKEN, POCKETBASE_SUPERUSER_*, GITHUB_APP_PRIVATE_KEY, OPENAI/ANTHROPIC/GOOGLE keys). - **WARN**: missing/extra custom domains (expected: showcase/dashboard/dojo/docs/hooks.copilotkit.ai for prod, .staging.copilotkit.ai for staging). - **IGNORE**: env-scoped URLs, volumes. ## Tests Minitest suite (stdlib) at `showcase/bin/spec/`: - `test_cli_parsing.rb` — argv parsing for each subcommand, dispatcher behavior, env aliases, `lint-prod --format` parsing. - `test_snapshot_roundtrip.rb` — YAML write/read, schema version validation, `find_service` helper. - `test_ghcr_digest.rb` — image-ref parsing across all shapes, digest resolution decision tree, 404 → nil, 5xx → raise. - `test_production_protection.rb` — staging bypasses, prod-without-yes aborts, prod+yes+non-interactive proceeds, typed-phrase prompt accept/reject. 27 runs, 70 assertions, 0 failures. ```sh ruby showcase/bin/spec/all_tests.rb ``` ## CI integration New workflow `.github/workflows/showcase_lint_prod.yml` runs on every PR that touches `showcase/**`: 1. `bin/railway lint-prod --exit-zero --format json` — checks that every prod service is digest-pinned. (Soft-skips with a warning if `RAILWAY_TOKEN` secret is unset.) 2. `ruby showcase/bin/spec/all_tests.rb` — runs the test suite. ### lint-prod is advisory during initial soak The lint-prod step currently passes `--exit-zero`, which makes the command exit 0 even when findings exist. The workflow is also resilient to snapshot/GraphQL errors: if `lint-prod` itself crashes for any reason, the workflow renders an "audit unavailable" block instead of failing the PR. Findings still print to the job log, so we can see drift, but they will not block PRs while we soak the check against real production state. **Plan to flip to enforcing:** 1. Merge this PR; let the advisory job run on every showcase PR for a few rounds. 2. Confirm the findings list stays clean (or fix any genuine drift we surface). 3. Remove `--exit-zero` (and the error-tolerant capture) from the workflow step in a follow-up one-liner PR to turn lint-prod into a hard CI gate. This avoids the failure mode where a brand-new check immediately blocks unrelated PRs because of pre-existing prod state we haven't audited yet. ### Visibility surfaces Every run renders the audit result in two places so people don't have to click into the job logs: 1. **`$GITHUB_STEP_SUMMARY`** — a structured markdown block at the top of every workflow run page. Shows on every event (`pull_request`, `push`, `workflow_dispatch`). Contains: one-line status, table of unpinned services (only — `pinned` services are not enumerated), Pacific-time run timestamp, finding count. 2. **Sticky PR comment** — on `pull_request` events, the workflow posts (or updates) a single comment per PR. The comment is keyed by the HTML marker `<!-- lint-prod-sticky-comment -->` so re-runs PATCH the same comment instead of creating duplicates. Plain `gh` CLI only — no third-party action. The workflow also writes the findings count to `$GITHUB_OUTPUT`, so a future Slack-alert step can compare against prior runs. If `lint-prod` itself fails (e.g. a snapshot/GraphQL error), both surfaces render an "audit unavailable" block with the captured error inside a `<details>` fold, rather than going blank. ## Test plan - [ ] CI passes (`Showcase: lint-prod (digest pinning)` job) - [ ] `showcase/bin/railway --help` lists all 9 subcommands - [ ] `showcase/bin/railway <sub> --help` works for every subcommand - [ ] Local: `RAILWAY_TOKEN=... showcase/bin/railway snapshot --env staging --dry-run` produces valid YAML - [ ] Local: `RAILWAY_TOKEN=... showcase/bin/railway env-diff staging production` produces a drift report - [ ] Local: `RAILWAY_TOKEN=... showcase/bin/railway lint-prod` returns 0 (or 1 if prod drift exists — informational) - [ ] Local: `RAILWAY_TOKEN=... showcase/bin/railway lint-prod --format json` emits valid JSON with `services`, `findings`, `timestamp` - [ ] CI run shows the audit block in the step summary - [ ] PR has a single sticky comment that updates (not duplicates) on re-runs
This commit is contained in:
Jordan Ritter
@@ -0,0 +1,173 @@
|
||||
name: "Showcase: lint-prod (digest pinning)"
|
||||
|
||||
# Runs `bin/railway lint-prod` on every PR that touches showcase/.
|
||||
#
|
||||
# Currently ADVISORY: the `--exit-zero` flag makes the step exit 0 even when
|
||||
# findings exist, so this workflow will not block PRs while we soak. Findings
|
||||
# still print to the step log so we can monitor drift. Once we have confidence
|
||||
# the findings are clean, remove `--exit-zero` to flip this to enforcing.
|
||||
#
|
||||
# Long-term contract: production must always be reproducible from a snapshot
|
||||
# (every service pinned to `ghcr.io/...@sha256:...`).
|
||||
#
|
||||
# Visibility surfaces:
|
||||
# - $GITHUB_STEP_SUMMARY: structured markdown table rendered at the top of
|
||||
# the workflow run page (every run, push or pull_request).
|
||||
# - Sticky PR comment: a single comment per PR, found+updated via an HTML
|
||||
# marker (<!-- lint-prod-sticky-comment -->). Only posted on
|
||||
# pull_request events; push events on main only write the step summary.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "showcase/**"
|
||||
- ".github/workflows/showcase_lint_prod.yml"
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: showcase-lint-prod-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
lint-prod:
|
||||
name: Lint production pinning (advisory)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Ruby
|
||||
uses: ruby/setup-ruby@afeafc3d1ab54a631816aba4c914a0081c12ff2f # v1.310.0
|
||||
with:
|
||||
ruby-version: "3.2"
|
||||
|
||||
- name: Lint production pinning (advisory)
|
||||
id: lint
|
||||
env:
|
||||
RAILWAY_TOKEN: ${{ secrets.RAILWAY_TOKEN }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
if [ -z "${RAILWAY_TOKEN:-}" ]; then
|
||||
echo "::warning::RAILWAY_TOKEN secret not configured; skipping lint-prod."
|
||||
echo "skipped=true" >> "$GITHUB_OUTPUT"
|
||||
echo "findings=0" >> "$GITHUB_OUTPUT"
|
||||
echo "errored=false" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
echo "skipped=false" >> "$GITHUB_OUTPUT"
|
||||
# Advisory mode: --exit-zero so findings don't block the PR while we soak.
|
||||
# Flip to enforcing by removing --exit-zero once findings are clean.
|
||||
# Also emit machine-readable JSON for the visibility renderer.
|
||||
#
|
||||
# We capture stderr + exit code so a snapshot/GraphQL error doesn't
|
||||
# abort the workflow before we can render a "audit unavailable" block.
|
||||
# The intent of advisory mode is "never block a PR on this check".
|
||||
rc=0
|
||||
ruby showcase/bin/railway lint-prod --exit-zero --format json \
|
||||
> lint-prod.json 2> lint-prod.err || rc=$?
|
||||
# Mirror to the job log for humans (best-effort; ignore errors).
|
||||
ruby showcase/bin/railway lint-prod --exit-zero || true
|
||||
if [ "${rc}" -ne 0 ] || [ ! -s lint-prod.json ]; then
|
||||
echo "::warning::lint-prod failed (rc=${rc}); rendering audit-unavailable block."
|
||||
echo "--- lint-prod stderr ---"
|
||||
cat lint-prod.err || true
|
||||
echo "errored=true" >> "$GITHUB_OUTPUT"
|
||||
echo "findings=0" >> "$GITHUB_OUTPUT"
|
||||
# Synthesize an empty payload so the renderer has something to chew on.
|
||||
ruby -rjson -rtime -e '
|
||||
err = File.exist?("lint-prod.err") ? File.read("lint-prod.err").strip : ""
|
||||
puts JSON.generate({"services" => [], "findings" => 0,
|
||||
"timestamp" => Time.now.utc.iso8601,
|
||||
"error" => err})
|
||||
' > lint-prod.json
|
||||
else
|
||||
echo "errored=false" >> "$GITHUB_OUTPUT"
|
||||
findings=$(ruby -rjson -e 'puts JSON.parse(File.read("lint-prod.json"))["findings"]')
|
||||
echo "findings=${findings}" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Render audit summary
|
||||
if: always() && steps.lint.outputs.skipped != 'true'
|
||||
id: render
|
||||
run: |
|
||||
set -uo pipefail
|
||||
# Render in America/Los_Angeles so the timestamp respects DST.
|
||||
export TZ="America/Los_Angeles"
|
||||
ruby <<'RUBY' > audit.md
|
||||
require "json"
|
||||
require "time"
|
||||
data = JSON.parse(File.read("lint-prod.json"))
|
||||
services = data["services"] || []
|
||||
findings = (data["findings"] || 0).to_i
|
||||
ts_utc = Time.parse(data["timestamp"] || Time.now.utc.iso8601)
|
||||
ts_pt = ts_utc.getlocal
|
||||
total = services.size
|
||||
err = data["error"].to_s
|
||||
out = +""
|
||||
out << "## Production Digest-Pinning Audit\n\n"
|
||||
if !err.empty?
|
||||
# Audit failed to run (e.g. snapshot/GraphQL error). Render a fallback
|
||||
# block instead of leaving the surface blank.
|
||||
out << "Audit could not run this round.\n\n"
|
||||
out << "<details><summary>Error</summary>\n\n```\n#{err}\n```\n\n</details>\n\n"
|
||||
elsif findings.zero?
|
||||
out << "All #{total} services digest-pinned.\n\n"
|
||||
else
|
||||
out << "#{findings} of #{total} service(s) not digest-pinned.\n\n"
|
||||
out << "| Service | Source.image | Status |\n"
|
||||
out << "|---|---|---|\n"
|
||||
services.each do |s|
|
||||
next if s["status"] == "pinned"
|
||||
src = s["source"].to_s.empty? ? "_(unset)_" : "`#{s['source']}`"
|
||||
out << "| #{s['name']} | #{src} | mutable-tag |\n"
|
||||
end
|
||||
out << "\n"
|
||||
end
|
||||
out << "_Run #{ts_pt.strftime('%Y-%m-%d %H:%M:%S %Z')} — #{findings} finding(s)._\n"
|
||||
puts out
|
||||
RUBY
|
||||
# Write to step summary (top of run page).
|
||||
cat audit.md >> "$GITHUB_STEP_SUMMARY"
|
||||
# Save body (with sticky marker) for the comment step.
|
||||
{
|
||||
echo "<!-- lint-prod-sticky-comment -->"
|
||||
cat audit.md
|
||||
} > comment.md
|
||||
|
||||
- name: Post/update sticky PR comment
|
||||
if: always() && github.event_name == 'pull_request' && steps.lint.outputs.skipped != 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
MARKER="<!-- lint-prod-sticky-comment -->"
|
||||
# Find existing sticky comment (returns id only).
|
||||
existing_id=$(gh api \
|
||||
"/repos/${REPO}/issues/${PR_NUMBER}/comments?per_page=100" \
|
||||
--jq ".[] | select(.body | contains(\"${MARKER}\")) | .id" \
|
||||
| head -1)
|
||||
# Build a JSON body file so we can PATCH/POST a multiline body safely.
|
||||
ruby -rjson -e 'puts JSON.generate({body: File.read("comment.md")})' \
|
||||
> comment.json
|
||||
if [ -n "${existing_id:-}" ]; then
|
||||
echo "Updating existing sticky comment id=${existing_id}"
|
||||
gh api -X PATCH \
|
||||
"/repos/${REPO}/issues/comments/${existing_id}" \
|
||||
--input comment.json > /dev/null
|
||||
else
|
||||
echo "Creating new sticky comment"
|
||||
gh pr comment "${PR_NUMBER}" --repo "${REPO}" --body-file comment.md
|
||||
fi
|
||||
|
||||
- name: Run bin/railway tests
|
||||
run: |
|
||||
ruby showcase/bin/spec/all_tests.rb
|
||||
@@ -0,0 +1,159 @@
|
||||
# `bin/railway`
|
||||
|
||||
Single-file Ruby tooling for showcase Railway operations.
|
||||
|
||||
## Why
|
||||
|
||||
The Showcase platform lives on Railway across two environments (staging and
|
||||
production). Day-to-day operations — promoting staging to production, pinning
|
||||
services to immutable image digests, rolling a bad deploy back, auditing drift
|
||||
between envs — used to require ad-hoc shell + GraphQL recipes. `bin/railway`
|
||||
makes those operations first-class CLI subcommands with consistent flags,
|
||||
exit codes, and production protection.
|
||||
|
||||
## Install
|
||||
|
||||
None. Requires system Ruby 3.x (stdlib only — no Bundler, no Gemfile).
|
||||
|
||||
```sh
|
||||
showcase/bin/railway --help
|
||||
```
|
||||
|
||||
## Auth
|
||||
|
||||
The tool reads a Railway API token from (in order):
|
||||
|
||||
1. `RAILWAY_TOKEN` environment variable
|
||||
2. `~/.railway/config.json` (the `token` field, or `user.token`)
|
||||
|
||||
It never invokes `railway login`, `railway logout`, or `op`. If neither source
|
||||
yields a token, it exits with code 2 and a clear error.
|
||||
|
||||
For GHCR digest resolution (`resolve-digest`, `pin`), set `GHCR_TOKEN` if you
|
||||
need to read private packages; public packages work anonymously via the GHCR
|
||||
`/token` endpoint.
|
||||
|
||||
## Subcommands
|
||||
|
||||
| Subcommand | Purpose |
|
||||
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `snapshot` | Capture an env's services + config into a YAML snapshot. |
|
||||
| `restore` | Restore an env to a snapshot (force-redeploy each service). |
|
||||
| `rollback` | Roll a single service back one deploy (or to a specific deployment id with `--to`). |
|
||||
| `rollback-commit` | Restore an env to the snapshot committed at a given git SHA. |
|
||||
| `promote` | Promote staging digests to production with prechecks. |
|
||||
| `pin` | Pin a service to a specific image digest. |
|
||||
| `env-diff` | Diff two envs; exits 1 on drift. |
|
||||
| `resolve-digest` | Resolve an image tag (e.g. `:latest`) to its `sha256:` digest. |
|
||||
| `lint-prod` | CI gate (advisory): warn if any prod service is not digest-pinned. `--exit-zero` for advisory mode, `--format json` for machine-readable output. |
|
||||
|
||||
Run any subcommand with `--help` for full flag list.
|
||||
|
||||
## Production protection
|
||||
|
||||
Every subcommand that mutates state requires both:
|
||||
|
||||
- `--yes` flag, **and**
|
||||
- typed confirmation of the literal string `production` on stdin
|
||||
|
||||
…before any production mutation runs. `--non-interactive` skips the prompt
|
||||
but still requires `--yes`. There is no way to mutate production without an
|
||||
explicit acknowledgement.
|
||||
|
||||
## Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | ------------------------------------------------------------------------ |
|
||||
| 0 | Clean / success |
|
||||
| 1 | Drift detected, findings reported, or promote refused for policy reasons |
|
||||
| 2 | Error (auth, network, GraphQL schema, refused confirmation, etc.) |
|
||||
|
||||
## Worked example: promote staging → production
|
||||
|
||||
```sh
|
||||
# 1. Audit drift first (read-only).
|
||||
showcase/bin/railway env-diff staging production
|
||||
# DRIFT: 3 finding(s)
|
||||
# service showcase-shell: digest sha256:abc != sha256:def
|
||||
# ...
|
||||
|
||||
# 2. Lint prod to confirm baseline is pinned.
|
||||
showcase/bin/railway lint-prod
|
||||
# OK: all production services digest-pinned.
|
||||
|
||||
# 3. Capture a "before" snapshot in case we need to roll back.
|
||||
showcase/bin/railway snapshot --env production --output before-promote.yaml
|
||||
|
||||
# 4. Run the promote with prechecks. Production confirmation prompt fires here.
|
||||
showcase/bin/railway promote --yes
|
||||
# Type 'production' to confirm promote: production
|
||||
# promoted showcase-shell -> ghcr.io/copilotkit/showcase-shell@sha256:def...
|
||||
# ...
|
||||
|
||||
# If anything goes sideways:
|
||||
showcase/bin/railway restore --env production --snapshot before-promote.yaml --yes
|
||||
```
|
||||
|
||||
## CI integration
|
||||
|
||||
`.github/workflows/showcase_lint_prod.yml` runs `bin/railway lint-prod` on
|
||||
every PR that touches `showcase/**`.
|
||||
|
||||
**Currently advisory** — the workflow passes `--exit-zero`, so findings print
|
||||
to the job log but do not fail the PR. This lets us soak the check against
|
||||
real production state before turning it into a hard gate. Once we've built
|
||||
confidence the findings are clean, remove `--exit-zero` from the workflow to
|
||||
flip the check to enforcing (exit 1 on drift).
|
||||
|
||||
Long-term contract: every production service must be pinned to an immutable
|
||||
`ghcr.io/...@sha256:...` digest, and the lint job will fail any PR that
|
||||
drifts away from that.
|
||||
|
||||
### Visibility surfaces
|
||||
|
||||
Two human-facing surfaces render the audit result every run:
|
||||
|
||||
1. **Workflow step summary** — the workflow writes a structured markdown
|
||||
block to `$GITHUB_STEP_SUMMARY` so the audit shows at the top of every
|
||||
run page (every event: `pull_request`, `push`, `workflow_dispatch`).
|
||||
2. **Sticky PR comment** — on `pull_request` events, the workflow posts (or
|
||||
updates) a single comment per PR. The comment is keyed by the HTML marker
|
||||
`<!-- lint-prod-sticky-comment -->` so re-runs update the same comment
|
||||
instead of creating duplicates.
|
||||
|
||||
Both surfaces show the same content: a one-line status, a table of the
|
||||
unpinned services (only — `pinned` services are not enumerated), and a
|
||||
Pacific-time run timestamp with the finding count.
|
||||
|
||||
### Machine-readable output
|
||||
|
||||
`lint-prod --format json` emits:
|
||||
|
||||
```json
|
||||
{
|
||||
"services": [
|
||||
{ "name": "...", "source": "...", "status": "pinned|mutable-tag" }
|
||||
],
|
||||
"findings": 3,
|
||||
"timestamp": "2026-05-27T18:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
The CI workflow uses this shape to render the step summary and PR comment.
|
||||
The `findings` count is also written to `$GITHUB_OUTPUT` so downstream jobs
|
||||
(e.g. a future Slack alert step) can compare against prior runs.
|
||||
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
ruby showcase/bin/spec/all_tests.rb
|
||||
```
|
||||
|
||||
Tests are minitest (stdlib). They cover:
|
||||
|
||||
- argv parsing per subcommand
|
||||
- snapshot YAML round-trip
|
||||
- GHCR digest-resolution decision tree (mocked HTTP)
|
||||
- production-protection prompt behavior
|
||||
|
||||
No Railway / GHCR network calls are made during tests.
|
||||
Executable
+1254
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,12 @@
|
||||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
# Entry point for bin/railway minitest suite. Discovers and runs every test_*.rb
|
||||
# in this directory.
|
||||
|
||||
$LOAD_PATH.unshift(File.expand_path("..", __dir__))
|
||||
$LOAD_PATH.unshift(__dir__)
|
||||
|
||||
require "minitest/autorun"
|
||||
|
||||
Dir.glob(File.join(__dir__, "test_*.rb")).sort.each { |f| require f }
|
||||
@@ -0,0 +1,11 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
# Helper that loads bin/railway as a library (so we can access Railway:: classes
|
||||
# without invoking the CLI).
|
||||
|
||||
require "minitest/autorun"
|
||||
|
||||
# Stub $PROGRAM_NAME so the bottom-of-file invocation guard is skipped.
|
||||
unless defined?(::Railway)
|
||||
load File.expand_path("../railway", __dir__)
|
||||
end
|
||||
@@ -0,0 +1,112 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require_relative "spec_helper"
|
||||
require "stringio"
|
||||
|
||||
class CLIParsingTest < Minitest::Test
|
||||
def test_usage_lists_all_nine_subcommands
|
||||
out = Railway.usage
|
||||
%w[snapshot restore rollback rollback-commit promote pin env-diff
|
||||
resolve-digest lint-prod].each do |sub|
|
||||
assert_includes out, sub, "usage missing subcommand: #{sub}"
|
||||
end
|
||||
end
|
||||
|
||||
def test_help_returns_zero
|
||||
rc = nil
|
||||
silence_io { rc = Railway.run(["--help"]) }
|
||||
assert_equal 0, rc
|
||||
end
|
||||
|
||||
def test_version_returns_zero
|
||||
rc = nil
|
||||
silence_io { rc = Railway.run(["--version"]) }
|
||||
assert_equal 0, rc
|
||||
end
|
||||
|
||||
def test_unknown_subcommand_returns_2
|
||||
rc = nil
|
||||
silence_io { rc = Railway.run(["definitely-not-a-cmd"]) }
|
||||
assert_equal 2, rc
|
||||
end
|
||||
|
||||
def test_snapshot_command_parses_env_and_output
|
||||
c = Railway::SnapshotCommand.new(["--env", "staging", "--output", "/tmp/x.yaml"])
|
||||
c.parser.parse!(c.argv)
|
||||
assert_equal "staging", c.options[:env]
|
||||
assert_equal "/tmp/x.yaml", c.options[:output]
|
||||
end
|
||||
|
||||
def test_restore_command_requires_env_and_snapshot
|
||||
c = Railway::RestoreCommand.new([])
|
||||
ex = nil
|
||||
silence_io { ex = assert_raises(SystemExit) { c.run } }
|
||||
assert_equal 2, ex.status
|
||||
end
|
||||
|
||||
def test_rollback_command_parses_to_flag
|
||||
c = Railway::RollbackCommand.new(["--env", "staging", "--service", "showcase-shell", "--to", "dep-123"])
|
||||
c.parser.parse!(c.argv)
|
||||
assert_equal "dep-123", c.options[:to]
|
||||
end
|
||||
|
||||
def test_envdiff_requires_two_args
|
||||
c = Railway::EnvDiffCommand.new(["staging"])
|
||||
ex = nil
|
||||
silence_io { ex = assert_raises(SystemExit) { c.run } }
|
||||
assert_equal 2, ex.status
|
||||
end
|
||||
|
||||
def test_promote_flags_parse
|
||||
c = Railway::PromoteCommand.new(["--include-startcommand", "--yes", "--dry-run"])
|
||||
c.parser.parse!(c.argv)
|
||||
assert c.options[:include_startcommand]
|
||||
assert c.options[:yes]
|
||||
assert c.options[:dry_run]
|
||||
end
|
||||
|
||||
def test_resolve_digest_requires_arg
|
||||
c = Railway::ResolveDigestCommand.new([])
|
||||
ex = nil
|
||||
silence_io { ex = assert_raises(SystemExit) { c.run } }
|
||||
assert_equal 2, ex.status
|
||||
end
|
||||
|
||||
def test_lint_prod_parses_format_and_exit_zero
|
||||
c = Railway::LintProdCommand.new(["--exit-zero", "--format", "json"])
|
||||
c.parser.parse!(c.argv)
|
||||
assert_equal true, c.instance_variable_get(:@exit_zero)
|
||||
assert_equal "json", c.instance_variable_get(:@format)
|
||||
end
|
||||
|
||||
def test_lint_prod_rejects_invalid_format
|
||||
c = Railway::LintProdCommand.new(["--format", "yaml"])
|
||||
assert_raises(OptionParser::InvalidArgument) { c.parser.parse!(c.argv) }
|
||||
end
|
||||
|
||||
def test_lint_prod_defaults_format_to_text
|
||||
c = Railway::LintProdCommand.new([])
|
||||
c.parser.parse!(c.argv)
|
||||
assert_equal "text", c.instance_variable_get(:@format)
|
||||
assert_equal false, c.instance_variable_get(:@exit_zero)
|
||||
end
|
||||
|
||||
def test_env_id_for_resolves_aliases
|
||||
assert_equal Railway::PRODUCTION_ENV_ID, Railway.env_id_for("production")
|
||||
assert_equal Railway::PRODUCTION_ENV_ID, Railway.env_id_for("prod")
|
||||
assert_equal Railway::STAGING_ENV_ID, Railway.env_id_for("staging")
|
||||
assert_equal Railway::STAGING_ENV_ID, Railway.env_id_for("stage")
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def silence_io
|
||||
orig_stdout, orig_stderr = $stdout, $stderr
|
||||
$stdout = StringIO.new
|
||||
$stderr = StringIO.new
|
||||
yield
|
||||
ensure
|
||||
$stdout = orig_stdout
|
||||
$stderr = orig_stderr
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,68 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require_relative "spec_helper"
|
||||
|
||||
class GHCRDigestTest < Minitest::Test
|
||||
# Fake HTTP layer for the GHCR client.
|
||||
class FakeHTTP
|
||||
def initialize(responses)
|
||||
@responses = responses
|
||||
end
|
||||
|
||||
def call(method:, url:, headers: {})
|
||||
key = [method, url]
|
||||
r = @responses[key] || @responses[url]
|
||||
raise "no fake response for #{key.inspect}" unless r
|
||||
r
|
||||
end
|
||||
end
|
||||
|
||||
def test_parse_image_ref_handles_all_shapes
|
||||
g = Railway::GHCR.new
|
||||
p1 = g.parse_image_ref("ghcr.io/copilotkit/showcase-shell:latest")
|
||||
assert_equal "ghcr.io", p1[:registry]
|
||||
assert_equal "copilotkit", p1[:org]
|
||||
assert_equal "showcase-shell", p1[:name]
|
||||
assert_equal "latest", p1[:tag]
|
||||
assert_nil p1[:digest]
|
||||
|
||||
p2 = g.parse_image_ref("ghcr.io/copilotkit/showcase-shell@sha256:abc")
|
||||
assert_equal "sha256:abc", p2[:digest]
|
||||
|
||||
p3 = g.parse_image_ref("ghcr.io/copilotkit/showcase-shell:latest@sha256:def")
|
||||
assert_equal "latest", p3[:tag]
|
||||
assert_equal "sha256:def", p3[:digest]
|
||||
end
|
||||
|
||||
def test_resolve_digest_returns_digest_from_header
|
||||
url = "https://ghcr.io/v2/copilotkit/showcase-shell/manifests/latest"
|
||||
fake = FakeHTTP.new(
|
||||
url => { status: 200, headers: { "docker-content-digest" => "sha256:beefcafe" }, body: "" },
|
||||
)
|
||||
g = Railway::GHCR.new(token: "x", http: fake)
|
||||
assert_equal "sha256:beefcafe", g.resolve_digest("ghcr.io/copilotkit/showcase-shell:latest")
|
||||
end
|
||||
|
||||
def test_resolve_digest_returns_existing_digest_immediately
|
||||
# When the ref already has @sha256:..., we don't hit the network at all.
|
||||
g = Railway::GHCR.new(token: "x", http: nil)
|
||||
assert_equal "sha256:abc",
|
||||
g.resolve_digest("ghcr.io/copilotkit/showcase-shell@sha256:abc")
|
||||
end
|
||||
|
||||
def test_resolve_digest_returns_nil_on_404
|
||||
url = "https://ghcr.io/v2/copilotkit/showcase-shell/manifests/nope"
|
||||
fake = FakeHTTP.new(url => { status: 404, headers: {}, body: "" })
|
||||
g = Railway::GHCR.new(token: "x", http: fake)
|
||||
assert_nil g.resolve_digest("ghcr.io/copilotkit/showcase-shell:nope")
|
||||
end
|
||||
|
||||
def test_resolve_digest_raises_on_5xx
|
||||
url = "https://ghcr.io/v2/copilotkit/showcase-shell/manifests/latest"
|
||||
fake = FakeHTTP.new(url => { status: 500, headers: {}, body: "boom" })
|
||||
g = Railway::GHCR.new(token: "x", http: fake)
|
||||
assert_raises(Railway::GHCR::Error) do
|
||||
g.resolve_digest("ghcr.io/copilotkit/showcase-shell:latest")
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,76 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require_relative "spec_helper"
|
||||
require "stringio"
|
||||
|
||||
class ProductionProtectionTest < Minitest::Test
|
||||
def test_staging_does_not_require_confirmation
|
||||
# staging is never a production env, so this returns true without prompting.
|
||||
assert_equal true, Railway.confirm_destructive!(
|
||||
env_label: "staging", action: "restore", yes: false, non_interactive: true,
|
||||
)
|
||||
end
|
||||
|
||||
def test_production_without_yes_aborts
|
||||
ex = nil
|
||||
capture_stderr do
|
||||
ex = assert_raises(SystemExit) do
|
||||
Railway.confirm_destructive!(env_label: "production", action: "restore",
|
||||
yes: false, non_interactive: true)
|
||||
end
|
||||
end
|
||||
assert_equal 2, ex.status
|
||||
end
|
||||
|
||||
def test_production_with_yes_and_non_interactive_proceeds
|
||||
# --yes + --non-interactive proceeds without prompting.
|
||||
result = capture_stderr do
|
||||
assert_equal true, Railway.confirm_destructive!(
|
||||
env_label: "production", action: "restore",
|
||||
yes: true, non_interactive: true,
|
||||
)
|
||||
end
|
||||
assert_includes result, "non-interactive"
|
||||
end
|
||||
|
||||
def test_production_with_yes_prompts_and_accepts_typed_phrase
|
||||
# Simulate the user typing 'production' on stdin.
|
||||
original_stdin = $stdin
|
||||
$stdin = StringIO.new("production\n")
|
||||
capture_stderr do
|
||||
assert_equal true, Railway.confirm_destructive!(
|
||||
env_label: "production", action: "restore",
|
||||
yes: true, non_interactive: false,
|
||||
)
|
||||
end
|
||||
ensure
|
||||
$stdin = original_stdin
|
||||
end
|
||||
|
||||
def test_production_with_yes_rejects_wrong_phrase
|
||||
original_stdin = $stdin
|
||||
$stdin = StringIO.new("yes\n")
|
||||
ex = nil
|
||||
capture_stderr do
|
||||
ex = assert_raises(SystemExit) do
|
||||
Railway.confirm_destructive!(env_label: "production",
|
||||
action: "restore",
|
||||
yes: true, non_interactive: false)
|
||||
end
|
||||
end
|
||||
assert_equal 2, ex.status
|
||||
ensure
|
||||
$stdin = original_stdin
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def capture_stderr
|
||||
original = $stderr
|
||||
$stderr = StringIO.new
|
||||
yield
|
||||
$stderr.string
|
||||
ensure
|
||||
$stderr = original
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,180 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require_relative "spec_helper"
|
||||
|
||||
# Verifies the GraphQL queries used by SnapshotCommand match Railway's
|
||||
# public schema shape (as of 2026-05). The mocks here mirror real Railway
|
||||
# responses; any drift between this file and the live schema means the
|
||||
# tool will fail at runtime — which is exactly the bug this PR fixes.
|
||||
class SnapshotGraphqlTest < Minitest::Test
|
||||
# Fake GraphQL client that returns canned responses keyed by query name.
|
||||
class FakeGQL
|
||||
def initialize(responses)
|
||||
@responses = responses
|
||||
@calls = []
|
||||
end
|
||||
|
||||
attr_reader :calls
|
||||
|
||||
def query(query_str, variables = {})
|
||||
@calls << [query_str, variables]
|
||||
# Match on the operation name (the line after `query `) so we can
|
||||
# serve different mocks for SERVICES_LIST_QUERY,
|
||||
# SERVICE_INSTANCE_QUERY, ENVIRONMENT_VARIABLES_QUERY.
|
||||
op = query_str[/query\s+(\w+)/, 1]
|
||||
response = @responses[op] || @responses[:default]
|
||||
raise "no fake response for op=#{op.inspect}" unless response
|
||||
response.respond_to?(:call) ? response.call(variables) : response
|
||||
end
|
||||
end
|
||||
|
||||
def services_list_response
|
||||
{
|
||||
"project" => {
|
||||
"id" => Railway::PROJECT_ID,
|
||||
"name" => "showcase",
|
||||
"services" => {
|
||||
"edges" => [
|
||||
{ "node" => { "id" => "svc-aimock", "name" => "aimock" } },
|
||||
{ "node" => { "id" => "svc-shell", "name" => "shell" } },
|
||||
],
|
||||
},
|
||||
},
|
||||
}
|
||||
end
|
||||
|
||||
def env_vars_response_with_per_service_keys
|
||||
{
|
||||
"environment" => {
|
||||
"id" => Railway::PRODUCTION_ENV_ID,
|
||||
"name" => "production",
|
||||
"variables" => {
|
||||
"edges" => [
|
||||
{ "node" => { "name" => "PORT", "serviceId" => "svc-aimock", "isSealed" => false } },
|
||||
{ "node" => { "name" => "NODE_OPTIONS","serviceId" => "svc-aimock", "isSealed" => false } },
|
||||
{ "node" => { "name" => "API_KEY", "serviceId" => "svc-shell", "isSealed" => true } },
|
||||
],
|
||||
},
|
||||
},
|
||||
}
|
||||
end
|
||||
|
||||
def service_instance_response(image:, start_cmd: nil, domains: [])
|
||||
{
|
||||
"serviceInstance" => {
|
||||
"id" => "inst-#{image[/sha256:[a-f0-9]+/] || 'tag'}",
|
||||
"serviceId" => "svc-aimock",
|
||||
"environmentId" => Railway::PRODUCTION_ENV_ID,
|
||||
"startCommand" => start_cmd,
|
||||
"source" => { "image" => image, "repo" => nil },
|
||||
"latestDeployment" => { "id" => "dep-1", "status" => "SUCCESS" },
|
||||
"domains" => {
|
||||
"customDomains" => domains.map { |d| { "id" => "cd-#{d}", "domain" => d } },
|
||||
"serviceDomains" => [],
|
||||
},
|
||||
},
|
||||
}
|
||||
end
|
||||
|
||||
def test_build_snapshot_uses_corrected_field_names_and_produces_pinned_entries
|
||||
fake = FakeGQL.new(
|
||||
"ProjectServices" => services_list_response,
|
||||
"EnvVariables" => env_vars_response_with_per_service_keys,
|
||||
"ServiceInstance" => lambda do |vars|
|
||||
if vars[:serviceId] == "svc-aimock"
|
||||
service_instance_response(
|
||||
image: "ghcr.io/copilotkit/showcase-aimock@sha256:cafef00d",
|
||||
start_cmd: "node /app/dist/cli.js",
|
||||
domains: ["aimock.showcase.copilotkit.ai"],
|
||||
)
|
||||
else
|
||||
service_instance_response(
|
||||
image: "ghcr.io/copilotkit/showcase-shell@sha256:beef1234",
|
||||
domains: [],
|
||||
)
|
||||
end
|
||||
end,
|
||||
)
|
||||
|
||||
cmd = Railway::SnapshotCommand.new(["--env", "production", "--dry-run"])
|
||||
cmd.instance_variable_set(:@gql, fake)
|
||||
|
||||
snap = cmd.build_snapshot(Railway::PRODUCTION_ENV_ID)
|
||||
|
||||
assert_equal 1, snap["version"]
|
||||
assert_equal 2, snap["services"].length
|
||||
|
||||
aimock = snap["services"].find { |s| s["name"] == "aimock" }
|
||||
assert_equal "ghcr.io/copilotkit/showcase-aimock@sha256:cafef00d", aimock["image"]
|
||||
assert_equal "sha256:cafef00d", aimock["digest"]
|
||||
assert_equal "ghcr.io/copilotkit/showcase-aimock", aimock["image_tag"]
|
||||
assert_equal "node /app/dist/cli.js", aimock["start_command"]
|
||||
assert_equal ["aimock.showcase.copilotkit.ai"], aimock["custom_domains"]
|
||||
assert_equal %w[NODE_OPTIONS PORT], aimock["env_keys"]
|
||||
assert_equal "dep-1", aimock["latest_deployment_id"]
|
||||
|
||||
shell = snap["services"].find { |s| s["name"] == "shell" }
|
||||
assert_equal ["API_KEY"], shell["env_keys"]
|
||||
assert_equal [], shell["custom_domains"]
|
||||
end
|
||||
|
||||
def test_lint_prod_marks_mutable_tag_when_image_is_unpinned
|
||||
fake = FakeGQL.new(
|
||||
"ProjectServices" => services_list_response,
|
||||
"EnvVariables" => env_vars_response_with_per_service_keys,
|
||||
"ServiceInstance" => lambda do |vars|
|
||||
# Both return an unpinned :latest tag.
|
||||
service_instance_response(
|
||||
image: "ghcr.io/copilotkit/#{vars[:serviceId]}:latest",
|
||||
)
|
||||
end,
|
||||
)
|
||||
snap_cmd = Railway::SnapshotCommand.new(["--env", "production", "--dry-run"])
|
||||
snap_cmd.instance_variable_set(:@gql, fake)
|
||||
snap = snap_cmd.build_snapshot(Railway::PRODUCTION_ENV_ID)
|
||||
|
||||
# All services are mutable-tag because none have @sha256:.
|
||||
snap["services"].each do |svc|
|
||||
refute svc["image"].include?("@sha256:"), "expected mutable tag for #{svc['name']}"
|
||||
assert_nil svc["digest"], "expected nil digest for #{svc['name']}"
|
||||
end
|
||||
end
|
||||
|
||||
def test_build_snapshot_queries_use_only_supported_fields
|
||||
# Regression guard: the previous version of this tool referenced
|
||||
# `Project.domains` and `Service.serviceInstances`, both of which do
|
||||
# NOT exist in Railway's public GraphQL schema. Ensure those tokens
|
||||
# never reappear in the query constants.
|
||||
sources = [
|
||||
Railway::SERVICES_LIST_QUERY,
|
||||
Railway::SERVICE_INSTANCE_QUERY,
|
||||
Railway::ENVIRONMENT_VARIABLES_QUERY,
|
||||
]
|
||||
sources.each do |q|
|
||||
refute_match(/project\s*\([^)]*\)\s*\{[^}]*\bdomains\b/m, q,
|
||||
"Project has no `domains` field — use serviceInstance.domains or the top-level domains query.")
|
||||
refute_match(/\bserviceInstances\b/m, q,
|
||||
"Service has no `serviceInstances` field — use serviceInstance(serviceId, environmentId) directly.")
|
||||
end
|
||||
end
|
||||
|
||||
def test_redeploy_mutation_uses_serviceInstanceRedeploy_not_serviceInstanceDeployV2_with_image
|
||||
# serviceInstanceDeployV2 has signature (commitSha, environmentId, serviceId).
|
||||
# It does NOT accept an `image` argument. Pinning must go through
|
||||
# serviceInstanceUpdate (source.image) + serviceInstanceRedeploy.
|
||||
refute_match(/serviceInstanceDeployV2\s*\([^)]*\bimage\b/m,
|
||||
Railway::RestoreCommand::UPDATE_IMAGE_MUTATION,
|
||||
"Restore must not call serviceInstanceDeployV2 with image arg.")
|
||||
assert_match(/serviceInstanceUpdate\s*\(/, Railway::RestoreCommand::UPDATE_IMAGE_MUTATION)
|
||||
assert_match(/source:\s*\{\s*image:/, Railway::RestoreCommand::UPDATE_IMAGE_MUTATION)
|
||||
assert_match(/serviceInstanceRedeploy\s*\(/, Railway::RestoreCommand::REDEPLOY_MUTATION)
|
||||
end
|
||||
|
||||
def test_deploymentRollback_mutation_has_no_selection_set_because_it_returns_boolean
|
||||
# deploymentRollback's return type is Boolean (scalar). GraphQL
|
||||
# forbids a selection set on scalar fields.
|
||||
refute_match(/deploymentRollback\s*\([^)]*\)\s*\{/m,
|
||||
Railway::RollbackCommand::ROLLBACK_MUTATION,
|
||||
"deploymentRollback returns Boolean; no selection set allowed.")
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,61 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require_relative "spec_helper"
|
||||
require "tempfile"
|
||||
require "stringio"
|
||||
|
||||
class SnapshotRoundtripTest < Minitest::Test
|
||||
def sample_snapshot
|
||||
{
|
||||
"version" => 1,
|
||||
"captured_at" => "2026-01-01T00:00:00Z",
|
||||
"project_id" => Railway::PROJECT_ID,
|
||||
"environment" => { "id" => Railway::STAGING_ENV_ID, "name" => "staging" },
|
||||
"services" => [
|
||||
{
|
||||
"name" => "showcase-shell",
|
||||
"service_id" => "svc-1",
|
||||
"image" => "ghcr.io/copilotkit/showcase-shell@sha256:abc",
|
||||
"image_tag" => "ghcr.io/copilotkit/showcase-shell",
|
||||
"digest" => "sha256:abc",
|
||||
"start_command" => "node server.js",
|
||||
"auto_updates_disabled" => nil,
|
||||
"latest_deployment_id" => "dep-1",
|
||||
"env_keys" => %w[KEY1 KEY2],
|
||||
"custom_domains" => ["showcase.staging.copilotkit.ai"],
|
||||
},
|
||||
],
|
||||
}
|
||||
end
|
||||
|
||||
def test_write_and_read_roundtrip
|
||||
snap = sample_snapshot
|
||||
Tempfile.create(["snap", ".yaml"]) do |f|
|
||||
Railway::SnapshotIO.write(f.path, snap)
|
||||
loaded = Railway::SnapshotIO.read(f.path)
|
||||
assert_equal snap["version"], loaded["version"]
|
||||
assert_equal "showcase-shell", loaded["services"][0]["name"]
|
||||
assert_equal "sha256:abc", loaded["services"][0]["digest"]
|
||||
end
|
||||
end
|
||||
|
||||
def test_read_rejects_wrong_schema_version
|
||||
bad = sample_snapshot
|
||||
bad["version"] = 999
|
||||
Tempfile.create(["snap", ".yaml"]) do |f|
|
||||
File.write(f.path, YAML.dump(bad))
|
||||
orig = $stderr
|
||||
$stderr = StringIO.new
|
||||
ex = assert_raises(SystemExit) { Railway::SnapshotIO.read(f.path) }
|
||||
$stderr = orig
|
||||
assert_equal 2, ex.status
|
||||
end
|
||||
end
|
||||
|
||||
def test_find_service_by_name
|
||||
snap = sample_snapshot
|
||||
svc = Railway.find_service(snap, "showcase-shell")
|
||||
assert_equal "svc-1", svc["service_id"]
|
||||
assert_nil Railway.find_service(snap, "does-not-exist")
|
||||
end
|
||||
end
|
||||
Reference in New Issue
Block a user