mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
feat(contracts): ADR-103 structured verdict handoffs — RiskDecision et al
Versioned, additive-only verdict envelopes for agent/MCP handoffs:
RiskDecision (quality gates), FindingVerdict (review findings with
adversarial refutations), CoverageGap (risk-weighted gaps).
- src/contracts/verdicts.ts: dependency-free validators (source of
truth) + mirrored draft-07 JSON Schemas; no zod/ajv runtime deps
(light-install posture preserved)
- quality_assess MCP boundary now attaches a validated riskDecision
envelope derived from the gate outcome (approve/block, escalate when
indeterminate) — additive, existing payload unchanged; direct and
wrapped handlers share the same mapToResult so MCP parity is inherent
- schemas/*.schema.json published via scripts/generate-verdict-schemas.mjs
for external ajv validation and workflow agent({schema}) enforcement
(consumed by improvement 5)
17 contract tests (goldens validate; mutations rejected with field-level
errors; builder always emits valid envelopes). E2E: envelope emitted by
the compiled boundary builder validates with stock ajv against the
published schema; mutated sample rejected. tsc + eslint clean.
Part of #520 (improvement 6/7, Tier 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,96 @@
|
||||
# ADR-103: Structured JSON-Schema Verdict Handoffs
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| **Decision ID** | ADR-103 |
|
||||
| **Status** | Implemented |
|
||||
| **Date** | 2026-06-10 |
|
||||
| **Author** | AQE Core (Fable 5 improvement initiative, issue #520) |
|
||||
| **Review Cadence** | 6 months |
|
||||
|
||||
---
|
||||
|
||||
## WH(Y) Decision Statement
|
||||
|
||||
**In the context of** handoffs between QE agents and at MCP tool boundaries, which today exchange prose and ad-hoc JSON,
|
||||
|
||||
**facing** unverifiable agent outputs — a quality gate's "decision" is whatever shape the producing code happened to emit, so consumers (workflows, dashboards, downstream agents) cannot validate, retry, or rank what they receive,
|
||||
|
||||
**we decided for** versioned, additive-only verdict envelopes (`RiskDecision`, `FindingVerdict`, `CoverageGap`) with dependency-free TypeScript validators as the source of truth (`src/contracts/verdicts.ts`), mirrored as published draft-07 JSON Schemas (`schemas/*.schema.json`), validated at the MCP boundary before emission,
|
||||
|
||||
**and neglected** adding zod/ajv as runtime dependencies (the package deliberately keeps installs light; the envelope subset is simple enough for hand-rolled validators) and restructuring existing MCP payloads (breaking consumers — the envelope is attached additively),
|
||||
|
||||
**to achieve** machine-verifiable handoffs that workflow `agent({schema})` calls can enforce with auto-retry (improvement 5) and external tooling can validate with stock ajv,
|
||||
|
||||
**accepting that** validators and schemas are maintained in one module and kept in lockstep by contract tests, and that only the quality-gate boundary emits an envelope so far (finding/coverage envelopes ship with their producers in improvements 5/7).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Part of the Fable 5 plan's "SynthLang" thread: replace prose handoffs with explicit constraint-bearing structures. ruflo's neural-trader pipeline (RegimeVerdict → SignalProposal → RiskDecision as typed gates) is the prior art. The first wired boundary is `quality_assess`: `mapToResult` derives a `RiskDecision` from the gate outcome (passed → approve, failed → block, indeterminate → escalate), validates it, and attaches it additively — both the direct and wrapped MCP handlers flow through the same config, so MCP parity is inherent.
|
||||
|
||||
## Options Considered
|
||||
|
||||
### Option 1: Dependency-free validators + published JSON Schemas (Selected)
|
||||
|
||||
**Pros:** zero new runtime deps; validators boundary-grade and unit-tested; schemas consumable by ajv and workflow structured-output enforcement; additive envelopes can't break existing consumers
|
||||
**Cons:** two artifacts to keep in sync (enforced by contract tests + the generator script)
|
||||
|
||||
### Option 2: zod + zod-to-json-schema (Rejected)
|
||||
|
||||
**Why rejected:** adds runtime dependencies to a package with a deliberate light-install posture (and a history of supply-chain audits); the schema subset needed here is trivial.
|
||||
|
||||
### Option 3: Extend ADR-075's framework type system (Rejected)
|
||||
|
||||
**Why rejected:** ADR-075 types describe test frameworks, not verdict envelopes; grafting envelopes there couples unrelated lifecycles. Relates-to, not part-of.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Relationship | ADR ID | Title | Notes |
|
||||
|--------------|--------|-------|-------|
|
||||
| Relates To | ADR-075 | Unified Test Framework Type System | Adjacent type system, deliberately not extended |
|
||||
| Relates To | ADR-054 | A2A Protocol | Verdicts ride inside A2A envelopes unchanged |
|
||||
| Relates To | ADR-074 | Loki-Mode Adversarial Gates | `FindingVerdict.refutations` carries refuter votes (improvement 5) |
|
||||
| Part Of | — | Fable 5 / ruflo-parity initiative | Tracking issue #520, improvement 6 |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
| Ref ID | Title | Type | Location |
|
||||
|--------|-------|------|----------|
|
||||
| — | Contracts + validators + schemas | Source | `src/contracts/verdicts.ts` |
|
||||
| — | Boundary wiring | Source | `src/mcp/handlers/domain-handler-configs.ts` (qualityAssess `mapToResult`) |
|
||||
| — | Published schemas | Artifact | `schemas/{risk-decision,finding-verdict,coverage-gap}.schema.json` |
|
||||
| — | Generator | Script | `scripts/generate-verdict-schemas.mjs` |
|
||||
| — | Contract tests (17) | Tests | `tests/unit/contracts/verdicts.test.ts` |
|
||||
|
||||
---
|
||||
|
||||
## Governance
|
||||
|
||||
| Review Board | Date | Outcome | Next Review |
|
||||
|--------------|------|---------|-------------|
|
||||
| AQE Core | 2026-06-10 | Implemented | 2026-12-10 |
|
||||
|
||||
---
|
||||
|
||||
## Status History
|
||||
|
||||
| Status | Date | Notes |
|
||||
|--------|------|-------|
|
||||
| Proposed | 2026-06-10 | From Fable 5 improvement plan (issue #520) |
|
||||
| Implemented | 2026-06-10 | Contracts + quality-gate boundary + published schemas + 17 contract tests; ajv round-trip verified (golden valid, mutated invalid) |
|
||||
|
||||
---
|
||||
|
||||
## Definition of Done Checklist
|
||||
|
||||
- [x] Evidence: 17 contract tests green; emitted envelope from the compiled boundary builder validates with stock ajv against the published schema; mutated sample rejected
|
||||
- [x] Criteria: 3 options compared; additive-only rule encoded (`additionalProperties: true`, envelope `contract` discriminator)
|
||||
- [x] Agreement: follows the improvement plan; ruflo typed-gate prior art
|
||||
- [x] Documentation: this ADR; generator workflow documented in the script header
|
||||
- [x] Review: verification record on issue #520
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"$id": "https://agentic-qe.dev/schemas/coverage-gap.schema.json",
|
||||
"title": "CoverageGap",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"contract",
|
||||
"file",
|
||||
"riskScore",
|
||||
"suggestedTests"
|
||||
],
|
||||
"additionalProperties": true,
|
||||
"properties": {
|
||||
"contract": {
|
||||
"const": "coverage-gap@1"
|
||||
},
|
||||
"file": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"rangeStart": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
},
|
||||
"rangeEnd": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
},
|
||||
"riskScore": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"maximum": 1
|
||||
},
|
||||
"suggestedTests": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"$id": "https://agentic-qe.dev/schemas/finding-verdict.schema.json",
|
||||
"title": "FindingVerdict",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"contract",
|
||||
"id",
|
||||
"title",
|
||||
"severity",
|
||||
"confidence",
|
||||
"evidence",
|
||||
"verdict",
|
||||
"refutations"
|
||||
],
|
||||
"additionalProperties": true,
|
||||
"properties": {
|
||||
"contract": {
|
||||
"const": "finding-verdict@1"
|
||||
},
|
||||
"id": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"title": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"file": {
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"enum": [
|
||||
"critical",
|
||||
"high",
|
||||
"medium",
|
||||
"low",
|
||||
"info"
|
||||
]
|
||||
},
|
||||
"confidence": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"maximum": 1
|
||||
},
|
||||
"evidence": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"verdict": {
|
||||
"enum": [
|
||||
"upheld",
|
||||
"refuted",
|
||||
"uncertain"
|
||||
]
|
||||
},
|
||||
"refutations": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"$id": "https://agentic-qe.dev/schemas/risk-decision.schema.json",
|
||||
"title": "RiskDecision",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"contract",
|
||||
"decision",
|
||||
"riskFactors",
|
||||
"confidence",
|
||||
"rationale"
|
||||
],
|
||||
"additionalProperties": true,
|
||||
"properties": {
|
||||
"contract": {
|
||||
"const": "risk-decision@1"
|
||||
},
|
||||
"decision": {
|
||||
"enum": [
|
||||
"approve",
|
||||
"block",
|
||||
"escalate"
|
||||
]
|
||||
},
|
||||
"riskFactors": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"confidence": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"maximum": 1
|
||||
},
|
||||
"rationale": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* ADR-103 — Publish the verdict JSON Schemas from the compiled contracts
|
||||
* module to schemas/*.schema.json. The TypeScript validators in
|
||||
* src/contracts/verdicts.ts are the source of truth; run this after
|
||||
* changing them (requires a prior `npx tsc` emit).
|
||||
*
|
||||
* Usage: node scripts/generate-verdict-schemas.mjs
|
||||
*/
|
||||
|
||||
import { mkdirSync, writeFileSync } from 'node:fs';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const root = join(dirname(fileURLToPath(import.meta.url)), '..');
|
||||
const { VERDICT_SCHEMAS } = await import(join(root, 'dist', 'contracts', 'verdicts.js'));
|
||||
|
||||
const outDir = join(root, 'schemas');
|
||||
mkdirSync(outDir, { recursive: true });
|
||||
|
||||
for (const [name, schema] of Object.entries(VERDICT_SCHEMAS)) {
|
||||
const file = join(outDir, `${name}.schema.json`);
|
||||
writeFileSync(file, JSON.stringify(schema, null, 2) + '\n');
|
||||
console.log(`wrote ${file}`);
|
||||
}
|
||||
@@ -0,0 +1,235 @@
|
||||
/**
|
||||
* Agentic QE v3 - Structured Verdict Contracts (ADR-103)
|
||||
*
|
||||
* Typed, versioned verdict envelopes for agent-to-agent and MCP-boundary
|
||||
* handoffs: RiskDecision (quality gates), FindingVerdict (review findings
|
||||
* with adversarial refutations), CoverageGap (risk-weighted gap reports).
|
||||
*
|
||||
* Validators are dependency-free and are the source of truth; the exported
|
||||
* JSON Schemas (published to schemas/*.schema.json by
|
||||
* scripts/generate-verdict-schemas.mjs) mirror them for external tooling
|
||||
* (ajv, workflow agent() schema option). Envelopes are versioned and
|
||||
* additive-only within a major version: consumers must tolerate unknown
|
||||
* fields, so schemas keep `additionalProperties: true`.
|
||||
*/
|
||||
|
||||
export const VERDICT_CONTRACT_VERSION = '1.0' as const;
|
||||
|
||||
// ============================================================================
|
||||
// Types
|
||||
// ============================================================================
|
||||
|
||||
export type RiskDecisionOutcome = 'approve' | 'block' | 'escalate';
|
||||
|
||||
export interface RiskDecision {
|
||||
/** Envelope discriminator + version */
|
||||
contract: 'risk-decision@1';
|
||||
decision: RiskDecisionOutcome;
|
||||
riskFactors: string[];
|
||||
/** 0..1 */
|
||||
confidence: number;
|
||||
rationale: string;
|
||||
}
|
||||
|
||||
export type FindingSeverity = 'critical' | 'high' | 'medium' | 'low' | 'info';
|
||||
export type FindingOutcome = 'upheld' | 'refuted' | 'uncertain';
|
||||
|
||||
export interface FindingVerdict {
|
||||
contract: 'finding-verdict@1';
|
||||
id: string;
|
||||
title: string;
|
||||
file?: string;
|
||||
severity: FindingSeverity;
|
||||
/** 0..1 */
|
||||
confidence: number;
|
||||
evidence: string[];
|
||||
verdict: FindingOutcome;
|
||||
/** One entry per refuter that voted to refute (empty when none) */
|
||||
refutations: string[];
|
||||
}
|
||||
|
||||
export interface CoverageGap {
|
||||
contract: 'coverage-gap@1';
|
||||
file: string;
|
||||
/** 1-based inclusive line range; omit for whole-file gaps */
|
||||
rangeStart?: number;
|
||||
rangeEnd?: number;
|
||||
/** 0..1 risk weighting */
|
||||
riskScore: number;
|
||||
suggestedTests: string[];
|
||||
}
|
||||
|
||||
export interface ValidationResult {
|
||||
valid: boolean;
|
||||
errors: string[];
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Validators (dependency-free, boundary-grade)
|
||||
// ============================================================================
|
||||
|
||||
function isRecord(v: unknown): v is Record<string, unknown> {
|
||||
return typeof v === 'object' && v !== null && !Array.isArray(v);
|
||||
}
|
||||
|
||||
function isStringArray(v: unknown): v is string[] {
|
||||
return Array.isArray(v) && v.every((x) => typeof x === 'string');
|
||||
}
|
||||
|
||||
function inUnitRange(v: unknown): v is number {
|
||||
return typeof v === 'number' && Number.isFinite(v) && v >= 0 && v <= 1;
|
||||
}
|
||||
|
||||
export function validateRiskDecision(value: unknown): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
if (!isRecord(value)) return { valid: false, errors: ['not an object'] };
|
||||
if (value.contract !== 'risk-decision@1') errors.push(`contract must be "risk-decision@1"`);
|
||||
if (!['approve', 'block', 'escalate'].includes(value.decision as string)) {
|
||||
errors.push('decision must be approve|block|escalate');
|
||||
}
|
||||
if (!isStringArray(value.riskFactors)) errors.push('riskFactors must be string[]');
|
||||
if (!inUnitRange(value.confidence)) errors.push('confidence must be a number in [0,1]');
|
||||
if (typeof value.rationale !== 'string' || value.rationale.length === 0) {
|
||||
errors.push('rationale must be a non-empty string');
|
||||
}
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
export function validateFindingVerdict(value: unknown): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
if (!isRecord(value)) return { valid: false, errors: ['not an object'] };
|
||||
if (value.contract !== 'finding-verdict@1') errors.push(`contract must be "finding-verdict@1"`);
|
||||
if (typeof value.id !== 'string' || value.id.length === 0) errors.push('id must be a non-empty string');
|
||||
if (typeof value.title !== 'string' || value.title.length === 0) errors.push('title must be a non-empty string');
|
||||
if (value.file !== undefined && typeof value.file !== 'string') errors.push('file must be a string when present');
|
||||
if (!['critical', 'high', 'medium', 'low', 'info'].includes(value.severity as string)) {
|
||||
errors.push('severity must be critical|high|medium|low|info');
|
||||
}
|
||||
if (!inUnitRange(value.confidence)) errors.push('confidence must be a number in [0,1]');
|
||||
if (!isStringArray(value.evidence)) errors.push('evidence must be string[]');
|
||||
if (!['upheld', 'refuted', 'uncertain'].includes(value.verdict as string)) {
|
||||
errors.push('verdict must be upheld|refuted|uncertain');
|
||||
}
|
||||
if (!isStringArray(value.refutations)) errors.push('refutations must be string[]');
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
export function validateCoverageGap(value: unknown): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
if (!isRecord(value)) return { valid: false, errors: ['not an object'] };
|
||||
if (value.contract !== 'coverage-gap@1') errors.push(`contract must be "coverage-gap@1"`);
|
||||
if (typeof value.file !== 'string' || value.file.length === 0) errors.push('file must be a non-empty string');
|
||||
for (const k of ['rangeStart', 'rangeEnd'] as const) {
|
||||
const v = value[k];
|
||||
if (v !== undefined && (!Number.isInteger(v) || (v as number) < 1)) {
|
||||
errors.push(`${k} must be a positive integer when present`);
|
||||
}
|
||||
}
|
||||
if (
|
||||
Number.isInteger(value.rangeStart) &&
|
||||
Number.isInteger(value.rangeEnd) &&
|
||||
(value.rangeEnd as number) < (value.rangeStart as number)
|
||||
) {
|
||||
errors.push('rangeEnd must be >= rangeStart');
|
||||
}
|
||||
if (!inUnitRange(value.riskScore)) errors.push('riskScore must be a number in [0,1]');
|
||||
if (!isStringArray(value.suggestedTests)) errors.push('suggestedTests must be string[]');
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Builders — derive contract envelopes at MCP boundaries
|
||||
// ============================================================================
|
||||
|
||||
/**
|
||||
* Derive a RiskDecision from a quality-gate outcome (quality_assess boundary).
|
||||
* Indeterminate gates (passed undefined) escalate rather than guess.
|
||||
*/
|
||||
export function buildRiskDecisionFromQualityGate(input: {
|
||||
passed?: boolean;
|
||||
qualityScore?: number;
|
||||
recommendations?: string[];
|
||||
}): RiskDecision {
|
||||
const { passed, qualityScore, recommendations = [] } = input;
|
||||
const decision: RiskDecisionOutcome =
|
||||
passed === true ? 'approve' : passed === false ? 'block' : 'escalate';
|
||||
return {
|
||||
contract: 'risk-decision@1',
|
||||
decision,
|
||||
riskFactors: recommendations.slice(0, 10),
|
||||
confidence: decision === 'escalate' ? 0.5 : 0.9,
|
||||
rationale:
|
||||
decision === 'escalate'
|
||||
? 'Quality gate outcome indeterminate — manual review required'
|
||||
: `Quality gate ${passed ? 'passed' : 'failed'}${
|
||||
typeof qualityScore === 'number' ? ` with score ${qualityScore}` : ''
|
||||
}`,
|
||||
};
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// JSON Schemas (draft-07) — mirrors of the validators above, published to
|
||||
// schemas/*.schema.json for ajv / workflow agent({schema}) consumers
|
||||
// ============================================================================
|
||||
|
||||
const unit = { type: 'number', minimum: 0, maximum: 1 } as const;
|
||||
const stringArray = { type: 'array', items: { type: 'string' } } as const;
|
||||
|
||||
export const RISK_DECISION_SCHEMA = {
|
||||
$schema: 'http://json-schema.org/draft-07/schema#',
|
||||
$id: 'https://agentic-qe.dev/schemas/risk-decision.schema.json',
|
||||
title: 'RiskDecision',
|
||||
type: 'object',
|
||||
required: ['contract', 'decision', 'riskFactors', 'confidence', 'rationale'],
|
||||
additionalProperties: true,
|
||||
properties: {
|
||||
contract: { const: 'risk-decision@1' },
|
||||
decision: { enum: ['approve', 'block', 'escalate'] },
|
||||
riskFactors: stringArray,
|
||||
confidence: unit,
|
||||
rationale: { type: 'string', minLength: 1 },
|
||||
},
|
||||
} as const;
|
||||
|
||||
export const FINDING_VERDICT_SCHEMA = {
|
||||
$schema: 'http://json-schema.org/draft-07/schema#',
|
||||
$id: 'https://agentic-qe.dev/schemas/finding-verdict.schema.json',
|
||||
title: 'FindingVerdict',
|
||||
type: 'object',
|
||||
required: ['contract', 'id', 'title', 'severity', 'confidence', 'evidence', 'verdict', 'refutations'],
|
||||
additionalProperties: true,
|
||||
properties: {
|
||||
contract: { const: 'finding-verdict@1' },
|
||||
id: { type: 'string', minLength: 1 },
|
||||
title: { type: 'string', minLength: 1 },
|
||||
file: { type: 'string' },
|
||||
severity: { enum: ['critical', 'high', 'medium', 'low', 'info'] },
|
||||
confidence: unit,
|
||||
evidence: stringArray,
|
||||
verdict: { enum: ['upheld', 'refuted', 'uncertain'] },
|
||||
refutations: stringArray,
|
||||
},
|
||||
} as const;
|
||||
|
||||
export const COVERAGE_GAP_SCHEMA = {
|
||||
$schema: 'http://json-schema.org/draft-07/schema#',
|
||||
$id: 'https://agentic-qe.dev/schemas/coverage-gap.schema.json',
|
||||
title: 'CoverageGap',
|
||||
type: 'object',
|
||||
required: ['contract', 'file', 'riskScore', 'suggestedTests'],
|
||||
additionalProperties: true,
|
||||
properties: {
|
||||
contract: { const: 'coverage-gap@1' },
|
||||
file: { type: 'string', minLength: 1 },
|
||||
rangeStart: { type: 'integer', minimum: 1 },
|
||||
rangeEnd: { type: 'integer', minimum: 1 },
|
||||
riskScore: unit,
|
||||
suggestedTests: stringArray,
|
||||
},
|
||||
} as const;
|
||||
|
||||
export const VERDICT_SCHEMAS = {
|
||||
'risk-decision': RISK_DECISION_SCHEMA,
|
||||
'finding-verdict': FINDING_VERDICT_SCHEMA,
|
||||
'coverage-gap': COVERAGE_GAP_SCHEMA,
|
||||
} as const;
|
||||
@@ -35,6 +35,11 @@ import {
|
||||
type SupportedLanguage,
|
||||
type TestFramework,
|
||||
} from '../../shared/types/test-frameworks.js';
|
||||
import {
|
||||
buildRiskDecisionFromQualityGate,
|
||||
validateRiskDecision,
|
||||
type RiskDecision,
|
||||
} from '../../contracts/verdicts.js';
|
||||
|
||||
const SUPPORTED_LANGUAGES = Object.keys(DEFAULT_FRAMEWORKS) as SupportedLanguage[];
|
||||
|
||||
@@ -92,6 +97,8 @@ export interface QualityAssessResult {
|
||||
recommendations: string[];
|
||||
duration: number;
|
||||
savedFiles?: string[];
|
||||
/** ADR-103: versioned, schema-validated gate verdict (additive) */
|
||||
riskDecision?: RiskDecision;
|
||||
}
|
||||
|
||||
export interface SecurityScanResult {
|
||||
@@ -551,16 +558,27 @@ export const qualityAssessConfig: DomainHandlerConfig<QualityAssessParams, Quali
|
||||
compiledContext: routingResult?.compiledContext,
|
||||
}),
|
||||
|
||||
mapToResult: (taskId, data, duration, savedFiles) => ({
|
||||
taskId,
|
||||
status: 'completed',
|
||||
qualityScore: (data.qualityScore as number) || 0,
|
||||
passed: (data.passed as boolean) || false,
|
||||
metrics: (data.metrics as Record<string, number>) || {},
|
||||
recommendations: (data.recommendations as string[]) || [],
|
||||
duration,
|
||||
savedFiles,
|
||||
}),
|
||||
mapToResult: (taskId, data, duration, savedFiles) => {
|
||||
// ADR-103: attach a schema-validated RiskDecision envelope at the MCP
|
||||
// boundary (additive — existing fields unchanged). Omit rather than
|
||||
// emit an invalid envelope.
|
||||
const riskDecision = buildRiskDecisionFromQualityGate({
|
||||
passed: typeof data.passed === 'boolean' ? data.passed : undefined,
|
||||
qualityScore: data.qualityScore as number | undefined,
|
||||
recommendations: (data.recommendations as string[]) || [],
|
||||
});
|
||||
return {
|
||||
taskId,
|
||||
status: 'completed',
|
||||
qualityScore: (data.qualityScore as number) || 0,
|
||||
passed: (data.passed as boolean) || false,
|
||||
metrics: (data.metrics as Record<string, number>) || {},
|
||||
recommendations: (data.recommendations as string[]) || [],
|
||||
duration,
|
||||
savedFiles,
|
||||
...(validateRiskDecision(riskDecision).valid ? { riskDecision } : {}),
|
||||
};
|
||||
},
|
||||
};
|
||||
|
||||
// ============================================================================
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
/**
|
||||
* Contract tests for verdict envelopes (ADR-103)
|
||||
*
|
||||
* Golden samples validate; mutated samples are rejected with field-level
|
||||
* errors; the quality-gate builder always emits a valid envelope.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import {
|
||||
validateRiskDecision,
|
||||
validateFindingVerdict,
|
||||
validateCoverageGap,
|
||||
buildRiskDecisionFromQualityGate,
|
||||
type RiskDecision,
|
||||
type FindingVerdict,
|
||||
type CoverageGap,
|
||||
} from '../../../src/contracts/verdicts';
|
||||
|
||||
const goldenRiskDecision: RiskDecision = {
|
||||
contract: 'risk-decision@1',
|
||||
decision: 'block',
|
||||
riskFactors: ['coverage below threshold', '3 critical complexity hotspots'],
|
||||
confidence: 0.9,
|
||||
rationale: 'Quality gate failed with score 61',
|
||||
};
|
||||
|
||||
const goldenFindingVerdict: FindingVerdict = {
|
||||
contract: 'finding-verdict@1',
|
||||
id: 'find-001',
|
||||
title: 'SQL injection in report filter',
|
||||
file: 'src/reports/filter.ts',
|
||||
severity: 'critical',
|
||||
confidence: 0.85,
|
||||
evidence: ['string concatenation at filter.ts:42', 'no parameterization'],
|
||||
verdict: 'upheld',
|
||||
refutations: [],
|
||||
};
|
||||
|
||||
const goldenCoverageGap: CoverageGap = {
|
||||
contract: 'coverage-gap@1',
|
||||
file: 'src/billing/refund.ts',
|
||||
rangeStart: 110,
|
||||
rangeEnd: 152,
|
||||
riskScore: 0.8,
|
||||
suggestedTests: ['refund over original amount', 'refund on voided invoice'],
|
||||
};
|
||||
|
||||
describe('validateRiskDecision', () => {
|
||||
it('should accept the golden sample', () => {
|
||||
expect(validateRiskDecision(goldenRiskDecision)).toEqual({ valid: true, errors: [] });
|
||||
});
|
||||
|
||||
it('should accept unknown additional fields (additive envelope)', () => {
|
||||
const result = validateRiskDecision({ ...goldenRiskDecision, futureField: 42 });
|
||||
|
||||
expect(result.valid).toBe(true);
|
||||
});
|
||||
|
||||
it('should reject an unknown decision value', () => {
|
||||
const result = validateRiskDecision({ ...goldenRiskDecision, decision: 'maybe' });
|
||||
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors.join()).toContain('decision');
|
||||
});
|
||||
|
||||
it('should reject confidence above 1', () => {
|
||||
const result = validateRiskDecision({ ...goldenRiskDecision, confidence: 1.2 });
|
||||
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors.join()).toContain('confidence');
|
||||
});
|
||||
|
||||
it('should reject a missing rationale', () => {
|
||||
const { rationale: _omitted, ...rest } = goldenRiskDecision;
|
||||
|
||||
expect(validateRiskDecision(rest).valid).toBe(false);
|
||||
});
|
||||
|
||||
it('should reject non-objects', () => {
|
||||
expect(validateRiskDecision('approve').valid).toBe(false);
|
||||
expect(validateRiskDecision(null).valid).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateFindingVerdict', () => {
|
||||
it('should accept the golden sample', () => {
|
||||
expect(validateFindingVerdict(goldenFindingVerdict)).toEqual({ valid: true, errors: [] });
|
||||
});
|
||||
|
||||
it('should reject a missing severity', () => {
|
||||
const { severity: _omitted, ...rest } = goldenFindingVerdict;
|
||||
|
||||
const result = validateFindingVerdict(rest);
|
||||
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors.join()).toContain('severity');
|
||||
});
|
||||
|
||||
it('should reject non-array evidence', () => {
|
||||
const result = validateFindingVerdict({ ...goldenFindingVerdict, evidence: 'just one string' });
|
||||
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors.join()).toContain('evidence');
|
||||
});
|
||||
|
||||
it('should reject an unknown verdict value', () => {
|
||||
expect(validateFindingVerdict({ ...goldenFindingVerdict, verdict: 'plausible' }).valid).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateCoverageGap', () => {
|
||||
it('should accept the golden sample', () => {
|
||||
expect(validateCoverageGap(goldenCoverageGap)).toEqual({ valid: true, errors: [] });
|
||||
});
|
||||
|
||||
it('should accept a whole-file gap without a range', () => {
|
||||
const { rangeStart: _s, rangeEnd: _e, ...rest } = goldenCoverageGap;
|
||||
|
||||
expect(validateCoverageGap(rest).valid).toBe(true);
|
||||
});
|
||||
|
||||
it('should reject negative riskScore', () => {
|
||||
expect(validateCoverageGap({ ...goldenCoverageGap, riskScore: -0.1 }).valid).toBe(false);
|
||||
});
|
||||
|
||||
it('should reject an inverted range', () => {
|
||||
const result = validateCoverageGap({ ...goldenCoverageGap, rangeStart: 200, rangeEnd: 100 });
|
||||
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors.join()).toContain('rangeEnd');
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildRiskDecisionFromQualityGate', () => {
|
||||
it('should approve on a passed gate and validate', () => {
|
||||
const decision = buildRiskDecisionFromQualityGate({ passed: true, qualityScore: 92 });
|
||||
|
||||
expect(decision.decision).toBe('approve');
|
||||
expect(validateRiskDecision(decision).valid).toBe(true);
|
||||
});
|
||||
|
||||
it('should block on a failed gate with recommendations as risk factors', () => {
|
||||
const decision = buildRiskDecisionFromQualityGate({
|
||||
passed: false,
|
||||
qualityScore: 61,
|
||||
recommendations: ['raise branch coverage'],
|
||||
});
|
||||
|
||||
expect(decision.decision).toBe('block');
|
||||
expect(decision.riskFactors).toEqual(['raise branch coverage']);
|
||||
expect(validateRiskDecision(decision).valid).toBe(true);
|
||||
});
|
||||
|
||||
it('should escalate when the gate outcome is indeterminate', () => {
|
||||
const decision = buildRiskDecisionFromQualityGate({});
|
||||
|
||||
expect(decision.decision).toBe('escalate');
|
||||
expect(decision.confidence).toBe(0.5);
|
||||
expect(validateRiskDecision(decision).valid).toBe(true);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user