The problem
Automations that demo well fail quietly in production. Nothing declares what happens when a node fails; external calls hang without a retry or a timeout; secrets get pasted straight into node parameters; outside data flows from a webhook into a CRM before anyone looked at it. A buyer of automation work is asked to trust a claim nobody can inspect — and most evaluation layers are narrated, not shown.
What we did
Built the evaluation layer we run before any agent engagement, as software that can be checked rather than described. Upload or paste an n8n workflow export and the engine runs six deterministic checks — error handling, retries and timeouts, error routing, input validation, credential hygiene, structural integrity — and scores the result against a quality bar that was committed to the repository before the check suite existed. If you also upload the previous export, the report diffs the two versions: added, removed and changed nodes (behaviour only — cosmetic moves never count) and exactly which checks flipped, in either direction. The finished report exports to a branded PDF from the browser's own print engine — the same data as on screen, with the full run fingerprint and disclaimers attached.
AI analysis is optional and deliberately fenced: four guardrail classes (input validation, output constraints, refusal behaviour, cost/token ceilings) surround one bounded call, its output is discarded if it fails schema validation, and the product works completely offline with zero model spend. The workflow itself is parsed and validated — never executed, never imported into n8n, never stored — and every run appends a structured record containing hashes, not content.
The same harness gates its own repository: fixtures must hold their verdicts in CI, and new expectations enter the graded suite only when a named human approves them. That eval-first gate is the working form of the claims on the AI agents service page.
What went wrong, and how we caught it
Eight failures are logged in real time in the repository; three travel well:
The evaluator was almost wrong about what starts a workflow. A broad webhook match classified respond-to-webhook action nodes as triggers. Caught by reading the live API response; fixed with an exact match and a test that names the single trigger.
The test was wrong, the engine was right. A regression-diff assertion expected a removed node in the wrong direction — present-now/absent-before is "added" by definition. Caught by the suite itself; the assertions were rewritten to cover both directions plus cosmetic-change-ignored.
A "working" UI that did nothing. Headless verification clicked the sample button and nothing happened: dev-server chunks were 403ing on the numeric loopback, the failed hydration then wiped the loaded state. Root-caused through network logs, fixed in config, and the fix is coded into the harness docs.
Results
There is no client, so there are no client results — the record states what the repository proves instead. At the final gate on 2026-08-12: 42 of 42 tests green, typecheck clean, production build clean, every route 200 in a production smoke test. The clean synthetic fixture scores 100/100 and clears the bar; the deliberately flawed one scores 0/100 with six named failures; a secret-bearing fixture fails critically while no secret value appears anywhere in the report; axe-core reports zero accessibility violations. All metrics above are tagged Projected because a concept build has no measured outcomes — the numbers are reproducible from the repo, which is the entire point.
Verification record
The evidence the claims above stand on, all dated 2026-08-12 and reproducible:
- Engine 0.3.0, quality bar v0.1.0 — the bar was committed before any result existed and is never adjusted after seeing a score.
- 42/42 tests across engine, regression-diff and audit batteries; 3 browser-exported PDFs content-verified; 7 interface surfaces checked (desktop and mobile, zero horizontal overflow).
- Typed refusal taxonomy — empty, non-JSON, non-workflow, oversized (2 MB ceiling) and unconfigured-AI inputs each get their own honest code; the 413-before-400 ordering is locked in tests.
- Run logs owned in-repo (
logs/runs.jsonl): timestamp, duration, sha-256 fingerprint, score, failed check ids — never workflow content. - HITL gate: engine-drafted expectations score nothing until a named human
approves them (
npm run review -- approve <file> --by <name>); anonymous approval is refused. - Prompts and rubrics published (
docs/prompts-rubrics.md,docs/bar.md) — the evaluation criteria are inspectable, not narrated. - Dual theme (2026-08-13) — the tool's UI now follows the reader's OS
scheme with a second hand-mapped palette on the same semantic tokens (no
inversion); all text pairs measured ≥5.5:1 in dark, and the print/PDF
export stays light regardless of OS scheme (
app/globals.css, dated comment). The three figures above were recaptured in dark the same day from the same production build; every Screenshot on this page now pairs a dark and a light variant at identical 1600×900 dimensions.
