Evidenced
A QA primitive for agents: it verifies a user outcome in a real browser and returns a structured, evidence-backed verdict (passed, failed, or inconclusive) an agent can loop on instead of a raw action log
Role
Full Stack Developer
Team
Solo
- Distinguishing failed (the app is wrong) from inconclusive (the environment prevented a verdict) so an agent loops on real defects, not on DNS failures or timeouts
- Five deterministic guards that rule before any LLM judgment, with a guard failure short-circuiting the model and the LLM never allowed to override a guard
- One hosted execution path shared by the MCP server and the dashboard, so both produce the same verdict from the same job, queue, worker, and code
- Envelope-encrypted per-org secrets decrypted only at enqueue, carried through Redis on the job, scrubbed from errors, and never logged or written into evidence
- URL allowlist checked at enqueue and re-checked in the worker before navigation, so an agent cannot widen its own target
- Open-core split where the commercial ee/ tier is reached only through a guarded dynamic import, keeping the OSS verdict loop functionally identical and unmetered
- Returning evidence (screenshots, DOM snapshots, console, network) by reference keeps verdicts compact so agents get a decision, not megabytes of artifacts
- A pure plan to execute to judge to assemble engine with no transport, storage, or queue imports, everything external supplied as an adapter
- Subpath exports keep concrete adapters (Puppeteer, OpenRouter, S3) out of the barrel so a types-only consumer never drags in an SDK
- Tenant isolation by construction: every row carries org_id, all access goes through forOrg(orgId), and orgId is resolved server-side, never from the request body
- Verdict schema validated on write, on read, and on receipt, with an additive-only schemaVersion
- Runtime open-core switch (BILLING_ENABLED / REQUIRE_BILLING) rather than a compile-time fork, so the seams resolve to no-ops when billing is off
Overview
Evidenced is a QA primitive for coding agents. Agents can write code but cannot reliably tell whether the result actually works for a user. Evidenced closes that loop: it drives a real browser toward a goal and returns a structured, goal-relative verdict (passed, failed, or inconclusive) that an agent can act on directly, rather than a raw action log it has to interpret.
Two entry points, an MCP server a coding agent calls and a dashboard a human uses, drive one hosted, server-side execution path, so both produce the same verdict from the same code. Evidenced is open-core and self-hostable under Apache-2.0, with a paid hosted tier living in ee/.
3
Verdict states
5
Deterministic guards
4
Stage engine pipeline
2
Entry points, one path
How It Works
- 1
Trigger a run
An agent (
evidenced goal:"...") or a human (dashboard "Run") hits the backend, the single front door. It authenticates the caller, enforces tenancy and the app's URL allowlist, gates on credits (hosted only), decrypts the run's secrets, and enqueues a validated job on BullMQ. - 2
Execute in isolation
The worker pulls the job, re-checks the allowlist, launches an isolated browser, and runs the engine. The only difference between an MCP run and a dashboard run is the recorded
source. - 3
Plan, execute, judge, assemble
The engine turns the goal and starting URL into an ordered journey, drives each step in the browser while capturing evidence, runs five deterministic guards, then consults the LLM only if no guard fired.
- 4
Return a verdict, loop
The verdict and evidence references land in Postgres and the object store. The agent reads the verdict and loops (fix, re-run); the human reviews it in the dashboard as the lifecycle streams back.
The engine is a pure pipeline with no transport, storage, or queue concerns; everything external is supplied by the worker as an adapter:
Key Features
Outcome-level verdicts
A verdict is relative to a goal, never a transcript the caller must parse.
failed means the app is wrong; inconclusive means the environment
(unreachable target, timeout) prevented a verdict. An agent loops only on
real defects.
Deterministic guards first
Five guards (HTTP status, URL assertion, element presence, console error, page load) run on every step before any LLM is consulted. A guard failure is conclusive and short-circuits the model, which can never override it.
Machine-readable failures
Every failed verdict carries a dossier: the failing step, the reason, a root-cause hypothesis, evidence references, and a suggested next action, so an agent gets a plan, not a mystery.
Context economy
Evidence (screenshots, DOM snapshots, console, network) is stored and returned by reference. Agents receive compact verdicts instead of megabytes of artifacts.
One execution path
MCP and dashboard runs share the same job, queue, worker, and verdict. No functional gap between hosted and self-hosted; only the commercial layer differs.
Secrets sealed by construction
Model keys and app credentials are sealed with a per-org data key wrapped by a boot-time KEK, decrypted only at enqueue, carried through Redis, and scrubbed from every error before it reaches a client.
Allowlist cannot be widened
The target URL is checked at enqueue in the backend and re-checked in the worker before navigation. An agent has no path to expand its own reach.
Pluggable adapters
Browser (Puppeteer), model (OpenRouter), resolution (DOM ladder), and storage (disk / S3 / R2) are all adapters behind interfaces. Swapping one is a config change in the composition root, not an engine change.
Architecture
A pnpm + Turborepo monorepo (Node 22, TypeScript, ESM throughout): five apps and four packages behind a single backend front door.
The commercial ee/ package is an optional dependency reached only through
a guarded dynamic import, gated by BILLING_ENABLED / REQUIRE_BILLING.
The switch is runtime, not compile-time, so the OSS build and the hosted
build run the identical verdict loop; with billing off the seams resolve to
no-ops and runs are unlimited.
An unreachable target (DNS failure, connection refused, timeout) yields
inconclusive, not failed: the environment prevented a verdict, the app
was never proven wrong. Collapsing these two would make an agent chase
phantom defects every time the network hiccuped.
packages/core exports the engine and adapter interfaces only. Concrete
adapters live behind subpath exports
(@evidenced/core/adapters/model/openrouter, etc.), so importing the
package for its types never drags Puppeteer or the model SDK into a
consumer that does not touch them.
Outcome
Evidenced demonstrates a production open-core agent-QA system: a pure four-stage browser verification engine, five deterministic guards gating the LLM, a single execution path shared by an MCP server and a human dashboard, per-org envelope-encrypted secrets, and strict server-side tenant isolation. The result an agent receives is a compact, schema-validated verdict with an evidence-backed failure dossier it can loop on, not an action log it has to reverse-engineer.