Concept Architecture
Read, prompt, execute
Every agent reads from one knowledge bank. Each run writes a scenario file back into it, and the next read starts from that state. Switch to automated to see the loop close. All figures on this page are illustrative.
Measured before it runs on its own
A synthetic entity with its answer key, thousands of scenarios cut from it, one hundred agents working the queue in isolated forks, every result graded. The read-out is a failure rate per task type with a confidence interval.
Synthetic entity
Truth first. The generator writes the correct accounting outcome, then renders the inputs from it: bank line, email, invoice PDF, transcript. One entity, 200 doors, 40 owners, 150 vendors, two years.
Scenario library
Thousands of snapshots of the bank at a moment plus one event. Replay of real months once they arrive, synthetic variations, adversarial cases built to break it. Each carries its answer key.
100 agents
Parallel workers pulling from one queue. Each run forks the knowledge bank, runs read, prompt and execute, and discards the fork. Same scenario, different prompt or model, for paired comparison.
Grader
Compares the execute cycle to the answer key field by field: entity, amount, date, account, class, action taken or correctly not taken. Logs the factors behind every failure.
Read-out
Failure rate per task type with a 95% interval. Automation switches on per task type when the upper bound clears the threshold, once.
Agent matrix · 100 workers, one queue
Two tiers of machines
- Decision
- Did the agent decide right. Runs at full parallelism on ordinary compute against a simulated ledger. Needs volume.
- Confirm
- Did the write land. A small pool of real QuickBooks Desktop VMs, 5 to 10, each restored from snapshot before every run. Needs fidelity, not volume.
- Threshold
- Recording tasks: upper bound of the interval below 1%. Money-moving tasks: zero failures in at least 3,000 scenarios, which bounds the rate under 0.1%.
- Live
- The matrix never touches the live file. After go-live it keeps grading a sample of live scenario files, so a slip shows in the same numbers.
Failure rate per task type · 95% interval
threshold 1.0% · scale 0 to 4%A/B · same scenarios, two prompt versions
| Pair outcome | Scenarios |
|---|---|
| Both pass | 1,151 |
| Both fail | 6 |
| v3 fails, v4 passes | 31 |
| v4 fails, v3 passes | 12 |
| Shared scenarios | 1,200 |
Only the 43 discordant pairs carry information. McNemar on 31 versus 12 gives p ≈ 0.004, so v4 replaces v3. Paired runs need a fraction of what two independent samples would.
After go-live · weekly sample on a control chart
Each week a sample of live scenario files is graded the same way. Limits at p̄ ± 3√(p̄(1−p̄)/n). A point above the upper limit triggers re-testing before anything else changes.
All figures on this page are illustrative. Real rates come from the matrix once the scenario library exists.