ASSAY

FILE 02 · EXHIBITS Case files, redacted where needed, dated always.

Here is what a
receipt looks like.

Two case files, both real. A is us. B is the first engagement — unsolicited, unpaid, on a launch-week model. Client exhibits will appear here only with written consent. Nobody made us publish it. We publish it because if our own tool won’t certify our own code, you should probably know that before you hire us.

Exhibit AWe ran it on ourselves.

REFUSED · 52.5%

Subject: vantage src/engines (65 files, 966 functions, 37,171 lines)  ·  Auditor: ASSAY (self-audit)  ·  Sealed: 2026-09-04

We built the tool. We ran it on itself. It refused to certify us. 30 critical security findings (56 quality notes), 4 HIGH severity issues blocking ship, 2 circular dependencies, 3 coupling issues, 3 god modules. AURORA verdict 52.5% — REJECTED. Published in full; below is the receipt as it went out.

RECEIPT · vantage self-auditRUN 2026-09-04 · bin/vantage.jsSUBJECT vantage src/engines @ live tree HARNESS bin/vantage.js analyze (same CLI clients run) SCORE 52.5 / 100 — REJECTED VERDICT REFUSED — fix HIGH severity issues before shipping FINDINGS (30 critical shown in ranks; 8 shown) ────────────────────────────────────────────────── concurrency.ts:236, :297, :325 Regex with nested quantifiers — evil-pattern ReDoS on crafted input. severity: HIGH contractdrift.ts:42, :195, :245, :285, :302 Regex with nested quantifiers — evil-pattern ReDoS on crafted input. severity: HIGH … plus 2 circular deps, 3 coupling issues, 3 god modules, 5 TODOs; full ledger in vantage-report.json NOTE Scanner code triggers scanner rules: regex literals inside our own detection engines fire the ReDoS detector. Some are true positives; some are detector-on- detector noise. All listed with file:line for the reader to judge.

Run 2026-09-04 · unedited CLI output

Exhibit B · ASSAY-001  Jev, by TypeSafe AI. Two claims, two corpora.

SPLIT · calibrated on 1 of 2 · 0 type errors

Subject: jev-latest, launched 2026-09-15  ·  Service: Claim Verification, unsolicited, unpaid  ·  Sealed: 2026-09-18  ·  Independently re-scored on a different model, blind

TypeSafe wrote that Jev returns calibrated probabilities and never makes type errors. We froze the protocol before we had an API key, ran it once on two human-labeled intent corpora we did not build, and published whatever came back. On CLINC150 the probabilities are calibrated (ECE 0.0204). On Banking77 they are not (ECE 0.0936, overconfident in every bin below 0.9). Across 8,576 responses, zero type errors.

RECEIPT · ASSAY-001 · claim verification PROTOCOL frozen 2026-09-17 · sha256 f8ce3f5b…a09b48 · two numbered amendments, published CLAIM 1 "calibrated probabilities" · pass ECE ≤ 0.05 Banking77 (77 intents, n=3,080) ECE 0.0936 DOES NOT HOLD CLINC150 (150+oos, n=5,496) ECE 0.0204 HOLDS CLAIM 2 "never makes type errors" · pass 0 violations 8,576 responses · 0 violations HOLDS CORPORA Banking77 @ github 57ec275d · CLINC150 @ hf 155b9c71 · hashed before first query RUN one pass per corpus · no retries · sealed b5c96bc1… / 5f5c58b2… CONTROLS uniform / overconfident / oracle / broken-schema — red and green before scoring RE-SCORE independent, different base model, blind — every field matched exactly DISCLOSED 513 responses with probabilities summing to 0.99 (2-decimal rounding; Amendment 2) 3 responses where choice ≠ argmax, all within 0.01 of a tie 4 CLINC items unscored: DNS failed on OUR laptop mid-run REPRODUCE github.com/jourdanlabs/assay-001 · full write-up: donttrustme.ai/assay-001.html

The limit: this applies to jev-latest on 2026-09-18, Choice questions with null option descriptions, these two corpora. Not a statement about any other task, corpus, day, or model.

Want your engagement
on this page someday? Or not.
Start an audit →

Publication policy: client exhibits will appear on this page only with the client’s written consent, or when the subject is ASSAY itself. Findings, dates, and hashes are unedited. Names are redacted at the client’s discretion. Exhibit B is an unsolicited verification of a public product; the subject was invited to reproduce it and had no edit or veto.