Validation picked the fraud model that had read the future
A naively joined fraud model beat the point-in-time one on validation, 0.746 to 0.681 AUC-PR, then scored 0.363 to 0.941 on the next two weeks. Why it happened.
16 min read
Topic
Applied ML & evaluation
Applied machine learning judged by its scorer: OCR extraction, short-text classification, embeddings, clustering, and measurements that changed the plan.
The model is rarely the hard part. The hard part is the scorer: deciding what “correct” means for a document class or a label before looking at the output, separating a refusal from a miss, and keeping a measurement honest when the data is small, skewed or synthetic. These notes are about that layer — evaluation sets, abstention, cost per decision, and the results that disproved the hypothesis they were meant to confirm.
7 posts
All writingThe agent got 48 of 54 right. Three of the wrong ones published themselves.
An analytics agent answered 48 of 54 questions correctly on a local model. What mattered was 3 of 37 auto-published answers wrong, and why no check saw them.
11 min read
The eval score fell 4.8 points. Twelve answers decided whether it meant anything.
A prompt change cost 4.8 points on an LLM judge, and 12 of 83 paired cases carried all the evidence. Why a release gate needs a paired test, not a delta.
13 min read
The classifier was 95% accurate. The trend it counted was a third too flat.
A 95%-accurate classifier counted a rise from 10% to 25% as 8% to 17%. Why counting labels flattens trends, and which estimators report them honestly.
14 min read
The best OCR model I tested does not ship. The scorer decided why.
Eight OCR engines on real clinic records, one fixed scorer. The strongest reader stays switched off: slower on the target box, and it cannot say when it is unsure.
15 min read
Label the string, not the row: a short-text triage lab
A synthetic 20,000-row multilingual lab: normalize to distinct keys, cluster them for one LLM call each, and why a cosine threshold needs a margin.
15 min read
The OCR plan said fix the photographs. Its first measurement withdrew it.
An OCR plan blamed skewed, creased phone photos for lost accuracy. Its first step, a per-defect cost table and a deskew test, showed capture cost nothing.
15 min read