Measured, not asserted

Does a score on this data mean anything?

This is a synthetic company ledger, the public sample on SAP ECC, with 188 planted, labelled fraud records in it. We handed the database to three real detectors without the answer key, then scored what they flagged. Run them below and watch what each one actually caught.

Fgeneric anomaly detection's grade

683false flags on a company with no fraud in it

98%of the planted fraud is findable from the data

1 seedregenerates every number on this page

Run the detectors replay of a recorded run

The same company, generated with fraud at medium intensity. The dashed line marks the 188 planted records a perfect detector would find with no false alarms.

actual fraud found false alarms 188 planted fraud records

Generic statistical anomaly detectionoutliers, digit tests, round numbers: the statistics a team tries first

grade F

found 48 of 188 476 false alarms

Found 48 of the 188 planted fraud records, with 476 false alarms.

26%

recall

precision

Textbook CAAT rulesthe audit analytics handbook: duplicates, thresholds, weekend postings

grade C

found 147 of 188 204 false alarms

Found 147 of the 188 planted fraud records, with 204 false alarms.

78%

recall

42%

precision

Targeted forensic rules (ceiling)rules written knowing which schemes were planted: a ceiling, not a model

grade C

found 185 of 188 251 false alarms

Found 185 of the 188 planted fraud records, with 251 false alarms.

98%

recall

42%

precision

Pressing run replays the recorded result of the real detector runs, read from the committed proof output. Nothing is simulated in your browser, which is why the numbers never change.

What was planted in this company

188

planted

Phantom revenue74

Self-approval (segregation of duties)37

Lapping17

Three-way match failures14

Duplicate invoices12

10 more scheme families34

The engine carries 25+ scheme families across seven business cycles; the public sample configuration plants these. Hover or tap anything on this page for the plain-English reading.

What the grades actually mean

A detector is scored on three numbers, and the grade is a band on the third. Here is each one in plain terms, worked on the textbook rules' real result from the SAP sample.

PrecisionOf everything a detector flags, the share that is actually fraud. The textbook rules flagged 351 records and 147 of those were planted fraud: precision 0.42. In practice, an investigator who opens 100 of its flags closes about 58 of them as innocent, and every one of those is paid working time.

RecallOf all the fraud that exists, the share the detector caught. This sample plants 188 fraud records and the textbook rules found 147: recall 0.78. The other 41 fraud records were never flagged at all, and in a real company they simply sail through.

F1One number balancing the two: the harmonic mean of precision and recall. It exists to punish cheating on one side. Flag every record in the ledger and recall is a perfect 1.00, but precision collapses and F1 collapses with it. The textbook rules' 0.42 precision and 0.78 recall combine to F1 0.54, which lands in the C band.

Grade	F1 band	What it means in practice	On this data
A	0.90 and above	Flags are almost always right and almost nothing slips through. Treat a grade A on a synthetic benchmark with suspicion: it usually means the answer key leaked into the data. Our own textbook rules scored A on the Oracle build before we fixed the engine, and the fix took it away.	no detector lands here on this data
B	0.75 to 0.89	A strong screen. Most flags are worth opening and most of the fraud is caught. A tuned, scheme-aware rule set on honest data lands here.	no detector lands here on this data
C	0.50 to 0.74	A usable screen that costs real work: roughly half the flags waste an investigator, or a meaningful share of the fraud is missed, or both at once.	Textbook CAAT rules on SAP ECC (F1 0.54) Targeted forensic rules (ceiling) on SAP ECC (F1 0.59) Textbook CAAT rules on Oracle Cloud (F1 0.66) Targeted forensic rules (ceiling) on Oracle Cloud (F1 0.65)
D	0.25 to 0.49	More wrong than right. The review pile is mostly noise and much of the fraud still escapes.	no detector lands here on this data
F	below 0.25	Statistically close to useless for this job. Most flags are noise and most fraud walks. Day-one statistics land here because behavioural fraud does not look like an outlier.	Generic statistical anomaly detection on SAP ECC (F1 0.13) Generic statistical anomaly detection on Oracle Cloud (F1 0.09)

The bands are fixed in the scorer: A at 0.90 or above, then B at 0.75, C at 0.50, D at 0.25, and F below that. It is the same scorer behind the product's score-a-model feature, so a model you test on a purchased dataset is graded on exactly this ladder.

How the test is run

No peekingA detector is handed one thing: a copy of the database in a directory where the answer key does not exist. Not looking is structural, not promised.

The fraud-free twinEach detector also runs against the same company generated with fraud set to none. Its answer key is empty, so every flag there is a false positive. That view is the cleanest statement of whether a rule finds fraud or finds noise.

Three tiers of skillDay-one statistics (outliers, digit tests, round numbers), then the audit-analytics handbook, then rules written knowing the scheme families. The last is a ceiling and is reported as one: it answers whether the fraud is findable at all, not how good anyone's model is.

The gradesA detector receives the grade band of its F1 score, the balance of precision and recall: A at 0.90 or above, then B at 0.75, C at 0.50, D at 0.25, and F below that. The same scorer drives the score-a-model feature in the product.

We attacked our own answer key first

Before publishing a single score, we ran two attacks on ourselves. First, the engine's ground-truth markers used to ship inside the data, where a two-line search could read the answers off the rows; evaluation mode now withholds them, and the package fails its own build if one survives. Second, we measured whether any fraud label was just a database query restated, which would make an innocent instance of that condition impossible and a detector incapable of being wrong.

The engine now records ground truth at the moment each scheme is planted, and the world carries innocent look-alikes of every fraud condition: receivables open simply because they are not yet due at the year-end extract, cheques voided for a misprint and reissued, rent and utilities invoiced with no purchase order, routine manual journals, deposits in transit. That is why the precision figures here are ordinary instead of perfect: on this data, a detector can be wrong. That is what makes the scores mean something.

Every number, in one table

The public sample, SAP ECC · 188 labelled fraud records

Detector	Precision	Recall	F1	Grade	Flags on the fraud-free twin
Generic statistical anomaly detection	0.09	0.26	0.13	F	683
Textbook CAAT rules	0.42	0.78	0.54	C	194
Targeted forensic rules (ceiling)	0.42	0.98	0.59	C	261

The same exercise on Oracle Cloud · 223 labelled fraud records

Detector	Precision	Recall	F1	Grade	Flags on the fraud-free twin
Generic statistical anomaly detection	0.06	0.22	0.09	F	917
Textbook CAAT rules	0.56	0.81	0.66	C	140
Targeted forensic rules (ceiling)	0.49	0.96	0.65	C	216

Check it yourself

The dataset scored above is the public sample configuration. Download the sample, run your own detector against the database, and score it against the answer key that ships in the same package. The proof run scores against the evaluation build of the same configuration, where the ground-truth markers are withheld; the public sample ships them openly so you can inspect the answers.

Download the sample How to evaluate on it Beat these detectors Create a free account

Engine v1.8.0. Every figure on this page is read from the committed output of the proof run, never typed in. Fraud levels, schemes and volumes are configurable; these numbers describe the public sample configuration at medium intensity.