Does a score on this data mean anything?
This is a synthetic company ledger, the public sample on SAP ECC, with 38 planted, labelled fraud records in it. We handed the database to three real detectors without the answer key, then scored what they flagged. Run them below and watch what each one actually caught.
The same company, generated with fraud at medium intensity. The dashed line marks the 38 planted records a perfect detector would find with no false alarms.
The engine carries 37 labelled fraud schemes, 35 across the seven business cycles and 2 in the intercompany group. The SAP world is the public sample, purchase to pay and order to cash at medium fraud, and it plants these. Hover or tap anything on this page for the plain-English reading.
What the grades actually mean
A detector is scored on three numbers, and the grade is a band on the third. Here is each one in plain terms, worked on the textbook rules' real result from the SAP sample.
| Grade | F1 band | What it means in practice | On this data |
|---|---|---|---|
| A | 0.90 and above | Flags are almost always right and almost nothing slips through. Treat a grade A on a synthetic benchmark with suspicion: it usually means the answer key leaked into the data. Our own textbook rules scored A on the Oracle build before we fixed the engine, and the fix took it away. | no detector lands here on this data |
| B | 0.75 to 0.89 | A strong screen. Most flags are worth opening and most of the fraud is caught. A tuned, scheme-aware rule set on honest data lands here. | no detector lands here on this data |
| C | 0.50 to 0.74 | A usable screen that costs real work: roughly half the flags waste an investigator, or a meaningful share of the fraud is missed, or both at once. | no detector lands here on this data |
| D | 0.25 to 0.49 | More wrong than right. The review pile is mostly noise and much of the fraud still escapes. | no detector lands here on this data |
| F | below 0.25 | Statistically close to useless for this job. Most flags are noise and most fraud walks. Day-one statistics land here because behavioural fraud does not look like an outlier. | Generic statistical anomaly detection on SAP ECC (F1 0.03) Textbook CAAT rules on SAP ECC (F1 0.12) Targeted forensic rules (ceiling) on SAP ECC (F1 0.18) Generic statistical anomaly detection on Oracle Cloud (F1 0.04) Textbook CAAT rules on Oracle Cloud (F1 0.15) Targeted forensic rules (ceiling) on Oracle Cloud (F1 0.15) |
The bands are fixed in the scorer: A at 0.90 or above, then B at 0.75, C at 0.50, D at 0.25, and F below that. It is the same scorer behind the product's score-a-model feature, so a model you test on a purchased dataset is graded on exactly this ladder.
How the test is run
We attacked our own answer key first
Before publishing a single score, we ran two attacks on ourselves. First, the engine's ground-truth markers used to ship inside the data, where a two-line search could read the answers off the rows; evaluation mode now withholds them, and the package fails its own build if one survives. Second, we measured whether any fraud label was just a database query restated, which would make an innocent instance of that condition impossible and a detector incapable of being wrong.
The engine now records ground truth at the moment each scheme is planted, and the world carries innocent look-alikes of nearly every fraud condition: receivables open simply because they are not yet due at the year-end extract, cheques voided for a misprint and reissued, rent and utilities invoiced with no purchase order, routine manual journals, deposits in transit. That is why the precision figures here are ordinary instead of perfect: on this data, a detector can be wrong. That is what makes the scores mean something. Three conditions are deliberate closed sets, where the join or the reconciliation is the analytic: segregation of duties, the ghost vendor and phantom inventory. A model scoring perfectly on those has rediscovered a query, not found a fraud.
Every number, in one table
| Detector | Precision | Recall | F1 | Grade | Flags on the fraud-free twin |
|---|---|---|---|---|---|
| Generic statistical anomaly detection | 0.02 | 0.16 | 0.03 | F | 422 |
| Textbook CAAT rules | 0.07 | 0.53 | 0.12 | F | 293 |
| Targeted forensic rules (ceiling) | 0.10 | 1.00 | 0.18 | F | 358 |
| Detector | Precision | Recall | F1 | Grade | Flags on the fraud-free twin |
|---|---|---|---|---|---|
| Generic statistical anomaly detection | 0.02 | 0.20 | 0.04 | F | 687 |
| Textbook CAAT rules | 0.08 | 0.66 | 0.15 | F | 437 |
| Targeted forensic rules (ceiling) | 0.08 | 1.00 | 0.15 | F | 639 |
The SAP dataset scored above is the public sample. Download it, run your own detector against the database, and score it against the answer key in the same package. The sample is built in evaluation mode, exactly as the proof run is: the ground-truth marker columns are withheld from the tables, the annotation layer sits in 01_Audit_Evidence/SME_Reasoning.csv, and the answer key ships in full.
Engine v1.15.0. Every figure on this page is read from the committed output of the proof run, never typed in. Fraud levels, schemes and volumes are configurable; the SAP numbers describe the public sample configuration at medium intensity, and the Oracle numbers the same company on Oracle Cloud with four cycles (purchase to pay, order to cash, cash and bank and record to report), which is not the public sample.