The evaluator's guide
How to evaluate a detector on this data
The working guide for data scientists and audit analytics teams. Everything below is self-serve on the free plan; nothing needs a call.
Your first hour
1. Get a datasetCreate a free account and generate one
(all three ERP platforms, up to 1,000 rows on the free plan), or start from the
public sample, a complete package you can download in a
minute.
2. Open the packageOne zip, four layers: an executive brief
with the reproducibility certificate, audit evidence including Ground_Truth_Labels.csv
(the answer key), a SQLite database with full referential integrity, and Excel workbooks
mirroring the real ERP tables. Data teams can also take the Lab export from any job card,
one JSONL file per table plus a dataset card and SHA-256 checksums.
3. Run your detectorPoint it at the database or the JSONL.
Its job is to produce the list of record ids it believes are fraudulent, for example
invoice document numbers or payment ids, exactly as they appear in the tables.
4. Score itIn the app, open Benchmark, pick your dataset or
an official track, and paste the flagged ids. You get precision, recall, F1, a grade on a
fixed ladder, and per-scheme recall, graded against the sealed answer key. Composite ids
like a ghost vendor's id pair match on either part.
5. Read the grade honestlyThe grade bands F1: A at 0.90 or
above, then B 0.75, C 0.50, D 0.25, F below. The full breakdown of what each band means in
practice, worked on real numbers, sits on the proof page.
Why a score here means something
No peeking, structurallyEvaluation mode withholds every
ground-truth marker from the database and workbooks and then verifies its own strip; a
surviving marker fails the build. The answer key ships separately for scoring. A blind
track goes further and withholds the key itself.
A detector can be wrongEvery fraud condition has innocent
look-alikes by design: receivables that are simply not yet due, cheques voided for a
misprint and reissued, rent with no purchase order, routine manual journals. Precision is
measurable because false positives exist.
The fraud-free twinRegenerate any company with fraud set to
none, same seed, empty answer key. Every flag on the twin is a false positive; that is
your detector's noise floor on an innocent company.
Frozen tracks and a public barFive benchmark tracks whose
configuration and seed never change, so scores stay comparable over time, with our own
three detectors published as the baselines to beat on
the leaderboard. Publishing your score is always your
choice.
Provenance and reproducibility, in one paragraph
Every field is produced by deterministic Python from accounting rules; no language model generates any data, ever. The same seed and configuration regenerate a package byte for byte, the certificate in every package lists the SHA-256 of every file, and reproducing a package is free and never spends a generation. The security page covers the rest: app.syntheticcfo.com/security.
For pipelines and agents
Python clientStandard library only, no dependencies.
Create an API key on your account page (Account, API access), then generate and download
without a browser. Install it, or take it as a single file with
curl -O https://app.syntheticcfo.com/sdk/syntheticcfo.py:pip install syntheticcfo
from syntheticcfo import Client
c = Client(api_key="scfo_...") # or set the SCFO_API_KEY env var
job = c.generate(platform="ORACLE", industry="RETAIL", fraud="high",
rows=600, modules=["P2P", "O2C"], seed=42, wait=True)
job.download("out/") # the four-layer audit package
job.lab_export("out/") # JSONL tables + answer key + dataset card
Raw HTTPAny client works. Send the key as an Authorization
Bearer header to the same endpoints the app itself uses: create a job, poll it, download
the package or the lab export, submit flagged ids for scoring. A key carries your plan and
monthly allowance, and it can never manage the account that owns it.
MCP serverFor agent frameworks, an MCP server drives the
engine programmatically: generate a dataset, inspect a package, read a table, query the
answer key, reproduce.