The evaluator's guide

How to evaluate a detector on this data

The working guide for data scientists and audit analytics teams. Everything below is self-serve and nothing needs a call; all of it runs on the free plan except the blind track and multi-year arcs, which open at Team.

Your first hour

1. Get a datasetCreate a free account and generate one, or start from the public sample, a complete package you can download in a minute. The Developer plan is free: all three platforms, the core four cycles, a scale of up to 1,000 base transactions per cycle, and 5 datasets a month. To try it with nothing installed, the starter notebook on Kaggle loads the sample and scores two detectors against its answer key in the browser.

2. Open the packageOne zip, four layers: an executive brief with the reproducibility certificate (the watermarked public sample leaves the certificate out, because the watermark changes its PDF and workbook bytes), audit evidence including Ground_Truth_Labels.csv (the answer key), a SQLite database in which every key between the tables it ships resolves, measured and printed in the package, and Excel workbooks mirroring the real ERP tables. Data teams can also take the Lab export from any job card, one JSONL file per table plus a dataset card and SHA-256 checksums.

3. Run your detectorPoint it at the database or the JSONL. Its job is to produce the list of record ids it believes are fraudulent, for example invoice document numbers or payment ids, exactly as they appear in the tables.

4. Score itIn the app, open Benchmark, pick your dataset or an official track, and paste the flagged ids. To run a detector on an official track, press Generate this track there first: it builds the track's data on your account, on every plan including the free one, for one generation each (on the free plan, one of the month's 5). You get precision, recall, F1, a grade on a fixed ladder, and per-scheme recall, graded against the dataset's answer key. A composite id matches as a whole and on the parts that are identifiers in their own right (a ghost vendor's vendor or employee id, the purchase order of a three-way-match line); a line item or a statement line number never scores on its own. On a multi-year arc (Team and above), prefix a flag with its year (FY2024:4500000123) to be graded inside that year.

5. Read the grade honestlyThe grade bands F1: A at 0.90 or above, then B 0.75, C 0.50, D 0.25, F below. The full breakdown of what each band means in practice, worked on real numbers, sits on the proof page.

Why a score here means something

No peeking, structurallyEvaluation mode withholds every ground-truth marker from the database and workbooks and then verifies its own strip; a surviving marker fails the build. The answer key ships in full, beside the tables, for scoring. A blind track (Team and above) goes further: its extra schemes are sealed on our server, out of the package, and scored there.

A detector can be wrongInnocent records that share a scheme's condition sit beside nearly every scheme, by design: receivables that are simply not yet due, cheques voided for a misprint and reissued, rent with no purchase order, routine manual journals. So a rule written on the condition alone is often wrong, and precision is measurable because false positives exist. Some stated rules still return exactly the planted records; the dataset card in each package's lab export says which, measured on that package. Three conditions are deliberate closed sets, where the join or the reconciliation is the analytic: segregation of duties, the ghost vendor and phantom inventory. A model scoring perfectly on those has rediscovered a query, not found a fraud. The published proof covers purchase to pay and order to cash on SAP ECC, the public sample, and four cycles on Oracle Cloud.

The fraud-free twinRegenerate any company with fraud set to none, same seed, empty answer key. Every flag on the twin is a false positive; that is your detector's noise floor on an innocent company.

Fixed tracks and a public barFive benchmark tracks whose configuration and seed are fixed, so every model graded on the same engine version sees the same bytes; when the engine moves, the tracks are regenerated on it, and scores from an earlier engine are listed apart, never silently compared. Our own three detectors are published as the baselines to beat on the leaderboard. Publishing your score is always your choice.

Provenance and reproducibility, in one paragraph

Every field is produced by deterministic Python from accounting rules; no language model generates any data, ever. The same seed and configuration regenerate a package byte for byte on the same engine version, every package generated on an account ships a certificate listing the SHA-256 of each of its files (the watermarked public sample leaves it out, because the watermark changes its PDF and workbook bytes). Reproducing a package on the engine version that built it is free and gives the same files; on another engine, or when the engine that built it is not on record, the same seed and configuration build a new dataset (its certificate names the engine), which counts as one generation on the free plan. The job card says which before you press Reproduce. The security page covers the rest: app.syntheticcfo.com/security.

From code, in five steps, nothing assumed

Rather not touch a terminal at all?Signed-in accounts get the API PLAYGROUND on the account page: pick a world, watch the exact code write itself, and press Run it now to execute the same request inside the platform - no install, no key, nothing below needed. The steps here are for wiring your own machine or pipeline.

1. Get PythonYou may already have it: open a terminal (next step) and type python --version; any 3.8 or newer answer means skip ahead. Otherwise download it from python.org and run the installer. On Windows, tick the box "Add python.exe to PATH" on the FIRST installer screen; it is what lets the commands below work anywhere.

2. Open a terminalWindows: press the Windows key, type cmd, press Enter. Mac: press Cmd and Space together, type Terminal, press Enter. The folder it opens in does not matter: every command below runs from anywhere, and downloads land in the folder the prompt shows (or wherever you point --out).

3. Install the clientRun once, in that terminal:

pip install syntheticcfo

If your workplace blocks the package indexThe platform serves the same version itself, so pip install https://app.syntheticcfo.com/sdk/syntheticcfo-0.2.2-py3-none-any.whl installs exactly the copy that matches this deployment.

4. Create your API keySign in to the app, open the menu, click your email to open Account, find the API access card, press Create key, and copy the key (it is shown exactly once). The card also hands you a ready-made generate command with your new key already inside it, so the next step is a paste.

5. GenerateOne line. It builds the world, waits, and saves the package zip into the folder you are standing in:

syntheticcfo generate --key scfo_PASTE_YOURS --platform oracle --industry retail --fraud high --rows 600

If something complains"pip is not recognized": reinstall Python with the PATH box ticked, or on Windows use py -m pip install syntheticcfo. "syntheticcfo is not recognized": python -m syntheticcfo generate ... does exactly the same thing everywhere. A 401 error means the key was pasted wrong or has been revoked: create a fresh one. Add --lab to also save the JSONL lab export, --seed 42 to make the world reproducible on purpose, and --evaluation-mode to withhold the ground-truth marker columns from the shipped data.

For pipelines and agents

Python clientThe same tool, from code. Standard library only, no dependencies; also served as a single file with curl -O https://app.syntheticcfo.com/sdk/syntheticcfo.py; source at github.com/syntheticcfo/syntheticcfo, and the sample's tables as JSON lines, built from the same seed and configuration (the listing names the engine version it was built on), at huggingface.co/datasets/syntheticcfo/synthetic-cfo-sample:

from syntheticcfo import Client

c = Client(api_key="scfo_...")            # or set the SCFO_API_KEY env var
job = c.generate(platform="ORACLE", industry="RETAIL", fraud="high",
                 rows=600, modules=["P2P", "O2C"], seed=42, wait=True)
job.download("out/")                      # the four-layer audit package
job.lab_export("out/")                    # JSONL tables + answer key + dataset card

Raw HTTPAny client works. Send the key as an Authorization Bearer header to the same endpoints the app itself uses: create a job, poll it, download the package or the lab export, submit flagged ids for scoring. A key carries your plan and monthly allowance, and it can never manage the account that owns it.

MCP serverFor agent frameworks, an MCP server drives the engine programmatically: generate a dataset, inspect a package, read a table, query the answer key, reproduce.

Start now

Create a free account Download the sample Read the proof See the leaderboard