Sets and datasets
Choose inline sets or chunked datasets and move between them.
A set is a small saved row collection you can inspect and reuse. A dataset stores data in chunks for larger processing jobs.
Save a set
From a generation preview or supported scenario result, choose the save-as-set action, name the set and review the rows. The result must be a non-empty array of row objects. A generation API response carries its rows in live_data; save that array, not the whole response.
Sets are capped at 10,000 rows and 5 MiB of serialized JSON. Oversized results are rejected rather than silently truncated. Use a dataset for larger data.
Lock a reviewed set when it should remain a stable regression fixture. A locked set rejects content, name and description changes and deletion until it is unlocked. Locking a set does not lock the templates or external systems that produced it.
Work with a dataset
Datasets have a lifecycle: building, ready or failed. A producer appends chunks while building, then seals the result before downstream processing. Inspect readiness before starting a transform, mask or load.
The API provides GET /datasets/{id}/preview, GET /datasets/{id}/sample and GET /datasets/{id}/stats. The statistics route is /stats, not /statistics. Previews and samples are bounded; a sample is not necessarily a uniform sample of the full dataset, and statistics can report partial coverage. Check the returned coverage or truncation indicators before making claims about the whole source.
Convert between them
POST /sets/{id}/to-datasetconverts a row-shaped set into a ready chunked dataset.POST /datasets/{id}/to-setconverts a ready dataset only when it fits the inline set limits.
The destination's write permission is required. Converting to a set reads the rows into memory, so it is not a route for bypassing set size limits.
Deliver or process data
The interactive export flow offers CSV, JSON, API and database targets for supported row collections. For longer pipelines, use data jobs and inspect their run counters and error outputs before considering delivery complete.
CSV exports and Excel
A CSV DataMaker exports is comma-delimited and begins with a sep=, line:
sep=,
id,company,country
1,"Acme, Inc.",GBExcel takes its delimiter from your operating system's list separator, not from
the file. On a locale where that separator is a semicolon, a comma-delimited
file opens with each row in a single column; the sep=, line tells Excel to
split on commas instead, so an exported file opens in columns with no Text to
Columns step.
A CSV you download also begins with a UTF-8 byte order mark, ahead of that
line. Excel reads an unmarked CSV in the system code page rather than as UTF-8,
so on a Windows install an accented or non-Latin value - Müller, Kraków, a
Japanese address - opens as mojibake. A downloaded file carries no encoding
header, so the mark is the only way to say what the bytes are. It is invisible
in Excel, and every standard reader drops it.
The mark is added as the file leaves, so it is on every CSV downloaded from DataMaker - the export dialog, a Set download, a plan run's artifacts, a file the agent produced in a chat, a scenario's deliverables, and each CSV inside a "download all" archive - and on none of the stored copies. That is deliberate: the agent reads its own earlier output back from storage between turns, and a mark written into the object would arrive as part of the first column's name. A file read through the API or the SDK is therefore unmarked.
The sep=, line is different: it is written by whatever produced the rows. It
is there on an export from the dialog, a Set download, and a plan run's
artifacts when the rows were handed over as JSON for the server to serialize -
a CSV body posted ready-made keeps its own first line, and a CSV written by a
chat script or a scenario's own Python carries whatever that code wrote. A file
rendered for a target through a file template follows
the target's format instead, with neither the hint nor the mark, because a
downstream system is parsing it. Check the first line rather than assuming it.
The line is an Excel convention rather than part of the CSV standard, so a script reading the file should skip it. In pandas:
pd.read_csv("export.csv", skiprows=1)On a downloaded file the byte order mark sits on that same first line, so
skipping the line removes both and the columns come back named correctly. A
reader that strips the sep=, line some other way should pass
encoding="utf-8-sig" as well, or its first column arrives with the mark
attached to the name.
DataMaker's own CSV uploads skip it for you, so an export can be edited and re-imported as it is. Files produced by a data job's file delivery are not affected: those follow the delimiter and layout configured on the file template, because a downstream system is parsing them.
Use a set as FlowProof inputs
Ask chat to prepare FlowProof inputs with their sources and operation checks. Open the saved Set's Automation inputs section: it shows each variable, its effective value, where it came from, and any test-case override. Choose the test case and operation you intend to run.
Values present means the data was found. Evidence passed means saved checks cover those values for that operation within their validity window. Saved execution notes counts previous results. These are separate: a previous invoice may have passed while today's posting period is closed.
When the selected operation has complete current evidence, Download FlowProof inputs provides inputs.values.yaml, provenance.json and instructions for an operator-controlled handoff. Flowproof 0.12.1 has no --vars flag, so the file is a handoff rather than something Flowproof loads. Credentials remain in the connection or runner environment.
To check the saved values inside DataMaker, choose the case, the operation and the connection to check against, then select Run validation. DataMaker runs a reviewed, read-only Flowproof 0.12.1 flow against that connection with exactly those values and the connection's stored credentials, shows its progress, and lets you cancel it. Only a complete report where every step passed counts as Passed. A wrong value is Failed and lists the failed steps. A run that is cancelled, times out or cannot reach the target is never shown as a pass. The result and its evidence appear in Validation history. Running a validation needs access to the Set, permission to read connections and to run data jobs, and a deployment with the validation worker.
The SAP sales-order check reads one order from an OData V2 sales-order service (such as API_SALES_ORDER_SRV) saved as an endpoint under your SAP connection. It compares the document id, sold-to party and total net amount as exact text: 0000004711 is not 4711, and 1250.00 is not 1250.0. Save expected values exactly as SAP returns them.
Missing values, undocumented sources and stale checks remain visible and prevent export and validation. Ask chat to resolve them through the connected system and save new evidence. For a locked Set, save a new candidate and review the changes. Exporting inputs does not run the test or recheck SAP automatically.