Python tips
Avoid common scenario data-shape, file and retry mistakes.
Check the actual result shape
dm.generate() and dm.generate_from_template_id() return an object with live_data and dependencies. Convert columns to rows before passing data to pandas, CSV writers or a row-oriented API. See the conversion example.
Use explicit paths
Read DATAMAKER_WORKSPACE_UPLOADS and write DATAMAKER_WORKSPACE_OUTPUTS. Do not assume the current working directory or a path on your laptop exists on a hosted runner. Check required inputs early and report their filenames without printing their contents.
Verify dependencies
Inspect the runner's Python and package versions when a script behaves differently locally. Declare anything beyond the preinstalled set in the scenario's # requirements: block; see Python environment. Do not assume pandas.read_excel() works without the required spreadsheet engine.
Preserve failure
Use HTTP timeouts and check status codes. Let unrecoverable exceptions fail the run; catching and printing every exception can make CI pass with missing data. Log counts and step names, not full credentials or raw personal data.
Design for reruns
A scenario starts from the beginning when rerun. After an ambiguous write failure, inspect the destination before repeating the request. Use the target's supported idempotency mechanism or a key-map workflow where appropriate. Cancellation and process termination may prevent cleanup code from finishing.
Use data jobs when you need chunk checkpoints, resume and tracked rejected rows.