Data jobs
Extract, transform, mask, load, cancel and resume chunked data processing.
Data jobs process datasets and retain progress. They are separate from Python scenario runs and have different status and resume semantics.
Build a pipeline
| Job kind | Input and purpose | Start route |
|---|---|---|
| Extraction | Read a configured source into a dataset. | POST /extractions |
| Transform | Apply an approved mapping plan to a ready dataset. | POST /transforms |
| Mask | Apply a masking policy to a ready dataset. | POST /masks |
| Load | Deliver a ready dataset to a supported target. | POST /loads |
| File load | Render a ready dataset through a file template, package it, upload it and submit it. | POST /file-loads |
| Poll | Wait on work already submitted to an external system, then read its result artifact back. | POST /polls |
Use the agent or the deployed API reference for each route's configuration. There is no generic POST /data-jobs start route.
Begin with a small bounded extraction. Inspect source columns and values, review the mapping or policy, then increase volume once the sample is correct. A transform requires an approved mapping and every mapping row to be marked mapped.
Chain the stages
POST /data-jobs/chain creates an ordered pipeline of between two and twenty jobs in one transaction. Only the first is enqueued; each later job runs when the job it depends on completes, and a step that names no input dataset inherits its upstream's output. Extract, mask and load becomes one call, with no dataset identifiers to carry between three.
Two properties are worth knowing before you review a chain:
- Approval is evaluated per step, at that step's own class, before anything is created. A chain cannot carry a write past a read-only grant. A write step without a granted approval refuses the whole chain with
409and creates no jobs at all. - A failure stops the chain rather than being skipped. An upstream ending failed or cancelled fails its dependents with a reason naming it, so nothing runs on data that never arrived.
Each job in the chain keeps its own status, checkpoint and counters. Inspect the step that failed rather than the chain as a whole.
Wait on external work
A poll job replaces holding a process open. It reads a status URL on an interval until the expression you declared as terminal holds, and checkpoints every attempt, so a restart continues the wait instead of beginning it again. That is what makes it usable for work outliving a scenario's 3,600-second ceiling, such as an Oracle Fusion ESS request. A failWhen expression ends the job as failed, and a maximum duration ends it with a distinct reason, so a poll never hangs indefinitely.
The status URL must share its endpoint's origin. That is checked when the job is created, because credentials are bound to an origin.
A poll can also ingest the result rather than only observing it: once the terminal expression holds it fetches the target's result artifact, parses it as CSV or JSON, quarantines each rejected source row with the vendor's error attached, and records the accepted rows in a key map so a file-loaded record can be rolled back exactly like an API-loaded one.
Track the result
GET /data-jobs/{id} exposes status and progress, including rows processed, failed and skipped, and any linked input, output or error dataset. Statuses are pending, running, completed, failed and cancelled.
A completed job can still have rejected rows. Inspect counters and the error dataset, not only the final status. Transformation and load errors can be quarantined for review. Mask jobs stop on unmaskable values; they do not persist those refused raw records as a quarantine output.
Cancel and resume
POST /data-jobs/{id}/cancel requests cancellation. Running work observes cancellation at processing boundaries, so an in-flight batch may already have reached its target.
Only failed or cancelled jobs can be resumed through POST /data-jobs/{id}/resume. Resume retains counters and the checkpoint; it does not start a clean run. Fix the cause first and inspect the target for partial writes. Scenario re-runs, by comparison, start Python from the beginning.
A file load checkpoints the stage it reached - render, package, upload, submit - so a resumed job does not upload a file the target already accepted or submit a request it already received.
Load approvals and duplicate prevention
Loads require a target-bound approval before a job is created. Standard, migration-scale and rollback approvals are distinct. The standard load ceiling is 100,000 input rows; migration mode requires its own approval and still batches individual writes.
A file load carries its own approval, separate again from a standard load, because it delivers a whole rendered file rather than individual records. A poll only reads, so it needs none - but it can quarantine and write key-map entries from the result it ingests, which is worth reviewing alongside the load that produced it.
Configure a key map so previously loaded source keys can be skipped. This reduces duplicate delivery but is not a blanket exactly-once guarantee after an uncertain network failure. Review target responses and key-map coverage.
After delivery, create a reconciliation report to compare expectations with the recorded outcomes.