Designing stateful orchestration for a long-running commercial process
A case told through a commercial-performance cycle: the pattern is stateful orchestration, not a monthly job.
The case in brief
Current reality
A recurring commercial process runs its dependent steps as one scheduled job, one long chain or several loosely coordinated automations—so when step seven fails, nobody can say what finished, what may run again or which external effects already committed.
What must become true
Every execution has one durable run identity and an explicit state. It starts once, runs its steps in order, waits and resumes, retries only what is safe, skips completed work, stops when continuing is unsafe, shows its progress and finishes with a definitive outcome—so a failure in step seven never means blindly rerunning steps one to six.
Design question
How can a long-running business process execute across dependent steps and external delays while remaining safe to resume, retry and operate?
01The reality · current state
What the automation does today.
Every period a commercial cycle prepares source data, computes analytical state, classifies accounts into segments, publishes the resulting signals to the CRM, activates follow-up, reconciles what happened and closes the run. The steps depend on each other and on work the cycle does not control: a data refresh, an analytical computation, a downstream platform that processes in its own time. The first version was one scheduled job that called every step in sequence. When it failed half-way, an administrator started it again from the beginning.
- Nobody can say which period a run belongs to, or whether it finished
- A restart after a failure repeats steps that already succeeded
- Accounts receive duplicate follow-up when a step runs twice
- The job starts before the data it depends on is ready
- A late callback is read as “never started”, so work is launched again
- Failures are visible only in a technical log, and a stuck run has no owner
- A test run and a production run look identical
02Why the naïve version fails
What the first automation cannot answer.
A naïve implementation runs the steps as one scheduled job, one long chain or several loosely coordinated automations. When something fails half-way, the questions that matter have no answer.
- Which steps finished?No durable record says so; the job log ends at the error.
- Which did not?A step that was running looks the same as one that never started.
- Which may safely run again?Some steps are harmless to repeat; others create work for real people.
- Which external operations already committed?The downstream platform may have accepted a batch the job never heard back about.
- Should the whole process restart?Restarting is the only recovery the design offers—and it repeats everything.
- Who owns a run that is stuck?Nobody is named; the first person to notice is the owner.
- What if a dependency finishes later?The job has already failed or moved on; nothing picks the work back up.
03Key dimension · Run identity, state machine, continuation & safe resume
One run, one state machine, one place to look
State determines what may execute next. The run record is the orchestration’s memory and its control plane: which period, which state, which step, what it waits for, what failed and who owns it.
- 01Prepare source dataCompleteRefresh confirmed for 2026-09
- 02Compute analytical stateCompleteStarted and completed markers recorded
- 03Classify accountsCompletePopulation = classified
- 04Publish operational signalsWaitingBatch 6 of 12 confirmed; the continuation checks again
- 05Activate actionsNot startedMay start only after step 4 is confirmed
- 06Reconcile outcomesNot startedOnce per run
- 07Close the runNot startedA person closes it
- Retry
A transient failure: retry the step or batch, not the run
- Intervention
Unsafe to continue: a named owner acts, then the run resumes
- Abort
Stopped on purpose, with a reason; nothing silently continues
- Complete
Every step done, reconciled and closed by a person
Waiting
A dependency has not finished yet. Waiting is a state, not a failure: the run leaves the transaction and continues later.
- ProcessingContinuation check finds the dependency completeTimer
- ReconcilingContinuation check finds activation confirmed downstreamTimer
- Intervention requiredDependency not ready within its budgetTimer
Entered from Preparing, Processing, Activating.
- Created
The run exists for its period and scope; inputs are frozen.
Can move to: Preparing (Start guard passes: no other run for this period and scope) - Preparing
Checking that the source data for the period is ready.
Can move to: Processing (Source data ready for the period); Waiting (Source refresh not finished); Aborted (Dry run meets a condition that makes continuing unsafe) - Processing
Computing, classifying and publishing—one step at a time.
Can move to: Waiting (Analytical computation still pending); Activating (Signals published and confirmed); Intervention required (Design or data error, e.g. population ≠ classified) - Waiting
A dependency has not finished yet. Waiting is a state, not a failure: the run leaves the transaction and continues later.
Can move to: Processing (Continuation check finds the dependency complete); Reconciling (Continuation check finds activation confirmed downstream); Intervention required (Dependency not ready within its budget) - Activating
Handing the published signals to action, batch by batch.
Can move to: Waiting (Downstream platform still processing); Intervention required (Unknown outcome on a batch—resolve before any retry) - Reconciling
Comparing what was expected with what exists downstream.
Can move to: Completed (Differences within tolerance; the owner closes the run); Intervention required (Differences beyond tolerance) - Completed
Every step done and reconciled; the owner closed the run.
No automatic way out - Intervention required
Continuing automatically would be unsafe; a named owner must act. The run holds at its step.
Can move to: Resuming (Owner fixed the cause); Aborted (Owner abandons the period’s run, with a reason) - Aborted
Stopped on purpose, with a reason. Completed steps stay recorded.
No automatic way out - Resuming
Continuing from the step that stopped, with the same run and effect keys.
Can move to: Processing (The stopped step was a processing step); Activating (Continue from the last confirmed batch)
- Run ID
- RUN-2609-ALL (example)
- Period / scope
- 2026-09 · all companies
- Started at
- 1 Oct, 06:10
- Current state
- Waiting
- Current step
- 4 · Publish operational signals
- Status
- Batch 6 of 12 confirmed
- Last transition
- Processing → Waiting, 07:42
- Outcome
- Open
- Failure reason
- None
- Owner / intervention
- Commercial operations · none required
The continuation does not own business logic. It finds unfinished runs, checks whether their dependency has moved, and wakes the state machine—which decides what may run next.
04Step contract
Each step with its mode, its actor and its guard.
The period opens and the run for that period and scope does not exist yet.
- Every eligible account has a current signal for the period
- Follow-up exists once per Account + segment + entry period
- The run ends Completed with a reconciliation report—or Aborted with a reason
- The business owner has the summary
SchedulerData platformCRMCampaign platform
- 01Prepare source data
Wait for the period’s refresh; it is a dependency, not a step the run executes.
- 02Compute analytical state
Compute performance to target for the period from frozen inputs.
- 03Classify accounts
Exactly one segment per eligible account.
- 04Publish operational signals
Publish the period’s version and apply it in batches; resume from the last confirmed batch.
- 05Activate actions
Hand the signals to closed-loop automation; follow-up is keyed, never repeated.
- 06Reconcile outcomes
Compare expected with actual signals and actions; classify the differences.
- 07Close the run
Review the summary and close the run—or abort it with a reason.
| Step | Starts when | Start guard | Waits for | On failure | Intervention owner |
|---|---|---|---|---|---|
| 1 · Prepare | Run created for the period | Inputs frozen once | The source refresh | Timeout → intervention (data team) | Data team |
| 2 · Compute | Step 1 complete | Started marker for this run | The analytical computation | Transient → retry; data error → intervention | Data platform team |
| 3 · Classify | Step 2 complete | Population = classified | — | Data error → stop before publishing | Sales operations |
| 4 · Publish | Step 3 complete | Version published once; batch cursor | Downstream batch confirmation | Unknown outcome → resolve by key, then retry | Integration operations |
| 5 · Activate | Step 4 confirmed | Account + segment + entry period | Downstream processing | Rejection → repair queue; the run goes on | Commercial operations |
| 6 · Reconcile | Step 5 complete | Once per run | — | Beyond tolerance → intervention | Commercial operations |
| 7 · Close | Reconciliation within tolerance | Closed once, by a person | The owner’s review | Owner closes or aborts with a reason | Commercial operations |
05Execution safety
If it can run twice, it is designed to run twice.
The cycle outlives one transaction, depends on work it does not control, touches systems whose effects a rerun cannot undo, and must resume at the step that stopped.
- Run key
- Period + scope (e.g. 2026-09 · all companies)
- Start once
- A second run for the same key is refused atomically
- Started marker
- Persisted before a step launches work; started is not completed
- Step guard
- Each step checks its own state for this run before it executes
- Effect keys
- Account + segment + entry period for every follow-up
- Continuation
- A recurring check wakes unfinished runs; it never creates one
- Dry run
- Same run, same checks, no side effects—and it aborts when continuing would be unsafe
- Abort
- A terminal state with a reason; completed steps stay recorded
06Retry, idempotency & safe failure
Retry belongs at the step boundary—and not every failure is a retry.
A failure is classified before anything runs again. “Exactly once” is not a delivery guarantee the platform provides: it is a business-effect property the design earns through durable state, step guards and idempotent boundaries.
Four classes of failure
| Class | Example | Behaviour | Where the run continues |
|---|---|---|---|
| Transient | A platform limit is reached mid-batch | Back off and retry the batch | At the batch cursor |
| Business rejection | Downstream refuses a record for a closed account | No retry; the record goes to a repair queue | The run goes on; the record returns when repaired |
| Unknown outcome | A call times out after the request has left | Resolve first: look up by the step’s key; retry only if nothing exists | At the same step, once the outcome is known |
| Design or data error | Classified accounts do not match the population | Stop before anything is published | Intervention, then resume—or abort |
If a step can be invoked twice
Every step decides in advance what a second invocation means.
| Situation | The second invocation | Because |
|---|---|---|
| The step already completed | Is ignored | The step guard reads Completed for this run |
| The step is still running | Is refused; the first continues | The started marker was persisted before any work began |
| The step stopped half-way | Resumes | The batch cursor records the last confirmed batch |
| The result already exists downstream | Detects it and records it | Every effect is looked up by its key before it is created |
| The computation runs again | Produces the same result | Inputs were frozen when the run was created |
| A callback never arrived | Does not assume the step never launched | Started and completed are different facts; the key is checked downstream |
A controlled abort beats a silent continuation
| Condition | Why continuing is unsafe | Response |
|---|---|---|
| A dry run reaches a step that would need a side effect it cannot fake | The rehearsal would stop being a rehearsal | Abort the dry run with a reason; report what would have happened |
| Inputs changed after the run was created | Later steps would disagree with earlier ones | Abort; start a new run for the same period |
| A run for the same period is already Completed | A second one would repeat real follow-up | Refuse to start |
07Failure & recovery
Not every failure is a retry.
- Source refresh late DependencyDependency not complete when the run startsWait as a state; the continuation checks again; escalate after the budgetData team
- Platform limit reached mid-batch Transient · retryLimit error on a batchBack off; continue from the cursorAutomation
- Run started twice DuplicateRun key already existsRefuse the second startNobody: absorbed
- Batch outcome unknown DependencyTimeout after the request leftLook up by key before any retry; intervention if still unclearIntegration operations
- Record refused downstream RejectionBusiness rejection on a recordRepair queue; the run continuesCommercial operations
- Classification incomplete ValidationPopulation ≠ classified countStop before publishing; interventionSales operations
08Observability
Business observability, not only logs.
An operator—not only an engineer—must be able to answer these questions from the run itself.
| Question | Answered by | Seen in |
|---|---|---|
| Which run is active? | Run ID, period and scope | Run list |
| Where is it? | Current state and current step | Run record |
| How long has it been there? | Last transition against the step’s budget | Ageing flag on the run |
| Which dependency is pending? | The waiting reason | Run record |
| What already executed? | Started and completed markers per step | Step history |
| What failed? | Failure class and reason | Run record and step history |
| Can it resume automatically? | The failure class: transient or not | Run record |
| Who owns intervention? | The step’s intervention owner | Run record and the owner’s queue |
| What will happen next? | The valid transitions from the current state | State machine |
09The trade-offs
Credible options, judged against these premises.
One scheduled job calling every step
Short, local, idempotent work that finishes in one go
Cost: Restart from zero; duplicates; no visibilityChained automations with no durable run state
Two or three local steps with no external waits
Cost: No run identity; failures break the chain silentlyDurable run, explicit state machine and continuation
Long, dependent, recurring work across systems
Cost: A run model, a continuation and a small run view to build and ownCriterion by criterion
The durable run earns its cost here because the process outlives one transaction, waits on work it does not control, touches systems whose effects a rerun cannot undo, and recurs every period. For short, local, idempotent work, one scheduled job remains the simpler right answer.
| Criterion | One scheduled job | Chained automations | Durable run + state machine |
|---|---|---|---|
| Recoverability | Restart from zero | Depends on where the chain broke | Resume at the stopped step |
| Visibility | A technical log | Scattered across automations | One run record |
| Rerun safety | Repeats everything | Unclear per link | Guards and keys per step |
| Long waits | Holds a transaction or fails | Lost between links | Waiting is a state |
| Dependency handling | Assumed ready | Implicit | Explicit, with a budget |
| Supportability | Whoever notices | Whoever built the link | Named owner per step |
| Implementation complexity | Low | Low to start, high to support | Higher: run model and continuation |
Design decision
Persist one durable orchestration run, model execution as an explicit state machine, make each step start-once and safely resumable, treat waiting and uncertain outcomes as states rather than failures, and use a lightweight continuation mechanism to advance incomplete runs.
A scheduler tells work when to wake up. A state machine tells it what it is allowed to do.
10The second layer
Questions that change the design.
Identity & start
- How is one run identified?
- How is a duplicate start prevented?
- How are dependencies represented?
Failure & resume
- What happens when step 4 fails?
- Can step 5 start twice?
- What does a late callback mean?
Completion & control
- When is the run complete?
- How does a dry run differ from production?
- Who sees run status—and who owns a stuck run?
11Decisions & outputs
What the work produces.
- 01Run model & run key
- 02State machine
- 03Step contract
- 04Continuation design
- 05Retry classes & idempotency boundaries
- 06Dry-run and abort semantics
- 07Run status view