Automation case / 01 · Run identity, state machine, continuation & safe resume

Designing stateful orchestration for a long-running commercial process

A case told through a commercial-performance cycle: the pattern is stateful orchestration, not a monthly job.

The case in brief

Current reality

A recurring commercial process runs its dependent steps as one scheduled job, one long chain or several loosely coordinated automations—so when step seven fails, nobody can say what finished, what may run again or which external effects already committed.

What must become true

Every execution has one durable run identity and an explicit state. It starts once, runs its steps in order, waits and resumes, retries only what is safe, skips completed work, stops when continuing is unsafe, shows its progress and finishes with a definitive outcome—so a failure in step seven never means blindly rerunning steps one to six.

Design question

How can a long-running business process execute across dependent steps and external delays while remaining safe to resume, retry and operate?

01The reality · current state

What the automation does today.

Every period a commercial cycle prepares source data, computes analytical state, classifies accounts into segments, publishes the resulting signals to the CRM, activates follow-up, reconciles what happened and closes the run. The steps depend on each other and on work the cycle does not control: a data refresh, an analytical computation, a downstream platform that processes in its own time. The first version was one scheduled job that called every step in sequence. When it failed half-way, an administrator started it again from the beginning.

  • Nobody can say which period a run belongs to, or whether it finished
  • A restart after a failure repeats steps that already succeeded
  • Accounts receive duplicate follow-up when a step runs twice
  • The job starts before the data it depends on is ready
  • A late callback is read as “never started”, so work is launched again
  • Failures are visible only in a technical log, and a stuck run has no owner
  • A test run and a production run look identical

02Why the naïve version fails

What the first automation cannot answer.

A naïve implementation runs the steps as one scheduled job, one long chain or several loosely coordinated automations. When something fails half-way, the questions that matter have no answer.

  • Which steps finished?No durable record says so; the job log ends at the error.
  • Which did not?A step that was running looks the same as one that never started.
  • Which may safely run again?Some steps are harmless to repeat; others create work for real people.
  • Which external operations already committed?The downstream platform may have accepted a batch the job never heard back about.
  • Should the whole process restart?Restarting is the only recovery the design offers—and it repeats everything.
  • Who owns a run that is stuck?Nobody is named; the first person to notice is the owner.
  • What if a dependency finishes later?The job has already failed or moved on; nothing picks the work back up.

03Key dimension · Run identity, state machine, continuation & safe resume

One run, one state machine, one place to look

State determines what may execute next. The run record is the orchestration’s memory and its control plane: which period, which state, which step, what it waits for, what failed and who owns it.

Run progress
  1. 01Prepare source dataCompleteRefresh confirmed for 2026-09
  2. 02Compute analytical stateCompleteStarted and completed markers recorded
  3. 03Classify accountsCompletePopulation = classified
  4. 04Publish operational signalsWaitingBatch 6 of 12 confirmed; the continuation checks again
  5. 05Activate actionsNot startedMay start only after step 4 is confirmed
  6. 06Reconcile outcomesNot startedOnce per run
  7. 07Close the runNot startedA person closes it
Ways out of the path
  • Retry

    A transient failure: retry the step or batch, not the run

  • Intervention

    Unsafe to continue: a named owner acts, then the run resumes

  • Abort

    Stopped on purpose, with a reason; nothing silently continues

  • Complete

    Every step done, reconciled and closed by a person

Run path
Failure
Recovery
Step 4 of 7
Run path

Waiting

A dependency has not finished yet. Waiting is a state, not a failure: the run leaves the transaction and continues later.

Can move to
  • ProcessingContinuation check finds the dependency completeTimer
  • ReconcilingContinuation check finds activation confirmed downstreamTimer
  • Intervention requiredDependency not ready within its budgetTimer

Entered from Preparing, Processing, Activating.

  1. Created · Run path

    The run exists for its period and scope; inputs are frozen.

    Can move to: Preparing (Start guard passes: no other run for this period and scope)
  2. Preparing · Run path

    Checking that the source data for the period is ready.

    Can move to: Processing (Source data ready for the period); Waiting (Source refresh not finished); Aborted (Dry run meets a condition that makes continuing unsafe)
  3. Processing · Run path

    Computing, classifying and publishing—one step at a time.

    Can move to: Waiting (Analytical computation still pending); Activating (Signals published and confirmed); Intervention required (Design or data error, e.g. population ≠ classified)
  4. Waiting · Run path

    A dependency has not finished yet. Waiting is a state, not a failure: the run leaves the transaction and continues later.

    Can move to: Processing (Continuation check finds the dependency complete); Reconciling (Continuation check finds activation confirmed downstream); Intervention required (Dependency not ready within its budget)
  5. Activating · Run path

    Handing the published signals to action, batch by batch.

    Can move to: Waiting (Downstream platform still processing); Intervention required (Unknown outcome on a batch—resolve before any retry)
  6. Reconciling · Run path

    Comparing what was expected with what exists downstream.

    Can move to: Completed (Differences within tolerance; the owner closes the run); Intervention required (Differences beyond tolerance)
  7. Completed · Run path · terminal

    Every step done and reconciled; the owner closed the run.

    No automatic way out
  8. Intervention required · Failure

    Continuing automatically would be unsafe; a named owner must act. The run holds at its step.

    Can move to: Resuming (Owner fixed the cause); Aborted (Owner abandons the period’s run, with a reason)
  9. Aborted · Failure · terminal

    Stopped on purpose, with a reason. Completed steps stay recorded.

    No automatic way out
  10. Resuming · Recovery

    Continuing from the step that stopped, with the same run and effect keys.

    Can move to: Processing (The stopped step was a processing step); Activating (Continue from the last confirmed batch)
The run record · Example values
Run ID
RUN-2609-ALL (example)
Period / scope
2026-09 · all companies
Started at
1 Oct, 06:10
Current state
Waiting
Current step
4 · Publish operational signals
Status
Batch 6 of 12 confirmed
Last transition
Processing → Waiting, 07:42
Outcome
Open
Failure reason
None
Owner / intervention
Commercial operations · none required

The continuation does not own business logic. It finds unfinished runs, checks whether their dependency has moved, and wakes the state machine—which decides what may run next.

04Step contract

Each step with its mode, its actor and its guard.

Trigger

The period opens and the run for that period and scope does not exist yet.

Done means
  • Every eligible account has a current signal for the period
  • Follow-up exists once per Account + segment + entry period
  • The run ends Completed with a reconciliation report—or Aborted with a reason
  • The business owner has the summary

Systems and partiesSchedulerData platformCRMCampaign platform

  1. 01
    Prepare source data

    Wait for the period’s refresh; it is a dependency, not a step the run executes.

    OrchestratedData platformGuardRefresh complete for the period
  2. 02
    Compute analytical state

    Compute performance to target for the period from frozen inputs.

    AutomatedData platformGuardNot yet computed for this run
  3. 03
    Classify accounts

    Exactly one segment per eligible account.

    AutomatedData platformGuardPopulation = classified
  4. 04
    Publish operational signals

    Publish the period’s version and apply it in batches; resume from the last confirmed batch.

    OrchestratedIntegration → CRMGuardBatch cursor
  5. 05
    Activate actions

    Hand the signals to closed-loop automation; follow-up is keyed, never repeated.

    AutomatedCRM automationGuardAccount + segment + entry period
  6. 06
    Reconcile outcomes

    Compare expected with actual signals and actions; classify the differences.

    AutomatedReconcilerGuardOnce per run
  7. 07
    Close the run

    Review the summary and close the run—or abort it with a reason.

    HumanCommercial operationsGuardReconciliation within tolerance
Step contract: when each step may start, its start guard, what it waits for, how it fails and who intervenes
StepStarts whenStart guardWaits forOn failureIntervention owner
1 · PrepareRun created for the periodInputs frozen onceThe source refreshTimeout → intervention (data team)Data team
2 · ComputeStep 1 completeStarted marker for this runThe analytical computationTransient → retry; data error → interventionData platform team
3 · ClassifyStep 2 completePopulation = classified—Data error → stop before publishingSales operations
4 · PublishStep 3 completeVersion published once; batch cursorDownstream batch confirmationUnknown outcome → resolve by key, then retryIntegration operations
5 · ActivateStep 4 confirmedAccount + segment + entry periodDownstream processingRejection → repair queue; the run goes onCommercial operations
6 · ReconcileStep 5 completeOnce per run—Beyond tolerance → interventionCommercial operations
7 · CloseReconciliation within toleranceClosed once, by a personThe owner’s reviewOwner closes or aborts with a reasonCommercial operations

05Execution safety

If it can run twice, it is designed to run twice.

Execution patternDurable run record + explicit state machine + start-once steps + lightweight continuation

The cycle outlives one transaction, depends on work it does not control, touches systems whose effects a rerun cannot undo, and must resume at the step that stopped.

Run key
Period + scope (e.g. 2026-09 · all companies)
Start once
A second run for the same key is refused atomically
Started marker
Persisted before a step launches work; started is not completed
Step guard
Each step checks its own state for this run before it executes
Effect keys
Account + segment + entry period for every follow-up
Continuation
A recurring check wakes unfinished runs; it never creates one
Dry run
Same run, same checks, no side effects—and it aborts when continuing would be unsafe
Abort
A terminal state with a reason; completed steps stay recorded

06Retry, idempotency & safe failure

Retry belongs at the step boundary—and not every failure is a retry.

A failure is classified before anything runs again. “Exactly once” is not a delivery guarantee the platform provides: it is a business-effect property the design earns through durable state, step guards and idempotent boundaries.

Four classes of failure

Failure classes: an example, the behaviour and where the run continues
ClassExampleBehaviourWhere the run continues
TransientA platform limit is reached mid-batchBack off and retry the batchAt the batch cursor
Business rejectionDownstream refuses a record for a closed accountNo retry; the record goes to a repair queueThe run goes on; the record returns when repaired
Unknown outcomeA call times out after the request has leftResolve first: look up by the step’s key; retry only if nothing existsAt the same step, once the outcome is known
Design or data errorClassified accounts do not match the populationStop before anything is publishedIntervention, then resume—or abort

If a step can be invoked twice

Every step decides in advance what a second invocation means.

Idempotency boundaries: the situation, what the second invocation does and the mechanism behind it
SituationThe second invocationBecause
The step already completedIs ignoredThe step guard reads Completed for this run
The step is still runningIs refused; the first continuesThe started marker was persisted before any work began
The step stopped half-wayResumesThe batch cursor records the last confirmed batch
The result already exists downstreamDetects it and records itEvery effect is looked up by its key before it is created
The computation runs againProduces the same resultInputs were frozen when the run was created
A callback never arrivedDoes not assume the step never launchedStarted and completed are different facts; the key is checked downstream

A controlled abort beats a silent continuation

Safe failure: the condition, why continuing is unsafe and the response
ConditionWhy continuing is unsafeResponse
A dry run reaches a step that would need a side effect it cannot fakeThe rehearsal would stop being a rehearsalAbort the dry run with a reason; report what would have happened
Inputs changed after the run was createdLater steps would disagree with earlier onesAbort; start a new run for the same period
A run for the same period is already CompletedA second one would repeat real follow-upRefuse to start

07Failure & recovery

Not every failure is a retry.

  • Source refresh late DependencyDependency not complete when the run startsWait as a state; the continuation checks again; escalate after the budgetData team
  • Platform limit reached mid-batch Transient · retryLimit error on a batchBack off; continue from the cursorAutomation
  • Run started twice DuplicateRun key already existsRefuse the second startNobody: absorbed
  • Batch outcome unknown DependencyTimeout after the request leftLook up by key before any retry; intervention if still unclearIntegration operations
  • Record refused downstream RejectionBusiness rejection on a recordRepair queue; the run continuesCommercial operations
  • Classification incomplete ValidationPopulation ≠ classified countStop before publishing; interventionSales operations

08Observability

Business observability, not only logs.

An operator—not only an engineer—must be able to answer these questions from the run itself.

Operator questions, what answers them and where it is seen
QuestionAnswered bySeen in
Which run is active?Run ID, period and scopeRun list
Where is it?Current state and current stepRun record
How long has it been there?Last transition against the step’s budgetAgeing flag on the run
Which dependency is pending?The waiting reasonRun record
What already executed?Started and completed markers per stepStep history
What failed?Failure class and reasonRun record and step history
Can it resume automatically?The failure class: transient or notRun record
Who owns intervention?The step’s intervention ownerRun record and the owner’s queue
What will happen next?The valid transitions from the current stateState machine

09The trade-offs

Credible options, judged against these premises.

Situational

One scheduled job calling every step

Short, local, idempotent work that finishes in one go

Cost: Restart from zero; duplicates; no visibility
Rejected

Chained automations with no durable run state

Two or three local steps with no external waits

Cost: No run identity; failures break the chain silently
Selected

Durable run, explicit state machine and continuation

Long, dependent, recurring work across systems

Cost: A run model, a continuation and a small run view to build and own

Criterion by criterion

The durable run earns its cost here because the process outlives one transaction, waits on work it does not control, touches systems whose effects a rerun cannot undo, and recurs every period. For short, local, idempotent work, one scheduled job remains the simpler right answer.

Orchestration options compared criterion by criterion
CriterionOne scheduled jobChained automationsDurable run + state machine
RecoverabilityRestart from zeroDepends on where the chain brokeResume at the stopped step
VisibilityA technical logScattered across automationsOne run record
Rerun safetyRepeats everythingUnclear per linkGuards and keys per step
Long waitsHolds a transaction or failsLost between linksWaiting is a state
Dependency handlingAssumed readyImplicitExplicit, with a budget
SupportabilityWhoever noticesWhoever built the linkNamed owner per step
Implementation complexityLowLow to start, high to supportHigher: run model and continuation

Design decision

Persist one durable orchestration run, model execution as an explicit state machine, make each step start-once and safely resumable, treat waiting and uncertain outcomes as states rather than failures, and use a lightweight continuation mechanism to advance incomplete runs.

Key principleA scheduler tells work when to wake up. A state machine tells it what it is allowed to do.

10The second layer

Questions that change the design.

Identity & start

  1. How is one run identified?
  2. How is a duplicate start prevented?
  3. How are dependencies represented?

Failure & resume

  1. What happens when step 4 fails?
  2. Can step 5 start twice?
  3. What does a late callback mean?

Completion & control

  1. When is the run complete?
  2. How does a dry run differ from production?
  3. Who sees run status—and who owns a stuck run?

11Decisions & outputs

What the work produces.

  1. 01Run model & run key
  2. 02State machine
  3. 03Step contract
  4. 04Continuation design
  5. 05Retry classes & idempotency boundaries
  6. 06Dry-run and abort semantics
  7. 07Run status view