Multi-step workflows

Multi-step workflows is the playbook for a business process that crosses several services or companies and outlives a request: an order that reserves stock, takes a UPI payment, gets a store to accept and finds a rider; a loan application that waits days for KYC. Every step can fail on its own, some wait for minutes or days, and the whole must end done or undone. Weak answers draw arrows between services. Strong ones say who owns the process, what happens when step 4 fails after step 2 moved money, and how a retry avoids charging twice.

Picture it first: a wedding planner’s diary

Your cousin’s wedding needs a venue, a caterer, a decorator and a band, booked in that order.

1. One shop does everything. One booking happens in full or not at all.
2. You phone each vendor yourself. You pay the caterer, fall ill before calling the band, and nobody knows what was done.
3. Vendors tell each other. Nobody sees the whole plan, and a change means calling everyone.
4. A planner with a diary. She books each vendor and writes it down. If the venue falls through, she cancels the caterer and loses the advance: an undo, not an eraser. Once the cards are posted, she only pushes forward.
5. The diary outlives the planner. If she falls ill, a colleague reads it and carries on. It holds reminders too: “if the pandit hasn’t confirmed by Friday, call him”.
6. Every payment quotes the booking number, so a transfer sent twice is spotted and returned.
7. In wedding season the agency runs 300 weddings at once and changes its checklist without confusing those half-planned.

Map it back: one shop is one database transaction; phoning vendors is synchronous calls in request code; vendors telling each other is choreography; the planner is a saga orchestrator with compensations and a pivot; the diary that survives her is durable execution with timers; the booking number is an idempotency key; wedding season is versioning, visibility and quotas.

Spotting the pattern

Listen for these cues:

• One business action, several owners: inventory, payments, the store, riders and notifications each own their data, or a step calls a bank, a KYC vendor or a partner hotel.
• Steps that fail independently, after earlier steps have changed something.
• Steps that wait: a UPI approval, a bank callback, a human sign-off, documents that take days, “cancel if unpaid in 10 minutes”.
• All or compensate: if the store rejects the order, release the stock and refund the money.
• “Where is order 81?” Support needs the stage of every instance.
• Retries that must not repeat effects: a timed-out payment call must not charge twice.

It does not apply when:

• Everything is in one database. Use a transaction (ACID transactions).
• It is one long piece of work, such as an export or a transcode: that is a long-running job.
• It is a fire-and-forget reaction, such as a confirmation SMS or an analytics event: publish an event and let subscribers react.

The running example is a quick-commerce and food platform averaging 10 million orders a day, with a design target of 2,000 orders a second at the festival-eve dinner peak. Each order goes: place → reserve stock at the dark store → collect payment by UPI (the customer approves in their UPI app, and the bank calls back) → store accepts and packs → assign a rider → deliver → settle with the store, with a refund if anything fails before the store commits. An order lives about 30 minutes, so at peak 2,000 a second × 1,800 s = 3.6 million orders are in flight (Little’s law). A slower cousin, loan onboarding, waits up to a week for documents and a video-KYC slot.

Level 0: one transaction

If stock, orders and the payment ledger sit in one PostgreSQL database, placing an order is one transaction:

BEGIN;
UPDATE stock SET qty = qty - 1
 WHERE store_id = 7 AND sku = 'atta-5kg' AND qty >= 1;   -- 0 rows: roll back
INSERT INTO orders (id, user_id, status) VALUES (81, 42, 'PLACED');
INSERT INTO ledger (order_id, account, amount_paise) VALUES (81, 'wallet:42', -64000);
COMMIT;

Either all three rows change or none do, and no other transaction sees a half-placed order. No compensation, no orchestrator, no events. When the data can live together, this is the best answer, and a service that owns all three tables is a perfectly good design (monolith vs microservices).

It stops working in three ways. The data splits across services, each with its own database. A step leaves your database: a bank, a store’s tablet, a rider’s phone can’t join your transaction. Or a step waits: holding row locks for the ten minutes a customer takes to approve a UPI request is the “long-lived transaction” problem Garcia-Molina and Salem described in 1987. Locks are held until commit, everyone else queues behind them, and deadlocks rise steeply with transaction size (with the fourth power of it, in the analysis they cite). Two-phase commit across services blocks while its coordinator is down, and banks don’t take part in yours (two-phase commit vs sagas).

You gain atomicity and isolation for free. You pay with every step’s data in one database and no step that waits. It breaks at the first second database or external system. Say: “If these tables can live in one database, one transaction beats any saga.”

Level 1: call each service in sequence

The checkout handler calls inventory, then payments, then the store, then rider dispatch, then notifications, and returns. It reads well and fails badly:

• Availability multiplies. Six calls at 99.9% each succeed together 0.999⁶ ≈ 99.4% of the time. At 2,000 orders a second, 0.6% is 12 orders a second, about 43,000 an hour, that fail somewhere in the middle.
• Waits don’t fit in a request. The customer may take minutes to approve the UPI collect request; no HTTP request should stay open that long (long-running jobs).
• A crash leaves no record. The pod dies after the charge and before the store call: the money is taken, the order goes nowhere, and nothing says which steps ran.
• Timeouts are ambiguous. The payment call timed out: did it charge? Retry and you may charge twice; don’t and you may lose a paid order.
• Inline compensation dies with the process: a try/except refund never runs if the process is gone.

You gain simple, readable code and an immediate answer. You pay with the product of every dependency’s availability and the sum of their latencies. It breaks at the first crash between two side effects. Say: “A chain of synchronous calls has no memory, so a crash after the charge strands the order; the process needs durable state.”

Level 2: choreography with events

Give each service its own step and let events carry the flow. Orders commits the order and an OrderPlaced event together through a transactional outbox; inventory reacts, reserves stock and publishes StockReserved; payments reacts and publishes PaymentCaptured or PaymentFailed; and so on (event-driven architecture). Compensation is choreographed too: inventory also listens for PaymentFailed and releases the stock.

Nothing coordinates, so nothing is a single point of failure, and each team owns its piece. Azure’s saga guidance lists the price, steep for a core flow:

• The flow lives nowhere. “Where is order 81?” means reading five services’ logs, and there is no one place to see the stage of 3.6 million orders.
• Cycles. Inventory consumes payment events and payments consumes inventory events, so neither can change its events alone.
• Hard to change. Adding a fraud check before payment touches the publisher, the subscriber and every compensation path.
• Nobody owns time. Whose job is “cancel if unpaid in 10 minutes”? Each service ends up with its own sweeper.
• Hard to test. An integration test needs every service running.

You gain loose coupling, independent teams and no coordinator to run. You pay with a process that exists only as the sum of subscriptions. It breaks at about the fifth step, the first timer, or the first support call asking where an order is stuck. Say: “I choreograph side effects such as SMS, loyalty points and analytics, but the core order flow needs an owner.”

Level 3: an orchestrated saga

A saga, in Garcia-Molina and Salem’s terms, is a sequence of local transactions T1 … Tn, each with a compensating transaction C1 … Cn that semantically undoes it. Either all of T1 … Tn complete, or T1 … Tj run followed by Cj … C1. Now give it an owner: an orchestrator that keeps one state machine per order in its own database, sends commands (reserve, charge, accept), records each reply and decides what comes next, including compensation.

Three ideas carry most of the interview.

Compensation is a new action, not a rollback. A refund is a second entry on the customer’s bank statement; releasing stock is an increment that may race with a new order. Compensations can fail too, so they must be retryable and idempotent (Level 5).

The pivot. Azure splits a saga into compensable steps, one pivot (the point of no return) and retryable steps after it. Once the store accepts and starts packing, you can’t un-pack: rider assignment, delivery and settlement are then retried until they succeed. Put cheap, likely-to-fail holds first: if you charge before checking the store can fill the order and 10% of orders fail there, you refund 10% of payments.

There is no isolation. Other sagas see intermediate states. Azure names lost updates, dirty reads (a coupon service sees “paid” for an order about to be refunded) and fuzzy reads, with countermeasures: a semantic lock, such as an order in a PENDING state that others treat as not final; commutative updates, such as qty = qty - 1 rather than setting a value; a pessimistic view, moving risky updates into retryable steps (award loyalty points after delivery); and rereading values with a version check before writing (contention and double booking).

Hand-rolled, the orchestrator is a table and a loop:

CREATE TABLE order_saga (
  order_id    BIGINT PRIMARY KEY,
  state       TEXT   NOT NULL,   -- RESERVING, CHARGING, ACCEPTING, COMPENSATING, DONE, CANCELLED
  attempt     INT    NOT NULL DEFAULT 0,
  deadline_at TIMESTAMPTZ,       -- when to time out the current step
  version     INT    NOT NULL    -- every transition is a compare-and-set on this
);

Commands leave through an outbox, replies move the row with UPDATE … WHERE version = :v, and a sweeper retries or compensates rows whose deadline_at has passed. The client gets the order ID at once and polls or is pushed the outcome.

You gain one owner, a visible state per order, explicit compensation and a sensible step order. You pay for an orchestrator service, its database and the state machine code. It breaks when you notice what you are building: timers, backoff, crash recovery, a status UI, and a way to change the state machine mid-flight. Fine for one flow; by the fifth you have written a workflow engine, badly. Say: “An order saga with an orchestrator: reserve and charge are compensable, the store’s acceptance is the pivot, and rider assignment and settlement are retried until done.”

Level 4: durable execution with a workflow engine

A workflow engine is that orchestrator, built once and run by someone else. You write the flow as ordinary code (Temporal, Cadence) or as a state-machine definition (AWS Step Functions), and the engine provides what Level 3 hand-rolled (workflow engines):

• An event history per workflow. Temporal appends every fact to an append-only log: started, activity scheduled, activity completed with its result, timer fired, signal received.
• Replay instead of saved variables. If a worker dies, another loads the history and re-runs the workflow code from the top; calls that already completed return their recorded results instead of running again. So workflow code must be deterministic: given the same history it must issue the same commands. Random numbers, the local clock and network calls break replay; put side effects in activities and use the SDK’s time and random APIs.
• Activities with timeouts and retry policies. Temporal retries activities by default, starting at 1 second, doubling, capped at 100 seconds, with unlimited attempts; workflows don’t retry by default.
• Durable timers. A timer of a second or a year is persisted and fires even if workers were down, and a waiting workflow uses no worker resources. “Cancel if unpaid in 10 minutes” and “close the loan application after 7 silent days” are one line each.
• Signals, queries and updates: the bank’s callback arrives as a signal, support reads state with a query, and a reviewer’s approval can be an update.

Here is the order saga as a Temporal-style workflow in Python, simplified:

@workflow.defn
class OrderWorkflow:
    def __init__(self) -> None:
        self.paid = False

    @workflow.signal
    def upi_paid(self) -> None:            # the bank's callback, delivered as a signal
        self.paid = True

    @workflow.run
    async def run(self, order: Order) -> str:
        opts = dict(start_to_close_timeout=timedelta(seconds=10))
        undo = []
        try:
            await workflow.execute_activity(reserve_stock, order, **opts)
            undo.append(release_stock)
            await workflow.execute_activity(send_upi_collect, order, **opts)
            await workflow.wait_condition(lambda: self.paid, timeout=timedelta(minutes=10))
            undo.append(refund_upi)
            await workflow.execute_activity(store_accept, order, **opts)   # the pivot
        except Exception:
            for step in reversed(undo):
                await workflow.execute_activity(step, order, **opts)
            return "cancelled"
        await workflow.execute_activity(assign_rider, order, **opts)       # retried until done
        return "accepted"

The ten-minute wait costs nothing while it waits, a crash anywhere resumes at the same line, and compensation is a loop over the steps that actually ran.

Know the limits. Temporal caps one workflow’s history at 51,200 events or 50 MB, warning from 10,240 events or 10 MB. An order uses a few dozen events; the trap is an entity workflow, such as one per rider shift receiving a signal per order, which must periodically Continue-As-New: same workflow ID, new run, fresh history. Step Functions Standard caps history at 25,000 events and payloads at 256 KiB, so pass IDs, not documents.

Step Functions comes in two kinds. Standard workflows run for up to a year with exactly-once workflow execution, keep history for 90 days, support callbacks for human or partner steps, and are billed per state transition. Express workflows run for at most five minutes, at-least-once (asynchronous) or at-most-once (synchronous), are billed per request and duration, and lack the callback pattern. A 30-minute order with a UPI wait needs Standard; a 300 ms checkout validation fits Express.

You gain durable state, timers, retries and history without writing them, and flow code that reads like the business process. You pay for an engine to run or rent, deterministic code, history limits and a new programming model. It breaks because the engine retries activities: a payment that succeeded at the bank but timed out on the way back runs again. Say: “Each order is a Temporal workflow with the order ID as workflow ID; activities have 10-second timeouts and retries, the UPI wait is a 10-minute durable timer, and compensations run in reverse.”

Level 5: make every step safe to repeat

Engines run steps at least once. An activity can succeed at the bank and still be retried, because the worker died before reporting, the reply was lost, or the timeout fired first. Even Step Functions Standard, which promises exactly-once workflow execution, re-runs a task whose Retry you configured. So exactly-once effects = at-least-once execution + idempotent steps (idempotency).

• One idempotency key per step, derived from the workflow. The charge activity sends o81:charge, built from the order ID and step name, never a fresh UUID per attempt. The payments service stores key → result in the same transaction as the charge and answers a repeat with the stored result.
• Timeouts at every layer. A 10-second start-to-close timeout on each call to the payment API, a schedule-to-close budget for the whole step including retries (say 5 minutes), heartbeats for long activities (long-running jobs), and a business deadline on the workflow itself: refund if not delivered in 90 minutes.
• Retry the transient, fail the permanent. Timeouts, 5xx and 429 get backoff with jitter (timeouts, retries and backoff). “Insufficient funds”, “store closed” and “item discontinued” are non-retryable: go straight to compensation instead of retrying a decline a hundred times.
• Plan for the compensation that fails. A refund API down for an hour must not leave the customer unpaid. Compensations retry with backoff for hours; past a limit they raise an alert and land in a manual repair queue, and a daily reconciliation compares orders with the payments ledger (design a payment system).

You gain retries that are always safe, so the engine can be aggressive about recovering. You pay for a key table in every service with side effects, careful error classification, and a human queue for the rare compensation that can’t complete. It breaks at scale: 3.6 million workflows in flight, a code change while they run, and a bill that grows with every step. Say: “Every activity sends a key made of the order ID and step name, declines are non-retryable and trigger compensation, and failed refunds retry for hours, then go to a repair queue.”

Level 6: operate workflows at scale

Throughput and quotas. At 2,000 orders a second with about 10 state transitions each, the engine sees 20,000 transitions a second. Step Functions Standard’s default token buckets refill at 5,000 transitions a second in N. Virginia, Oregon and Ireland and 800 in other Regions such as Mumbai, StartExecution refills at 300 or 150 a second, and an account and Region may hold 1,000,000 open Standard executions; we need 3.6 million. So raise quotas, spread orders across accounts or cells, or end the workflow at rider assignment and track delivery separately. Self-hosted Temporal scales by adding servers and database capacity.

Cost. Step Functions Standard costs $0.000025 per transition in US East, retries included. 10 million orders a day is 300 million a month; × 10 transitions = 3 billion × $0.000025 = $75,000 a month. An Express sub-flow for a 2-second checkout check costs 300 million × $1 per million = $300 in requests plus at most 300 million × 2 s × 0.0625 GB × $0.00001667 ≈ $625 in duration: under $1,000. Self-hosted Temporal swaps per-transition fees for servers, a database and people (cost-aware design).

Versioning. A deploy lands while 3.6 million orders are mid-flight, and replaying an old history against reordered code fails with a non-determinism error. Temporal recommends Worker Versioning, which keeps each execution on the build it started with, or patching, a branch that old histories take one way and new ones the other.

Visibility. Make the workflow ID the order ID, so “where is order 81?” is one lookup, and index status, city and store as search attributes, so “orders stuck in ACCEPTING for 5 minutes in Pune” is a query and an alert.

People and partners. A human approval or a partner callback is a signal or update in Temporal, or a task token in Step Functions Standard, always paired with a timer that escalates or cancels.

Protect downstreams. Cap activity concurrency per dependency so retries can’t exceed the payment gateway’s rate limit, and give each dependency its own task queue, so a slow bank can’t starve rider assignment (bulkheads).

Keep choreography where it fits. The workflow emits OrderConfirmed and OrderDelivered through an outbox; SMS, loyalty points, analytics and search indexing subscribe. They need no compensation or deadline, so they stay out of the orchestrator.

You gain a platform that survives peaks, deploys and audits. You pay for quota planning, versioning discipline, dashboards and a real bill. Say: “About 20,000 transitions a second and 3.6 million open orders exceed Step Functions’ defaults and would cost about $75,000 a month, so I’d run Temporal, version workers, and keep notifications choreographed.”

Comparing the rungs

Which classic problems use it

Common mistakes

• Two-phase commit across microservices and a bank that will never join it.
• Holding a database transaction or an HTTP request open while a customer or a bank responds.
• A new idempotency key per attempt, which makes every retry a new charge.
• Compensations that assume they succeed and have no retry, alert or repair queue.
• The pivot in the wrong place: charging before checking the store can fill the order, so refunds become routine.
• Random numbers, clock reads or HTTP calls in workflow code.
• Choreographing the core flow, so nobody owns timeouts or knows where an order is.
• Deploying changed workflow code without versioning.
• One workflow per entity that never Continues-As-New until it hits the history limit.

Cheat sheet

• Chain availability = product of the steps’: 0.999⁶ ≈ 99.4%.
• Instances in flight = start rate × average lifetime: 2,000/s × 1,800 s = 3.6 million.
• Engine load = start rate × transitions per instance: 2,000 × 10 = 20,000 a second.
• Standard cost = instances × transitions × $0.000025 (US East).
• Exactly-once effect = at-least-once execution + an idempotency key per step.

How to say it in the interview

“Stock, payments, stores and riders are separate services, and UPI approval can take minutes, so neither one transaction nor a chain of calls works. I’d run each order as a saga in a workflow engine, with the order ID as the workflow ID. Reserve stock and charge are compensable; the store’s acceptance is the pivot; rider assignment and settlement are retried until done. The UPI wait is a 10-minute durable timer and the bank’s callback a signal. Every activity carries an idempotency key built from the order ID and step; declines are non-retryable and trigger compensation in reverse; failed refunds retry, then go to a repair queue. Notifications just subscribe to the workflow’s events.”

Why not two-phase commit? It blocks every participant while the coordinator is down, holds locks across a minutes-long UPI wait, and the bank and the store’s tablet can’t join it anyway. A saga trades isolation for availability, and I handle that with a PENDING state and version checks.

The payment call timed out. Did we charge? We don’t need to know. The retry sends the same key, o81:charge, and the payments service returns the stored result if the first call succeeded, or charges now if it didn’t.

The refund keeps failing. Now what? Compensations are retryable and idempotent, so they retry with backoff for hours. Past a limit they alert and go to a manual repair queue, and daily reconciliation catches anything else.

How do you change the workflow with 3.6 million orders in flight? Worker Versioning keeps each running order on the build it started with while new orders use the new one; for small changes, a patch branch. Orders finish within an hour, so the old build drains quickly.

Choreography or orchestration? Orchestration for the core flow, which has deadlines, compensation and a stage everyone asks about; choreography for side effects that only react to what happened.

Where to go next

• Distributed transactions: two-phase commit vs sagas: the theory behind this playbook.
• Workflow engines (Temporal): how durable execution works inside.
• Transactional outbox: commands and events that always match the database.
• Event sourcing and CQRS: the event-history idea applied to your own data.
• Event-driven architecture: choreography and its trade-offs.
• Long-running jobs: when the work is one long step, not many.
• Contention and double booking: holds and version checks for the steps that compete.
• Finding service boundaries with DDD: fewer cross-service flows start with better boundaries.
• Serverless architecture: Step Functions with Lambda.

References and further reading

1. Sagas (Hector Garcia-Molina and Kenneth Salem, Princeton tech report, published at SIGMOD 1987)cs.princeton.eduLong-lived transactions that run for hours or days, why holding their locks until commit hurts (deadlock frequency grows with the fourth power of transaction size), and the saga as a sequence of transactions with compensations.
2. Azure Architecture Center: Saga distributed transactions patternlearn.microsoft.comCompensable, pivot and retryable transactions; choreography versus orchestration trade-offs including cyclic dependencies; lost updates, dirty reads and fuzzy reads; semantic locks, commutative updates, pessimistic view and reread values.
3. Pattern: Saga (microservices.io, Chris Richardson)microservices.ioAn order created in a PENDING state, choreography- and orchestration-based examples, the lack of automatic rollback and isolation, and clients that poll or are notified of a saga’s outcome.
4. Temporal documentation: Workflow Execution limitsdocs.temporal.ioEvent History capped at 51,200 events or 50 MB, with warnings from 10,240 events or 10 MB, and no limit on the number of concurrent workflow executions.
5. Temporal documentation: Workflow Definitiondocs.temporal.ioDeterminism and replay, why random numbers and local clocks break it, non-determinism errors from code changes, and Worker Versioning (recommended) or patching.
6. Temporal documentation: Retry Policiesdocs.temporal.ioActivities retry by default (1 s initial interval, coefficient 2.0, maximum interval 100 s, unlimited attempts) while workflows don’t, plus non-retryable errors.
7. Temporal documentation: Timersdocs.temporal.ioPersisted timers from one second to years that fire after outages, with workers using no resources while they wait.
8. AWS Step Functions: Choosing workflow typedocs.aws.amazon.comStandard (up to one year, exactly-once, billed per state transition, 90-day history) versus Express (five minutes, at-least-once or at-most-once, no callback pattern), and idempotent starts by execution name.
9. AWS Step Functions: Service quotasdocs.aws.amazon.com25,000-event history, 256 KiB payloads, 1,000,000 open Standard executions per account and Region, and state-transition and StartExecution token buckets by Region.
10. AWS Step Functions pricingaws.amazon.com$0.000025 per Standard state transition in US East including retries, 4,000 free a month, and Express at $1.00 per million requests plus $0.00001667 per GB-second.