Workflow engines (Temporal)

A workflow engine runs a business process that spans many services and lasts minutes to months (take payment, reserve stock, wait two days for the seller, ship) as one piece of code that survives crashes, deploys and outages. Temporal, the open-source successor to Uber’s Cadence, is the reference design: it records every step in an event history and rebuilds a crashed workflow by replaying it. In an interview it answers “this process has many steps, takes hours, and must never get stuck halfway”, provided you can say what it guarantees and what it does not.

Picture it first: the wedding planner’s file

A planner is running the Sharma family’s wedding in Delhi. Her checklist: book the hall, book the caterer, pay the advance, book the band, confirm everything three days before. Whenever something happens she adds a line to the family’s file: “10:40, hall booked, receipt H-17”.

In week two she falls ill. A colleague reads the file from the first line and follows the same checklist. Hall and caterer are booked, so she calls neither; the band has no line, so she starts there. One entry reads “called caterer to pay the advance” with no “paid” after it, so she calls again quoting the booking reference, and the caterer does not take the money twice. “Call the decorator back in 3 days” is in the file, so it survives the handover. If the band pulls out for good, she unwinds in reverse: caterer, then hall.

Map it back: the checklist is the workflow definition, the Sharma wedding is a workflow execution with workflow ID sharma-wedding, the file is the event history, each call is an activity, the planner is a worker, reading from line one is replay, the booking reference is an idempotency key, “call back in 3 days” is a timer, a family call saying “add 50 guests” is a signal, and the unwinding is compensation.

Why not a status column and a cron job?

Take a Swiggy-style food order. The customer pays; the restaurant must accept within 5 minutes or the order is refunded; riders get 30 seconds each to accept; then pickup, delivery and a rating request an hour later. Six services, a dozen ways to fail.

The hand-rolled version: a status column, a cron job that sweeps for stuck rows, a retry table, a queue per step with a dead-letter queue, and timeouts coded in three services. It works until:

• A crash lands between two writes. The service sets status = PAID and dies before enqueuing “notify restaurant”. Nothing moves until the sweeper notices, if it checks that state at all: the dual-write problem.
• Two sweeper instances grab the same row, and two riders get the same offer.
• Nobody can answer “where is order 42, and why?” The answer is spread over rows, queue offsets and logs.

Durable execution turns this inside out. You write the whole process as one function; the engine records the outcome of every step, timer and message in a durable log, and if the process dies it rebuilds the function’s state from that log and continues. The general pattern is in multi-step workflows; this page is about the engine.

Data model: workflows, activities and the history

Workflow definition and execution

A workflow definition is a deterministic function written with a Temporal SDK (Go, Java, Python, TypeScript, .NET and others). A workflow execution is one run of it, for one order or one user, identified by a workflow ID you choose, usually a business key such as order-42, plus a generated run ID, inside a namespace (the unit of isolation and retention). At most one execution per workflow ID is open at a time.

Activities

An activity is any code with side effects: call a payment gateway, write to PostgreSQL, send an SMS, call an LLM. A worker runs it and its result goes into the history. A failed activity is retried under a retry policy that by default waits 1 second, doubles up to 100 seconds, and never gives up. Four timeouts bound it:

• Start-To-Close: one attempt. The only way the service notices a worker that died mid-activity, so always set it.
• Schedule-To-Close: the whole activity, retries included; bound retrying with this, not an attempt count.
• Schedule-To-Start: time waiting in the queue; infinite by default.
• Heartbeat: a long activity pings with progress; a missed heartbeat fails the attempt early, and the retry resumes from the recorded progress.

Without Start-To-Close or Schedule-To-Close the service rejects the activity.

Task queues and workers

Workers are your own processes. Each long-polls a named task queue for workflow tasks (run the workflow code until it blocks) and activity tasks (run one activity). Workers ask only when they have capacity, so an overloaded fleet builds a backlog instead of falling over.

Messages, timers and long lives

• Signals: asynchronous writes into a running workflow (“the seller packed the parcel”), with no result.
• Queries: reads of workflow state that never touch the history.
• Updates: synchronous writes that return a result or a validation error; GA since server 1.26.2 (December 2024).
• Timers: durable sleeps from a second to years. A waiting workflow holds no worker resources.
• Child workflows: separately tracked executions, for fan-out such as 50 transcodes.
• Continue-as-new: close the run and start a fresh one with the same workflow ID, an empty history and the state you pass in, so a workflow can live for years.
• Schedules: start workflows on a calendar or interval, replacing cron.

An order workflow in the Python SDK, with a retry policy, durable waits and compensation:

import asyncio
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError

with workflow.unsafe.imports_passed_through():
    from shop.activities import (Order, reserve_stock, release_stock, charge_card,
                                 refund_card, create_shipment, send_rating_sms)

STEP = dict(
    start_to_close_timeout=timedelta(seconds=30),        # notices a dead worker
    retry_policy=RetryPolicy(
        initial_interval=timedelta(seconds=1), backoff_coefficient=2.0,
        maximum_interval=timedelta(minutes=2),
        non_retryable_error_types=["CardDeclined", "OutOfStock"]),
)

@workflow.defn
class OrderWorkflow:
    def __init__(self) -> None:
        self.packed = False

    @workflow.signal
    def seller_packed(self) -> None:                       # sent by the seller app
        self.packed = True

    @workflow.run
    async def run(self, order: Order) -> str:
        undo = []                                          # compensations, newest last
        try:
            await workflow.execute_activity(reserve_stock, order, **STEP)
            undo.append(release_stock)
            await workflow.execute_activity(charge_card, order, **STEP)
            undo.append(refund_card)
            # durable timer: the seller has 48 hours to pack
            await workflow.wait_condition(lambda: self.packed, timeout=timedelta(hours=48))
            await workflow.execute_activity(create_shipment, order, **STEP)
        except (ActivityError, asyncio.TimeoutError):
            for step in reversed(undo):
                await workflow.execute_activity(step, order, **STEP)
            return "cancelled"
        await workflow.sleep(timedelta(days=7))           # survives crashes and deploys
        await workflow.execute_activity(send_rating_sms, order, **STEP)
        return "completed"

Using the business key as the ID makes the start idempotent: a retried checkout attaches to the running order.

from temporalio.common import WorkflowIDConflictPolicy

await client.start_workflow(OrderWorkflow.run, order, id=f"order-{order.id}",
                            task_queue="orders",
                            id_conflict_policy=WorkflowIDConflictPolicy.USE_EXISTING)

The event history

Every execution has an event history, an append-only, durably stored log: WorkflowExecutionStarted; WorkflowTaskScheduled, Started and Completed each time a worker advances the code; ActivityTaskScheduled, ActivityTaskStarted, ActivityTaskCompleted (or Failed, TimedOut); TimerStarted, TimerFired; WorkflowExecutionSignaled; WorkflowExecutionCompleted. While an activity retries, only ActivityTaskScheduled is recorded (event history). It is source of truth, audit log and debugger in one.

Durable execution: replay and determinism

When a worker runs a workflow task, the SDK turns each call in your code into a command (ScheduleActivityTask, StartTimer, CompleteWorkflowExecution); the service records them as events and dispatches the work. Each new event, such as a finished activity or a signal, triggers another workflow task. Workers cache executions between tasks, so normally nothing is replayed.

Now a worker crashes. The next workflow task goes to another worker with the full history, and that worker runs the function from the first line. Each command is matched against the event in the same position: reserve_stock matches event 5, and the result recorded in event 7 is returned at once, without calling the stock service. At the first step the history has never seen, the worker issues a new command and the workflow is live again. It is event sourcing for a running program: the history is the state, and local variables are a projection rebuilt by replay.

The determinism rule

Replay works only if the same code, given the same history, issues the same commands in the same order. So workflow code must never:

• read the clock: use workflow.now(), the time recorded for the current workflow task;
• draw random numbers or UUIDs: use workflow.random() and workflow.uuid4();
• do I/O (HTTP, databases, files): put it in an activity;
• start threads or rely on global mutable state.

The Python SDK’s sandbox blocks many of these, and the SDKs ship a replayer for testing new code against recorded histories.

Changing code while workflows are running

If an order was sleeping on a timer and new code calls an activity first, replay finds ScheduleActivityTask where the history says TimerStarted, and the workflow stalls with a non-determinism error. Changing durations or arguments is safe; adding, removing or reordering steps is not. Two tools:

• Patching: if workflow.patched("fraud-check"): takes the new path in new executions and the old path when replaying old ones; delete it once old executions finish.
• Worker Versioning (GA in March 2026, server 1.29.1 or later), now the docs’ default recommendation: Pinned workflows finish on the worker build they started on; Auto-Upgrade workflows move to the newest build and still need patches. Upgrade on continue-as-new (public preview) lets a long-lived workflow switch builds at each new run.

Guarantees, precisely

A workflow execution runs to completion despite failures. Worker crashes, deploys and even a Temporal Service outage only pause it, provided the persistence store survives; per the docs, a workflow fails only when your code throws.

At most one open execution per workflow ID. Starting order-42 while it runs fails with “already started” (the default conflict policy, Fail), returns the running execution (Use Existing) or replaces it (Terminate Existing). For closed executions a reuse policy (Allow Duplicate by default, Allow Duplicate Failed Only, Reject Duplicate) decides, within the retention period. So “start order 42” is idempotent.

Activities run at least once. A worker can finish the work and die before reporting it; the service cannot tell, so when Start-To-Close expires it retries, perhaps while the first attempt still runs behind a network split. Every external effect needs an idempotency key, stable across attempts and enforced by the service called. The docs suggest run ID plus activity ID; a business key such as order-42:charge also survives rerunning the whole workflow.

@activity.defn
async def charge_card(order: Order) -> str:
    info = activity.info()                       # same workflow_id on every attempt
    return await gateway.charge(order.card, order.amount_paise,
                                idempotency_key=f"{info.workflow_id}:charge")

Replay never repeats a completed step; it returns the recorded result.

Timers fire after their duration, never before, and late if workers are down or backlogged. Treat durations as minimums.

Messages are durable. An accepted signal or update is an event in the history, and history order never changes once written. Queries are not recorded and need a live worker.

Each step is one transaction inside the service. New events, mutable state and the internal task that dispatches the next step are written together in the workflow’s shard: a transactional outbox, built in.

How it scales

Four services and a database

The Temporal Service splits into four services that scale independently:

• Frontend: a stateless gateway that authenticates, rate-limits and routes each call by its workflow ID.
• History: owns workflow state, spread over a fixed number of history shards by hashing namespace and workflow ID. A shard holds its executions’ histories, mutable state, timers and internal task queues, one persistence operation at a time.
• Matching: hosts the task queues workers poll; 4 partitions per queue by default.
• Worker service: runs Temporal’s own system workflows, such as schedules.

Underneath sit the persistence store (Cassandra, PostgreSQL or MySQL) and a visibility store for searching executions (Elasticsearch or SQL). The docs cite a deployment of 5 Frontend, 15 History, 17 Matching and 3 Worker service hosts.

1. The order service starts order-42.
2. The Frontend routes it to the History host owning that ID’s shard.
3. History appends events and state in one write; visibility updates later.
4. History adds a workflow task to the orders queue.
5. A worker long-polls (through the Frontend), gets the task and runs the code until it blocks.
6. It returns commands or results via the Frontend, and the loop repeats.

History shards: a one-way door

You choose numHistoryShards before the database is created, and it never changes: more shards later means a new cluster and a migration. Clusters run with anything from 1 to 128K shards; start with about one History process per 500 shards and add hosts as load grows. It is database sharding with a final shard count, so size it for years ahead.

Workers are yours, and the database is the ceiling

Workers scale horizontally and are your capacity problem (Temporal Cloud never runs them): when they fall behind, schedule-to-start latency climbs. The service’s ceiling is its database, since every state transition is a write. A rough model for a Flipkart-style sale at 2,000 orders/s, each with 4 activities, 1 timer and 1 signal:

• Events: 1 start + 7 workflow tasks × 3 + 4 activities × 3 + 2 for the timer + 1 signal + 1 completion = 38 events per order, so 2,000 × 38 ≈ 76,000 events/s, written in batches of a few events per workflow task.
• Temporal Cloud bills actions (starts, activities, timers, signals; replay is free): 1 + 4 + 1 + 1 = 7 per order, so 2,000 × 7 = 14,000 actions/s, 28 times a namespace’s default floor of 500. On-demand capacity follows the previous 7 days of use, so provision before the sale.

Lineage. Maxim Fateev led Amazon’s Simple Workflow Service and Samar Abbas built Azure’s Durable Task Framework; at Uber they built Cadence (open-sourced 2017), then founded Temporal in October 2019 as an MIT-licensed fork (their story). Uber’s 2023 Cadence v1.0 post reported 12 billion executions a month across over 1,000 services.

Limits and numbers

From the Temporal docs as of October 2026 (server 1.32). Per-execution limits apply everywhere; self-hosted clusters can tune most through dynamic config, Temporal Cloud cannot.

A derived number worth carrying: a loop running one activity per iteration adds about 6 events each time (3 for the activity, 3 for the workflow task after it), so it hits the warning after about 10,240 / 6 ≈ 1,700 iterations and termination after about 51,200 / 6 ≈ 8,500.

A saga, worked through

Book a Goa trip: hold a flight, book a hotel, charge the card. If the card is declined, the flight and hotel must be released. That is a saga (Garcia-Molina and Salem, 1987): steps T1 … Tn, each with a compensating step Ci, so that either all Ti complete or T1 … Tj complete followed by Cj … C1 (see two-phase commit vs sagas). In Temporal compensations are ordinary code, like the undo list above, and durable: if a worker dies mid-compensation, the next one carries on.

What the engine does not fix:

• Compensations fail too. Retry them generously, then alert a human; a signal resumes the workflow after a manual fix.
• Compensation is not rollback. A refund is a new transaction the customer sees, and an SMS cannot be unsent: put hard-to-undo steps last.
• There is no isolation. Others can see the held room; where that matters, hold with an expiry (contention and double booking) and confirm at the end.
• Some steps have no undo. After the point of no return (ticket issued), retry forward instead.

When to pick it, and when not

Step Functions bills Standard workflows per state transition and caps their history at 25,000 events; Express per execution and duration. Airflow is built for scheduled batch workflows: right for a nightly ETL, wrong for a per-order flow.

Pick a workflow engine when a process spans services, waits on time or people, must never stop halfway, and someone will ask “where is order 42, and why?”.

Pick something else when:

• It is one step, or fire-and-forget: a message queue with retries and a dead-letter queue is enough.
• It is a high-volume event stream (clicks, GPS pings): use Kafka and a stream processor, not dozens of database writes per event.
• It is on a latency-critical path: every step is a round trip to the service and its database.
• Team independence beats the end-to-end view: choreograph with events and an outbox (event-driven architecture), accepting that tracing a stuck order means joining logs.
• Nobody can operate it: use a managed service, or keep the state machine small.

Common interview usage

Going deeper: failure modes

Non-determinism after a deploy

A reordered step or a datetime.now() in workflow code passes every unit test, then stalls thousands of running executions after the deploy. Replay production histories in CI, ship changes behind patches or pinned Worker Versioning, and ramp them like any deployment; reset can rewind a stuck execution to an earlier workflow task once the code is fixed.

Histories that never stop growing

Polling loops, subscriptions and chat sessions add events forever; a loop is terminated after roughly 8,500 iterations. Continue-as-new every few hundred iterations, or when the SDK says it is suggested.

Big payloads

Arguments and results live in the history: 2 MB per payload, 50 MB per history. A PDF or a long LLM transcript passed between activities fails or bloats every replay; store it in blob storage and pass the key.

Duplicate side effects

A non-idempotent activity double-charges after a worker crash. Key every external write, and give each activity one external effect, so a retry repeats only the failed step.

Timeouts that are missing or wrong

A 2-hour Start-To-Close with no heartbeat on a transcode means a crashed worker is noticed after 2 hours; heartbeat every few seconds with a 30-second heartbeat timeout. Too short a Start-To-Close kills healthy slow attempts and piles retries on the downstream, a small retry storm.

Workers that can’t keep up

Too few workers show up as rising schedule-to-start latency, not errors. Autoscale on it, and give slow or fragile activities their own rate-limited task queue (backpressure).

Hot workflows

One execution lives on one shard and advances one workflow task at a time, so thousands of signals per second into one “flash-sale counter” workflow will not keep up, and the 10,000-signal cap ends it anyway. Keep one workflow per entity; put counters in Redis or the database.

The database as the ceiling

When persistence latency rises, every workflow slows. Watch persistence_latency, scale the database and History hosts together, and remember the shard count is final. And don’t start a workflow per click: at 100,000 clicks/s that is 100,000 actions/s before any activity runs. Streams carry events; workflows run processes.

Cheat sheet

How to say it in the interview

“The order lifecycle is a Temporal workflow with ID order-<id>, so a retried checkout can’t start it twice. It calls activities to reserve stock, charge and ship, each with a 30-second Start-To-Close and exponential retries, and waits on a durable 48-hour timer for the seller. Every step lands in the event history, so if a worker dies another replays it without repeating finished steps. Activities are at-least-once, so the charge carries order-<id>:charge as the gateway’s idempotency key. If a step fails for good, compensations run in reverse: refund, then release stock.”

Why not a status column, a cron job and a queue? That is the same state machine built by hand, spread over rows, queues and retry tables: a crash between two writes strands an order. The engine owns state, timers and retries in one place, at the cost of operating it and the determinism rule.

A worker crashes in the middle of charging the card. What happens? The history has no completion, so when Start-To-Close expires Temporal retries on another worker. The card may already be charged; the idempotency key makes the gateway return the first result, and the workflow records one completed charge.

How do you change workflow code while two million orders are in flight? Running executions replay against new code, so added or reordered steps cause non-determinism errors. Pin workflows with Worker Versioning so old orders finish on the old build, or wrap the change in a patch, and replay production histories in CI first.

Can a workflow run for a year? Yes: there is no time limit. History size is the limit (51,200 events or 50 MB), so a subscription workflow continues-as-new each billing cycle, which also moves it onto newer code.

Temporal or Step Functions? Step Functions is a managed JSON state machine on AWS, capped at one year and 25,000 events, billed per state transition: good AWS glue at modest volume. Temporal keeps the flow in ordinary code with no duration limit: better for complex, long-lived processes, at the cost of running it or paying for Temporal Cloud.

Where to go next

• Multi-step workflows and long-running jobs: the patterns a workflow engine implements.
• Distributed transactions: two-phase commit vs sagas: when compensation beats locking.
• Idempotency: making at-least-once steps safe.
• Timeouts, retries and backoff with jitter: the policies behind activity retries.
• Transactional outbox and event sourcing and CQRS: two ideas Temporal builds in.
• Kafka and ZooKeeper and etcd: the alternatives for streams and coordination.
• Design a payment system, design food delivery and design a job scheduler: where workflows show up in full designs.

References and further reading

1. Temporal documentation: Workflow Definitiondocs.temporal.ioDeterministic constraints, which API calls produce Commands, how replay matches Commands to Events, non-determinism errors, and patching versus Worker Versioning.
2. Temporal documentation: Detecting Activity failuresdocs.temporal.ioStart-To-Close, Schedule-To-Close and Schedule-To-Start timeouts with their defaults, heartbeats, heartbeat throttling and why the server cannot see a dead worker.
3. Temporal documentation: Retry Policiesdocs.temporal.ioThe default activity retry policy (1 s, ×2.0, 100 × initial, unlimited attempts), non-retryable errors, and why workflows do not retry by default.
4. Temporal documentation: Activity Definitiondocs.temporal.ioWhy activities must be idempotent, the crash-before-report case, and Run ID plus Activity ID as an idempotency key enforced by the called service.
5. Temporal documentation: Temporal Serverdocs.temporal.ioFrontend, History, Matching and Worker services, history shards fixed for the life of a cluster, 1 to 128K shards, one History process per 500 shards, retention defaults.
6. Temporal Cloud documentation: System limitsdocs.temporal.ioEvent history (51,200 events or 50 MB), payload (2 MB) and gRPC (4 MB) limits, 2,000 pending operations, signal and update caps, 500 actions/s default, 30-day retention.
7. Temporal documentation: Workflow Id and Run Iddocs.temporal.ioAt most one open execution per Workflow Id in a namespace, and the Workflow Id reuse and conflict policies with their defaults.
8. Cadence: The Only Workflow Platform You’ll Ever Need (Uber Engineering, 2023)uber.comCadence v1.0, workflows as code instead of DSLs, and its scale at Uber: over 1,000 services, 12 billion executions and 270 billion actions a month.
9. AWS Step Functions Developer Guide: Choosing workflow typedocs.aws.amazon.comStandard (one year, exactly-once, priced per state transition) versus Express (five minutes, at-least-once or at-most-once, priced per execution and duration).
10. Sagas (Garcia-Molina and Salem, SIGMOD 1987)cs.cornell.eduThe original definition: a long-lived transaction as a sequence of transactions, each with a compensating transaction run to amend a partial execution.