System Type: Workflow Orchestration

Patterns, failure modes, and anti-patterns for long-running workflow and process orchestration systems.

Core Patterns

Orchestration vs Choreography

Orchestration. A central coordinator owns the workflow definition and explicitly directs each step. The coordinator calls services, waits for results, handles failures, and decides what happens next. The entire flow is visible in one place. When to use. Complex multi-step processes (5+ steps). Flows with conditional branching, retries, timeouts, or compensation logic. When you need a single place to understand what the process does. When you need to answer "where is this order in the pipeline?" without querying every service. When to avoid. Simple event reactions (service A publishes, service B reacts). When the coordinator becomes a bottleneck or single point of failure you can't tolerate. When teams owning individual steps need to evolve independently without touching the orchestrator.

Choreography. Each service listens for events and decides what to do next. No central coordinator. The process is emergent from the event chain. When to use. Loosely coupled domains where services genuinely don't need to know about each other. Simple fan-out patterns (order placed → send email, update analytics, reserve inventory). When the number of steps is small (≤3) and failure handling is straightforward. When to avoid. When you catch yourself drawing the event flow on a whiteboard and it takes more than 60 seconds to explain. When debugging requires tracing events across 6 services to figure out why an order is stuck. When compensation logic is non-trivial — choreographed compensation is a nightmare to reason about.

The honest tradeoff. Choreography feels elegant and decoupled until something goes wrong. Orchestration feels heavy and centralized until you need to debug a production incident at 2am. Most teams that start with choreography for complex flows end up building an ad-hoc orchestrator anyway — they just don't call it that, and it's worse than a purpose-built one.

State Machines

What it is. Model workflow as a finite set of states with explicit transitions between them. Each transition has a trigger (event, condition, timeout) and may have side effects. When to use. Workflows with well-defined states and transitions: order processing, approval flows, document lifecycle, account provisioning. When you need to enforce that certain transitions are illegal (e.g., can't ship before payment). When auditors ask "what are all the possible states this entity can be in?" When to avoid. Workflows where the number of states explodes combinatorially (parallel execution with many branches). Purely sequential pipelines where a state machine adds ceremony without value. Key discipline. Every state machine needs an explicit terminal state and a timeout for every non-terminal state. A state machine without timeouts is a state machine that accumulates orphaned instances forever.

DAG-Based Workflows

What it is. Model workflow as a directed acyclic graph where nodes are tasks and edges are dependencies. Tasks with no unmet dependencies can execute in parallel. When to use. Data pipelines, build systems, ETL jobs, ML training pipelines — anywhere the work is a dependency graph with parallelism opportunities. When tasks are largely independent and the critical path optimization matters. When to avoid. Workflows with cycles (approval → rejection → resubmission). Workflows where the graph shape depends on runtime results (conditional branching makes DAGs awkward). Long-running workflows with human interaction — DAG engines are typically designed for batch, not for processes that pause for days. Key systems. Airflow, Prefect, Dagster, Argo Workflows. Each has different opinions about dynamic DAGs, task isolation, and scheduling.

Step Functions / Serverless Workflows

…