Hi,

Four weeks ago, I looked into Temporal.io as a platform for building and orchestration reliable workflows. I looked at it through the lens of distributed systems, combining it with concepts from Martin Kleppmann’s Designing Data-Intensive Applications (for reference, DDIA). In this post, I team up with Claude Code and Codex to rebuild these core architectural ideas using LangGraph and Postgres. LangGraph is built specifically for stateful, multi-agent systems. My goal here is to dig deeper into orchestration patterns and gain a clearer understanding of what it takes to run agentic systems in production. My long-term goal is exploring the fault-tolerant guarantees of Temporal with the agentic flexibility of LangGraph as a combination for modern enterprise workflows.

Here I use my TicketGraph project for the deep dive, which is a rewrite of my ticketflow project without Temporal.io. The project implements a simple ticket system where a mock agent resolves support tickets with a conditional human-in-the-loop step. I keep the structure as same as in my previous post, but I wanted to get a better understanding of the concepts.

Project introduction

A support ticket is not a good fit for a single request/response handler. The system has to call an unreliable agent, retry transient failures, wait for human approval, execute a refund, send a reply and answer status checks while the process is still running.

I picked LangGraph here as it seems the most used framework for workflows and I want to get my hands on it. The ticket flow is as follows: classify the ticket, draft a reply, wait for human approval, then either send the reply and execute the approved action or reject the proposed resolution. The flow also includes fallback queues in case an agent is not available.

The service is split into five components, where Postgres is used as workflow checkpointer, a hand-built task queue and the read model:

  • FastAPI app: Accepts ticket requests and starts, queries and updates workflows.
  • Workflow Runner: Leases runnable workflows, advances the graph, dispatches tasks at-least-once.
  • Agent worker: Polls the agent queue with a rate limit and the agent-fallback queue.
  • Fallback worker: Polls the agent-fallback queue.
  • Side-effect worker: sends replies, executes refund at-most-once. Upserts results by ticket ID.

Related to DDIA (Chapter 1: Reliable, Scalable and Maintainable Applications), we use Postgres as backbone for reliability. Reliability is not about avoiding faults, but also deciding which faults the system can tolerate. Those five components do not fail in the same way. If the API is down, no new tickets can be created and status checks fail, but running workflows continue. If a worker is down, work is waiting in Postgres and resumes when the worker returns. If the Agent worker is overloaded or unavailable, agent calls can wait, time out and move to a fallback queue. If Postgres goes down, the system has lost its durable coordinator. So, a dead worker is a fault the system can absorb. A dead Postgres DB is a system outage. Separating the LLM worker is not required for correctness, but it creates an independent failure domain for a rate limited dependency.

Durable execution: remembering where the ticket is

The first problem is not the agent call itself. The first problem is remembering where the ticket is when the process running it disappears. A ticket can be in one of several states:

  • waiting for an LLM
  • retrying after a transient failure
  • waiting for human approval
  • waiting for long-running execution
  • executing a refund
  • sending a final reply

If this process lives only in a Python request handler or an in-memory background task, a worker restart turns into a bug, where the ticket may be stuck, repeated, forgotten or be completed twice.

This needs durable execution, where the ticket progress is recorded independently of the currently running process. Here we need to display the workflow as a checkpointed state machine. The workflow code advances one state at a time and after every step LangGraph writes that state to Postgres. The runner leases a run, reloads the latest checkpoint, advances the graph by one step and persists the result. A crashed process leaves the last checkpoint untouched and a different runner reloads it and continues.

Another important point is, that waiting for human approval also looks like ordinary async code:

1
2
3
4
5
6

    async def prepare_approval(state: TicketState) -> TicketState:
        _ = state
        return {
            "status": TicketStatus.AWAITING_APPROVAL,
            "wakeup_at": await timestamp_after(APPROVAL_TIMEOUT),
        }

and

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

    async def await_approval(state: TicketState) -> TicketState:
        ticket = state.get("ticket")
        reply = state.get("draft")
        wakeup_at = state.get("wakeup_at")
        assert ticket is not None and reply is not None and wakeup_at is not None

        resume = interrupt(
            {
                "kind": "approval_required",
                "ticket_id": ticket.id,
                "wakeup_at": wakeup_at.isoformat(),
                "draft": reply.model_dump(mode="json"),
            }
        )
        if not isinstance(resume, dict):
            raise ValueError("approval resume payload must be an object")

        kind = resume.get("kind")
        if kind == "timeout":
            return {"status": TicketStatus.ESCALATED}
        if kind != "decision":
            raise ValueError("approval resume kind must be 'decision' or 'timeout'")

        decision = ApprovalDecision.model_validate(resume.get("decision"))
        if decision.approved:
            return {"decision": decision}
        return {"decision": decision, "status": TicketStatus.REJECTED}

This does not hold a worker thread for the full APPROVAL_TIMEOUT. prepare_approval records a durable timer and await_approval suspends the graph. The worker process can disappear in the meantime, but the workflow state still exists in the durable checkpoint and the runner only resumes the run once the timer is due or an approval signal arrives.

Approval reaches the workflow through a durable signal, written by the corresponding HTTP request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

@app.post("/tickets/{ticket_id}/approval")
async def submit_approval(
    request: Request, ticket_id: str, decision: ApprovalDecision
) -> dict[str, TicketStatus]:
    """Accept a human approval decision by writing a durable workflow signal."""
    signal_id = db.add_pending_signal_if_waiting(
        ticket_id,
        APPROVAL_DECISION_SIGNAL,
        decision.model_dump(mode="json"),
        waiting_status=TicketStatus.AWAITING_APPROVAL,
        pool=request.app.state.pool,
    )
    if signal_id is None:
        raise _approval_conflict()
    return {"status": TicketStatus.AWAITING_APPROVAL}

This is where the Postgres version diverges from Temporal. In Temporal, approval is a workflow update, where it runs inside the workflow, validates the decision and returns the resulting status synchronously. Here it is a durable signal row. The endpoint does not run workflow logic and does not know the resulting status. The actual transition happens later, when the runner consumes the signal and advances the graph.

This reframes the classic finishing race condition. A late approval could arrive while the workflow is executing a terminal side effect. In Temporal you often close this by writing the terminal status before the final activities, so the update validator rejects approvals that arrive while they run. This codebase closes it structurally instead, because the finalize worker actually writes the terminal status last:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

async def run_finalize(
    task: db.QueuedTask, activities: TicketActivities
) -> TicketResult:
    if task.task_type != "finalize_ticket":
        raise ValueError(f"unexpected side-effect task_type {task.task_type!r}")

    ticket = Ticket.model_validate(task.payload["ticket"])
    action = ProposedAction.model_validate(task.payload["action"])
    result = TicketResult.model_validate(task.payload["result"])

    refund_executed = False
    if result.status == TicketStatus.RESOLVED and action.type == ActionType.REFUND:
        assert action.refund_amount is not None
        await activities.execute_refund(
            ticket.id, action.refund_amount, attempt=task.attempts
        )
        refund_executed = await activities.refund_recorded(ticket.id)

    await activities.send_reply(ticket, result.reply_text, attempt=task.attempts)
    result = result.model_copy(update={"refund_executed": refund_executed})
    await activities.record_result(result)
    return result

The lesson from the Temporal version still holds, that races in a single thread are ordering problems and you fix them by placing the state transition right and not using a lock. Here, only the location moves. In Temporal it’s a status write in the event history. Here it’s two separate facts: workflow_run.status says whether approvals are still accepted and the refund/reply/result ledgers say which side effects have run. Finalize can fail and retry, so each question has to be answerable on its own.

This connects directly to DDIA (Chapter 7: Transactions). The run behaves like a small state machine whose every advance is a database transaction. When the runner resumes a run, one transaction advances workflow_run.status, releases the lease and marks the consumed signal. So a crash can never leave the status moved but the signal un-consumed or vice versa. The checkpoint with the task queue play the role of a write-ahead log. The important decisions and effects are recorded durably before the workflow is allowed to proceed, which is exactly what lets a new process pick the ticket up and finish it.

Timeouts, rate limits, backpressure and fallback routing

Another problem is deciding how long the workflow should wait for an event like an agent/LLM response. This sounds like one question, but in a distributed system it is several different questions.

Timeouts

  • Did the task wait too long before any worker picked it up?
  • Did the worker start the task but take too long to finish?
  • Did the worker stop making progress while the task was running?

Temporal gives each its own activity timeout with schedule_to_start, start_to_close and heartbeat_timeout. In this codebase it is handled differently, for example there is no heartbeat.

“Was the task ever picked up?” is the durable schedule-to-start timer. When the workflow dispatches an agent task, it uses a wake-up AGENT_SCHEDULE_TO_START_S:

1
2
3
4
5
6
7
8
9

async def enqueue_agent_task(
    *, task_type: str, workflow_id: str, payload: dict[str, Any]
) -> datetime:
    key = f"{workflow_id}:{task_type}"
    wakeup_at = await enqueue_task(
        queue_name=config.AGENT_TASK_QUEUE,
        ...
        wakeup_after=timedelta(seconds=config.AGENT_SCHEDULE_TO_START_S),
    )

If the timer fires and the task is still pending, the task gets rerouted to the the fallback queue. This is the only question that produces its own signal.

“Did a started attempt finish?” and “is the worker still alive?” are not handled as separate questions. Both are answered by the lease. Dequeue hands a worker a task with a fixed 30-second lease:

1
2
3
4
5
6

    UPDATE task_queue
       SET status = 'leased',
           lease_owner = %s,
           lease_expires_at = now() + interval '30 seconds',
           attempts = attempts + 1
     WHERE id = ( ... FOR UPDATE SKIP LOCKED LIMIT 1 )

There is no heartbeat and no lease renewal, since the lease is set once and never extended. Also, there is no separate wall-clock cap on the agent call itself. The 30-second lease is the only bound. If a worker does not complete within the lease, the reclaim_expired returns the task to pending and it is redelivered. A crashed worker and a slow worker are indistinguishable. When both let the lease lapse, the trigger reclaim and the second one means the same task can run twice. That is the cost of dropping heartbeats and it is why every terminal effect is idempotent rather than guarded by an execution timeout.

So the three timeouts are collapsed into two signals with fallback routing and a redelivery in case of an expired lease. The correctness is covered further downstream by idempotency instead of by timing.

Rate limits and backpressure

Those timeouts are implemented from the workflow’s point of view. But there is another issue: throughput. Even if every activity eventually succeeds, the system should not start unlimited LLM calls. This matters because the LLM provider has a global rate limit and each worker process also has limited local capacity.

  • How many LLM calls may the whole system start per second?
  • How many activities can this worker handle at the same time?

Temporal answers both with worker options of max_task_queue_activities_per_second and max_concurrent_activities. This codebase answers them with two constants on the agent worker:

1
2

AGENT_MAX_PER_SECOND = float(os.environ.get("AGENT_MAX_PER_SECOND", "10.0"))
AGENT_MAX_CONCURRENT = int(os.environ.get("AGENT_MAX_CONCURRENT", "20"))

AGENT_MAX_CONCURRENT bounds how many tasks one worker runs at a time. The worker keeps a set of in-flight tasks and only leases a new one while it has room:

1
2
3
4

while len(in_flight) < max_concurrent:
    if bucket is not None:
        await bucket.acquire()
    in_flight.add(asyncio.create_task(process_one_task(...)))

This protects the worker host. It also composes the way you’d expect, so three workers at max_concurrent=20 run up to 60 tasks at once.

Rate is a token bucket that spaces task starts at AGENT_MAX_PER_SECOND. This is meant to protect the shared LLM budget. Temporal’s per-second limit lives on the task queue, but this token bucket is an in-process object created per worker:

1

bucket = TokenBucket(max_per_second) if max_per_second is not None else None

Nothing coordinates it across processes. So the rate limit holds for a single primary agent worker, but it does not support horizontal scaling. A truly global cap would need a shared limiter (a Postgres-backed token bucket) or a single worker. This is one of the things Temporal hands you for free that a hand-built queue does not.

Backpressure falls out of the queue for free. It happens when tickets arrive faster than the agent workers can drain them and because the queue is just a Postgres table, the signal is literally a row count:

1

SELECT count(*) FROM task_queue WHERE queue_name = 'ticketflow-agent' AND status = 'pending';

If the agent worker stops, the tasks stay pending in task_queue. When a worker starts again it resumes polling with dequeue and the workflows continue. The task_queue table plays the same role as Temporal’s task queue. It is a durable buffer between tickets and the rate-limited LLM tier. A growing count of pending agent tasks is the backpressure signal and the schedule-to-start timer from the previous section is the automated reaction to it.

Fallback routing

The schedule_to_start timeout is also used as a routing signal. If an agent activity waits too long in the primary task queue, the workflow treats this as “the primary LLM worker cannot get to this task in time.” Instead of failing the whole ticket, it reruns the agent call on the fallback queue.

In Temporal this is a try/except around the activity that inspects the error. Here there is no exception to catch and the timeout is the durable wakeup_at timer firing, delivered to the parked workflow as a resume value. await_agent_task checks for it and redispatches:

1
2
3
4
5

resume = interrupt({...})           # park until result or timer
if _is_timeout(resume):
    redispatched = await redispatch_agent_task(...)
    resume = interrupt({...})       # park again, now on the fallback queue
return resume

Redispatch is a queue switch, not a retry of the same row. It cancels the still-pending primary task and enqueues a new one on the fallback queue under a distinct idempotency key:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

redispatched = await taskqueue.acancel_pending(conn, key, reason="redispatched to fallback")
if redispatched:
    await taskqueue.aenqueue(
        conn,
        queue_name=config.FALLBACK_TASK_QUEUE,
        task_type=task_type,
        workflow_id=workflow_id,
        payload=payload,
        idempotency_key=f"{key}:fallback",
    )

cancel_pending only succeeds if the task is still pending. If a primary worker already picked it up, the workflow keeps awaiting the primary and the reroute happens only when the task truly was never started. The {key}:fallback key keeps the fallback enqueue from colliding with the original.

The fallback tier is a cheaper, weaker model, so its answers come back with lower confidence. The approval rule escalates anything below CONFIDENCE_THRESHOLD = 0.75, so a low-confidence fallback answer lands in the human approval inbox on its own. The degradation is explicit and reviewable, not hidden.

There’s a retry question here too and it’s simpler than in Temporal. Retry and routing are separate mechanisms keyed on separate signals:

  • Transient failure (AgentOverloadedError) → the task queue retries with exponential backoff (attempts, available_at), declaratively, as data.
  • Schedule-to-start timeout (never picked up) → the graph redispatches to the fallback queue.
  • Exhausted or permanent failure → the graph escalates to a human (_is_task_failed → ESCALATED).

So retry stays in the queue and routing stays in the graph. The workflow owns only the routing decision and it stays small.

This connects to DDIA (Chapter 8, The Trouble with Distributed Systems and Chapter 11, Stream Processing). Waiting is not neutral: a task may be slow because a worker is overloaded, a queue is growing or a dependency is down. Here, schedule-to-start makes overload observable, the task_queue absorbs backpressure and the fallback queue turns a blocked primary tier into degraded service instead of a full failure. Postgres provides the durable queue; the application still decides what “too slow” means and what happens next.

Idempotency: exactly-once as a useful lie

The goal is to make failed tasks safe to retry without applying the same effect twice. In this project, the dangerous case is a crash during a refund. A naive retry could refund the customer twice.

The refund activity writes to Postgres and delivery is at-least-once. A worker leases the finalize_check task and if it executes the refund but crashes before acknowledging the task, the lease expires. The reclaim_expired returns the task to pending and it is redelivered. The second attempt reaches the same table, but the refund effect is keyed by ticket_id. The insert uses ON CONFLICT (ticket_id) DO NOTHING, so the duplicate is ignored and the customer is refunded once.

That is the practical version of exactly-once. The code may run more than once, but the external effect happens once. The task queue gives at-least-once execution and idempotency makes the effect effectively-once.

The activity does not ask, “did I run before?” Instead, the effect is modeled with an idempotency key. refund_attempts logs every attempt, while refunds records the business effect. Both are committed in one transaction, so the log of what was tried and the record of what happened do not disagree. The same two-table pattern backs the customer reply (reply_attempts / sent_replies) and the terminal result is an idempotent upsert into ticket_results. Every terminal effect is at-most-once, not just the refund.

This is DDIA’s point about retries and uncertainty. The caller cannot always know whether the callee did nothing, did the work and crashed or did the work and lost the reply. Idempotency does not remove that uncertainty. It makes it harmless.

Payloads outlive code: schema evolution in workflow histories

The next problem is that workflow data lives longer than the code that created it. A ticket can sit parked for a long time waiting for a human or pending in the queue under backpressure and a deploy can land in that window. When the ticket resumes, new code has to read a payload that old code wrote.

Temporal surfaces this through history replay. This system has no replay, but the same exposure lives in two durable places, the task queue (task_queue.payload) and the LangGraph checkpoint (the parked TicketState). Both are deserialized with Pydantic model_validate when a worker leases a task or the runner resumes a run and the agent and side-effect workers validate task.payload against the current models.

The failure mode is concrete. Add a new required field to Classification and the next worker that validates an old task.payload raises a ValidationError. The task can’t be processed, the run can’t advance and the ticket is stuck. No line of code is at fault, only a shape mismatch between old data and a new model.

The lesson is a versioning rule. Only add fields with defaults, never add required fields and don’t remove or rename existing ones. Each direction breaks a different compatibility. A new required field breaks backward compatibility and new code can’t read old payloads that lack it. Removing or renaming a field breaks forward compatibility and old code still draining the queue can’t read payloads that new code wrote.

The project follows the rule. The model-provenance field was added as model: str = “primary”, so payloads that predate it still deserialize cleanly. A regression test pins this down by validating old-shape payloads against the current models, so a future breaking change fails a test instead of silently failing a parked ticket.

This is DDIA (Chapter 4: Encoding and Evolution) in practice, where encoded data outlives the code that wrote it.

Derived data and two read paths

The system gets asked two different kinds of questions and they do not need the same storage:

  • Which tickets are waiting for approval right now?
  • What happened to ticket X?

The first is a search across many tickets. The API serves it from the workflow_run table, which carries a status column with a secondary index. The runner writes each state change into that column when it advances a run (save_run), so list_tickets answers “who is awaiting approval?” with one indexed query instead of opening every ticket’s state. This is the role Temporal’s visibility store and search attributes play.

The second question is historical and ticket-specific. The source of truth is the workflow’s own state in the LangGraph checkpoint. For a live ticket, get_ticket reads it directly with aget_state. When the ticket finishes, its outcome is materialized once into the Postgres ticket_results table and get_ticket falls back to that (readmodel.load_result) once retention has pruned the checkpoint with returning 404 only when neither store knows the ticket.

This is DDIA’s (Chapter 11: Stream Processing) materialized views and CQRS at toy scale. You don’t serve every read from the source of truth. You derive read-optimized projections from a log and route each question to the view built for it, while accepting that the views can lag and must be rebuildable from the log.

Where would this break in production

The project is useful as a lab, but it is not production-ready. A reliable system should be clear about the faults it handles and the faults it does not handle yet.

The first gap is heartbeats. The 30-second lease is the only liveness signal and it is never renewed. A real LLM call slower than the lease gets reclaimed and redelivered while still running. It is safe only because effects are idempotent. The fix is a heartbeat that extends the lease during the call.

The second gap is the fallback path. The primary call has a schedule-to-start budget, but the fallback await lacks a timer. If both agent workers are down, the ticket hangs forever. Production needs a deadline on the fallback to end in escalation.

The third gap is the rate limit scalability. The token bucket is per-process, so it holds only with a single primary worker. Three workers have 3× the intended rate. A real deployment needs a shared limiter.

The fourth gap is the missing atomicity of state and projection. The checkpoint, the outbox and the workflow_run projection commit in separate transactions. Correctness relies on idempotency keys and reconcile_orphaned_runs. This is a load-bearing reconciler and weaker than the atomic history with state Temporal gives you.

The fifth gap is deployment. A single Postgres in Docker Compose is fine for experiments, but has no backups, replication or failover.

There are also ordinary application gaps with refund_amount as a float and the approval endpoint without authentication. Those were out of scope for the exercise, but they would not be acceptable production defaults.

This connects back to DDIA (Chapter 1: Reliability) that states that reliability is not a vague promise that nothing will fail, but starts with knowing your fault assumptions. Which failures the system absorbs, which ones degrade service and which ones are still outages.

Conclusion

This project focuses on the system design aspect of AI engineering. I took Temporal as the benchmark and rebuilt its durable execution on Postgres, with DDIA for the concepts behind each primitive. Temporal showed what reliable orchestration should do and DDIA explained why. By rebuilding it on Postgres I had to get a better understanding about leases, idempotency keys, durable timers and backpressure. From my current point of view, this is really helpful, even with the help of LLMs.

I want to build more systems like this to develop a better understanding of system design in AI engineering, but also about the strength and weaknesses of different frameworks and tools. At the same time, it is important not to forget the ML part. LLM calls are probabilistic and non-deterministic. They are not just ordinary API calls, even if they often are wrapped to look like one.

That distinction matters. The engineering layer can make an AI system more reliable around the model with retries, timeouts, idempotency, queues, observability and human approval. But it does not make the model itself deterministic or correct. Good AI engineering has to hold both ideas at once: treat the surrounding system like a distributed system and treat the model output like an uncertain prediction.

Thank you for your attention.