Most teams start building agents the same way they build any other web feature: wrap the model in a route handler, parse the request, wait for the response.
This works until the agent does something interesting. Then the request times out, or the process restarts halfway through, or the model decides to call thirty tools instead of three, and nothing anywhere has a record of how far it got.
The handler is answering three questions at once, and they have very little to do with each other.
- Where should this work run? A queue answers this. It takes a unit of work, stores it durably, and hands it to a worker with capacity.
- What should happen next, given what already happened? A workflow engine answers this. It keeps the history of a run and picks the next step by reading it.
- What should I do next to accomplish the goal? Only the agent answers this, at runtime, and it can answer differently every time.
An HTTP handler answers the first question by accident, wherever the request happened to land. It answers the second by not asking it, since the only history is a call stack that dies with the process. The third it hands to the model. At demo scale that is fine, because all three collapse into one process with one lifetime. Production pulls them apart, and most of the difficulty in running agents lives in the seams.
What follows adds one layer at a time. Each layer answers a question the one below it could not, and each one creates the problem that forces the next.
The agent loop
Traditional workflow code knows its own shape before it runs:
Agent code does not:
The iteration count, the tools called, and the order they run in are all chosen at runtime by the model. A run might finish in three steps or thirty, fan out across several tools, wait for a human, produce a large intermediate artifact, or stop early.
The run therefore outlives the request that started it, and needs its own lifetime and its own identity. It also means you cannot reason about a run's failures in advance, because you do not know its shape in advance. You have to record what happened while it happens, or you will have nothing to work from when the process dies.
Move the run off the request
Separate the run from the request first. The handler creates a durable record, hands the work to something else, and returns.
A worker picks the job up, marks the run as running, executes it, and writes the outcome back. The client has a run ID immediately, and polls, subscribes, or waits for a callback.
That answers the first question. A queue is a durable buffer between the thing that creates work and the thing that performs it, and it owns the decision of where work runs: which worker, when, at what concurrency, and how many times to try again. The last of those has a price.
At-least-once delivery becomes your problem
Most production queues are at-least-once, which means a job may run more than once. That is a deliberate trade to avoid losing work, and it becomes part of your correctness model as soon as an agent can call tools, write records, send messages, or provision resources.
Here is a worker that crashes between two lines:
The fix is an idempotency boundary around the effect. Check whether it already happened, perform it, record that it happened.
Every side-effecting tool call an agent can make needs the same treatment, keyed on something stable that survives the retry:
Retrying is only a recovery strategy when the operation being retried is safe to repeat. Otherwise it is a way to cause the original problem a second time.
What a queue cannot tell you
A queue knows that a job exists. It does not know that the job is step three of a run, that step three depended on step two, that a human approval is pending, or that four sibling branches have to finish before anything can be synthesized.
One background job is fine. Twenty dependent steps with retries, two branches, and a pause for approval is a process, and the queue will not track it for you. Writing the code that does is how teams end up maintaining a homegrown workflow engine.
Give the run a memory
A workflow engine answers the second question. It stores the history of a run — which steps started, completed, failed, retried, timed out, or waited on input — and decides what happens next by reading that history. A crash stops the process without restarting the run.
Every workflow system splits your code the same way. A coordinator decides what happens next. Steps do the work: model calls, tool executions, writes. The names change between engines. The division does not: decisions in one layer, effects in the other.
The coordinator has to be deterministic
Recovery works by replay. After a crash the engine re-executes the coordinator from the top, substituting recorded results for the steps that already ran. That only lands the run where it left off if the coordinator makes the same decisions given the same history.
It is easy to break this by accident:
On the first pass the loop runs for a minute. On replay an hour later, Date.now() is already past deadline before the first iteration, so the coordinator skips every recorded step and reports the run as abandoned even though most of its work succeeded. The code and the history now disagree about what happened, and the engine cannot detect that they do.
Anything capable of answering differently on a second pass belongs in a step: model calls, the clock, random numbers, reads from external services. A step runs once, and every later replay reads its recorded result.
Durable execution tells you where a run was when it died. It does not give you exactly-once effects. A step that crashed after its side effect but before its result was written will run again, so the idempotency from the previous section is still load-bearing.
Compensation
The history a workflow keeps is also what makes it possible to undo a run.
Take an agent that charges a card, provisions a resource, and sends a confirmation email, then fails permanently on its fourth step. No database transaction rolls those three back. The money has already moved. Restarting from the beginning charges the card twice, which is the problem idempotency was supposed to prevent.
The standard answer is the saga pattern. For every side-effecting action, define a compensating action that reverses it: a charge by a refund, a provisioned resource by a deprovision, a sent email by a correction. When a run fails permanently, walk the completed steps backward and run their compensations.
Look at the query that drives it. Compensation is only possible because something recorded which steps completed and in what order, which is exactly what a workflow engine already stores. You can build this against a hand-rolled table instead, and then you are maintaining a step history without the engine that comes with one.
Compensations fail on their own too. A refund API goes down as easily as a charge API. Give the compensation chain a bounded retry, a dead-letter path for the ones that will not complete, and a way for a person to finish the job by hand. Without those, a failed run can end in a state nobody has a name for: some effects reversed, some not, and no record of which.
Compensation is also the wrong response to most partial failures. If four of five parallel sub-orders succeeded, unwinding the four is usually worse for everyone than dropping the fifth. It earns its complexity when partial success is genuinely unacceptable, like an approval granted on incomplete information or a charge with nothing delivered against it. For everything else, recording what succeeded and stopping is the better outcome.
Let the agent branch
The third question belongs to the agent, and once the answer can be "several things at once," a run stops being a line.
Fan-out is the common shape. A lead agent splits a goal into independent subtasks, dispatches them, and synthesizes what comes back. It buys parallelism and a fresh context window per branch, which suits research, document analysis, and other breadth-first work.
Dispatching the workers is the easy half, and the queue already does it. The join is the hard half. What happens when one branch fails, how many failures are tolerable, whether to synthesize partial results, how long to wait before giving up: none of those are questions about where work runs, so none of them are questions a queue can answer.
At scale the two layers stack. The workflow coordinates the process and the queue distributes the execution.
Promise.allSettled keeps one failed branch from poisoning the join, and plan.minResults puts the partial-failure policy in writing. Leaving that threshold out does not remove the policy. It makes the policy whatever the first unhandled exception happens to do.
Fan-out works when subtasks are independent and mostly read-only. It degrades when they are tightly coupled or share state that keeps changing, since the branches then need to see each other's writes and the parallelism was never real. Every additional agent also costs tokens and coordination, so deciding how many to run matters more than the mechanics of spawning them.
Contention
Fan-out has a failure mode that does not appear at small widths. A system that runs fine with a handful of parallel subagents can fall over once the fan-out grows, with provider capacity to spare, because the branches are not independent in the ways that matter. They share a rate limit, a connection pool, and a retry schedule. When the shared resource pushes back, every branch backs off together and then retries together.
Treat providers, databases, and model APIs as shared resources the whole run has to fit inside. Cap concurrency below what the provider allows rather than at it, put a token bucket in front of anything with a published limit, and jitter the backoff so the branches stop synchronizing.
What a platform can take over
Everything above is platform-agnostic. You can assemble it from any queue, any workflow engine, and any worker fleet. What is left to decide is how much of it you want to operate.
On Render, the first two layers collapse into one dependency. Your web service is the API. Render Workflows is the queue, the workers, the retries, the isolation, and the per-run history. You define tasks as TypeScript or Python functions. There is no queue to provision, no worker fleet to keep warm, and no orchestrator cluster to run.
Here is the loop from the top of this post, written as tasks. The agent books travel, so a decision it makes at runtime moves real money.
Those task() wrappers are the entire execution plane. When runTrip calls decide(...) or bookSegment(...), that is not a local function call. Each one starts a chained task run on its own on-demand instance, with its own retry policy and compute profile, and Promise.all is the fan-out. Three wrappers replace the API, queue, and worker plumbing from earlier in this post. There is no message schema, no polling loop, and no worker registration to write.
One agent run is also one tree of task runs. runTrip is the root and every decide, bookSegment, and cancelBookings chains off it, with status, retries, and logs recorded per run, so "what did the agent do" is a question the dashboard can answer. If your application needs its own record, store the root run ID and work from there.
What the platform does not supply is any of the correctness work still visible in that file. The idempotency key on bookSegment is there because the retry policy directly above it guarantees the task will sometimes run twice. cancelBookings exists because Workflows will tell you a run failed and will not tell you that three flights are still booked under the customer's name. Both are the patterns from earlier in this post, and both are still application code.
Triggering a run
Starting a run, from your web service or anywhere else, is one SDK call:
startTask is the enqueue. The platform stores the work durably, dispatches it, and exposes its status, which is the whole boundary from the first pattern in one line. runTask blocks until completion instead, though at an API request boundary startTask is the right fit.
Other primitives compose in when a run needs them. A Cron Job calling startTask gives you scheduled agents, and Key Value is where the token bucket from the contention section lives when parallel runs share a rate limit.
What Workflows does not do
Workflows removes queue and worker plumbing. It does not remove idempotency, compensation, artifact storage, trace propagation, approval boundaries, cost controls, or rate limiting.
That boundary is moving. Stronger durability guarantees, first-class idempotency, and native approval gates are on the roadmap. The patterns in this post will not change when those ship, but more of them will become configuration instead of code you maintain.
Workflows are in Beta, with current limitations around supported languages, scheduling, Blueprint support, and HIPAA-compatible hosts.
What stays yours
Pulling the three questions apart does not answer any of them. It puts each one somewhere you can see it, which is a different and smaller claim.
Where state lives, which effects are safe to repeat, how much partial success is enough to synthesize, when a person has to approve, how a run is traced from the API call through the last tool execution: those decisions are about your application, and a platform can make them cheap to implement without making them for you.
What you get for the trouble is a run you can ask questions about later. When something goes wrong at 3am, "what did this agent do, and how far did it get" has an answer, because one of the layers was writing it down while it happened.
Render Workflows are in Beta. If you want to run these patterns without building the infrastructure behind them, start with the Workflows docs.