Yesterday we evaluated multi-agent systems end to end, from task success to handoffs to per-agent cost. Day 29 takes everything from this series into production: the architecture, the readiness checklist, and the failure handling, security, and cost controls that keep agents dependable under real traffic.

Save this sheet. The rest of this note walks through it section by section.
What changes in production
A prototype serves one developer making a handful of requests. Production brings high traffic and concurrent users, uptime expectations, latency targets, real costs, security and compliance requirements, the need to monitor everything, data consistency, and failures you have to recover from automatically.
Agents add their own twist. Each request can trigger many model and tool calls, so load multiplies in ways a normal web service does not. Your model provider's rate limits and every third-party API's quotas become part of your capacity plan. The sheet puts it well: great agents are built, production systems are engineered.
The production architecture
Users / applications
-> API gateway and load balancer
-> Agent orchestrator (planner / router)
-> Agent workers (execute tasks)
-> Tools and services (internal / external)
Shared layer: vector DB, database, cache (Redis), message queue (Kafka / SQS),
object storage (S3 / GCS), secrets manager, config service
Platform: CI/CD, containers (Docker / Kubernetes), infrastructure as code, autoscaling
Observability layer: logs, metrics, traces, alerts, dashboards
This is the deployment architecture from Day 26, hardened. The new pieces are the ones that absorb load and failure: a load balancer in front, a queue between the orchestrator and workers, a cache in front of expensive calls, and autoscaling driven by queue depth rather than CPU.
Scaling strategies
| Strategy | What it does |
|---|---|
| Horizontal scaling | Add more agent workers and services |
| Async execution | Use queues for long-running tasks |
| Caching | Cache prompts, embeddings, and responses |
| Load balancing | Distribute traffic evenly |
| Stateless design | Keep agents stateless wherever possible |
| Microservices | Split capabilities into independent services |
| Cost optimization | Right-size models, cache, and batch calls |
Two notes from experience. Scale workers on queue depth and age of the oldest message, because agent workers spend most of their time waiting on model calls, so CPU looks idle while users wait. And cache carefully: embeddings and tool results for stable data cache well; full responses only cache well for repeated, non-personal questions.
Concurrency with backpressure in code
The sheet's example runs agent tasks concurrently with asyncio and a thread pool. For I/O-bound work like model and API calls, plain asyncio with a concurrency limit and retries is usually simpler and safer:
import asyncio, random
MAX_CONCURRENCY = 20 # stay under provider rate limits
sem = asyncio.Semaphore(MAX_CONCURRENCY)
async def with_retry(fn, *args, attempts=4, base=0.5):
for attempt in range(attempts):
try:
return await asyncio.wait_for(fn(*args), timeout=30)
except (asyncio.TimeoutError, TransientError):
if attempt == attempts - 1:
raise # let the caller dead-letter it
await asyncio.sleep(base * 2 ** attempt + random.uniform(0, base))
async def run_agent(task):
async with sem:
return await with_retry(agent_step, task)
async def main(tasks):
return await asyncio.gather(*(run_agent(t) for t in tasks), return_exceptions=True)
The semaphore is backpressure: it stops a traffic spike from turning into a storm of rate-limit errors. Exponential backoff with jitter keeps retries from synchronizing. return_exceptions=True keeps one failure from cancelling the batch, so failed tasks can go to a dead letter queue for inspection.
State and observability under load
The data and state guidance on the sheet: short-term memory in session, long-term memory in a vector store, summarize and prune regularly; ACID databases for critical data, event sourcing where you need an audit trail, careful handling of concurrent updates; versioned knowledge sources, incremental updates, and freshness monitoring. The data flow runs ingestion, processing, storage, retrieval, and use by the agent.
For monitoring, the four pillars are logs (what happened), metrics (how the system is performing), traces (why it happened), and alerts (what needs attention). The key signals to watch: request volume, p95 and p99 latency, error rate, token usage, cost per request, queue depth, tool success rate, agent success rate, and user feedback. Cost per request and agent success rate are the two that standard infrastructure dashboards miss.
Design for failure
| Technique | Why it matters for agents |
|---|---|
| Retries with exponential backoff | Model and API calls fail transiently |
| Circuit breakers | Stop hammering a failing tool or provider |
| Timeouts on every external call | One hung call should not stall a whole run |
| Fallback models and responses | Degrade to a smaller model or a safe canned answer |
| Dead letter queues | Keep failed tasks for inspection and replay |
| Graceful degradation | Return a partial result rather than nothing |
| Health checks and self-healing | Replace unhealthy workers automatically |
| Disaster recovery and backups | Recover state, memory, and indexes |
Combine these with idempotent operations from Day 26, and a retry becomes safe instead of dangerous.
Security and cost
Security is a foundation, not a feature. The sheet lists authentication and authorization for users, services, and APIs, input sanitization and output filtering, secrets management with tools like Vault or AWS Secrets Manager, encryption in transit and at rest, audit logs and access tracking, compliance with frameworks like GDPR, SOC 2, or HIPAA where they apply, and data minimization and retention policies.
Cost drivers are model usage, token count, tool calls, data transfer, storage, and infrastructure. The levers: use the right model for each task, cache prompts, embeddings, and responses, batch where possible, compress or summarize context, monitor token usage, and set budgets, alerts, and quotas per tenant or feature. Small savings per request add up at scale.
The readiness checklist
Before launch, confirm: SLAs and SLOs defined, error handling and retries, idempotent operations, input validation and guardrails, auth and access control, secrets and API key management, rate limiting and throttling, data privacy and compliance, automated tests and evaluations, a rollback and disaster recovery plan, and cost monitoring and budgets.
Then the deployment gate: architecture and security reviewed, performance and load tested, failover tested, monitoring, logs, and traces verified, cost optimized, backups configured, runbooks prepared, access controls set, data policies defined, documentation ready, stakeholders informed. Check twice, deploy once.
Key Takeaways
- Each agent request fans out into many calls, so provider rate limits are part of capacity planning.
- Put a queue between orchestrator and workers, and autoscale on queue depth, not CPU.
- Use bounded concurrency, timeouts, and backoff with jitter on every external call.
- Design for failure with circuit breakers, fallbacks, dead letter queues, and idempotency.
- Track cost per request and agent success rate alongside standard infrastructure metrics.
FAQs
What usually breaks first when an agent scales?
Often rate limits and latency from model providers, followed by cost. Load test with realistic multi-step requests, not single calls.
Do I need Kubernetes to scale agents?
No. Managed container platforms and serverless queues work well for many teams. Kubernetes helps when you need fine control over many services.
How do I keep costs predictable?
Set per-request and per-tenant budgets, route simple tasks to smaller models, cache stable results, and alert on cost per request.
Up Next
Day 30 closes the series with The Future of AI Agents: a look back at the full journey, the trends shaping what comes next, and what builders should focus on. If you are preparing an agent for real traffic, let's talk.




Letters to the editor
No letters yet. Questions and counterpoints are welcome.