AI Agent Safety & Guardrails | 30 Days of AI Agents, Day 22

Yesterday was about evaluating agents: test sets, metrics, judges, and the loop that keeps quality from drifting. Evaluation tells you how an agent behaves. Day 22 is about constraining that behavior: the guardrails that keep an agent inside safe boundaries even when its input is hostile or its reasoning goes sideways.

Save this sheet. The rest of this note walks through it section by section.

What agent safety means

Agent safety means the agent behaves as intended, avoids harm, follows your rules, and operates within boundaries you defined. In practice that covers five goals: prevent harmful outputs, reduce hallucinations, protect users and their data, stay compliant, and earn trust.

Agents raise the stakes compared to chatbots because they act. A chatbot that hallucinates gives a wrong answer. An agent that hallucinates can issue a refund, delete a record, or email a customer. So the question is not only "what will it say?" but "what is it allowed to do?" The sheet's phrasing is a good design rule: safety first, intelligence second, trust always.

How guardrails wrap the agent

The flow on the sheet places a checkpoint at each stage of a request:

User input
  -> Input guardrails    (validate, filter, sanitize)
  -> Agent reasoning     (plan the next steps)
  -> Action guardrails   (check tools, API calls, permissions)
  -> Generate output
  -> Output guardrails   (safety, quality, policy)
  -> Safe response
Across every step: monitoring and logging, plus human oversight for risky cases

The key idea is defense in depth. No single layer is reliable on its own. Prompt injection detection misses novel attacks, so the action layer must still block tools the agent should never call. Output checks miss subtle errors, so high-impact actions still go to a human. Each layer assumes the one before it can fail.

The five types of guardrails

Type	What it checks	Examples
Input	What comes in	Prompt injection, jailbreaks, PII detection, profanity, relevance
Action	What the agent does	Tool and API allowlists, permission checks, rate limits, data access control, cost limits
Output	What goes out	Toxicity, bias, hallucination checks, quality and formatting rules
Context	What the agent sees and remembers	Data filtering, memory limits, sensitive data handling, retention policy
Workflow	How the run progresses	Step validation, loop protection, failure handling, escalation rules

If you only build one layer well, make it the action layer. Input and output checks are probabilistic: they classify text and sometimes get it wrong. Action guardrails can be deterministic. An allowlist either contains a tool or it does not. A permission check either passes or it does not. Code that runs outside the model cannot be talked out of its rules.

Worked example with a travel booking agent

The sheet walks a booking agent through each layer:

Stage	What it checks	Example
Input guardrails	Malicious prompts or sensitive data	Block "Ignore previous instructions..."
Action guardrails	Allowed tools and permissions	Allow the Flight API, block the Delete Booking API
Agent reasoning	Plans stay within scope	Search flights, check availability, calculate price
Output guardrails	Safety and accuracy	No PII leakage, no hallucinated prices
Monitoring	Errors and anomalies	Alert on high error rates or unusual tool usage
Human oversight	Uncertain or high-risk cases	Refund requests over $300 go to a human

Notice that "no hallucinated prices" is enforceable in code: every price in the response must match a price returned by a tool in this run. Wherever you can turn a policy into a comparison against tool output, do it. That is far more reliable than asking a second model whether the answer looks right.

A small guardrail layer in code

The sheet uses the Guardrails AI library. Library APIs change, so here is the same idea in plain Python you can adapt to any framework.

import re

class GuardrailViolation(Exception):
    pass

INJECTION_PATTERNS = [r"ignore (all |any )?previous instructions", r"reveal .*system prompt"]
ALLOWED_TOOLS = {"search_flights", "get_booking", "create_refund"}
REFUND_LIMIT = 300  # above this, a human approves

def check_input(text):
    for pattern in INJECTION_PATTERNS:
        if re.search(pattern, text, re.IGNORECASE):
            raise GuardrailViolation("possible prompt injection")

def check_action(tool, args, user):
    if tool not in ALLOWED_TOOLS:
        raise GuardrailViolation(f"tool not allowed: {tool}")
    if "booking_id" in args and not user.owns(args["booking_id"]):
        raise GuardrailViolation("booking belongs to another user")
    if tool == "create_refund" and args["amount"] > REFUND_LIMIT:
        return "escalate_to_human"
    return "allow"

def check_output(text, prices_from_tools):
    for price in re.findall(r"\$\d+(?:\.\d{2})?", text):
        if price not in prices_from_tools:
            raise GuardrailViolation(f"unverified price: {price}")

The regex input check is deliberately simple and will miss things; treat it as a cheap first filter and pair it with a classifier. The action and output checks are the ones doing real work, because they do not depend on the model's judgment.

Risks worth designing against

The sheet lists the usual threats: prompt injection and jailbreaks, data leakage and PII exposure, hallucinations, toxic or biased outputs, unauthorized actions and tool misuse, over-reliance on agent decisions, infinite loops and runaway agents, insecure integrations, and cost abuse.

Two get underestimated. Indirect prompt injection arrives through content the agent reads, such as a web page, an email, or a document, not through the user's message. Anything retrieved should be treated as untrusted data. Runaway loops are a cost and safety problem at once: cap steps, tool calls, and spend per run, and fail closed when a cap is hit.

The best practices follow from this: defense in depth, least-privilege access for tools and data, human review for critical decisions, regular red-team testing, design for failure with timeouts and safe defaults, and honesty with users about what the agent cannot do.

Tools and the safety checklist

The sheet lists Guardrails AI, NVIDIA NeMo Guardrails, Meta's Llama Guard, Azure AI Content Safety, the OpenAI Moderation API, and LangChain integrations. Use them for classification problems like toxicity, harmful content, and PII. Keep authorization and business rules in your own code.

Then work the checklist: define safety goals, identify risks, implement input, action, and output guardrails, set up monitoring, define escalation rules, test with red-team prompts, review false positives, update policies, and document decisions. The challenges are real too. Overly strict guardrails block legitimate users, and threats evolve, so review blocked requests regularly.

Key Takeaways

• Agents act, so safety is about what they are allowed to do, not only what they say.
• Layer input, action, output, context, and workflow guardrails; assume each layer can fail.
• Put your strongest, deterministic checks on actions: allowlists, ownership checks, limits.
• Verify facts like prices against tool outputs in code instead of asking another model.
• Cap steps and spend per run, and route high-impact actions to a human.

FAQs

Is a strong system prompt enough to keep an agent safe?

No. Instructions in a prompt can be overridden by injected text. Enforce critical rules in code that runs outside the model.

Where should human approval sit?

On irreversible or high-value actions such as refunds, deletions, and external messages, and on cases where the agent's confidence is low.

How do I avoid guardrails that annoy legitimate users?

Log every block, review false positives weekly, and tune thresholds against real traffic rather than guesses.

Up Next

Day 23 covers Advanced RAG Techniques: hybrid search, query rewriting, reranking, contextual compression, and agentic retrieval that plans and verifies its own lookups. If you want guardrails designed into an agent your team is building, talk to us.