# AI Customer Support Copilot | 30 Days of Building AI Systems, Day 3

> Build an AI customer support copilot that triages tickets, retrieves KB articles, and drafts cited replies for human agents to review, edit, and send.

Source: https://codeitronics.com/blogs/ai-systems-day-3-support-copilot (published 2026-11-21)

Yesterday we built document chat over private files. Today we put that retrieval to work inside a support team: an AI customer support copilot that reads each ticket, finds the right knowledge base articles, and drafts a reply that a human agent reviews, edits, and sends.

![AI Customer Support Copilot — CodeITronics infographic](https://pub-206c6a53642c4bdb86310f0393204592.r2.dev/blogs/series/30-days-of-building-ai-systems/48-ai-systems-day-3-support-copilot.png)

*Save this sheet. The rest of this note walks through it section by section.*

## Table of Contents

* [What We Are Building](#what-we-are-building)
* [System Architecture](#system-architecture)
* [Key Components](#key-components)
* [Data Flow Step by Step](#data-flow-step-by-step)
* [A Minimal Tech Stack](#a-minimal-tech-stack)
* [Example Input and Output](#example-input-and-output)
* [Production Concerns](#production-concerns)
* [Extensions Worth Adding](#extensions-worth-adding)
* [Key Takeaways](#key-takeaways)
* [FAQs](#faqs)
* [Up Next](#up-next)

## What We Are Building

The copilot works for the support agent, not instead of them. It understands the incoming ticket, searches the knowledge base, suggests a response with references, and learns from what agents actually send. The agent stays in control of every word the customer sees.

That boundary is deliberate. Later in this series we build an autonomous support agent that resolves tickets on its own, and a customer-facing chatbot. The copilot comes first because it's the safest place to start: mistakes get caught by a person before they reach a customer, and every edit the agent makes becomes training signal.

It fits anywhere tickets repeat: e-commerce order and refund questions, SaaS billing and feature issues, technical helpdesks, IT and employee support, and telecom.

## System Architecture

```
ticket (chat / email / portal)
  -> ingestion (API or webhook)
  -> ticket understanding: intent, entities, urgency
  -> knowledge retrieval (RAG over KB articles + resolved tickets)
  -> suggested response with references
  -> agent review: edit, approve, send
  -> customer reply + feedback captured
  -> interaction stored for learning
```

The loop at the end is what makes it a copilot rather than a template engine. Every suggestion has an outcome: sent as-is, edited, or discarded. Store all three.

## Key Components

| Component        | Job                                                                         |
| ---------------- | --------------------------------------------------------------------------- |
| Ticket ingestion | Receive tickets from your helpdesk by webhook or API                        |
| Intent detection | Classify category, urgency, and extract entities like order IDs             |
| Vector DB        | Index KB articles and past resolved tickets (Pinecone, Chroma)              |
| LLM              | Draft the reply grounded in retrieved articles                              |
| Knowledge base   | The source of truth, in Confluence, Notion, or a docs site                  |
| Agent interface  | Shows the draft, its sources, and edit controls inside the agent's workflow |
| Feedback loop    | Captures edits, ratings, and outcomes                                       |

The agent interface decides whether this gets used. If agents have to switch tabs to see the suggestion, adoption drops. Put it in the helpdesk sidebar they already live in.

## Data Flow Step by Step

1. A customer creates a ticket through chat, email, or a portal.
2. The webhook delivers it to the copilot service.
3. The LLM extracts structured fields: intent, category, urgency, and entities.
4. Retrieval searches KB articles and similar resolved tickets, using the extracted category as a filter.
5. The LLM drafts a reply that cites the articles it used.
6. The agent reviews, edits, and sends from their normal interface.
7. The system records the final text and any rating, then stores the interaction.

Step 3 is worth making strict, because everything downstream filters and routes on it. Ask for JSON and validate it:

```python
from pydantic import BaseModel
from typing import Literal

class TicketInsight(BaseModel):
    intent: str
    category: Literal["account", "billing", "order", "technical", "other"]
    urgency: Literal["low", "medium", "high"]
    entities: dict[str, str]          # e.g. {"order_id": "A-1042"}
    needs_human_judgment: bool        # refunds, legal, angry customers

insight = TicketInsight.model_validate_json(llm_json(TRIAGE_PROMPT, ticket_text))
```

And step 7 is where learning comes from. A simple, honest signal is how much the agent changed the draft:

```python
import difflib

def edit_ratio(suggested: str, sent: str) -> float:
    return 1 - difflib.SequenceMatcher(None, suggested, sent).ratio()
```

A ratio near zero means the agent sent the suggestion almost untouched. Track it by category and you quickly see where the copilot helps and where it gets in the way.

## A Minimal Tech Stack

| Layer          | Choice                                                                     |
| -------------- | -------------------------------------------------------------------------- |
| Agent UI       | React or Next.js sidebar app inside the helpdesk, or Streamlit for a pilot |
| Orchestration  | LangChain or LlamaIndex, or plain code                                     |
| LLM            | A current OpenAI GPT or Anthropic Claude model                             |
| Embeddings     | OpenAI text-embedding-3                                                    |
| Vector DB      | Pinecone or Chroma                                                         |
| Knowledge base | Confluence, Notion, or your docs site, synced on a schedule                |
| Deployment     | Vercel, Render, or AWS                                                     |

## Example Input and Output

The sheet's ticket: subject "Unable to reset password", body "I tried to reset my password but didn't receive the email. It's been 15 minutes." The copilot tags it category Account, priority Medium, channel Email.

The draft it hands the agent thanks the customer, asks them to check spam, gives three steps (request the reset again, confirm the email address on file, reply if it still doesn't arrive so the team can reset it manually), and lists two references: the "Reset Password Guide" and "Troubleshooting Email Issues" articles.

The references are for the agent, not decoration. They let the agent confirm the steps in seconds, and if an article is wrong or outdated, the agent's edit tells you which article to fix.

## Production Concerns

### Evaluation

Before launch, replay a few hundred historical tickets and compare drafts against what your best agents actually sent. Have senior agents grade a sample for accuracy, tone, and policy compliance. After launch, watch edit ratio, discard rate, and handle time by category. If a category shows heavy edits, look at its KB articles before you touch the prompt.

### Cost

Every ticket costs one triage call, one retrieval, and one draft. Run triage on a small model. Generate drafts on demand when the agent opens the ticket rather than for every ticket on arrival, so tickets closed by macros or merged as duplicates cost nothing.

### Failure Modes

* **Outdated KB articles.** The copilot will confidently quote a policy you changed last quarter. Sync the KB on a schedule and show article dates in the sidebar.
* **Rubber-stamping.** Agents under time pressure stop reading drafts. Flag high-risk categories (refunds, cancellations, legal threats) for mandatory review, and never pre-fill them.
* **Leaking internal notes.** Retrieval over past tickets can surface internal comments. Index only public replies, or strip internal notes during ingestion.
* **Wrong customer context.** Retrieving another customer's resolved ticket is fine for patterns but dangerous for specifics. Redact names, emails, and account details in the indexed history.

## Extensions Worth Adding

The sheet lists auto-resolving repetitive tickets, multi-language support, sentiment detection, voice support through speech-to-text, and proactive assistance. Sentiment detection is the quickest win: route frustrated customers to senior agents. Auto-resolution is a different system with different risks, and it's where Day 13 picks up.

## Key Takeaways

* A copilot keeps a human between the model and the customer. That makes it the safest first support AI to ship.
* Structured triage (intent, category, urgency, entities) drives retrieval filters and routing, so validate it.
* Show sources next to every draft so agents can verify quickly and flag bad articles.
* Edit ratio per category is a cheap, honest signal of where the copilot helps.

## FAQs

### Should the copilot learn from agent edits automatically?

Use edits to find weak KB articles and to pick good few-shot examples first. Fine-tuning on edits can come later, once you have enough clean, reviewed examples.

### How is this different from helpdesk macros?

Macros are fixed templates the agent chooses. The copilot picks the relevant knowledge, adapts the reply to the specific ticket, and shows its sources.

### Where do we start if our knowledge base is thin?

Index resolved tickets with good replies alongside the articles you have. Then use the copilot's misses to decide which articles to write next.

## Up Next

Tomorrow is Day 4, the AI Meeting Notes System: a post-meeting pipeline that takes a recording, transcribes it, and turns it into a summary, decisions, and action items with owners. If your support team wants a copilot grounded in your own knowledge base, [let's talk](/contact).
