# AI Document Chat System | 30 Days of Building AI Systems, Day 2

> Build an AI document chat system: ingestion, chunking, embeddings, filtered retrieval, and answers with page-level citations, plus evaluation and failure modes.

Source: https://codeitronics.com/blogs/ai-systems-day-2-document-chat (published 2026-11-20)

Yesterday we built a research assistant that retrieves from the open web. Today we point the same retrieval core at your own files: an AI document chat system where users upload PDFs, DOCX, and text files, ask questions in plain language, and get answers that cite the exact document and page.

![AI Document Chat System — CodeITronics infographic](https://pub-206c6a53642c4bdb86310f0393204592.r2.dev/blogs/series/30-days-of-building-ai-systems/47-ai-systems-day-2-document-chat.png)

*Save this sheet. The rest of this note walks through it section by section.*

## Table of Contents

* [What We Are Building](#what-we-are-building)
* [System Architecture](#system-architecture)
* [Key Components](#key-components)
* [Data Flow Step by Step](#data-flow-step-by-step)
* [A Minimal Tech Stack](#a-minimal-tech-stack)
* [Example Input and Output](#example-input-and-output)
* [Production Concerns](#production-concerns)
* [Extensions Worth Adding](#extensions-worth-adding)
* [Key Takeaways](#key-takeaways)
* [FAQs](#faqs)
* [Up Next](#up-next)

## What We Are Building

A document chat system accepts multiple uploads, extracts and understands their content, retrieves the passages relevant to each question, and answers with citations. It also remembers the conversation, so "what about the refund section?" works as a follow-up.

The use cases on the sheet tell you who pays for this: legal teams reading contracts, students working through papers and textbooks, companies with internal knowledge bases and SOPs, developers navigating technical documentation, and HR teams answering policy questions from handbooks.

The difference from Day 1 is that the corpus is private and persistent. That shifts the hard problems from source quality to ingestion quality, access control, and conversation handling.

## System Architecture

The pipeline on the sheet has two halves that run at different times.

```
INGEST (once per upload)
  documents -> text extraction -> chunking -> embeddings -> vector DB

QUERY (every message)
  question + chat history -> standalone query -> retrieve top-K
    -> LLM answer with citations -> update chat history
```

Keeping these separate matters operationally. Ingestion is slow, retryable, and belongs in a background job. Queries must be fast, so they should never wait on parsing a 200-page PDF.

## Key Components

| Component        | Job                                                                                  |
| ---------------- | ------------------------------------------------------------------------------------ |
| Document loaders | Read PDF, DOCX, and TXT into text plus page metadata (PyPDF, Docx2txt, Unstructured) |
| Text splitters   | Cut text into chunks that keep sentences and headings intact                         |
| Embeddings       | Turn chunks and questions into vectors                                               |
| Vector DB        | Store chunks with metadata and search them (Pinecone, Chroma)                        |
| Retriever        | Fetch the top-K chunks, filtered to documents the user is allowed to see             |
| LLM              | Write the answer from retrieved chunks, with citations                               |
| Memory           | Hold chat history so follow-ups make sense                                           |

The quiet hero here is metadata. Every chunk should carry its document ID, file name, page number, section heading, and owner. Citations, filtering, deletion, and access control all depend on it.

## Data Flow Step by Step

1. **Upload.** The user uploads PDF, TXT, or DOCX files. Store the originals in object storage and record a document row in your database.
2. **Extract and process.** Pull text page by page so page numbers survive. Flag pages that come back empty, since they are usually scans that need OCR.
3. **Chunk and embed.** Split by structure first (headings, paragraphs), then by size. Embed each chunk.
4. **Store.** Write vectors plus metadata to the vector DB, namespaced by user or workspace.
5. **Ask.** The user sends a question.
6. **Retrieve.** Rewrite the question into a standalone query using recent history, then retrieve the most relevant chunks, filtered to the user's documents.
7. **Answer.** The LLM answers from those chunks with file-and-page citations, and the exchange is appended to chat history.

Step 6 hides the most common bug in document chat. A follow-up like "and the second one?" embeds terribly on its own, so rewrite it before retrieving:

```python
def standalone_query(history, question):
    if not history:
        return question
    recent = "\n".join(f"{m['role']}: {m['content']}" for m in history[-6:])
    prompt = (
        "Rewrite the final question so it makes sense without the conversation. "
        "Return only the rewritten question.\n\n"
        f"Conversation:\n{recent}\n\nFinal question: {question}"
    )
    return llm(prompt, model=SMALL_MODEL)

def retrieve(user_id, query, k=6):
    return index.query(
        vector=embed(query),
        top_k=k,
        filter={"owner_id": user_id},   # never search other users' files
        include_metadata=True,
    )
```

The `llm`, `embed`, and `index` helpers are thin wrappers around your provider and vector DB clients. The filter is the line that matters most: retrieval without an ownership filter is a data leak waiting for its first multi-tenant customer.

## A Minimal Tech Stack

| Layer         | Choice                                                 |
| ------------- | ------------------------------------------------------ |
| Frontend      | Streamlit for a prototype, Next.js for a product       |
| Orchestration | LangChain for loaders and splitters, or plain code     |
| LLM           | A current OpenAI GPT or Anthropic Claude model         |
| Embeddings    | OpenAI text-embedding-3                                |
| Vector DB     | Pinecone or Chroma                                     |
| Loaders       | PyPDF, Docx2txt, Unstructured                          |
| Deployment    | Render, Railway, or Vercel, with ingestion on a worker |

## Example Input and Output

On the sheet, a user uploads four files (a research PDF, a company policy PDF, a product guide DOCX, and a notes file) and asks: "What are the key benefits mentioned in the product guide?"

The answer lists the benefits (easy to use, highly scalable, secure by design, 24/7 customer support, seamless integrations) and cites Product\_Guide.docx pages 3 and 5, plus Company\_Policy.pdf page 2.

Two details are worth copying. First, the answer cites pages, not just files, which lets the reader verify in seconds. Second, the user named "the product guide" but the answer still drew on the policy document. That's fine if the policy genuinely supports a claim, but when a user names a document, consider boosting or filtering retrieval toward it so answers stay on the source they asked about.

## Production Concerns

### Evaluation

Write 30 to 50 question and answer pairs against a fixed document set, including questions whose answer isn't in the documents. Measure retrieval separately from generation: did the right chunk land in the top-K, and given the right chunk, was the answer faithful? Those two numbers tell you whether to fix chunking or prompting.

### Cost

Ingestion cost scales with document volume and is paid once. Query cost scales with chunk count per answer and history length. Trim history to recent turns, keep top-K modest, and use a small model for query rewriting.

### Failure Modes

* **Scanned PDFs.** Text extraction returns nothing. Detect it and route those pages through OCR.
* **Tables.** Naive splitting shreds them. Extract tables separately or keep them whole as single chunks.
* **Confident non-answers.** The answer isn't in the documents, but the model answers anyway from general knowledge. Instruct it to say so, and include those cases in your eval set.
* **Stale vectors.** A user deletes or replaces a file, but its chunks stay searchable. Delete by document ID on every change.
* **Cross-tenant leakage.** Covered above, and it's the one failure you can't apologize your way out of.

## Extensions Worth Adding

The sheet lists folder-level chat, OCR for images and tables, advanced metadata filtering, multi-user chat and sharing, and exporting a conversation to PDF or Markdown. Metadata filtering is the one to build first, because it improves both relevance and security.

## Key Takeaways

* Document chat is two pipelines: a slow ingestion job and a fast query path. Keep them separate.
* Metadata on every chunk (document, page, section, owner) powers citations, filtering, deletion, and access control.
* Rewrite follow-up questions into standalone queries before retrieving.
* Evaluate retrieval and answer faithfulness separately, and include questions the documents can't answer.

## FAQs

### How big should my chunks be?

Start with a few hundred tokens and a small overlap, split on headings and paragraphs first. Then tune against your eval set, since dense legal text and loose meeting notes behave differently.

### Do I still need RAG if the model has a huge context window?

For one short document, you can often put the whole thing in context. For many documents, many users, or frequent questions, retrieval is cheaper, faster, and gives you precise citations.

### How do I handle DOCX files without page numbers?

DOCX has no fixed pages until it's rendered. Cite section headings or paragraph ranges instead, or convert to PDF at upload time if page-level citations are a hard requirement.

## Up Next

Tomorrow is Day 3, the AI Customer Support Copilot. It's a system that sits next to human support agents, reads each ticket, retrieves the right knowledge base articles, and drafts a reply the agent reviews before sending. If you want a document chat system built on your own content, [talk to us](/contact).
