Building a Production-Ready RAG Application: Architecture, Challenges, and Lessons Learned

That’s where Retrieval-Augmented Generation (RAG) comes in. Instead of forcing a model to memorize information during training, RAG allows it to retrieve relevant knowledge from your own documents at query time and generate grounded answers.

In this article, we’ll walk through a fully local, no-paid-API RAG system built with:

• FastAPI for APIs and WebSockets
• PostgreSQL for metadata and chat history
• Qdrant for vector search
• Redis for caching and Celery messaging
• Celery for asynchronous document processing
• Ollama running:
  • nomic-embed-text for embeddings
  • llama3.2 for answer generation

The entire stack runs locally through Docker, making it ideal for organizations that require privacy, low operating costs, and complete control over their data.

Why Build a Local RAG System?

Many RAG implementations rely on external APIs from OpenAI, Anthropic, or Cohere. While convenient, this introduces several challenges:

• Recurring API costs
• Data privacy concerns
• Internet dependencies
• Vendor lock-in
• Rate limits

A local architecture solves these problems.

Benefits include:

• ✅ No token costs
• ✅ Documents never leave your environment
• ✅ Full customization
• ✅ Offline capabilities
• ✅ Predictable latency

For internal knowledge bases, educational systems, legal documents, research papers, and enterprise manuals, local RAG can be a very attractive solution.

High-Level Architecture

The system follows a classic Retrieval-Augmented Generation pipeline.

┌──────────────────────────────────────────────────────────────┐
│                    DOCUMENT INGESTION FLOW                   │
└──────────────────────────────────────────────────────────────┘
                 ┌─────────────────┐
                 │   Admin Upload  │
                 │      PDF        │
                 └────────┬────────┘
                          │
                          ▼
          ┌─────────────────────────────────┐
          │ FastAPI Document Service        │
          │ • Store PDF                     │
          │ • Create document record        │
          │ • Queue processing job          │
          └──────────────┬──────────────────┘
                         │
                         ▼
              ┌─────────────────────┐
              │ Redis + Celery      │
              │ Async Processing    │
              └─────────┬───────────┘
                        │
                        ▼
            ┌──────────────────────────┐
            │ PyMuPDF Extraction       │
            │ • Layout analysis        │
            │ • Reading order recovery │
            │ • Header/Footer removal  │
            └──────────┬───────────────┘
                       │
                       ▼
            ┌──────────────────────────┐
            │ Structure-Aware Chunking │
            │                          │
            │ Parent: 1400 tokens      │
            │ Child : 400 tokens       │
            │ Overlap: 60 tokens       │
            └──────────┬───────────────┘
                       │
                       ▼
            ┌──────────────────────────┐
            │ Ollama Embeddings        │
            │ nomic-embed-text         │
            │ (768 dimensions)         │
            └──────────┬───────────────┘
                       │
          ┌────────────┴───────────────┐
          ▼                            ▼
┌──────────────────┐      ┌────────────────────┐
│ PostgreSQL       │      │ Qdrant             │
│                  │      │ Vector Database    │
│ Documents        │      │                    │
│ Parent Chunks    │      │ Child Embeddings   │
│ Metadata         │      │ Similarity Search  │
└──────────────────┘      └────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│                    QUESTION ANSWERING FLOW                   │
└──────────────────────────────────────────────────────────────┘
              ┌──────────────────────┐
              │    User Question     │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │ Conversation Context │
              │ + Chat History       │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │ Query Embedding      │
              │ (Ollama)             │
              └──────────┬───────────┘
                         │
          ┌──────────────┴──────────────┐
          ▼                             ▼
┌──────────────────────┐    ┌──────────────────────┐
│ Qdrant Search        │    │ PostgreSQL BM25      │
│ Semantic Retrieval   │    │ Keyword Retrieval    │
└──────────┬───────────┘    └──────────┬───────────┘
           │                           │
           └─────────────┬─────────────┘
                         ▼
              ┌──────────────────────┐
              │ Reciprocal Rank      │
              │ Fusion (RRF)         │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │ Top-K Context Chunks │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │ Ollama llama3.2      │
              │ Grounded Generation  │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │ Streaming Response   │
              │ WebSocket Tokens     │
              └──────────────────────┘

The workflow separates document processing from user chat, ensuring responsive conversations while heavy indexing happens asynchronously.

Conversation flow

Service Breakdown

The core infrastructure consists of six services.

Service	Responsibility
FastAPI	Uploads, chat APIs, WebSockets
Celery Worker	PDF processing
PostgreSQL	Metadata, chats, chunks
Qdrant	Vector similarity search
Redis	Cache and broker
Ollama	Embeddings and LLM inference

Docker networking allows services to talk to each other by service name:

QDRANT_URL=http://qdrant:6333
OLLAMA_URL=http://ollama:11434

A useful Docker tip:

Inside containers:

http://ollama:11434

From the host machine:

http://localhost:11434

If a container needs to reach software running directly on macOS:

host.docker.internal

Setting Up the Environment

After cloning the project:

docker compose up -d --build

Download the two local models:

docker compose exec ollama ollama pull nomic-embed-text
docker compose exec ollama ollama pull llama3.2

Run migrations:

docker compose exec api alembic upgrade head

Verify containers:

docker compose ps

Understanding Embeddings

Before discussing indexing, let’s quickly explain embeddings.

An embedding converts text into a numerical vector.

Example:

“What is RAG?”

===> [0.24, -0.13, 0.88, …]

Documents with similar meaning produce vectors located near each other in vector space.

This project uses: nomic-embed-text

Therefore: EMBEDDING_DIMENSIONS=768

match the Qdrant collection configuration.

If dimensions mismatch, indexing and searches will fail.

PDF Processing Pipeline

Once a PDF is uploaded, a Celery worker takes over.

Document lifecycle:

queued
   ▼
extracting
   ▼
indexing
   ▼
saving
   ▼
ready/failed

This decouples heavy processing from API requests and improves scalability.

A user doesn’t need to wait for indexing to finish.

Smarter PDF Extraction

A common mistake in RAG systems is treating a page as a giant blob of text.

That destroys structure.

This project instead uses PyMuPDF layout-aware extraction.

The extractor captures:

1/ Page number
2/ Bounding coordinates
3/ Text blocks
4/ Reading order

It also:

• Detects headings
• Recognizes paragraphs
• Finds lists
• Detects question blocks
• Attempts table identification
• Removes repeating headers and footers

Parent-Child Chunking Strategy

Chunking is arguably the most important part of a RAG system.

Too small:

• Missing context

Too large:

• Retrieval becomes noisy

This project uses a two-level chunking design.

Parent Chunks

1400 tokens

Stored in PostgreSQL.

Purpose:

• Preserve semantic sections
• Retain structure
• Support auditing

Child Chunks

400 tokens

Stored in Qdrant.

Purpose:

• Retrieval
• Embedding generation

Overlap

60 tokens

Why?

Without overlap:

Sentence A ends...
Sentence B starts...

could be split apart, causing information loss.

Overlap helps maintain continuity.

Why Hybrid Retrieval Beats Pure Vector Search

Many beginners assume vector search alone is enough.

Consider:

RW Question 3

The term itself may be critically important.

Vector similarity may not prioritize it strongly.

This system combines:

Dense Retrieval

Using Qdrant:

Meaning-based search

Great for:

• Concepts
• Semantic similarity
• Natural language questions

BM25 Retrieval

Using PostgreSQL:

• Keyword search

Great for:

• IDs
• Names
• Exact labels
• Technical terms

Reciprocal Rank Fusion (RRF)

After obtaining two rankings, the system merges them using:

Reciprocal Rank Fusion

Formula:

Advantages:

• Simple
• Effective
• Robust
• Industry-proven

Rather than choosing vector search or keyword search, RRF lets both contribute.

In practice, RRF often improves retrieval quality substantially over either method alone.

Grounded Generation with Ollama

After retrieval:

• Top K chunksShow more lines

are sent to:

• llama3.2

The prompt explicitly instructs the model:

Answer only from provided context.

This reduces hallucinations and keeps answers grounded in uploaded documents.

For exam-style PDFs, the prompt prioritizes:

Answer:
Key:
Solution:

Conversation-Aware Retrieval

One underrated feature of modern RAG systems is conversational memory.

Suppose a user asks:

Tell me about RW Question 3.

Followed by:

Give me more details.

The second query alone lacks context.

This project solves that by using:

• Recent chat history
• Previous user topics
• Conversation-specific retrieval

The retriever can resolve references like:

it
this
that section
more information

making follow-up interactions feel much more natural.

Why Caching Matters

LLM systems frequently repeat work.

Two users may ask:

What is Question 3?

multiple times.

Redis caches:

Embeddings

24 hours

Retrieval Results

5 minutes

Benefits:

• Reduced Qdrant queries
• Lower retrieval latency

An important architectural detail:

The system maintains a corpus version.

Whenever a new PDF becomes active:

Cache keys become invalid automatically.

Newest-Document Architecture

One particularly elegant design decision is that chat only searches the latest successfully indexed document.

Upon upload:

1. Existing vector collection deleted
2. Older documents archived
3. New document indexed
4. Retrieval restricted to ready documents

Benefits:

• Predictable search scope
• No document contamination
• Simpler retrieval logic

Historical documents remain available for auditing while keeping the active knowledge base clean.

Current Limitations

No system is perfect.

Current constraints include:

Scanned PDFs

The extractor currently works best with:

Selectable text PDFs

Image-only documents require OCR.

Possible improvements:

• Tesseract OCR
• PaddleOCR
• Azure Document Intelligence
• DocTR

Complex Tables

Table extraction is heuristic.

Future enhancements could include:

• Camelot
• Tabula
• LayoutParser
• Unstructured.io

Multi-Document Retrieval

The current design intentionally focuses on:

• Newest document only

Future versions could support:

• Collections
• Knowledge bases
• Multi-document ranking

Final Thoughts

This project demonstrates that we don’t need expensive APIs to build a capable Retrieval-Augmented Generation platform. By combining FastAPI, PostgreSQL, Qdrant, Redis, Celery, Ollama, and hybrid retrieval, we can create a fully local RAG solution that is private, scalable, and surprisingly effective.

The strongest aspects of the architecture are:

• Structure-aware PDF extraction
• Parent-child chunking
• Hybrid retrieval (Vector + BM25)
• Reciprocal Rank Fusion
• Conversation-aware search
• Streaming responses
• Automatic cache invalidation
• Zero external AI costs

For teams building internal copilots, document assistants, enterprise search, educational Q&A systems, or research knowledge bases, this architecture provides an excellent foundation that can evolve into a production-grade AI platform.