What Does a Production RAG System Need? Vector Search, Chunking, Guardrails, and Cost Controls

Building a toy Retrieval-Augmented Generation (RAG) prototype takes less than an afternoon using LangChain or LlamaIndex. Taking that system into production—where it must process thousands of enterprise documents, enforce multi-tenant access controls, eliminate hallucinations, and keep API token bills under control—is a completely different engineering challenge.

Most naive RAG implementations fail in real-world usage because they treat retrieval as a simple black-box vector search. Here is the architectural anatomy of what a true production RAG system requires.

The Production RAG Architecture

┌─────────────────┐       ┌──────────────────────────────────────────────────────────────┐
│ Enterprise Docs │ ────► │ Ingestion Pipeline: Parse ➔ Semantic Chunking ➔ Embeddings   │
└─────────────────┘       └──────────────────────────────┬───────────────────────────────┘
                                                         │
                                                         ▼
                                       ┌──────────────────────────────────┐
                                       │ PostgreSQL + pgvector (HNSW Index│
                                       │ Hybrid Search + RBAC Metadata)   │
                                       └─────────────────┬────────────────┘
                                                         │
User Query ──► [Query Expansion] ──► [Hybrid Retrieval] ─┴─► [Cross-Encoder Reranker]
                                                                     │
                                                                     ▼
                                                   [Strict Context Window Assembly]
                                                                     │
                                                                     ▼
                                                   [LLM Generation + Streaming SSE]
                                                                     │
                                                                     ▼
                                                   [Hallucination & Guardrail Check]
                                                                     │
                                                                     ▼
                                                   Verified Answer with Citations

1. Ingestion: Why Fixed-Character Chunking Fails

The most frequent cause of poor RAG response quality is poor document chunking. Naive approaches split documents every 500 characters or 200 tokens. This inevitably cuts sentences in half, separates tables from their column headers, and strips contextual meaning.

Production Chunking Best Practices:

1. Semantic Chunking: Split text on structural document boundaries: Markdown headings, paragraph breaks, or list items.

2. Document Hierarchy Preservation: Prepend parent metadata (e.g. document title, section heading, author) to each chunk so the vector embedding retains thematic context.

3. Table & Schema Extraction: Tables should be parsed into structured Markdown or JSON format before chunking so numerical comparisons are not scrambled.

2. Retrieval: Vector Search Alone Is Not Enough

Pure vector search (cosine similarity on dense embeddings) excels at capturing conceptual similarity, but it notoriously fails on exact keyword lookups: SKU numbers, acronyms, product model codes, and person names.

The Solution: Hybrid Search with Reciprocal Rank Fusion (RRF)

In a production database like PostgreSQL with pgvector, combine two search techniques:

1. Dense Vector Search: Using HNSW (Hierarchical Navigable Small World) indexes to capture semantic intent.

2. Sparse Lexical Search: Using PostgreSQL Full-Text Search (tsvector with ts_rank_cd) for exact keyword matches.

3. Reciprocal Rank Fusion (RRF): Merging both result lists mathematically to surface chunks that rank highly across both vectors and keywords.

-- Hybrid Vector + Lexical Search Query with pgvector
WITH semantic_search AS (
  SELECT id, rank() OVER (ORDER BY embedding <=> $1) AS rank
  FROM document_chunks
  WHERE tenant_id = $2
  LIMIT 20
),
keyword_search AS (
  SELECT id, rank() OVER (ORDER BY ts_rank_cd(text_search_vector, query) DESC) AS rank
  FROM document_chunks, to_tsquery('english', $3) query
  WHERE text_search_vector @@ query AND tenant_id = $2
  LIMIT 20
)
SELECT COALESCE(s.id, k.id) as chunk_id,
       COALESCE(1.0 / (60 + s.rank), 0.0) + COALESCE(1.0 / (60 + k.rank), 0.0) AS rrf_score
FROM semantic_search s
FULL OUTER JOIN keyword_search k ON s.id = k.id
ORDER BY rrf_score DESC
LIMIT 5;

3. Role-Based Access Control (RBAC) at the Retrieval Layer

In corporate environments, document permissions are paramount. An employee in Marketing must never retrieve financial compensation records or executive board minutes, regardless of their query.

Filtering permissions *after* retrieval is insecure and degrades search quality. In a production architecture, RBAC metadata must be filtered directly inside the SQL index query:

4. Hallucination Defense & Guardrails

To ensure production dependability, implement automated guardrails:

1. Strict Context Adherence: Instruct the model to cite specific chunk IDs and explicitly state *"I cannot find this information in the provided documents"* when evidence is lacking.

2. Re-ranking with Cross-Encoders: Pass the top 15 retrieved chunks through a lightweight reranker (such as Cohere Rerank or BGE-Reranker) to filter out irrelevant noise before spending LLM prompt tokens.

3. Automated Citation Validation: Post-process the generated response to verify that every cited claim maps to an actual retrieved chunk.

5. Cost & Latency Governance

Unmonitored LLM integrations lead to massive cloud bills and frustrated users waiting 10+ seconds for answers.

Ready to Build a Production AI or RAG Solution?

Need an experienced engineer to design, build, and deploy an enterprise-ready RAG search system or AI feature?