A RAG chatbot answers questions using your content — docs, a knowledge base, product data — instead of whatever the model memorized during training. It’s the most common way teams ship a useful, grounded AI assistant. In this guide I’ll build a production-minded RAG chatbot with Next.js, Postgres + pgvector, and an LLM, end to end, with the parts that actually bite you in production: chunking, retrieval quality, streaming, cost, and caching.
What RAG actually is (and when you need it)
RAG — Retrieval-Augmented Generation — means you retrieve relevant chunks of your own data first, then hand them to the LLM as context so it answers from facts you control. It’s the fix for two LLM weaknesses: it doesn’t know your private/internal data, and it confidently makes things up.
Reach for RAG when answers must come from a specific corpus — support docs, internal wikis, a product catalog. If you only need general reasoning, you don’t need RAG. If you need fresh, proprietary, or citable answers, you do.
The architecture in five steps
Every RAG pipeline is the same shape. Get this mental model and the code writes itself:
- Ingest — load your documents.
- Chunk — split them into passages small enough to embed and retrieve precisely.
- Embed & store — turn each chunk into a vector and save it in pgvector.
- Retrieve — embed the user’s question and fetch the nearest chunks.
- Generate — give those chunks to the LLM as context and stream the answer.
Steps 1–3 run offline (ingestion); steps 4–5 run per request (query time).
Set up pgvector
Store embeddings next to your data in the Postgres you already run — no separate vector database needed for most apps.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
source text NOT NULL, -- which doc it came from
content text NOT NULL, -- the chunk text
embedding vector(1536) -- text-embedding-3-small dims
);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);If pgvector vs a dedicated vector DB is a real question for you, I went deep on the trade-offs in pgvector vs Pinecone.
Chunk your documents
Chunking is where most RAG quality is won or lost. Chunks that are too big bury the answer in noise; too small and they lose context. A good default: ~500–800 tokens with a little overlap so sentences aren’t cut mid-thought.
export function chunk(text: string, size = 700, overlap = 100): string[] {
const words = text.split(/\s+/);
const chunks: string[] = [];
for (let i = 0; i < words.length; i += size - overlap) {
chunks.push(words.slice(i, i + size).join(' '));
}
return chunks;
}Common mistake: splitting on a fixed character count and slicing words in half. Split on whitespace (or better, sentence boundaries) and keep a small overlap.
Embed and store (ingestion)
Run this once per document (and again when content changes). Each chunk becomes a 1536-dim vector you can search by similarity.
import OpenAI from 'openai';
import { sql } from './db';
import { chunk } from '../lib/chunk';
const openai = new OpenAI();
export async function ingest(source: string, text: string) {
const chunks = chunk(text);
// Embed in one batched call — cheaper and faster than one-by-one.
const { data } = await openai.embeddings.create({
model: 'text-embedding-3-small',
input: chunks,
});
for (let i = 0; i < chunks.length; i++) {
await sql`
INSERT INTO chunks (source, content, embedding)
VALUES (${source}, ${chunks[i]}, ${JSON.stringify(data[i].embedding)})
`;
}
}Tip: batch your embedding calls. Embedding 200 chunks in one request is dramatically cheaper and faster than 200 requests.
Retrieve the right context
At query time, embed the question and pull the closest chunks with the pgvector <=> cosine-distance operator. Top 4–6 is usually plenty.
export async function retrieve(question: string, k = 5) {
const { data } = await openai.embeddings.create({
model: 'text-embedding-3-small',
input: question,
});
const queryVec = JSON.stringify(data[0].embedding);
return sql`
SELECT content, source, 1 - (embedding <=> ${queryVec}) AS score
FROM chunks
ORDER BY embedding <=> ${queryVec}
LIMIT ${k}
`;
}Best practice: keep a relevance floor. If the top score is low, the corpus probably doesn’t contain the answer — say so instead of forcing a guess.
Generate the answer with the LLM
Now assemble a tight prompt: a system instruction that tells the model to answer only from the provided context, the retrieved chunks, and the question. Stream the response so the UI feels instant.
export async function* answer(question: string) {
const docs = await retrieve(question);
const context = docs.map((d) => `[${d.source}] ${d.content}`).join('\n\n');
const stream = await openai.chat.completions.create({
model: 'gpt-4o-mini',
stream: true,
messages: [
{
role: 'system',
content:
'Answer using ONLY the context below. If the answer is not in ' +
'the context, say you do not know. Cite the [source] you used.\n\n' +
context,
},
{ role: 'user', content: question },
],
});
for await (const part of stream) {
yield part.choices[0]?.delta?.content ?? '';
}
}That "answer only from context, otherwise say you don’t know" instruction is the single most important line for trustworthiness — it’s what stops hallucinations.
Wire it into Next.js
Expose it as a streaming Route Handler. The App Router streams a ReadableStream straight to the browser, so tokens appear as they’re generated.
import { answer } from '@/lib/answer';
export async function POST(req: Request) {
const { question } = await req.json();
const stream = new ReadableStream({
async start(controller) {
const encoder = new TextEncoder();
for await (const token of answer(question)) {
controller.enqueue(encoder.encode(token));
}
controller.close();
},
});
return new Response(stream, {
headers: { 'Content-Type': 'text/plain; charset=utf-8' },
});
}On the client, read the stream and append tokens as they arrive — a dozen lines with fetch + response.body.getReader(). The result is a chatbot that starts answering in well under a second.
Production concerns most tutorials skip
Cost
Embeddings are cheap; generation is not. Use a small, fast model (like gpt-4o-mini) for most queries and reserve a larger model for hard ones. Cap the number and size of retrieved chunks — context tokens are billed on every call.
Caching
Cache embeddings for repeated/identical inputs, and cache full answers for common questions. If you serve RAG from Next.js, understand the caching layers first — I broke them down in Next.js App Router caching, finally explained.
Reliability
LLM and embedding APIs rate-limit and occasionally fail. Wrap calls in retries with backoff, and make any ingestion webhook idempotent so a retry never double-inserts chunks.
Guardrails
Keep the "answer only from context" rule, show citations, and add a relevance threshold. A chatbot that admits "I don’t know" beats one that invents a confident wrong answer.
Common mistakes to avoid
- Chunks too large — the answer drowns in irrelevant text. Smaller, overlapping chunks retrieve better.
- No "I don’t know" path — without it the model hallucinates when the corpus lacks the answer.
- Re-embedding everything on every deploy — embed on content change, not on every build.
- Ignoring retrieval quality — if retrieval returns junk, no prompt or model can save the answer. Fix retrieval first.
Summary
A RAG chatbot is five steps: ingest, chunk, embed, retrieve, generate. Next.js streams it, pgvector stores it, and the LLM grounds its answer in chunks you control. The hard parts aren’t the API calls — they’re chunking, retrieval quality, the "don’t know" guardrail, and cost. Get those right and you have an assistant that’s genuinely useful instead of confidently wrong.
If you’re weighing where to store the vectors, start with pgvector vs Pinecone — most teams need less infrastructure than they think.
Retrieval is the product. The model just phrases what your retrieval found.