Build a RAG chatbot with Next.js, pgvector and an LLM

A RAG chatbot answers questions from your content — docs, a knowledge base, product data — rather than from whatever the model absorbed during training. It is the usual way teams ship an assistant that is grounded in something they control.

This is the whole pipeline in Next.js with Postgres, pgvector and an LLM, including the parts that bite in production rather than in a demo: chunking, retrieval quality, the "I don't know" path, streaming and cost.

When you need it, and when you do not

Retrieval-Augmented Generation means you fetch relevant chunks of your own data first, then hand them to the model as context so the answer comes from facts you chose. It addresses two LLM weaknesses at once: the model does not know your private data, and it will invent an answer rather than decline.

Reach for it when answers must come from a specific corpus. Support docs, an internal wiki, a product catalogue. If what you need is general reasoning, RAG adds latency, cost and a retrieval system you now have to maintain, and buys you nothing.

The shape is always the same

Five steps, and once the mental model is there the code mostly writes itself.

Ingest your documents. Chunk them into passages small enough to retrieve precisely. Embed each chunk and store the vector. Then, per request, embed the question and fetch the nearest chunks, and give those to the model as context while streaming the answer back.

The first three run offline. The last two run per request, which is the division that matters for cost and latency.

pgvector setup

Embeddings live next to your data in the Postgres you already run. Most applications never need a separate vector database.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id         bigserial PRIMARY KEY,
  source     text NOT NULL,          -- which doc it came from
  content    text NOT NULL,          -- the chunk text
  embedding  vector(1536)            -- text-embedding-3-small dims
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

Build the HNSW index after the first bulk ingest rather than before it. Inserting into an existing index is slower than building it once over populated data, and on a first load of any size that difference is minutes.

If pgvector versus a dedicated vector database is a live question, I went through the trade-offs in pgvector vs Pinecone.

Chunking is where quality is won

This is the step people tune last and should tune first. Chunks that are too large bury the answer in noise, and the retrieval score drops because most of the chunk is irrelevant to the question. Too small and they lose the context that made them meaningful.

Around 500 to 800 tokens with a little overlap is a reasonable default, so sentences are not cut mid-thought.

// Note: `size` and `overlap` are in words, not tokens. English runs roughly
// 1.3 tokens per word, so 550 words ≈ 700 tokens. Check against your tokenizer
// if you are near a context limit.
export function chunk(text: string, size = 550, overlap = 80): string[] {
  const words = text.split(/\s+/).filter(Boolean);
  const chunks: string[] = [];
  for (let i = 0; i < words.length; i += size - overlap) {
    chunks.push(words.slice(i, i + size).join(' '));
  }
  return chunks;
}

The word-versus-token distinction trips people up, including the earlier version of this post. If your chunker counts words and your limits are in tokens, you are off by about 30% in the direction that breaks things.

Splitting on a fixed character count is the other common mistake, because it slices words and sometimes UTF-8 sequences in half. Split on whitespace at minimum, sentence boundaries if the corpus rewards it.

Ingestion

Once per document, and again when the content changes.

import OpenAI from 'openai';
import { sql } from './db';
import { chunk } from '../lib/chunk';

const openai = new OpenAI();

export async function ingest(source: string, text: string) {
  const chunks = chunk(text);

  // One batched embedding call: 200 chunks in one request is far cheaper
  // and faster than 200 requests.
  const { data } = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: chunks,
  });

  // One multi-row insert, not a loop of single inserts — the loop costs you
  // a round trip per chunk, which on a large document dwarfs the API call.
  const rows = chunks.map((content, i) => ({
    source,
    content,
    embedding: JSON.stringify(data[i].embedding),
  }));

  await sql`
    INSERT INTO chunks (source, content, embedding)
    SELECT source, content, embedding::vector
    FROM jsonb_to_recordset(${JSON.stringify(rows)}::jsonb)
      AS t(source text, content text, embedding text)
  `;
}

Batch the embedding call and batch the insert. Advice to batch the first while looping the second is advice that has not been run, and the insert loop is usually the slower half on a document of any size.

Make ingestion idempotent while you are here. Delete existing chunks for a source before inserting, or key on a content hash, because the re-ingest path will run twice at some point and duplicated chunks degrade retrieval quietly by crowding the top-k with copies of the same passage.

Retrieval, with a floor that actually exists

Embed the question, pull the closest chunks with pgvector's <=> cosine distance operator. Five is usually plenty.

The relevance floor is the part most tutorials mention and then omit from the code, which makes it useless. If the best match is weak, the corpus probably does not contain the answer, and the system needs to know that before the model is asked to produce one.

const MIN_SCORE = 0.3; // tune against your own corpus — this is not a universal number

export async function retrieve(question: string, k = 5) {
  const { data } = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: question,
  });
  const queryVec = JSON.stringify(data[0].embedding);

  const rows = await sql`
    SELECT content, source, 1 - (embedding <=> ${queryVec}) AS score
    FROM chunks
    ORDER BY embedding <=> ${queryVec}
    LIMIT ${k}
  `;

  return rows.filter((r) => r.score >= MIN_SCORE);
}

Tune MIN_SCORE by running your real questions and looking at the scores of answers you know are right, then at the scores of questions the corpus genuinely cannot answer. The gap between those two distributions is your threshold, and it is corpus-specific enough that borrowing someone else's number is guesswork.

Generation

A system instruction that confines the model to the context, the retrieved chunks, the question. Stream it so the UI responds immediately.

export async function* answer(question: string) {
  const docs = await retrieve(question);

  // Nothing cleared the floor: decline rather than asking the model to guess.
  if (docs.length === 0) {
    yield "I don't have anything in my sources that answers that.";
    return;
  }

  const context = docs.map((d) => `[${d.source}] ${d.content}`).join('\n\n');

  const stream = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    stream: true,
    messages: [
      {
        role: 'system',
        content:
          'Answer using ONLY the context below. If the answer is not in ' +
          'the context, say you do not know. Cite the [source] you used.\n\n' +
          context,
      },
      { role: 'user', content: question },
    ],
  });

  for await (const part of stream) {
    yield part.choices[0]?.delta?.content ?? '';
  }
}

Two guardrails there, and they do different jobs. The early return handles the case where retrieval found nothing worth sending, which saves you a generation call you were going to regret. The system instruction handles the case where retrieval found something close but not actually relevant, which the floor cannot catch.

Keeping both matters. The instruction alone still sends weak context and invites a confident answer built on it.

The Next.js route

A streaming Route Handler. The App Router streams a ReadableStream straight through, so tokens appear as they are produced.

import { answer } from '@/lib/answer';

export async function POST(req: Request) {
  const { question } = await req.json();

  const stream = new ReadableStream({
    async start(controller) {
      const encoder = new TextEncoder();
      try {
        for await (const token of answer(question)) {
          controller.enqueue(encoder.encode(token));
        }
      } catch (err) {
        controller.enqueue(encoder.encode('\n\n[error generating answer]'));
      } finally {
        controller.close();
      }
    },
  });

  return new Response(stream, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

The try/finally is not decoration. Without it, an error mid-stream leaves the controller open and the client hanging on a response that will never close, which presents as a chatbot that has stopped mid-sentence with no error anywhere.

On the client, fetch plus response.body.getReader() and append tokens as they arrive. A dozen lines, and the result starts answering well under a second.

Then production arrives

Embeddings are cheap and generation is not, so that is where the attention goes. A small fast model handles most queries, with a larger one reserved for the hard ones if you can classify them. Cap the number and size of retrieved chunks, because context tokens are billed on every single call and a generous k is a permanent tax. I costed this properly in what a RAG answer costs.

Cache embeddings for repeated inputs and full answers for common questions. If you are serving from Next.js, note that the caching defaults changed in 15 and changed again in 16, so check the current model rather than assuming.

Both the LLM and the embedding API will rate-limit you and will occasionally fail. Retries with backoff around every call, and any ingestion webhook made idempotent so a retry never double-inserts, which is the same pattern as idempotent APIs and webhooks.

And show citations. They are the cheapest trust mechanism available, and they let a user check the one thing you cannot guarantee.

The mistakes that cost the most

Oversized chunks, where the answer drowns in surrounding text and the similarity score drops because most of the chunk is about something else.

No "I don't know" path, which guarantees hallucination on every question the corpus does not cover, and those are the questions users ask most.

Re-embedding everything on deploy rather than on content change, which is a bill for no benefit.

And ignoring retrieval quality, which is the one that wastes months. If retrieval returns junk, no prompt and no larger model recovers it. Measure retrieval on its own, before you tune anything downstream of it.

What I would get right first

Five steps: ingest, chunk, embed, retrieve, generate. Next.js streams it, pgvector stores it, the model phrases what retrieval found.

The API calls are the easy part and they are what most guides spend their length on. The work is in chunking, in retrieval quality, in having a real "I don't know" path rather than a sentence in a prompt, and in not letting context tokens quietly become your largest bill.

If I were starting one tomorrow I would spend the first day on retrieval alone, with no model in the loop at all. Write twenty real questions, run them against your corpus, and read the top five chunks for each by hand. If the right passage is not in there, nothing you do afterwards matters, and you will have found that out on day one instead of after the prompt engineering.

If you are weighing where to put the vectors, start with pgvector vs Pinecone. Most teams need less infrastructure than they think.