Vector Search

ជំពូកនេះមិនទាន់ត្រូវបានបកប្រែជាភាសាខ្មែរនៅឡើយទេ — សូមអានជាភាសាអង់គ្លេសខាងក្រោម។

Table of Contents

  1. From a Loop to a Store
  2. Brute-Force Search
  3. The VectorStore Class
  4. A Tiny Store, Saved and Loaded
  5. The Indexer: Load, Chunk, Embed, Store
  6. Searching the Club Documents
  7. Top-k: How Many Results?
  8. Score Thresholds
  9. Metadata Filters
  10. Only Re-embed What Changed
  11. Safety Checks
  12. How Slow Is Brute Force?
  13. Approximate Nearest Neighbour Search
  14. When You Need a Real Vector Database
  15. Recipe: The store and indexer Modules
  16. Try It Yourself
  17. Quick Review: Common Questions

1. From a Loop to a Store

At the end of the last chapter you ranked chunks with a loop inside a demo script. It worked, but a script is not a component. A real RAG program needs a place to keep chunks and vectors together, a way to ask for the closest ones, and a way to save the result so you do not rebuild everything on every start. That component is a vector store.

Think of the index at the back of a textbook. You do not read the whole book to find "photosynthesis". You open the index, find the page numbers, and go there. A vector store is the index of your documents, but it is looked up by meaning-numbers instead of by exact word.

In the pipeline, this chapter covers the last box of ingest and the second box of the query side:

ingest:  load -> chunk -> embed -> STORE
query:   embed question -> SEARCH -> prompt -> generate -> cite

You will write two small modules:

  • src/store.js: the VectorStore class (add, search, save, load).
  • src/indexer.js: buildIndex(dir), which runs load, chunk, embed and store in one call, and keeps an index.json on disk.

They use loadDocs and chunkDocs from Loading and Chunking and embed from Embeddings in Practice.

Setup. Run the mock server in another terminal (node mock-llm.js, port 8787). If you use a different port, set LLM_BASE_URL for your scripts, for example LLM_BASE_URL=http://localhost:8850 node demo-index.js.

How do you find the closest vectors? The simplest way is to compare the question vector with every stored vector, then sort by score and keep the best few. That is called brute force (or "exact" or "flat") search. It has no clever data structure at all:

  1. Embed the question into a vector q.
  2. For each chunk, compute score = cosine(q, vector).
  3. Keep the ones that pass your rules (a minimum score, a filter).
  4. Sort from high to low and return the first k.

It is slow in theory, because the work grows with the number of chunks. But it is exact (it never misses the true best match), very easy to write and debug, and for a few thousand chunks it is fast enough, as you will measure in section 12. Many real projects never need more. Start here.

One trick makes it faster. If all stored vectors have length 1, then cosine similarity is just the dot product (see the previous chapter). So the store normalises every vector once when you add it, and search only multiplies and adds.

3. The VectorStore Class

Here is the store. Read it slowly, it is only about 50 lines:

// src/store.js
import { readFile, writeFile } from "node:fs/promises";

const unit = (v) => {
  const len = Math.sqrt(v.reduce((s, x) => s + x * x, 0)) || 1;
  return v.map((x) => Number((x / len).toFixed(6))); // 6 decimals keeps index.json small
};

export class VectorStore {
  chunks = [];
  vectors = [];
  model = null; // which embedding model made these vectors (set by the indexer)

  get size() {
    return this.chunks.length;
  }

  get dimensions() {
    return this.vectors[0]?.length ?? 0;
  }

  add(chunks, vectors) {
    if (chunks.length !== vectors.length) {
      throw new Error(`got ${chunks.length} chunks but ${vectors.length} vectors`);
    }
    vectors.forEach((v, i) => {
      if (this.dimensions && v.length !== this.dimensions) {
        throw new Error(`vector ${i} has ${v.length} dimensions, the store has ${this.dimensions}`);
      }
      this.chunks.push(chunks[i]);
      this.vectors.push(unit(v)); // unit length: cosine becomes a plain dot product
    });
  }

  // Brute force: score every chunk, keep the best k.
  search(vector, { k = 4, minScore = 0, filter } = {}) {
    if (this.size && vector.length !== this.dimensions) {
      throw new Error(`query has ${vector.length} dimensions, the store has ${this.dimensions}`);
    }
    const q = unit(vector);
    const hits = [];
    for (let i = 0; i < this.size; i++) {
      if (filter && !filter(this.chunks[i])) continue;
      let score = 0;
      for (let d = 0; d < q.length; d++) score += q[d] * this.vectors[i][d];
      if (score >= minScore) hits.push({ chunk: this.chunks[i], score });
    }
    return hits.sort((a, b) => b.score - a.score).slice(0, k);
  }

  async save(path) {
    const data = { version: 1, model: this.model, dimensions: this.dimensions, chunks: this.chunks, vectors: this.vectors };
    await writeFile(path, JSON.stringify(data));
  }

  static async load(path) {
    const data = JSON.parse(await readFile(path, "utf8"));
    if (data.version !== 1) throw new Error(`unknown index version: ${data.version}`);
    const store = new VectorStore();
    store.model = data.model;
    store.add(data.chunks, data.vectors);
    return store;
  }
}

Points to notice:

  • Chunks and vectors live in two arrays with the same index: vectors[i] belongs to chunks[i]. add checks that the counts match.
  • The store checks dimensions on add and on search. This is the guard we asked for in the last chapter: mixing models fails loudly.
  • search returns { chunk, score } objects, best first. It takes three options, explained in sections 7 to 9: k, minScore and filter.
  • filter is a function that receives a chunk and returns true to keep it. Because it runs before scoring, filtered-out chunks never appear in the results, whatever their score.
  • model is saved in the file. Anyone who loads the index later knows which model to use for questions.
  • JSON is a plain text file, so you can open it in your editor. It is not compact or fast, but it needs no library.

4. A Tiny Store, Saved and Loaded

First test the store with three hand-written chunks, before using the whole corpus:

// demo-store.js
import { VectorStore } from "./src/store.js";
import { embed } from "./src/embed.js";

const chunks = [
  { id: "fee", text: "Membership costs 10 US dollars per term." },
  { id: "seats", text: "The room has 24 seats and a projector." },
  { id: "loan", text: "A laptop can be borrowed for up to 7 days." },
];
const store = new VectorStore();
store.add(chunks, await embed(chunks.map((c) => c.text)));
console.log("size:", store.size, " dimensions:", store.dimensions);

const [q] = await embed(["How much does membership cost?"]);
for (const { chunk, score } of store.search(q, { k: 2 })) {
  console.log(score.toFixed(3), chunk.id);
}

await store.save("tiny-index.json");
const again = await VectorStore.load("tiny-index.json");
const [hit] = again.search(q, { k: 1 });
console.log("after save + load:", hit.chunk.id, hit.score.toFixed(3));
// output
// size: 3  dimensions: 256
// 0.471 fee
// 0.000 seats
// after save + load: fee 0.471

All of this output is stable on the mock. Two things to see:

  • We asked for k: 2, and the second result has score 0.000. With the default minScore of 0, a chunk with no similarity at all is still allowed in, so you always get k results when the store has k chunks. Setting a higher minScore removes such useless results (section 8).
  • After saving and loading, the same search gives the same hit and score. Delete tiny-index.json afterwards.

5. The Indexer: Load, Chunk, Embed, Store

Now the function that builds the real index from a folder of documents. buildIndex(dir, options) does the whole ingest in one call. It takes indexPath (default index.json), a model, and any options for chunkDocs (such as strategy or maxChars):

// src/indexer.js
import { createHash } from "node:crypto";
import { loadDocs } from "./load.js";
import { chunkDocs } from "./chunk.js";
import { embed } from "./embed.js";
import { embedModel } from "./config.js";
import { VectorStore } from "./store.js";

const hashOf = (model, text) => createHash("sha256").update(`${model}\n${text}`).digest("hex").slice(0, 16);

// load -> chunk -> embed -> store. Only text we have not seen before is sent to the API.
export async function buildIndex(dir, { indexPath = "index.json", model = embedModel, ...chunkOptions } = {}) {
  const chunks = chunkDocs(await loadDocs(dir), chunkOptions).map((c) => ({ ...c, hash: hashOf(model, c.text) }));

  // Vectors from the previous index, keyed by content hash.
  const old = new Map();
  try {
    const previous = await VectorStore.load(indexPath);
    if (previous.model === model) previous.chunks.forEach((c, i) => old.set(c.hash, previous.vectors[i]));
  } catch {} // no index yet, or an unreadable one: start fresh

  const todo = chunks.filter((c) => !old.has(c.hash));
  const fresh = await embed(todo.map((c) => c.text), { model });
  todo.forEach((c, i) => old.set(c.hash, fresh[i]));

  const store = new VectorStore();
  store.model = model;
  store.add(chunks, chunks.map((c) => old.get(c.hash)));
  store.stats = { chunks: chunks.length, embedded: todo.length, reused: chunks.length - todo.length };
  await store.save(indexPath);
  return store;
}

Every chunk gets a hash: a short fingerprint of its model name and text (section 10 explains why). The stats object is a convenience so you can print how much work was really done. Run it:

// demo-index.js
import { stat } from "node:fs/promises";
import { buildIndex } from "./src/indexer.js";

console.time("build");
const store = await buildIndex("docs", { indexPath: "index.json" });
console.timeEnd("build");
console.log(store.stats);
console.log("size:", store.size, "chunks, dimensions:", store.dimensions, ", model:", store.model);
console.log("index.json:", (await stat("index.json")).size, "bytes");

node demo-index.js
node demo-index.js
# output
# build: 23.406ms
# { chunks: 33, embedded: 33, reused: 0 }
# size: 33 chunks, dimensions: 256 , model: mock-embed
# index.json: 30789 bytes
# build: 4.504ms
# { chunks: 33, embedded: 0, reused: 33 }
# size: 33 chunks, dimensions: 256 , model: mock-embed
# index.json: 30789 bytes

The stats, the 33 chunks and the file size are stable. The build: times vary by machine. The first run embedded all 33 chunks. The second run embedded 0: it read index.json, saw that every chunk's hash was already there, and reused the old vectors. Against a paid API, that second run is free.

6. Searching the Club Documents

Open the file to see what is stored. This is the start of index.json (shortened):

{"version":1,"model":"mock-embed","dimensions":256,"chunks":[{"id":"about.md#1","source":"about.md","heading":"About Mekong Code Club","text":"Mekong Code Club is a student coding club in Phnom Penh. ...","hash":"c00b554b8ec761d9"}, ...],"vectors":[[0,0,0,...], ...]}

Each chunk keeps its id, source, heading and text. These are the metadata you made in the chunking chapter, and later they become the citations in the answer.

Now search. We load the saved index (no re-embedding needed), embed each question with the same model, and print the top three:

// demo-search.js
import { VectorStore } from "./src/store.js";
import { embed } from "./src/embed.js";

const store = await VectorStore.load("index.json");

export async function show(question, options = {}) {
  const [q] = await embed([question], { model: store.model });
  const hits = store.search(q, options);
  console.log(`\n${question}  ${JSON.stringify({ ...options, filter: options.filter ? "fn" : undefined })}`);
  if (hits.length === 0) console.log("  (no hits)");
  for (const { chunk, score } of hits) {
    console.log(`  ${score.toFixed(3)}  ${chunk.id.padEnd(16)} ${chunk.heading}`);
  }
}

await show("How much does membership cost per term?", { k: 3 });
await show("How long can I borrow a laptop?", { k: 3 });
await show("How long does the Saturday workshop last?", { k: 3 });
await show("How long is each project presentation?", { k: 3 });
await show("Who do I talk to about a scholarship?", { k: 3 });
// output
//
// How much does membership cost per term?  {"k":3}
//   0.435  membership.md#1  Membership fee
//   0.217  membership.md#4  Refunds
//   0.143  membership.md#2  Scholarships
//
// How long can I borrow a laptop?  {"k":3}
//   0.480  faq.md#2         Do I need my own laptop?
//   0.293  laptops.md#1     Who can borrow
//   0.274  laptops.md#3     Rules
//
// How long does the Saturday workshop last?  {"k":3}
//   0.209  laptops.md#2     How long
//   0.186  projects.md#4    Sharing your work
//   0.167  schedule.md#1    Saturday workshops
//
// How long is each project presentation?  {"k":3}
//   0.316  projects.md#1    Term projects
//   0.292  projects.md#2    Showcase Day
//   0.192  conduct.md#2     Academic honesty
//
// Who do I talk to about a scholarship?  {"k":3}
//   0.340  membership.md#2  Scholarships
//   0.158  mentors.md#4     Mentor meeting
//   0.129  conduct.md#4     What happens if rules are broken

These results are stable on the mock. Let us read them honestly:

  • Fee and scholarship questions work well. The right chunk is first, with a clear gap to the next one.
  • The laptop question is only half right. The three results are about laptops, but the chunk with the answer (laptops.md#2, "How long", up to 7 days) is not among them. The mock matched the shared words "laptop" and "borrow", and short chunks score higher.
  • The workshop question has the wrong order. The right chunk, schedule.md#1, is third. A laptop chunk mentions "Saturday" and "workshop" too.
  • The project question is split. The answer (7 minutes) is in projects.md#2, which is second behind a more general chunk.

Real embeddings do better on paraphrases, but you should still expect imperfect rankings, and that is exactly why retrieval needs measuring. The next section shows the first knob you can turn.

7. Top-k: How Many Results?

k is how many top results you keep. Too small and you may cut off the right chunk; too large and you fill the prompt with noise, which costs money and can confuse the model. Here is where the correct chunk actually ranks for four questions, when we ask for all 33:

// demo-topk.js : where does the right chunk land, and how big must k be?
import { VectorStore } from "./src/store.js";
import { embed } from "./src/embed.js";

const store = await VectorStore.load("index.json");
const cases = [
  ["How long can I borrow a laptop?", "laptops.md#2"],
  ["When is the Saturday workshop?", "schedule.md#1"],
  ["How much does membership cost per term?", "membership.md#1"],
  ["Who do I talk to about a scholarship?", "membership.md#2"],
];
for (const [question, wanted] of cases) {
  const [q] = await embed([question]);
  const ranked = store.search(q, { k: store.size });
  const rank = ranked.findIndex((h) => h.chunk.id === wanted) + 1;
  console.log(`rank ${String(rank).padStart(2)}  (${wanted})  ${question}`);
}
// output
// rank  4  (laptops.md#2)  How long can I borrow a laptop?
// rank  3  (schedule.md#1)  When is the Saturday workshop?
// rank  1  (membership.md#1)  How much does membership cost per term?
// rank  1  (membership.md#2)  Who do I talk to about a scholarship?

With k = 1 you would find the answer for only two of the four questions. With k = 3 for three of four. With k = 4, all four. That is why the default in our store is k = 4: it is a trade-off, not a rule. In RAG a good k is usually small, a handful of chunks, because every chunk you add costs tokens and gives the model more to get distracted by. The right number depends on your chunk size, because k chunks of 200 tokens is a different prompt from k chunks of 1,000 tokens. You choose it by measuring (see the evaluation chapter), not by guessing.

8. Score Thresholds

Top-k always returns something, even when the documents have no answer. Ask about a topic the handbook does not cover and you still get the "closest" chunks. If you pass them to the model, you invite it to make something up. A score threshold (minScore) says: "if nothing is at least this similar, return nothing".

Let us look at the best score for questions with and without an answer:

// demo-threshold.js : what is the top score for answerable vs unanswerable questions?
import { VectorStore } from "./src/store.js";
import { embed } from "./src/embed.js";

const store = await VectorStore.load("index.json");
const questions = [
  ["answerable", "How much does membership cost per term?"],
  ["answerable", "How long can I borrow a laptop?"],
  ["answerable", "When is the Saturday workshop?"],
  ["answerable", "How many minutes does each team present?"],
  ["answerable", "Who is the club treasurer?"],
  ["unanswerable", "Does the club teach Rust?"],
  ["unanswerable", "Who is the vice president?"],
  ["unanswerable", "What is the Wi-Fi password?"],
  ["unanswerable", "What is the weather in Phnom Penh tomorrow?"],
];
for (const [kind, question] of questions) {
  const [q] = await embed([question]);
  const [best] = store.search(q, { k: 1 });
  console.log(kind.padEnd(13), best.score.toFixed(3), best.chunk.id.padEnd(16), question);
}
// output
// answerable    0.435 membership.md#1  How much does membership cost per term?
// answerable    0.480 faq.md#2         How long can I borrow a laptop?
// answerable    0.295 laptops.md#2     When is the Saturday workshop?
// answerable    0.522 projects.md#2    How many minutes does each team present?
// answerable    0.394 projects.md#4    Who is the club treasurer?
// unanswerable  0.378 projects.md#3    Does the club teach Rust?
// unanswerable  0.239 laptops.md#1     Who is the vice president?
// unanswerable  0.333 conduct.md#3     What is the Wi-Fi password?
// unanswerable  0.167 about.md#1       What is the weather in Phnom Penh tomorrow?

Look at the scores. The answerable questions score between 0.295 and 0.522. The unanswerable ones between 0.167 and 0.378. These ranges overlap: "Does the club teach Rust?" (0.378) scores higher than "When is the Saturday workshop?" (0.295). (Also notice that three of the "answerable" questions put the wrong chunk first: laptop, Saturday workshop and treasurer. The score says "this is similar", not "this is correct".) So what does a threshold do? Count how many questions survive at several values:

// demo-sweep.js : try several thresholds on the same questions
import { VectorStore } from "./src/store.js";
import { embed } from "./src/embed.js";

const store = await VectorStore.load("index.json");
const answerable = ["How much does membership cost per term?", "How long can I borrow a laptop?", "When is the Saturday workshop?", "How many minutes does each team present?", "Who is the club treasurer?"];
const unanswerable = ["Does the club teach Rust?", "Who is the vice president?", "What is the Wi-Fi password?", "What is the weather in Phnom Penh tomorrow?"];

async function best(question) {
  const [q] = await embed([question]);
  return store.search(q, { k: 1 })[0].score;
}
const a = await Promise.all(answerable.map(best));
const u = await Promise.all(unanswerable.map(best));

console.log("minScore  answerable kept   unanswerable wrongly kept");
for (const t of [0.1, 0.2, 0.3, 0.35, 0.4]) {
  const kept = a.filter((s) => s >= t).length;
  const wrong = u.filter((s) => s >= t).length;
  console.log(`${t.toFixed(2).padStart(8)}  ${kept} of ${a.length}            ${wrong} of ${u.length}`);
}
// output
// minScore  answerable kept   unanswerable wrongly kept
//     0.10  5 of 5            4 of 4
//     0.20  5 of 5            3 of 4
//     0.30  4 of 5            2 of 4
//     0.35  4 of 5            1 of 4
//     0.40  3 of 5            0 of 4

This is a trade-off, with no perfect value. Set the threshold low and you let the unanswerable questions through, so the model may be asked to answer from irrelevant text. Set it high and you reject real questions: at 0.40 we refuse two of the five answerable questions (because the correct chunk simply scores low) and we finally catch all four unanswerable ones.

What to take from this:

  • A threshold is specific to one embedding model, one chunking strategy and your documents. A value that works with one model is meaningless on another, since each model has its own score range. So you cannot copy "0.75" from a blog post.
  • Pick it by measuring on real questions: collect some answerable and some unanswerable ones and look at both groups, as above. The evaluation chapter turns this into code.
  • Treat the threshold as a cheap first filter, not a safety net. The prompt in the next chapter also tells the model to say "I don't know" when the passages do not contain the answer, so there are two layers.
  • With real embeddings the two groups usually separate better than the mock's do, but not perfectly. Expect some overlap and plan for it.

9. Metadata Filters

Sometimes you know where the answer must be. A user asks "in the laptop policy, how many days can I keep a laptop?". Or your app serves several courses, and a student should only search their own course. A metadata filter restricts the search to chunks whose metadata match. In our store the filter option is a function that sees each chunk:

// demo-filter.js
import { VectorStore } from "./src/store.js";
import { embed } from "./src/embed.js";

const store = await VectorStore.load("index.json");
const [q] = await embed(["How many days can I keep a club laptop?"]);
const print = (label, hits) => {
  console.log(label, hits.length === 0 ? "(no hits)" : "");
  for (const h of hits) console.log(`  ${h.score.toFixed(3)}  ${h.chunk.id.padEnd(16)} ${h.chunk.heading}`);
};

print("no filter:", store.search(q, { k: 3 }));
print("only laptops.md:", store.search(q, { k: 3, filter: (c) => c.source === "laptops.md" }));
print("only membership.md:", store.search(q, { k: 3, filter: (c) => c.source === "membership.md" }));
print("nothing allowed:", store.search(q, { k: 3, filter: () => false }));
// output
// no filter: 
//   0.495  laptops.md#3     Rules
//   0.372  faq.md#2         Do I need my own laptop?
//   0.298  about.md#1       About Mekong Code Club
// only laptops.md: 
//   0.495  laptops.md#3     Rules
//   0.280  laptops.md#2     How long
//   0.227  laptops.md#1     Who can borrow
// only membership.md: 
//   0.145  membership.md#1  Membership fee
//   0.108  membership.md#4  Refunds
//   0.098  membership.md#3  How to pay
// nothing allowed: (no hits)

The filter changed which chunks were even considered. Restricting to laptops.md removed the FAQ and "About" chunks, so the laptop chunks filled all three slots. (The mock still puts "Rules" ahead of "How long", the same word-matching problem as before.)

Use filters for things that are facts about the chunk: the source file, a language, a course, a date range, a document version, or who is allowed to see it. That last one is important: access control must happen inside the search, before any text reaches the model. If you retrieve a secret chunk and then ask the model to "not mention it", you have already leaked it into the prompt. The last chapter comes back to this.

Filtering after you take the top k is a common mistake. If you get the top 4 and then remove the forbidden ones, you may have nothing left. Our filter runs before scoring, so k always counts allowed chunks.

10. Only Re-embed What Changed

Your documents change. Someone edits the fee. A new file arrives. A chunk is deleted. You do not want to pay to embed 10,000 unchanged chunks because one paragraph changed.

The idea is a content hash: a short fingerprint calculated from the text (and the model name). If the text is the same, the hash is the same; if even one character changes, the hash changes. In buildIndex each chunk gets hash = sha256(model + "\n" + text) (we keep 16 characters of it). Then:

  • Same hash found in the old index -> reuse the old vector, no API call.
  • Hash not found -> embed it.
  • Old hash that no chunk has any more -> simply not copied. The new store is built from the current chunks, so deleted and edited-away text disappears from the index automatically.
  • Different model than the old index -> the hash includes the model, and the indexer ignores an index made by another model, so everything is re-embedded.

Test it in a copy of the docs, so the original stays untouched:

// demo-incremental.js : what does buildIndex re-embed after a change?
import { cp, readFile, writeFile } from "node:fs/promises";
import { buildIndex } from "./src/indexer.js";

await cp("docs", "docs-edit", { recursive: true });
const opts = { indexPath: "index-edit.json" };

console.log("1. first build        ", (await buildIndex("docs-edit", opts)).stats);
console.log("2. nothing changed    ", (await buildIndex("docs-edit", opts)).stats);

// Change one number in one paragraph.
const file = "docs-edit/membership.md";
await writeFile(file, (await readFile(file, "utf8")).replace("10 US dollars", "12 US dollars"));
console.log("3. fee edited         ", (await buildIndex("docs-edit", opts)).stats);

// Add a new file.
await writeFile("docs-edit/parking.md", "# Parking\n\n## Bicycles\n\nBicycles can be parked in the ground floor lobby.\n");
console.log("4. one file added     ", (await buildIndex("docs-edit", opts)).stats);

// Change the embedding model: every old vector is unusable.
console.log("5. new embedding model", (await buildIndex("docs-edit", { ...opts, model: "mock-embed-v2" })).stats);
// output
// 1. first build         { chunks: 33, embedded: 33, reused: 0 }
// 2. nothing changed     { chunks: 33, embedded: 0, reused: 33 }
// 3. fee edited          { chunks: 33, embedded: 1, reused: 32 }
// 4. one file added      { chunks: 34, embedded: 1, reused: 33 }
// 5. new embedding model { chunks: 34, embedded: 34, reused: 0 }

This output is stable. Editing one paragraph re-embedded exactly one chunk. Adding a one-chunk file re-embedded one. Changing the model re-embedded all 34. (The mock ignores the model name, so the "new" model gives the same numbers, but our code cannot know that, and with a real provider it would be a different vector space.) Delete docs-edit and index-edit.json afterwards.

Be aware of what this does not do. If you change the chunking settings (for example a new size), the chunk texts change, so most hashes change and most chunks are re-embedded. That is correct, because they really are new chunks. Also, our chunk ids like membership.md#2 are positions, so inserting a section near the top of a file renumbers the chunks below it. The vectors are reused (the hash is of the text), but if you stored ids somewhere else, for example in an evaluation file, check them after big edits. Using the heading in your notes is more stable than the number.

11. Safety Checks

The store protects you from the silent bugs we worried about:

// demo-errors.js
import { VectorStore } from "./src/store.js";

const store = new VectorStore();
store.add([{ id: "a" }], [[1, 0, 0]]);
for (const attempt of [
  () => store.add([{ id: "b" }], [[1, 0]]),       // wrong dimensions
  () => store.add([{ id: "c" }, { id: "d" }], [[0, 1, 0]]), // lengths differ
  () => store.search([1, 0, 0, 0]),                // query from another model
]) {
  try { attempt(); } catch (err) { console.log(err.message); }
}
console.log("still", store.size, "chunk in the store");
// output
// vector 0 has 2 dimensions, the store has 3
// got 2 chunks but 1 vectors
// query has 4 dimensions, the store has 3
// still 1 chunk in the store

Each mistake raises a clear error and the store stays unchanged. This catches two common causes of strange results: a query embedded with a different model than the index, and a half-finished batch. It cannot catch two different models with the same number of dimensions. For that, the index remembers its model name (store.model), and the indexer refuses to reuse an index made by another model. In your own app, compare store.model with the model you use for questions.

12. How Slow Is Brute Force?

"Compare with every vector" sounds expensive. Let us measure it with random vectors of 256 dimensions, the same as the mock. We build stores of different sizes and time one search:

// demo-speed.js : how slow is brute force? Random vectors, 256 dimensions.
import { VectorStore } from "./src/store.js";

const DIM = 256;
const randomVector = () => Array.from({ length: DIM }, () => Math.random() - 0.5);

for (const n of [1_000, 10_000, 100_000]) {
  const store = new VectorStore();
  const chunks = Array.from({ length: n }, (_, i) => ({ id: `c${i}` }));
  store.add(chunks, chunks.map(randomVector));
  const q = randomVector();
  store.search(q); // warm-up
  const t0 = performance.now();
  for (let i = 0; i < 5; i++) store.search(q, { k: 4 });
  const ms = (performance.now() - t0) / 5;
  console.log(`${String(n).padStart(7)} chunks: ${ms.toFixed(1)} ms per search`);
}
// output (the times VARY with your computer)
//    1000 chunks: 0.5 ms per search
//   10000 chunks: 3.2 ms per search
//  100000 chunks: 31.2 ms per search

These times are from one laptop run, and yours will differ. The pattern is what matters: time grows in step with the number of chunks. Ten times more chunks takes about ten times longer. For a club handbook, a school's notes or a company wiki of some thousands of chunks, brute force on a normal computer is far faster than the request to the language model, which takes a large part of a second or more.

Real vectors have more dimensions than 256 (more work per comparison), and memory is a limit too: the whole index sits in RAM as JavaScript arrays, and JSON.parse of a big index.json is slow and memory-hungry. So this design is for small to medium corpora. When does it stop being enough? Let us look at the alternative.

When there are millions of vectors, comparing against every one is too slow. Approximate nearest neighbour (ANN) search gives up a little exactness for a lot of speed. It builds an extra structure at ingest time so that a query only visits a small part of the data.

You do not need to implement one, but the idea of the popular approach, HNSW (hierarchical navigable small world graphs), is easy to picture. Imagine finding a shop in a big city with no map. You start at the nearest main road (a few well-known places far apart), drive toward the right district, then go to a smaller street, then walk the last streets. Each step makes you closer. HNSW links vectors into layers of such roads: the top layer has few points and long jumps, the bottom layer has all points and short jumps. A search enters at the top and keeps moving to the neighbour closest to the question, going down layer by layer.

Other methods exist too. Some group vectors into clusters and only search the few clusters nearest the question. Some compress vectors to take less memory, at the cost of some precision. They share the same trade-off, which is:

  • Faster and smaller than brute force on large data.
  • Approximate: occasionally the real best match is missed. Most systems let you tune how hard to look (more effort, better recall, slower).
  • More moving parts: an index to build, settings to tune, and often extra work when data is updated or deleted.

The words to remember are recall (what fraction of the true nearest neighbours the approximate search actually returns) and latency (how long a search takes). With a few thousand chunks you do not need this, and brute force gives exact answers with no tuning.

14. When You Need a Real Vector Database

Our JSON file store is a good teaching tool and fine for a prototype or a small private tool. You should think about something stronger when:

  • You have many thousands of chunks or more, and brute-force time or memory starts to hurt (measure, as in section 12).
  • Many users search at the same time. A plain script that loads the whole index in each process does not scale well.
  • Your data changes often: documents are added, edited and deleted all day, and you need those changes to appear quickly and safely.
  • You need filters at scale, such as per-user permissions, plus durability, backups and monitoring like any other production data.

There are several kinds of tools to look at (this is a list of categories, not a recommendation, and I am not comparing speed or quality):

  • A vector extension in a relational database. Your documents, users and vectors sit in one database, so filters and permissions are ordinary queries. This is attractive if you already run such a database.
  • A dedicated vector database: a server built for storing vectors, filtering and ANN search.
  • A search engine with vector support. It can mix keyword search and vector search in one place, which is useful for the hybrid search of the Improving Retrieval chapter.
  • A library or file-based store that runs inside your program, a step above a JSON file, without a separate server.

Whatever you choose, the interface stays the same as our store: add chunks with vectors and metadata, search with a vector, k and a filter, get chunks with scores. That is why VectorStore exposes only add, search, save and load. If you replace it later, the rest of the program, the prompt building and the answer code, does not have to change. Try to avoid choosing a heavy tool before you have measured that you need one. A system you can understand end to end is easier to fix.

15. Recipe: The store and indexer Modules

src/store.js is the file shown in section 3 and src/indexer.js is the file in section 5. Here they are together, as the final versions for this chapter. Nothing else in the project changes.

// src/store.js
import { readFile, writeFile } from "node:fs/promises";

const unit = (v) => {
  const len = Math.sqrt(v.reduce((s, x) => s + x * x, 0)) || 1;
  return v.map((x) => Number((x / len).toFixed(6))); // 6 decimals keeps index.json small
};

export class VectorStore {
  chunks = [];
  vectors = [];
  model = null; // which embedding model made these vectors (set by the indexer)

  get size() {
    return this.chunks.length;
  }

  get dimensions() {
    return this.vectors[0]?.length ?? 0;
  }

  add(chunks, vectors) {
    if (chunks.length !== vectors.length) {
      throw new Error(`got ${chunks.length} chunks but ${vectors.length} vectors`);
    }
    vectors.forEach((v, i) => {
      if (this.dimensions && v.length !== this.dimensions) {
        throw new Error(`vector ${i} has ${v.length} dimensions, the store has ${this.dimensions}`);
      }
      this.chunks.push(chunks[i]);
      this.vectors.push(unit(v)); // unit length: cosine becomes a plain dot product
    });
  }

  // Brute force: score every chunk, keep the best k.
  search(vector, { k = 4, minScore = 0, filter } = {}) {
    if (this.size && vector.length !== this.dimensions) {
      throw new Error(`query has ${vector.length} dimensions, the store has ${this.dimensions}`);
    }
    const q = unit(vector);
    const hits = [];
    for (let i = 0; i < this.size; i++) {
      if (filter && !filter(this.chunks[i])) continue;
      let score = 0;
      for (let d = 0; d < q.length; d++) score += q[d] * this.vectors[i][d];
      if (score >= minScore) hits.push({ chunk: this.chunks[i], score });
    }
    return hits.sort((a, b) => b.score - a.score).slice(0, k);
  }

  async save(path) {
    const data = { version: 1, model: this.model, dimensions: this.dimensions, chunks: this.chunks, vectors: this.vectors };
    await writeFile(path, JSON.stringify(data));
  }

  static async load(path) {
    const data = JSON.parse(await readFile(path, "utf8"));
    if (data.version !== 1) throw new Error(`unknown index version: ${data.version}`);
    const store = new VectorStore();
    store.model = data.model;
    store.add(data.chunks, data.vectors);
    return store;
  }
}

// src/indexer.js
import { createHash } from "node:crypto";
import { loadDocs } from "./load.js";
import { chunkDocs } from "./chunk.js";
import { embed } from "./embed.js";
import { embedModel } from "./config.js";
import { VectorStore } from "./store.js";

const hashOf = (model, text) => createHash("sha256").update(`${model}\n${text}`).digest("hex").slice(0, 16);

// load -> chunk -> embed -> store. Only text we have not seen before is sent to the API.
export async function buildIndex(dir, { indexPath = "index.json", model = embedModel, ...chunkOptions } = {}) {
  const chunks = chunkDocs(await loadDocs(dir), chunkOptions).map((c) => ({ ...c, hash: hashOf(model, c.text) }));

  // Vectors from the previous index, keyed by content hash.
  const old = new Map();
  try {
    const previous = await VectorStore.load(indexPath);
    if (previous.model === model) previous.chunks.forEach((c, i) => old.set(c.hash, previous.vectors[i]));
  } catch {} // no index yet, or an unreadable one: start fresh

  const todo = chunks.filter((c) => !old.has(c.hash));
  const fresh = await embed(todo.map((c) => c.text), { model });
  todo.forEach((c, i) => old.set(c.hash, fresh[i]));

  const store = new VectorStore();
  store.model = model;
  store.add(chunks, chunks.map((c) => old.get(c.hash)));
  store.stats = { chunks: chunks.length, embedded: todo.length, reused: chunks.length - todo.length };
  await store.save(indexPath);
  return store;
}

A small note on this design: index.json is generated from your documents, so do not edit it by hand, and it is normally not committed to version control (add it to .gitignore). Deleting it is always safe; the next buildIndex rebuilds it. The catch {} in the indexer hides a missing or corrupt file on purpose, since "start fresh" is the right response to both.

16. Try It Yourself

  1. Change k. In demo-search.js, ask the laptop question with k from 1 to 6. At which k does laptops.md#2 first appear? What would that mean for the token cost of your prompt?
  2. Find your own threshold. Add three more answerable and three more unanswerable questions to demo-sweep.js. Do the best scores still separate the two groups? Which value of minScore would you choose, and what would you give up?
  3. Filter by heading. Search "who can join" with a filter that keeps only chunks whose heading includes "Who". Then write a filter that keeps chunks from two files at once. What happens if filter returns undefined instead of false?
  4. Prove the hash. Edit one word in laptops.md inside a copy of docs and rebuild the index twice, printing store.stats. Then change the chunking with { strategy: "size", size: 300, overlap: 30 } and rebuild. How many chunks are embedded, and why is it not zero?
  5. Break the model rule. Build the index, then load it and search with a 3-number vector. Read the error message. Now write a helper searchText(store, question, options) that embeds the question with store.model and then searches, so callers cannot forget the model.
  6. Scale it. Change DIM in demo-speed.js to 1024 and n up to 200,000 (watch your memory). How does the time per search change? At what size would you want something other than brute force for your own use case, and why?

17. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Do I really need a vector database?

Not for a small project. For a few thousand chunks, a plain array and brute-force search on one machine is exact, fast and simple, and you can fully understand it. You need a vector database when the corpus grows large, many users search at once, or the data changes constantly and you need permissions, backups and monitoring. Measure first, then decide.

What is the difference between exact and approximate search?

Exact (brute force) compares your question with every vector, so it always finds the true closest ones, but takes longer as data grows. Approximate search builds a helper structure so it only checks a small part of the data, which is much faster on large collections, but it may sometimes miss a true best match. The trade-off is speed against accuracy.

What value should I use for the minimum score?

There is no universal number. Scores depend on the embedding model, the chunk size and your documents. Collect real questions, some that the documents can answer and some that they cannot, look at the best score for each group, and pick a value that keeps most good questions while refusing most bad ones. Check it again whenever you change the model or the chunking.

Why does the first result sometimes not contain the answer?

Similarity is not the same as "contains the answer". A chunk can be about the right topic and still not hold the fact you need, and short chunks that repeat the question's words can score high. That is why we retrieve several chunks (top-k), and why later chapters add keyword search and re-ranking. Always look at the actual retrieved text when debugging.

Why save the model name inside the index?

Because vectors from different models cannot be compared. If you rebuild your app with another embedding model and forget to re-index, the queries and the index live in different spaces and the results become random, usually without any error message. Saving the model name lets your code notice the mismatch instead of silently giving wrong answers.

Is the metadata filter a security feature?

It can be part of one, if you do it correctly: filter by permission before the chunks reach the model, on the server, using the identity of the logged-in user. It is not secure if the filter value comes from the browser, if you filter after retrieval, or if you rely on the model to hide things. Treat any text that enters the prompt as visible to the user.

How do I delete a document from the index?

In our design, remove the file and run buildIndex again. The new store is built only from current chunks, so the old ones are gone. Real vector databases have delete operations too, but check how they handle it, since some mark data as deleted and clean it up later. If privacy rules require you to erase something, verify it is really gone, including from backups and caches.

Where to go next. You now have a working retrieval layer: documents become chunks, chunks become vectors in a saved index, and a question finds its closest chunks with scores, a limit, a threshold and filters. In the next chapter, Retrieve and Generate, you connect it to the model: build the prompt from the retrieved chunks, make the answer cite its sources, refuse when nothing is relevant, and defend against a document that tries to give the model instructions.

មុនEmbeddings in Practiceបន្ទាប់Retrieve and Generate

ទិញកាហ្វេជូនខ្ញុំ

ចូលចិត្តមេរៀននេះមែនទេ? សូមពិចារណាទិញកាហ្វេជូនខ្ញុំមួយកែវ ដើម្បីគាំទ្រការបង្កើតមាតិកាឥតគិតថ្លៃបន្ថែមទៀត!