Embeddings in Practice

Table of Contents

1. What an Embedding Does in RAG
2. The Embeddings Endpoint
3. Reading the Response
4. Cosine Similarity: Scoring a Pair
5. The Honest Limit of the Mock's Embeddings
6. What a Real Embedding Model Does Differently
7. Version 1: One Text per Request
8. Version 2: Batching
9. Version 3: A Cache
10. The Cache Key: Text and Model
11. Same Model for Documents and Questions
12. Normalising: Why Length Does Not Matter
13. Embedding the Club Chunks
14. Cost and Size
15. Rate Limits and Retries
16. A Cache on Disk
17. Other Languages, Including Khmer
18. Recipe: The embed Module
19. Try It Yourself
20. Quick Review: Common Questions

1. What an Embedding Does in RAG

In Loading and Chunking you cut the club handbook into 33 chunks. Now we need a way to ask: "which chunk is about this question?" Comparing words one by one is fragile. People say "fee", "price" and "cost" for the same thing.

An embedding is a list of numbers that stands for a piece of text. The embedding model is trained so that texts with similar meaning get vectors that point in similar directions. You met the idea in Embeddings and Attention, where you made four-number vectors by hand. A real embedding model makes the numbers for you, from any text, with hundreds or thousands of dimensions.

Think of a library. Every book gets a shelf position that depends on its topic. To find books about "growing rice", you do not read every book. You walk to the right shelf and look nearby. Embeddings give every chunk a position, and search means "walk to where the question sits and see what is close".

In the RAG pipeline this chapter is the embed step:

ingest:  load -> chunk -> EMBED -> store
query:   EMBED question -> search -> prompt -> generate -> cite

The same embed() function is used twice: once for every chunk (at ingest time), and once for every question (at query time). Getting this function right, cheap and reliable is the goal of this chapter.

Setup. Keep working in your raglab project from the previous chapters. Start the mock server in a second terminal with node mock-llm.js. The examples here assume it listens on port 8787, the default. (If you need another port, start it with PORT=8850 node mock-llm.js and run your scripts with LLM_BASE_URL=http://localhost:8850.) src/config.js already has the new embedModel setting, read from LLM_EMBED_MODEL, with the default mock-embed.

2. The Embeddings Endpoint

Providers that offer embeddings use a call that looks a lot like the chat call: a POST to /v1/embeddings with a model name and the text. The input can be one string, or an array of strings to embed many texts in one request. Try it with curl. The vector has 256 numbers, so we only look at the end of the answer:

curl -s http://localhost:8787/v1/embeddings \
  -H "authorization: Bearer test-key" \
  -H "content-type: application/json" \
  -d '{"model":"mock-embed","input":["Membership costs 10 US dollars per term.","The room has 24 seats."]}' \
  | tail -c 130
# output
# 0,0,0,0,0,0,0,0,0,0,0.5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]}],"usage":{"prompt_tokens":16,"total_tokens":16}}

Now the same call from JavaScript. The code is the same shape as the chat call from the chatbot series:

// demo-bare.js
import { baseUrl, apiKey, embedModel } from "./src/config.js";

const res = await fetch(`${baseUrl}/v1/embeddings`, {
  method: "POST",
  headers: { "content-type": "application/json", authorization: `Bearer ${apiKey}` },
  body: JSON.stringify({ model: embedModel, input: "Membership costs 10 US dollars per term." }),
});
const data = await res.json();

console.log(Object.keys(data));
console.log(data.model, data.data.length, data.data[0].index);
console.log("dimensions:", data.data[0].embedding.length);
console.log("first 8 numbers:", data.data[0].embedding.slice(0, 8));
console.log("non-zero slots:", data.data[0].embedding.filter((x) => x !== 0).length);
console.log(data.usage);
// output
// [ 'model', 'data', 'usage' ]
// mock-embed 1 0
// dimensions: 256
// first 8 numbers: [
//   0, 0, 0, 0,
//   0, 0, 0, 0
// ]
// non-zero slots: 6
// { prompt_tokens: 10, total_tokens: 10 }

All of this output is stable: the mock gives the same vector for the same text every time. (A real model usually does too, up to tiny rounding differences, but check the documentation of yours.)

3. Reading the Response

Look at what came back:

• data is an array, one item per input text. Each item has an index (its position in your input array) and the embedding. Real providers can return the items in a different order than you sent them, so always sort by index or use it to place the vector. Our code will do this.
• embedding is a list of numbers. Its length is the number of dimensions. The mock has 256. Real models use their own fixed number, often hundreds or a few thousand, and it is written in the model's documentation. All vectors from one model have the same length.
• usage tells you how many tokens you were charged for. Embeddings are billed by input tokens, with no output tokens.

Now the honest part about the mock. Its vectors are not learned. The server does this: it lowercases the text, drops common words ("the", "is", "how"), cuts a trailing plural "s", then hashes each remaining word into one of 256 slots and counts. Finally it divides by the length, so the vector has length 1. That is why our sentence about membership has exactly 6 non-zero slots: after "us" is dropped as a common word, it has 6 content words (membership, cost, 10, dollar, per, term), one slot each.

This is a bag of words. It is fast and free and perfect for a tutorial, because you can predict its behaviour. But it can only match shared words. Section 5 shows what that means.

4. Cosine Similarity: Scoring a Pair

To compare two vectors we use cosine similarity, which measures the angle between them. You built it in the previous series; the project has it in src/embed.js as cosine(a, b). The score means:

• 1.0: the vectors point in exactly the same direction (very similar).
• 0.0: they are unrelated (at right angles).
• Negative: they point away from each other. This is possible with real models, but the mock never produces it (it only counts words).

Let us score a question against two sentences. One answers the question, one does not:

// demo-cosine.js
import { embed, cosine } from "./src/embed.js";

const [q, fee, seats] = await embed([
  "How much does membership cost per term?",
  "Membership costs 10 US dollars per term.",
  "The room has 24 seats and a projector.",
]);
console.log("question vs fee sentence :", cosine(q, fee).toFixed(3));
console.log("question vs seats sentence:", cosine(q, seats).toFixed(3));
console.log("same text vs itself       :", cosine(fee, fee).toFixed(3));
const norm = (v) => Math.sqrt(v.reduce((s, x) => s + x * x, 0));
console.log("length of a vector        :", norm(fee).toFixed(3));
// output
// question vs fee sentence : 0.730
// question vs seats sentence: 0.000
// same text vs itself       : 1.000
// length of a vector        : 1.000

The fee sentence scores high (0.730), the seats sentence scores 0. A text is always 1.000 against itself. The last line shows the vectors have length 1, which matters in section 12.

The exact numbers here depend on the mock's word list. Treat them as stable on the mock, but not meaningful in general: you can never say "0.73 is a good score" for all models. Each model has its own range.

5. The Honest Limit of the Mock's Embeddings

Here is the experiment that matters most in this chapter. We compare pairs of words and sentences that mean the same thing, using different words:

// demo-caveat.js
import { embed, cosine } from "./src/embed.js";

const pairs = [
  ["fee", "fee"],
  ["fee", "price"],
  ["fee", "cost"],
  ["How much is the fee?", "The fee is 10 US dollars."],
  ["How much is the price?", "The fee is 10 US dollars."],
  ["laptop", "computer"],
  ["laptops", "laptop"],
];
for (const [a, b] of pairs) {
  const [va, vb] = await embed([a, b]);
  console.log(cosine(va, vb).toFixed(3).padStart(6), " ", JSON.stringify(a), "vs", JSON.stringify(b));
}
// output
//  1.000   "fee" vs "fee"
//  0.000   "fee" vs "price"
//  0.000   "fee" vs "cost"
//  0.408   "How much is the fee?" vs "The fee is 10 US dollars."
//  0.000   "How much is the price?" vs "The fee is 10 US dollars."
//  0.000   "laptop" vs "computer"
//  1.000   "laptops" vs "laptop"

Read it line by line:

• fee vs fee is 1.000. Same word, same slot.
• fee vs price and fee vs cost are 0.000. Any person knows these mean nearly the same. The mock cannot know, because it never learned meaning, only which slot a spelling lands in.
• laptop vs computer is 0.000 for the same reason.
• laptops vs laptop is 1.000, but only because the server strips a trailing "s". That is a hand-written rule, not understanding.
• The question "How much is the price?" scores 0 against the sentence that says "fee" and "10 US dollars". A user who asks about the price would get nothing from our handbook.

Say this plainly to anyone you teach: the mock matches shared words, not meaning. It is a stand-in so that you can run everything for free. It is also deliberately limited, so that you feel the problem that real embeddings solve.

6. What a Real Embedding Model Does Differently

A real embedding model is a neural network trained on a huge amount of text, so that texts used in similar situations end up with similar vectors. In practice you can expect:

• Synonyms and paraphrases match. "How much is the price to join?" should land close to "Membership costs 10 US dollars per term", even though almost no words are shared.
• Word order and context count a little. "dog bites man" and "man bites dog" use the same words but are not identical to a good model.
• Some models handle many languages in one shared space, so a question in one language can find a passage in another. Section 17 comes back to this.
• Scores are never exactly 0 or 1. Unrelated texts often still get a small positive score, and the typical range differs from model to model.

It is still not magic. A real model can also be fooled: it may rank a chunk that talks about the topic above the chunk that contains the answer, and it may be weak on names, product codes and numbers. That is why Improving Retrieval adds keyword search next to embeddings. We will not claim numbers for any specific model here. The only honest way to know how a model works on your documents is to measure it, which is the job of the evaluation chapter.

Everything you build in this series works with a real model. You only change three environment variables: LLM_BASE_URL, LLM_API_KEY and LLM_EMBED_MODEL, plus whatever name your provider gives the embedding model. The code does not change, as long as the provider's embeddings endpoint accepts the same request shape. Many do, but read the documentation.

7. Version 1: One Text per Request

Now build embed() step by step. The simplest version embeds one text per request, using the same fetch as before and the readError helper from src/client.js:

// src/embed-v1.js : the simplest possible version, one text per request
import { baseUrl, apiKey, embedModel } from "./config.js";
import { readError } from "./client.js";

export async function embedOne(text, model = embedModel) {
  const res = await fetch(`${baseUrl}/v1/embeddings`, {
    method: "POST",
    headers: { "content-type": "application/json", authorization: `Bearer ${apiKey}` },
    body: JSON.stringify({ model, input: text }),
  });
  if (!res.ok) throw await readError(res);
  const data = await res.json();
  return data.data[0].embedding;
}

It works, but each call is a full round trip over the network. To embed 33 chunks you would make 33 requests. For 33 chunks on your own machine that is fast. For 30,000 chunks against a real server it is slow, and it may hit rate limits.

8. Version 2: Batching

The endpoint accepts an array. So send many texts in one request. This is called batching. Two details:

• Providers limit how many texts, and how many tokens, one request may hold. So we send batches of a fixed size (we use 64) instead of everything at once. Check your provider's limit and lower batchSize if needed.
• We sort the response by index, so the vectors line up with the texts even if the server reorders them.

// src/embed-v2.js : batching, many texts per request
import { baseUrl, apiKey, embedModel } from "./config.js";
import { readError } from "./client.js";

async function embedBatch(texts, model) {
  const res = await fetch(`${baseUrl}/v1/embeddings`, {
    method: "POST",
    headers: { "content-type": "application/json", authorization: `Bearer ${apiKey}` },
    body: JSON.stringify({ model, input: texts }),
  });
  if (!res.ok) throw await readError(res);
  const data = await res.json();
  return data.data.sort((a, b) => a.index - b.index).map((d) => d.embedding);
}

export async function embedMany(texts, { model = embedModel, batchSize = 64 } = {}) {
  const vectors = [];
  for (let i = 0; i < texts.length; i += batchSize) {
    vectors.push(...(await embedBatch(texts.slice(i, i + batchSize), model)));
  }
  return vectors;
}

9. Version 3: A Cache

Version 2 still has a waste. Run your ingest script twice and you pay for the same 33 chunks twice. Also, many chunks repeat (a footer, a copied paragraph), and users ask the same question again and again.

A cache remembers answers we already have. Ours is a Map from the text to its vector. Before calling the API, we look at which texts are missing, embed only those, and then build the result from the cache. Duplicates are removed with a Set so the same text is sent at most once per call. This is the final embed() in the recipe in section 18. First let us measure all three versions on the real club chunks:

// demo-requests.js : count HTTP requests for each version
import { loadDocs } from "./src/load.js";
import { chunkDocs } from "./src/chunk.js";
import { embedOne } from "./src/embed-v1.js";
import { embedMany } from "./src/embed-v2.js";
import { embed } from "./src/embed.js";

let requests = 0;
const realFetch = globalThis.fetch;
globalThis.fetch = (...args) => { requests += 1; return realFetch(...args); };

const chunks = chunkDocs(await loadDocs("docs"));
const texts = chunks.map((c) => c.text);
console.log("chunks:", texts.length);

async function run(label, fn) {
  requests = 0;
  const t0 = performance.now();
  const vectors = await fn();
  const ms = Math.round(performance.now() - t0);
  console.log(`${label.padEnd(22)} ${vectors.length} vectors, ${requests} requests, ${ms} ms`);
}

await run("v1 one by one", async () => {
  const out = [];
  for (const t of texts) out.push(await embedOne(t));
  return out;
});
await run("v2 batches of 10", () => embedMany(texts, { batchSize: 10 }));
await run("v2 batches of 64", () => embedMany(texts, { batchSize: 64 }));
await run("v3 cache, first time", () => embed(texts));
await run("v3 cache, second time", () => embed(texts));
await run("v3 cache, one new text", () => embed([...texts, "A brand new sentence."]));
// output
// chunks: 33
// v1 one by one          33 vectors, 33 requests, 38 ms
// v2 batches of 10       33 vectors, 4 requests, 3 ms
// v2 batches of 64       33 vectors, 1 requests, 2 ms
// v3 cache, first time   33 vectors, 1 requests, 2 ms
// v3 cache, second time  33 vectors, 0 requests, 0 ms
// v3 cache, one new text 34 vectors, 1 requests, 0 ms

The request counts are stable: 33, 4, 1, 1, 0, 1. The milliseconds vary on every run and machine. Against a real server over the internet, each request costs far more than the 1 ms the mock spends, so the difference between 33 requests and 1 is much larger.

The second-to-last line is the big one: the second call costs zero requests. The last line shows that one new text sends just one request, because the 33 old ones come from the cache.

10. The Cache Key: Text and Model

A cache must not return a wrong answer. Our key is `${model}\n${text}` : the model name, a newline, then the text. It must include the model. A vector from model A is not valid for model B (they live in different "spaces"), so the same text under two models needs two cache entries. Look at what happens when the model or the text changes:

// demo-cache-key.js
import { embed } from "./src/embed.js";

let requests = 0;
const realFetch = globalThis.fetch;
globalThis.fetch = (...args) => { requests += 1; return realFetch(...args); };

const text = ["Membership costs 10 US dollars per term."];
await embed(text);                                  console.log("first call           ->", requests, "request");
await embed(text);                                  console.log("same text, same model ->", requests, "requests in total");
await embed(text, { model: "mock-embed-v2" });      console.log("same text, new model  ->", requests, "requests in total");
await embed(["Membership costs 12 US dollars per term."]); console.log("edited text          ->", requests, "requests in total");

try {
  await embed([""]);
} catch (err) {
  console.log(err.name, err.status, err.message);
}
// output
// first call           -> 1 request
// same text, same model -> 1 requests in total
// same text, new model  -> 2 requests in total
// edited text          -> 3 requests in total
// ApiError 400 model and a non-empty string (or array of strings) input are required

This is also the rule for your stored index in the next chapter:

• If a chunk's text changes (the fee goes from 10 to 12 dollars), its old vector is stale. Embed it again.
• If you change the embedding model, every vector is stale. You must re-embed the whole corpus. (A model with a new name, a new version, or even a different dimension setting counts as a new model.)
• If the text did not change and the model did not change, do not pay again.

The last lines also show an edge case: an empty string is rejected with a 400 error. Real providers do this too. Skip empty chunks before you embed, or filter them in the chunker.

11. Same Model for Documents and Questions

This is the most important rule: use the same embedding model for the chunks and for the questions. Each model builds its own "map". A vector from one model has no meaning on another model's map, as if you looked up a street in Phnom Penh using the coordinates of a map of Siem Reap.

The scary part is that nothing crashes. If two models have the same number of dimensions, cosine still returns a number, and the results are just random. If the dimensions differ, JavaScript still does not complain:

// demo-guard.js : what happens if vectors from two different models meet?
import { cosine } from "./src/embed.js";

const fromModelA = [0.1, 0.2, 0.3];       // pretend: a 3-dimension model
const fromModelB = [0.5, 0.1, 0.2, 0.9];  // pretend: a 4-dimension model
console.log("cosine, different sizes:", cosine(fromModelA, fromModelB));

// A tiny guard you can put in front of any comparison.
function safeCosine(a, b) {
  if (a.length !== b.length) throw new Error(`dimension mismatch: ${a.length} vs ${b.length}. Same model for documents and queries?`);
  return cosine(a, b);
}
try { safeCosine(fromModelA, fromModelB); } catch (e) { console.log(e.message); }
// output
// cosine, different sizes: 0.6343350474165464
// dimension mismatch: 3 vs 4. Same model for documents and queries?

cosine() printed a perfectly normal-looking number for a meaningless comparison. A silent wrong answer is the worst kind of bug. In the next chapter the VectorStore will check dimensions and refuse bad input, and the saved index will remember which model made it. Make that a habit: store the model name next to the vectors.

Be careful with one more trap. If your provider has separate models or modes for documents and for queries (some do), follow its documentation for which to use where. The rule "same model family, used as documented" is what counts.

12. Normalising: Why Length Does Not Matter

A vector has a direction and a length. Cosine similarity only looks at the direction. That is useful: a long chunk and a short chunk about the same topic should still match.

Many embedding models return vectors already normalised to length 1. Then cosine similarity equals the plain dot product (multiply matching numbers, add them up), which is cheaper to compute. Check it:

// demo-normalise.js
import { embed, cosine } from "./src/embed.js";

const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);
const length = (v) => Math.sqrt(dot(v, v));

const [a, b] = await embed(["Membership costs 10 US dollars per term.", "How much does membership cost per term?"]);
console.log("length of a:", length(a).toFixed(4), " length of b:", length(b).toFixed(4));
console.log("dot product:", dot(a, b).toFixed(6));
console.log("cosine     :", cosine(a, b).toFixed(6));

// Make b 5 times longer. Direction is the same, so cosine must not change.
const bigB = b.map((x) => x * 5);
console.log("after scaling b by 5 -> dot:", dot(a, bigB).toFixed(6), " cosine:", cosine(a, bigB).toFixed(6));
// output
// length of a: 1.0000  length of b: 1.0000
// dot product: 0.730297
// cosine     : 0.730297
// after scaling b by 5 -> dot: 3.651484  cosine: 0.730297

For unit-length vectors, dot and cosine are the same number. Make the vector 5 times longer and the dot product jumps to 3.65, while cosine stays 0.730297. So: if you are not sure your model normalises, use cosine() (it divides by the lengths), or normalise the vectors yourself once when you store them. The VectorStore in the next chapter does the second one, so it can use the fast dot product at search time.

13. Embedding the Club Chunks

Time to use it for real. We embed all 33 chunks, then rank them against a few questions with cosine similarity, printing the top 3. (In the next chapter we move this ranking into a proper VectorStore.)

// demo-search.js : embed every chunk, rank them for a question
import { loadDocs } from "./src/load.js";
import { chunkDocs } from "./src/chunk.js";
import { embed, cosine } from "./src/embed.js";

const chunks = chunkDocs(await loadDocs("docs"));
const vectors = await embed(chunks.map((c) => c.text));
console.log(`${vectors.length} vectors x ${vectors[0].length} dimensions`);

async function top(question, n = 3) {
  const [q] = await embed([question]);
  const ranked = chunks
    .map((chunk, i) => ({ chunk, score: cosine(q, vectors[i]) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, n);
  console.log(`\n${question}`);
  for (const { chunk, score } of ranked) {
    console.log(`  ${score.toFixed(3)}  ${chunk.source} > ${chunk.heading}`);
  }
}

await top("How much does membership cost?");
await top("How long can I borrow a laptop?");
await top("When is the Saturday workshop?");
await top("How much is the price to join?");
await top("What is the Wi-Fi password?");
// output
// 33 vectors x 256 dimensions
//
// How much does membership cost?
//   0.187  membership.md > Membership fee
//   0.167  faq.md > Is the club connected to a school?
//   0.149  conduct.md > Be kind
//
// How long can I borrow a laptop?
//   0.480  faq.md > Do I need my own laptop?
//   0.293  laptops.md > Who can borrow
//   0.274  laptops.md > Rules
//
// When is the Saturday workshop?
//   0.295  laptops.md > How long
//   0.263  projects.md > Sharing your work
//   0.236  schedule.md > Saturday workshops
//
// How much is the price to join?
//   0.154  about.md > Who can join
//   0.120  faq.md > How do I contact the club?
//   0.099  schedule.md > Holidays
//
// What is the Wi-Fi password?
//   0.333  conduct.md > Safety online
//   0.289  about.md > About Mekong Code Club
//   0.281  membership.md > Membership fee

This output is stable on the mock. Read it with a critical eye, because it teaches a lot about toy embeddings:

• Fee question: the right chunk is first, but its score is low (0.187), and the second and third places are random chunks. Chunks with many words dilute a few matching words.
• Laptop question: the best chunk is the FAQ entry "Do I need my own laptop?", because it is short and has the word "laptop" prominently. The chunk with the real answer ("How long", up to 7 days) is not even in the top 3. Shared words beat the answer.
• Saturday workshop: the right chunk (schedule.md > Saturday workshops) is only third. A laptop chunk that happens to say "Saturday ... workshop" came first.
• "price to join": the mock cannot connect "price" with "fee". The membership chunk is not in the top 3 at all (section 5).
• Wi-Fi password: this is not in the documents, yet the top chunk has a score of 0.333, higher than the correct chunk got for the fee question. A high score does not mean the answer is there. Keep this in mind for score thresholds in the next chapter.

None of this is a bug in your code. It is what a bag-of-words model does, and a real embedding model would do better on the synonym and paraphrase cases. But you will still see rankings that are slightly wrong with real models too, and the habit of looking at the top results and their scores is how you find problems.

14. Cost and Size

Embedding is cheap compared with generating text, but it is not free with a paid provider, and it adds up on a large corpus. The mock tells you how many tokens the 33 chunks use (it estimates tokens as characters divided by 4, a rough rule of thumb). We use an example price, not a real one:

// demo-cost.js
import { loadDocs } from "./src/load.js";
import { chunkDocs } from "./src/chunk.js";
import { baseUrl, apiKey, embedModel } from "./src/config.js";

const chunks = chunkDocs(await loadDocs("docs"));
const res = await fetch(`${baseUrl}/v1/embeddings`, {
  method: "POST",
  headers: { "content-type": "application/json", authorization: `Bearer ${apiKey}` },
  body: JSON.stringify({ model: embedModel, input: chunks.map((c) => c.text) }),
});
const { usage } = await res.json();
const chars = chunks.reduce((n, c) => n + c.text.length, 0);
console.log("chunks:", chunks.length, " characters:", chars);
console.log("tokens reported by the server:", usage.total_tokens);

// EXAMPLE price only. Look up your provider's real price.
const PRICE_PER_MILLION = 0.02; // US dollars per 1,000,000 tokens (example)
const cost = (tokens) => (tokens / 1_000_000) * PRICE_PER_MILLION;
console.log("this corpus      :", cost(usage.total_tokens).toFixed(6), "USD");
const tokensPerChunk = usage.total_tokens / chunks.length;
for (const n of [1_000, 100_000, 1_000_000]) {
  console.log(`${String(n).padStart(9)} chunks:`, `${(n * tokensPerChunk).toLocaleString("en-US", { maximumFractionDigits: 0 })} tokens`, "->", cost(n * tokensPerChunk).toFixed(2), "USD");
}
// output
// chunks: 33  characters: 6148
// tokens reported by the server: 1547
// this corpus      : 0.000031 USD
//      1000 chunks: 46,879 tokens -> 0.00 USD
//    100000 chunks: 4,687,879 tokens -> 0.09 USD
//   1000000 chunks: 46,878,788 tokens -> 0.94 USD

(The token counts are the mock's own estimate and are stable. The price is an example I made up to show the arithmetic. Real prices differ by provider and model and change over time. Look them up.)

What to learn from this:

• Cost is tokens x price. Tokens depend on how much text you embed, so chunk overlap has a cost: with 20% overlap, you embed about 20% more text.
• You pay to embed each chunk once, if you cache and only re-embed changed text. The cache from section 9 is also a cost control.
• You also pay a little for every question, which is a short text. That is usually small next to the cost of generating the answer.
• Some providers offer free or local embedding models you run on your own computer. That is an option if privacy or cost is a problem, but quality and speed depend on the model and your hardware.

Vectors also take space. Each number is typically stored as a 4-byte float. As an example calculation: 1,000,000 chunks x 1,024 dimensions x 4 bytes is about 4 GB. Our toy index (33 chunks, 256 dimensions) is only a few tens of kilobytes. In the next chapter you will see why that matters for choosing where to store vectors.

15. Rate Limits and Retries

Providers limit how fast you can call them. When you go over, you get 429 Too Many Requests, often with a Retry-After header. Servers also have bad moments (5xx errors). In the chatbot series you saw how readError turns a failed response into an ApiError with status and retryAfter. For a big ingest you want retries with a growing wait (called exponential backoff): wait a little, then twice as long, then four times as long.

The mock does not rate-limit embeddings, so we test the logic with a fake function that fails twice with a 429 and then succeeds:

// demo-retry.js
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// Retry on rate limits (429) and server errors (5xx), with growing waits.
export async function withRetry(fn, { tries = 4, baseMs = 200 } = {}) {
  for (let attempt = 1; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const retryable = err.status === 429 || err.status >= 500;
      if (!retryable || attempt === tries) throw err;
      const wait = err.retryAfter ? err.retryAfter * 1000 : baseMs * 2 ** (attempt - 1);
      console.log(`attempt ${attempt} failed (${err.status}), waiting ${wait} ms`);
      await sleep(wait);
    }
  }
}

// Test it with a fake call that fails twice with 429, then works.
let calls = 0;
const flaky = async () => {
  calls += 1;
  if (calls <= 2) throw Object.assign(new Error("slow down"), { status: 429 });
  return "vectors!";
};
console.log(await withRetry(flaky));
console.log("total calls:", calls);
// output
// attempt 1 failed (429), waiting 200 ms
// attempt 2 failed (429), waiting 400 ms
// vectors!
// total calls: 3

To use it, wrap the batch call, for example await withRetry(() => embedBatch(batch, model)). Retry only errors that can go away (429, 5xx). A 400 or 401 will fail again, so it should stop at once. The Retry-After header, when the server sends it, is better than your own guess, so the code prefers it. Our main embed() keeps things small and does not include this wrapper, but add it before you ingest a large corpus from a paid API.

Two more habits for big ingests: send batches one after another (as our loop does), not 500 requests at the same time, and print progress so you can see the job is alive.

16. A Cache on Disk

The cache in embed() lives in memory, so it disappears when the program ends. Run the ingest again tomorrow and you pay again. A simple fix is to save the cache in a file. Here is a small wrapper, kept apart from embed() so you can see the idea. The key is a hash of the model and the text, so the file does not repeat long texts as keys:

// src/disk-cache.js (idea): remember vectors between runs
import { readFile, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import { embed } from "./embed.js";
import { embedModel } from "./config.js";

const key = (model, text) => createHash("sha256").update(`${model}\n${text}`).digest("hex");

export async function embedWithDiskCache(texts, { path = "embed-cache.json", model = embedModel } = {}) {
  let cache = {};
  try { cache = JSON.parse(await readFile(path, "utf8")); } catch {} // no file yet: start empty

  const missing = texts.filter((t) => !(key(model, t) in cache));
  if (missing.length > 0) {
    const vectors = await embed(missing, { model });
    missing.forEach((t, i) => { cache[key(model, t)] = vectors[i]; });
    await writeFile(path, JSON.stringify(cache));
  }
  return { vectors: texts.map((t) => cache[key(model, t)]), embedded: missing.length };
}

// demo-disk-cache.js
import { loadDocs } from "./src/load.js";
import { chunkDocs } from "./src/chunk.js";
import { embedWithDiskCache } from "./src/disk-cache.js";

const texts = chunkDocs(await loadDocs("docs")).map((c) => c.text);
const { vectors, embedded } = await embedWithDiskCache(texts);
console.log(`vectors: ${vectors.length}, newly embedded this run: ${embedded}`);

node demo-disk-cache.js
node demo-disk-cache.js
# output
# vectors: 33, newly embedded this run: 33
# vectors: 33, newly embedded this run: 0

The second run paid nothing. This is a teaching sketch: it rewrites the whole file each time and never removes old entries, so it is fine for a small project and not for a big one. You do not need it in raglab, because in the next chapter the index file itself keeps the vectors together with a fingerprint of each chunk's text and model, which gives you the same saving in a better place. Delete embed-cache.json when you are done with this experiment.

17. Other Languages, Including Khmer

Your users may ask in Khmer about documents written in English, or the other way around. Whether this works depends on the model:

• A multilingual embedding model is trained on many languages and places the same idea close together whatever the language. Then a Khmer question can find an English chunk. Whether a particular model supports Khmer, and how well, is something you must check and test. Do not assume. Many models list their supported languages in their documentation, and Khmer is often covered less well than large languages.
• An English-only model will give poor, noisy results for Khmer text.
• Chunking matters too (see the chunking chapter): Khmer has no spaces between words, so use character-based sizes.

The mock cannot do any of this. Here is what it does with a Khmer question about the fee, against the same question in English:

// demo-multilingual.js
import { embed, cosine } from "./src/embed.js";

const [en, km, kmSame, other] = await embed([
  "How much is the membership fee?",
  "តើថ្លៃសមាជិកភាពប៉ុន្មាន?",
  "តើថ្លៃសមាជិកភាពប៉ុន្មាន?",
  "ម៉ោងប៉ុន្មានដែលសិក្ខាសាលាចាប់ផ្តើម?",
]);
console.log("English vs Khmer (same question):", cosine(en, km).toFixed(3));
console.log("Khmer vs the identical Khmer      :", cosine(km, kmSame).toFixed(3));
console.log("Khmer question vs other Khmer text:", cosine(km, other).toFixed(3));
console.log("non-zero slots in the Khmer vector:", km.filter((x) => x !== 0).length);
// output
// English vs Khmer (same question): 0.000
// Khmer vs the identical Khmer      : 1.000
// Khmer question vs other Khmer text: 0.441
// non-zero slots in the Khmer vector: 9

Read this carefully. The English and Khmer versions of the same question score 0.000, as expected for a word-matching toy. But the Khmer question scores 0.441 against a different Khmer question about the workshop time. That is accidental: the mock's word splitter cuts Khmer text at the vowel and sign marks, so it sees little fragments, and the two sentences share some. This is noise, not meaning. To test cross-language search you need a real multilingual model, and the way to know if it is good enough for Khmer is a small test set of Khmer questions with known answers, as in the evaluation chapter.

Also remember that the model that writes the answer must handle the language you want, and the chunks it reads must be in a language it understands. Retrieval working across languages does not by itself guarantee a good Khmer answer.

18. Recipe: The embed Module

Here is the full final src/embed.js. It uses config.js and client.js from the chatbot series, and it has no other dependencies:

// src/embed.js
import { baseUrl, apiKey, embedModel } from "./config.js";
import { readError } from "./client.js";

const cache = new Map(); // text -> vector, so we never pay twice for the same text

async function embedBatch(texts, model) {
  const res = await fetch(`${baseUrl}/v1/embeddings`, {
    method: "POST",
    headers: { "content-type": "application/json", authorization: `Bearer ${apiKey}` },
    body: JSON.stringify({ model, input: texts }),
  });
  if (!res.ok) throw await readError(res);
  const data = await res.json();
  return data.data.sort((a, b) => a.index - b.index).map((d) => d.embedding);
}

// embed(["text", ...]) -> [[number, ...], ...] in the same order.
export async function embed(texts, { model = embedModel, batchSize = 64 } = {}) {
  const missing = [...new Set(texts.filter((t) => !cache.has(`${model}\n${t}`)))];
  for (let i = 0; i < missing.length; i += batchSize) {
    const batch = missing.slice(i, i + batchSize);
    const vectors = await embedBatch(batch, model);
    batch.forEach((t, j) => cache.set(`${model}\n${t}`, vectors[j]));
  }
  return texts.map((t) => cache.get(`${model}\n${t}`));
}

export const cosine = (a, b) => {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2; }
  return dot / (Math.sqrt(na) * Math.sqrt(nb) || 1);
};

The files src/embed-v1.js, src/embed-v2.js and src/disk-cache.js were only for learning. You can delete them, or keep them as notes. Nothing else in the series imports them.

19. Try It Yourself

1. Prove the cache key. Change embed() so the cache key leaves out the model (use only t). Then run demo-cache-key.js. Which line of the output changes, and why is that dangerous when you switch to a real provider?
2. Find a collision. The mock puts every word in one of 256 slots, so two different words can share a slot. Write a loop that embeds many random words and finds two different words with cosine 1.0. What does this tell you about 256 dimensions for a real vocabulary?
3. Batch sizes. Run embedMany with batch sizes 1, 5, 10, 33 and 100 on the club chunks and count requests. Which sizes give the same request count, and why? Then guess how a provider's per-request token limit should change your choice.
4. Worst question. Write five questions about the club that you think a person finds easy. Rank the chunks for each as in section 13 and note where the right chunk lands. Which questions fail on the mock, and which shared word would fix each one?
5. Cost sheet. Using the token count from demo-cost.js and an example price of your choice, estimate the cost of embedding a 10,000-page company handbook (assume 400 tokens per page). Then recompute it with 20% chunk overlap. Label your price as an example.
6. Retry for real. Wrap embedBatch with withRetry, then write a small fake server (use node:http) that answers 429 with Retry-After: 1 the first time and 200 after. Confirm your code waits about one second.

20. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now turn any text into a vector and score how close two vectors are, with batching and a cache to keep it cheap. In the next chapter, Vector Search, you move the ranking loop into a proper VectorStore, save the index to a file, re-embed only what changed, and learn about top-k, score thresholds, metadata filters and when you need a real vector database.