Loading and Chunking Documents

Table of Contents

1. Why We Cut Documents Into Chunks
2. Loading Documents: loadDocs
3. Chunk Size Is a Trade-Off
4. Cutting Blindly, and Why It Hurts
5. Fixed-Size Chunks With Overlap
6. Splitting on Meaning: Headings
7. chunkDocs: One Function for Both Strategies
8. Comparing Strategies on a Real Question
9. Metadata: Remember Where Each Chunk Came From
10. Tokens Versus Characters
11. Khmer and Other Languages Without Spaces
12. PDFs, HTML and Other Formats
13. Recipe: The load and chunk Modules
14. Try It Yourself
15. Quick Review: Common Questions

1. Why We Cut Documents Into Chunks

In Why RAG? you saw the pipeline: load, chunk, embed, store, then search at question time. This chapter builds the first two steps.

Why cut at all? Because the unit you search is the unit you get back. Imagine a library where you can only ask for whole books. You ask "what is the refund rule?" and the librarian hands you the 300-page handbook. The answer is somewhere inside, but you must pay (in tokens) to carry the whole book to the model, and the model must find the sentence.

Now imagine the library catalogues every paragraph on its own card. You get the one card about refunds. That is what chunking does: it turns long documents into small, self-contained passages, so that retrieval can return just the right one.

A good chunk has three properties:

• Self-contained. Read alone, it still makes sense.
• About one thing. A chunk about both refunds and laptops will match neither question well.
• Labelled. We know which file and which section it came from, so we can show a citation.

Most of the art of chunking is balancing the first two. We build two modules: src/load.js reads files, and src/chunk.js cuts them. Keep the mock server from chapter 1 running; this chapter does not need it, but chapter 3 will.

2. Loading Documents: loadDocs

Loading is the boring part, and it is where many real bugs hide. The job: read every .md file in a folder and return a list of documents with a stable shape.

// src/load.js
import fs from "node:fs";
import path from "node:path";

// loadDocs("docs") -> [{ id, source, title, text }]
export function loadDocs(dir) {
  return fs
    .readdirSync(dir)
    .filter((name) => name.endsWith(".md"))
    .sort()
    .map((name) => {
      const text = fs.readFileSync(path.join(dir, name), "utf8").replace(/\r\n/g, "\n").trim();
      const title = text.match(/^# (.+)$/m)?.[1].trim() ?? name.replace(/\.md$/, "");
      return { id: name, source: name, title, text };
    });
}

A few small decisions:

• .sort() makes the order the same on every computer. (readdirSync does not promise an order. Reproducible order means reproducible chunk ids and reproducible test results.)
• .replace(/\r\n/g, "\n") turns Windows line endings into \n, so the rest of our code only handles one kind of newline.
• The title is the first line that starts with # . If a file has none, we fall back to the filename.
• source is the file name. It becomes the citation later.

Try it:

// demo-load.js
import { loadDocs } from "./src/load.js";

const docs = loadDocs("docs");
for (const d of docs) console.log(d.id.padEnd(16), String(d.text.length).padStart(4), "chars  ", d.title);
console.log("total:", docs.reduce((n, d) => n + d.text.length, 0), "chars");
// output
// about.md          869 chars   About Mekong Code Club
// conduct.md        835 chars   Code of Conduct
// faq.md            882 chars   Frequently Asked Questions
// laptops.md        850 chars   Laptop Loan Policy
// membership.md     827 chars   Membership and Fees
// mentors.md        909 chars   Mentors
// projects.md       985 chars   Projects and Showcase Day
// schedule.md       894 chars   Weekly Schedule
// total: 7051 chars

The numbers are stable. (They are a few characters smaller than the 7,059 in chapter 1, because loadDocs trims the trailing newline.)

3. Chunk Size Is a Trade-Off

How big should a chunk be? There is no perfect answer, only a trade-off.

Too small (say one sentence, 100 characters). The chunk loses its surroundings. "The fee pays for snacks, printing, and the Wi-Fi bill." is a fine sentence, but alone you cannot tell which fee. Pronouns like "it" and "they" point to nothing. The model gets facts without context.

Too big (say a whole chapter, 5,000 characters). The chunk is about many things, so its single "meaning" is a blur, and it matches many questions only weakly. It also wastes tokens: you pay to send 5,000 characters when 200 held the answer, and the model must read through all of it.

A common starting point is a few hundred characters up to around a thousand, and then you measure (chapter 7). The right size depends on your documents and your questions. A FAQ with one-line answers wants small chunks. A legal contract with long clauses wants bigger ones. Treat any number in a tutorial as a starting guess, including ours.

4. Cutting Blindly, and Why It Hurts

The simplest chunker slices the text every N characters. Here it is on our membership document, 200 characters at a time:

// demo-blind.js: cut at exactly 200 characters, no care for meaning
import { loadDocs } from "./src/load.js";

const text = loadDocs("docs").find((d) => d.source === "membership.md").text;
for (let i = 0; i < 3; i++) {
  console.log(`--- piece ${i + 1}`);
  console.log(text.slice(i * 200, (i + 1) * 200));
}
// output
// --- piece 1
// # Membership and Fees
//
// ## Membership fee
//
// Membership costs 10 US dollars per term. A term lasts about four months.
// The fee pays for snacks, printing, and the Wi-Fi bill. There is no fee
// for the first
// --- piece 2
// month, so new students can try the club before paying.
//
// ## Scholarships
//
// No student is turned away because of money. Students who cannot pay can
// apply for a free scholarship membership by talking to t
// --- piece 3
// he club treasurer,
// Dara, after any Saturday session. Scholarship students have exactly the
// same rights as paying members.
//
// ## How to pay
//
// Pay the treasurer in cash, or send the fee by bank transfer. T

Look at the damage:

• Piece 1 ends in the middle of "for the first month", and piece 2 starts with "month, so new students...". The fact "no fee for the first month" is torn in two.
• Piece 2 stops halfway through "talking to t", and piece 3 begins with "he club treasurer". Even a single word was split.
• The heading ## Scholarships ended up in piece 2, but its first sentences are cut off from the name of the person who handles it (Dara) in piece 3. A question about scholarships would find a chunk that does not contain the answer.

So we need two improvements: cut at natural boundaries, and overlap neighbouring chunks a little so that a fact near a cut appears whole in at least one of them.

5. Fixed-Size Chunks With Overlap

Our chunkBySize(text, { size, overlap }) works like this:

1. Take a window of size characters.
2. If there is a paragraph break, line break or sentence end in the second half of the window, cut there instead, so the chunk ends at a natural place. (A cut earlier than half would make chunks too small.)
3. Start the next chunk overlap characters before the end of this one, moved forward to the start of a word.

Overlap is like reading a book with the last two lines of the previous page repeated at the top of the next page: you never lose the thread at the page turn. Let us see 300-character chunks without and with an overlap of 60:

// demo-size.js
import { loadDocs } from "./src/load.js";
import { chunkBySize } from "./src/chunk.js";

const text = loadDocs("docs").find((d) => d.source === "membership.md").text;
for (const overlap of [0, 60]) {
  const pieces = chunkBySize(text, { size: 300, overlap });
  console.log(`\n=== size 300, overlap ${overlap}: ${pieces.length} pieces`);
  pieces.forEach((p, i) => console.log(`--- piece ${i + 1} (${p.length} chars)\n${p}`));
}
// output
//
// === size 300, overlap 0: 3 pieces
// --- piece 1 (271 chars)
// # Membership and Fees
//
// ## Membership fee
//
// Membership costs 10 US dollars per term. A term lasts about four months.
// The fee pays for snacks, printing, and the Wi-Fi bill. There is no fee
// for the first month, so new students can try the club before paying.
//
// ## Scholarships
// --- piece 2 (263 chars)
// No student is turned away because of money. Students who cannot pay can
// apply for a free scholarship membership by talking to the club treasurer,
// Dara, after any Saturday session. Scholarship students have exactly the
// same rights as paying members.
//
// ## How to pay
// --- piece 3 (289 chars)
// Pay the treasurer in cash, or send the fee by bank transfer. The club
// does not accept payment through social media messages. Always ask for a
// receipt.
//
// ## Refunds
//
// If you leave in the first two months of a term, you can ask for half of
// your fee back. After two months there are no refunds.
//
// === size 300, overlap 60: 4 pieces
// --- piece 1 (271 chars)
// # Membership and Fees
//
// ## Membership fee
//
// Membership costs 10 US dollars per term. A term lasts about four months.
// The fee pays for snacks, printing, and the Wi-Fi bill. There is no fee
// for the first month, so new students can try the club before paying.
//
// ## Scholarships
// --- piece 2 (276 chars)
// students can try the club before paying.
//
// ## Scholarships
//
// No student is turned away because of money. Students who cannot pay can
// apply for a free scholarship membership by talking to the club treasurer,
// Dara, after any Saturday session. Scholarship students have exactly the
// --- piece 3 (269 chars)
// any Saturday session. Scholarship students have exactly the
// same rights as paying members.
//
// ## How to pay
//
// Pay the treasurer in cash, or send the fee by bank transfer. The club
// does not accept payment through social media messages. Always ask for a
// receipt.
//
// ## Refunds
// --- piece 4 (180 chars)
// media messages. Always ask for a
// receipt.
//
// ## Refunds
//
// If you leave in the first two months of a term, you can ask for half of
// your fee back. After two months there are no refunds.

Now chunks end at line breaks, not in the middle of words. With overlap 0 we get 3 pieces, and with overlap 60 we get 4. You can see the overlap in action: piece 2 begins with "students can try the club before paying." (the end of piece 1), and piece 3 begins "any Saturday session. ..." (the end of piece 2).

Look at what is still wrong. In both runs, the ## Scholarships heading is the last line of piece 1, while its text is in piece 2. A size chunker does not know what a heading is. The heading, which is the best label of what the text is about, is separated from the text it names.

Overlap has a cost: it repeats text, so you embed and store more. With 20% overlap you store about 20% more. Too much overlap (50%) makes neighbouring chunks nearly identical, so one search can return the same sentence three times. A common starting range is roughly 10 to 20 percent of the chunk size. The constraint overlap < size is checked in the code, because otherwise the loop would never move forward.

6. Splitting on Meaning: Headings

Handbooks, wikis, documentation and README files already contain the boundaries a human would choose: headings. Each ## section is usually one topic. Splitting at headings gives self-contained chunks that are about one thing and labelled, with no overlap needed.

chunkByHeading(doc) walks the lines. When it sees a line that starts with ## , it closes the current section and starts a new one, keeping the heading text. Text before the first ## (such as an introduction under the # title) belongs to the document title.

// demo-heading.js
import { loadDocs } from "./src/load.js";
import { chunkByHeading, chunkDocs } from "./src/chunk.js";

const docs = loadDocs("docs");
const membership = docs.find((d) => d.source === "membership.md");
for (const s of chunkByHeading(membership)) console.log(`[${s.heading}] ${s.text.length} chars`);

const chunks = chunkDocs(docs);
console.log("\nheading strategy:", chunks.length, "chunks");
console.log(chunks.find((c) => c.id === "membership.md#2"));
const lengths = chunks.map((c) => c.text.length);
console.log("shortest", Math.min(...lengths), "longest", Math.max(...lengths),
  "average", Math.round(lengths.reduce((a, b) => a + b) / lengths.length));
// output
// [Membership fee] 212 chars
// [Scholarships] 248 chars
// [How to pay] 150 chars
// [Refunds] 125 chars
//
// heading strategy: 33 chunks
// {
//   id: 'membership.md#2',
//   source: 'membership.md',
//   heading: 'Scholarships',
//   text: 'No student is turned away because of money. Students who cannot pay can\n' +
//     'apply for a free scholarship membership by talking to the club treasurer,\n' +
//     'Dara, after any Saturday session. Scholarship students have exactly the\n' +
//     'same rights as paying members.'
// }
// shortest 88 longest 295 average 186

Four sections, one per heading, each a clean paragraph or two. The whole corpus becomes 33 chunks, between 88 and 295 characters, 186 on average. The chunk membership.md#2 is the Scholarships section, whole, with Dara's name inside it. This is the kind of chunk we want.

But headings can fail. A document may have no headings at all (a transcript, a long email). A single section may be huge (a 20-page "## Policies"). And headings alone do not help a plain-text file. So we need a fallback: if a heading section is longer than maxChars, split that section by size. A made-up long section shows how:

// demo-fallback.js
import { chunkDocs } from "./src/chunk.js";

// A made-up document: one "## " section that is far too long.
const paragraph = "The club keeps a record of every session. Mentors write a short note after each workshop.";
const doc = {
  source: "long.md",
  title: "Long Notes",
  text: "# Long Notes\n\n## Session notes\n\n" + Array(30).fill(paragraph).join("\n\n"),
};
const chunks = chunkDocs([doc], { maxChars: 600, overlap: 80 });
console.log(chunks.length, "chunks");
for (const c of chunks) console.log(c.id, `[${c.heading}]`, c.text.length, "chars");
// output
// 6 chunks
// long.md#1 [Session notes] 587 chars
// long.md#2 [Session notes] 579 chars
// long.md#3 [Session notes] 572 chars
// long.md#4 [Session notes] 579 chars
// long.md#5 [Session notes] 572 chars
// long.md#6 [Session notes] 215 chars

One heading section turned into six chunks of roughly 600 characters, and all six keep the heading Session notes. That repeated heading is useful metadata for citations (section 9).

7. chunkDocs: One Function for Both Strategies

chunkDocs(docs, options) is the function the rest of the series calls. It takes the documents from loadDocs and returns chunks of one shape:

{ id: "membership.md#2", source: "membership.md", heading: "Scholarships", text: "..." }

The options:

The id is the file name plus the chunk's position in that file (membership.md#2), so it is human readable and stable as long as the document does not change. With the "size" strategy the heading of every chunk is just the document title, because the size chunker does not know about sections. That difference matters for citations, and it is one reason to prefer headings when your documents have them.

Note that chunkByHeading skips sections with no text, so you never embed an empty chunk, and that loadDocs already turned Windows line endings into \n, so the heading test startsWith("## ") is reliable.

8. Comparing Strategies on a Real Question

Do not argue about chunk size in the abstract. Measure it. Here we use the crude keyword search from chapter 1 (so this chapter needs no server) and ask "Who can I ask for a scholarship?" with four chunkings. For each, we print the best-scoring chunk, its score, its size, and whether it contains the answer ("Dara").

// demo-compare.js
import { loadDocs } from "./src/load.js";
import { chunkDocs } from "./src/chunk.js";

// Keyword-overlap search, as in naive-rag.js, so we only compare the chunks.
const words = (s) => new Set(s.toLowerCase().match(/\p{L}{3,}/gu) ?? []);
function top(chunks, question) {
  const q = words(question);
  return chunks
    .map((c) => ({ c, score: [...words(c.text)].filter((w) => q.has(w)).length }))
    .sort((a, b) => b.score - a.score)[0];
}

const docs = loadDocs("docs");
const question = "Who can I ask for a scholarship?";
const strategies = {
  "size 150": { strategy: "size", size: 150, overlap: 0 },
  "size 400": { strategy: "size", size: 400, overlap: 40 },
  "heading": { strategy: "heading" },
  "size 3000": { strategy: "size", size: 3000, overlap: 0 },
};
for (const [name, options] of Object.entries(strategies)) {
  const chunks = chunkDocs(docs, options);
  const best = top(chunks, question);
  console.log(`${name.padEnd(10)} ${String(chunks.length).padStart(2)} chunks, best: ${best.c.id} (${best.score}, ${best.c.text.length} chars)`);
  console.log("   has the answer (Dara)?", best.c.text.includes("Dara"));
}
// output
// size 150   56 chunks, best: about.md#6 (3, 122 chars)
//    has the answer (Dara)? false
// size 400   24 chunks, best: membership.md#2 (5, 393 chars)
//    has the answer (Dara)? true
// heading    33 chunks, best: membership.md#2 (4, 248 chars)
//    has the answer (Dara)? true
// size 3000   8 chunks, best: membership.md#1 (5, 827 chars)
//    has the answer (Dara)? true

Read the four lines:

• size 150 produced 56 tiny chunks. The best match is an irrelevant piece of about.md, and the answer is missing. Too small.
• size 400 works. The best chunk (membership.md#2) holds the answer, but it is 393 characters, because the size chunker did not stop at the heading and swept in neighbouring text.
• heading finds the same section at 248 characters: the answer, and less text to pay for.
• size 3000 is bigger than every file, so each file is one chunk. It finds the answer inside the whole 827-character membership file. It "works" here only because our files are tiny. With real documents, this is the "too big" case.

Be careful with the conclusion. This is one question, one corpus and a crude scorer, so it shows how chunking can change retrieval, not which strategy is best. In chapter 7 you will measure with a proper question set, and you will see the numbers move when you change chunking.

9. Metadata: Remember Where Each Chunk Came From

Every chunk carries source and heading next to its text. This is cheap to store and very valuable:

• Citations. The answer can say "(source: membership.md, section Scholarships)" and the user can open the file and check. Chapter 5 puts this into the prompt.
• Filtering. Later you can search only chunks from some files, or only the files a user may read (chapters 4 and 8).
• Debugging. When an answer is wrong, the first question is "which chunks were retrieved?" and metadata tells you at a glance.
• Updating. When faq.md changes, you know which chunks to re-embed and which to leave alone.

You can add more as your data grows: the page number of a PDF, a URL, an author, a "last updated" date, a language, a document version or an access level. Decide early what you might want to cite or filter by, because adding metadata later means re-running the pipeline.

One more trick used in practice is to prepend the heading to the text that gets embedded (for example Scholarships: No student is turned away...), so that the topic words travel with the chunk. We keep the heading separate here so that you can see the effect clearly when you experiment.

10. Tokens Versus Characters

Models do not read characters or words. They read tokens, small pieces of text from the model's own vocabulary (see How LLMs Work). Context limits and prices are counted in tokens. But our chunk sizes are in characters, which are easier to count and need no special library.

For English, a handy rule of thumb is that one token is about four characters. It is only a rule of thumb. Each model has its own tokenizer, code and numbers usually take more tokens per character, and languages other than English (including Khmer) often take noticeably more tokens per character, so a Khmer chunk costs more than an English chunk of the same length. Do not trust a rule of thumb for money: when it matters, count with your provider's tokenizer or read the usage field the API returns.

Here is a rough count for our corpus under two chunkings. Remember that the / 4 is an estimate:

// demo-tokens.js
import { loadDocs } from "./src/load.js";
import { chunkDocs } from "./src/chunk.js";

// A rough rule of thumb for English: 1 token is about 4 characters.
// Real tokenizers differ, and Khmer usually needs more tokens per character.
const roughTokens = (text) => Math.ceil(text.length / 4);

const docs = loadDocs("docs");
const all = docs.reduce((n, d) => n + roughTokens(d.text), 0);
console.log("whole corpus: about", all, "tokens");

for (const [name, options] of Object.entries({
  "heading": { strategy: "heading" },
  "size 400": { strategy: "size", size: 400, overlap: 40 },
})) {
  const chunks = chunkDocs(docs, options);
  const total = chunks.reduce((n, c) => n + roughTokens(c.text), 0);
  console.log(`${name.padEnd(9)} ${chunks.length} chunks, about ${total} tokens in total, ${Math.round(total / chunks.length)} per chunk`);
}
// output
// whole corpus: about 1767 tokens
// heading   33 chunks, about 1547 tokens in total, 47 per chunk
// size 400  24 chunks, about 1917 tokens in total, 80 per chunk

Two things to notice. First, the heading strategy stores fewer total tokens (1,547) than the whole corpus (1,767), because it drops the # Title and ## lines and does not overlap. Second, the size strategy stores more than the corpus (1,917) because of overlap. A chunk of about 47 tokens is small; about 80 tokens is still small. Real chunk sizes are usually larger because real documents have longer sections, and our club handbook is deliberately tiny.

If a chunk is too big for your embedding model (they also have an input limit, measured in tokens), it fails or gets cut off. Choose a chunk size comfortably below the limit and check your provider's documentation.

11. Khmer and Other Languages Without Spaces

English puts spaces between words, so "cut at a space" is a safe place to cut. Khmer does not. Words in a sentence run together, and spaces mostly mark the end of a phrase or clause. This changes how you chunk.

Here is a short Khmer sample (about the club fee and the Saturday workshop; the wording is simple sample text, so ask a Khmer speaker to review anything you publish) and three ways of measuring it:

// demo-khmer.js
const km = "ថ្លៃសមាជិកភាពគឺ ១០ ដុល្លារក្នុងមួយឆមាស។ សិក្ខាសាលាប្រចាំសប្តាហ៍ធ្វើឡើងរៀងរាល់ថ្ងៃសៅរ៍។";

console.log("characters (.length):", km.length);
console.log("split on spaces:", km.split(" ").length, "pieces");

const words = [...new Intl.Segmenter("km", { granularity: "word" }).segment(km)]
  .filter((s) => s.isWordLike).map((s) => s.segment);
console.log("Intl.Segmenter words:", words.length);
console.log(words.join(" | "));

const graphemes = [...new Intl.Segmenter("km", { granularity: "grapheme" }).segment(km)];
console.log("visible characters (graphemes):", graphemes.length);

// A blind cut can land inside one visible character.
console.log(JSON.stringify(km.slice(0, 5)), "<- cut after 5 UTF-16 units");
console.log(JSON.stringify(km.slice(0, 6)));

console.log(JSON.stringify(km.slice(0, 2)), "<- a consonant cut off from its subscript");

import { chunkBySize } from "./src/chunk.js";
const long = Array(3).fill(km).join(" ");
for (const piece of chunkBySize(long, { size: 60, overlap: 10 })) {
  console.log(piece.length, JSON.stringify(piece));
}
// output
// characters (.length): 86
// split on spaces: 4 pieces
// Intl.Segmenter words: 14
// ថ្លៃ | សមាជិកភាព | គឺ | ១០ | ដុល្លារ | ក្នុង | មួយ | ឆមាស | សិក្ខាសាលា | ប្រចាំ | សប្តាហ៍ | ធ្វើឡើង | រៀងរាល់ថ្ងៃ | សៅរ៍
// visible characters (graphemes): 44
// "ថ្លៃស" <- cut after 5 UTF-16 units
// "ថ្លៃសម"
// "ថ្" <- a consonant cut off from its subscript
// 39 "ថ្លៃសមាជិកភាពគឺ ១០ ដុល្លារក្នុងមួយឆមាស។"
// 56 "ងមួយឆមាស។ សិក្ខាសាលាប្រចាំសប្តាហ៍ធ្វើឡើងរៀងរាល់ថ្ងៃសៅរ៍។"
// 49 "ថ្ងៃសៅរ៍។ ថ្លៃសមាជិកភាពគឺ ១០ ដុល្លារក្នុងមួយឆមាស។"
// 56 "ងមួយឆមាស។ សិក្ខាសាលាប្រចាំសប្តាហ៍ធ្វើឡើងរៀងរាល់ថ្ងៃសៅរ៍។"
// 49 "ថ្ងៃសៅរ៍។ ថ្លៃសមាជិកភាពគឺ ១០ ដុល្លារក្នុងមួយឆមាស។"
// 56 "ងមួយឆមាស។ សិក្ខាសាលាប្រចាំសប្តាហ៍ធ្វើឡើងរៀងរាល់ថ្ងៃសៅរ៍។"

What this teaches (the numbers are stable and do not depend on the mock; the word split depends on the Unicode data built into your Node.js, so a very different Node version could split words a little differently):

• Splitting on spaces is wrong. It finds only 4 "words" in a sentence that has 14 real words. A word-based chunker (for example "300 words per chunk") would think a Khmer page is tiny.
• Character-based sizes are safer. .length counts characters, so "500 characters" means roughly the same amount of text whatever the language. (Strictly, JavaScript's .length counts UTF-16 code units, which is close enough for size limits.)
• Intl.Segmenter can find word and character boundaries. It is built into modern Node.js and uses dictionaries for languages like Khmer. We use its grapheme mode, which finds visible characters. This sample has 86 code units but only 44 visible characters, because Khmer builds one visible character out of several pieces (a consonant, a subscript mark, a vowel sign).
• Do not cut inside a visible character. slice(0, 2) returns a consonant joined to the start of a subscript, which is a broken character. Our chunkBySize therefore moves a cut back (and an overlap start forward) to a grapheme boundary, which is what snapToGrapheme in the recipe does.
• Khmer has its own full stop, ។. chunkBySize treats it like the English . when choosing where to end a chunk. In the last lines of the output, the chunks end right after ។.

What if there is no full stop and no space at all, the worst case? Then all we can do is cut at a visible-character boundary:

// demo-khmer-cut.js
import { chunkBySize } from "./src/chunk.js";

// No full stop and no spaces, so the only choice is where to cut.
const word = "សិក្ខាសាលាប្រចាំសប្តាហ៍";
for (const piece of chunkBySize(word, { size: 7, overlap: 2 })) {
  console.log(piece.length, piece);
}
console.log("blind slice:", JSON.stringify(word.slice(0, 7)), JSON.stringify(word.slice(7, 14)));
// output
// 6 សិក្ខា
// 7 សាលាប្រ
// 4 ចាំស
// 7 សប្តាហ៍
// blind slice: "សិក្ខាស" "ាលាប្រច"

The last line shows the problem we avoided. A blind slice at 7 separates a consonant from its vowel sign ("ស" ends the first piece and "ា" starts the second), which breaks the character. Our chunks do not do that. They still may cut in the middle of a word (for example ចាំស is the end of one word and the start of the next), because without a word dictionary we do not know where words end. For most retrieval this is acceptable, since embeddings and overlap soften it, but if you chunk lots of Khmer text, test it with real Khmer questions (chapter 7) and consider cutting at the ។ or at spaces (phrase breaks) only, and using Intl.Segmenter word boundaries to place cuts.

The same ideas apply to other languages written without spaces, such as Thai, Lao, Burmese, Chinese and Japanese. Whether the embedding model understands Khmer well is a separate question for chapter 3.

12. PDFs, HTML and Other Formats

Real organisations do not keep their knowledge in tidy Markdown. They have PDFs, Word files, slides, web pages and spreadsheets. The rule is simple to say and hard to do well: convert everything to plain text first, then feed it to the same chunker. The chunker should never care where the text came from.

Be honest about how messy this is:

• PDF is a page-drawing format, not a text format. A PDF stores "draw these letters at these coordinates". Extraction has to guess the reading order. Two-column pages, headers and footers, page numbers, footnotes and hyphenated words are often mixed into the text in the wrong order.
• Tables lose their rows and columns, and numbers end up detached from their labels. A table of fees can turn into a stream of numbers with no meaning.
• Scanned PDFs are pictures. There is no text to extract. You need OCR (optical character recognition), which makes mistakes, especially with low-quality scans, handwriting and less common scripts like Khmer.
• Fonts and encodings. Some PDFs, including ones with Khmer text, use fonts that extract as wrong or missing characters, and what you see on screen is not what you copy out. Always check a sample.
• HTML has the opposite problem: lots of text you do not want, such as menus, cookie banners, ads and footers. You must strip these, keep the main content, and keep the headings (they are valuable boundaries).
• Word, slides and spreadsheets have structure (headings, slides, sheets and cells) that you can use for chunking if the converter keeps it.

Tools exist for each format, and their quality varies a lot with your particular files, so try a few on your own documents, read the extracted text yourself, and fix what is broken before building anything on top. The most valuable five minutes in a RAG project is opening the extracted text and checking it. If the text is garbage, every later step is garbage too.

A practical approach is to write one small converter per format that outputs Markdown with # and ## headings (so our heading chunker works unchanged), saves it next to the original, and records the original file name and page in the metadata. Then loadDocs only has to read .md files, which is exactly what we built.

13. Recipe: The load and chunk Modules

These are the two modules chapter 2 owns. Neither imports anything except Node's built-in fs and path (and chunk.js imports nothing).

src/load.js

// src/load.js
import fs from "node:fs";
import path from "node:path";

// loadDocs("docs") -> [{ id, source, title, text }]
export function loadDocs(dir) {
  return fs
    .readdirSync(dir)
    .filter((name) => name.endsWith(".md"))
    .sort()
    .map((name) => {
      const text = fs.readFileSync(path.join(dir, name), "utf8").replace(/\r\n/g, "\n").trim();
      const title = text.match(/^# (.+)$/m)?.[1].trim() ?? name.replace(/\.md$/, "");
      return { id: name, source: name, title, text };
    });
}

src/chunk.js

// src/chunk.js

const graphemes = new Intl.Segmenter(undefined, { granularity: "grapheme" });

// Move `pos` to a boundary between visible characters, so a cut never lands
// inside one (this matters for Khmer, Hindi, emoji and so on). `dir` -1 moves
// back, +1 moves forward. `start` must be a known boundary at or before pos.
function snapToGrapheme(text, start, pos, dir = -1) {
  if (pos >= text.length) return pos;
  let before = start;
  for (const { index } of graphemes.segment(text.slice(start, pos + 16))) {
    const at = start + index;
    if (at <= pos) before = at;
    if (at >= pos) return dir < 0 && at > pos ? before : at;
  }
  return pos;
}

// Cut text into pieces of about `size` characters. Each piece starts
// `size - overlap` characters after the previous one, so neighbours share
// `overlap` characters. We prefer to end a piece at a paragraph or sentence
// break instead of in the middle of a word.
export function chunkBySize(text, { size = 500, overlap = 50 } = {}) {
  if (overlap >= size) throw new Error("overlap must be smaller than size");
  const pieces = [];
  let start = 0;
  while (start < text.length) {
    let end = Math.min(start + size, text.length);
    if (end < text.length) {
      const window = text.slice(start, end);
      const cut = Math.max(
        window.lastIndexOf("\n\n"),
        window.lastIndexOf(". "),
        window.lastIndexOf("។"), // Khmer full stop
        window.lastIndexOf("\n"),
      );
      if (cut > size / 2) end = start + cut + 1; // only cut early if it is not too early
      else end = snapToGrapheme(text, start, end, -1);
    }
    const piece = text.slice(start, end).trim();
    if (piece) pieces.push(piece);
    if (end >= text.length) break;
    let next = Math.max(end - overlap, start + 1);
    const gap = text.slice(next - 1, end).search(/\s/); // begin at a word start...
    next = gap === -1 ? snapToGrapheme(text, start, next, 1) : next + gap; // ...if there are spaces
    start = Math.max(next, start + 1);
  }
  return pieces;
}

// Split a document at "## " headings. Text before the first "## " belongs
// to the document title. Returns [{ heading, text }].
export function chunkByHeading(doc) {
  const sections = [];
  let heading = doc.title;
  let lines = [];
  const flush = () => {
    const text = lines.join("\n").trim();
    if (text) sections.push({ heading, text });
  };
  for (const line of doc.text.split("\n")) {
    if (line.startsWith("## ")) {
      flush();
      heading = line.slice(3).trim();
      lines = [];
    } else if (!line.startsWith("# ")) {
      lines.push(line);
    }
  }
  flush();
  return sections;
}

// chunkDocs(docs, options) -> [{ id, source, heading, text }]
export function chunkDocs(docs, { strategy = "heading", size = 500, overlap = 50, maxChars = 1200 } = {}) {
  const chunks = [];
  for (const doc of docs) {
    const sections =
      strategy === "heading"
        ? chunkByHeading(doc).flatMap((s) =>
            s.text.length > maxChars
              ? chunkBySize(s.text, { size: maxChars, overlap }).map((text) => ({ heading: s.heading, text }))
              : [s],
          )
        : chunkBySize(doc.text, { size, overlap }).map((text) => ({ heading: doc.title, text }));
    sections.forEach((s, i) => {
      chunks.push({ id: `${doc.source}#${i + 1}`, source: doc.source, heading: s.heading, text: s.text });
    });
  }
  return chunks;
}

Notes on the details:

• chunkBySize prefers, in the second half of the window, the last paragraph break, then a sentence end (. or ។), then a line break.
• snapToGrapheme uses Intl.Segmenter, which is built into Node.js. When there are spaces to use, the next chunk starts at a word; otherwise (as in Khmer) it starts at a visible-character boundary.
• In heading mode, text under the # title but before the first ## becomes its own chunk with the title as its heading.
• The # title line and the ## heading lines are not repeated in the chunk text. The heading is in the heading field instead.

14. Try It Yourself

1. Tune the size. Run demo-compare.js with sizes of 100, 200, 800 and 1500, and with different overlaps. Ask two more questions of your own ("How many laptops?", "How long is the presentation?"). Which size is best for which question?
2. Overlap test. Take a sentence in the middle of membership.md and write a size/overlap pair so that the sentence is split in two chunks with overlap 0, but whole in at least one chunk with some overlap. How much overlap did you need?
3. Chunk by paragraph. Write chunkByParagraph(doc) that splits on blank lines and keeps the nearest heading. Compare its chunk count and sizes to the heading strategy. When would it be better?
4. Add ###. Documents often have sub-headings. Extend chunkByHeading so that ### also starts a new chunk, and the heading becomes "Parent > Child". Test it with a document you write.
5. Make a Khmer document. Write docs/km-handbook.md with a title, two ## sections and several sentences in Khmer (a Khmer speaker can check it). Run chunkDocs on it with size of 60 and print the chunks. Do any chunks start or end in a strange place? Why?
6. Convert a real file. Take one real PDF or web page you own, extract its text with any tool you like, and save it as a .md file in a scratch docs folder. Open the result in your editor. What did the conversion get wrong?

15. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now read a folder of documents and cut it into labelled chunks of a sensible size. But a chunk is still just text, and the search in chapter 1 could only match exact words. In the next chapter, Embeddings in Practice, you turn every chunk into a list of numbers that represent its meaning, and measure how close two pieces of text are.