What Is an LLM?

Table of Contents

1. The Autocomplete You Already Use
2. The Whole Idea in One Loop
3. What "Large Language Model" Means
4. What an LLM Is Not
5. From Autocomplete to Chatbot
6. Why One Simple Loop Can Look Clever
7. How This Series Is Organized
8. Set Up the llmlab Folder
9. The Shared Corpus
10. A Hand-Written Autocomplete Table
11. Recipe: The Autocomplete Module
12. What the Toy Gets Wrong
13. Try It Yourself
14. Quick Review: Common Questions

1. The Autocomplete You Already Use

Open the messages app on your phone and type:

I am going to the

Above the keyboard you will see three suggestions, perhaps market, shop, and office. You did not teach your phone those words. It guessed them from patterns: after "going to the", people very often type one of those. Tap one and the phone shows three new guesses for the next word. Do that again and again and you can write a whole sentence by tapping suggestions.

That is the seed of everything in this series. A large language model (LLM) is a much, much more powerful version of that same trick. You give it some text. It predicts what text should come next. You add that piece to the text, and it predicts again.

Three questions people ask right away:

• Is a chatbot really just autocomplete? The core loop is the same, but the scale and the training make a huge difference. Section 6 and later chapters explain how.
• Then how does it answer questions it has never seen? It does not look up answers. It continues text in a way that fits patterns it learned from a very large amount of writing.
• Is it thinking? Honest answer: scientists and engineers disagree about the right word. We come back to this in the last chapter. For now, watch what it does before arguing about what it is.

2. The Whole Idea in One Loop

Here is the idea in four steps. Read them slowly, because every chapter is a closer look at one of them.

1. Read the text so far (the prompt).
2. Predict how likely each possible next piece of text is.
3. Pick one piece, usually a likely one, and append it to the text.
4. Repeat from step 1 until you have enough text or the model says it is finished.

As a diagram:

  "The coffee"  --->  [ model ]  --->  "shop" is likely
       ^                                    |
       |                                    v
       +----------  "The coffee shop"  <----+
                   (append, then repeat)

In JavaScript, the loop is only a few lines. This is pseudo-code: the shape is real, but predictNext is not defined yet.

let text = "The coffee";
for (let step = 0; step < 5; step++) {
  const next = predictNext(text); // the hard part: the model
  text = text + " " + next;       // append
}

Everything difficult about an LLM is hidden inside predictNext. The loop around it is almost trivial. In this chapter we write a predictNext by hand, with a lookup table. In chapter 3 the computer builds that table by counting. In chapter 5 it learns it by gradual adjustment. In real LLMs, predictNext is a huge neural network, but the loop around it looks exactly like the one above.

3. What "Large Language Model" Means

Take the three words one by one.

Language. The model works on text: words, code, numbers, emojis, anything you can write. Some models also accept images or sound, but the language part is the heart.

Model. A model is a simplified copy of something real, made so you can make predictions. A weather model predicts tomorrow's rain. A language model predicts the next piece of text. In code, a model is usually numbers plus a recipe for combining them. The numbers are called parameters (or weights). Training means adjusting those numbers until the predictions get good.

Large. Two things are large: the model and the training text.

• The model has a very large count of parameters, often billions. Each one is just a number, but together they store patterns about language.
• It was trained on a very large amount of text, such as books, websites, and code, far more than any person could read in many lifetimes.

How large is "large"? It keeps changing, and we will not quote numbers that will be out of date next year. The rule of thumb: the tiny model we build in this series has a few dozen numbers, a real LLM has billions, so the real one is millions of times bigger. Our toy shows the idea, not the strength.

4. What an LLM Is Not

Because LLMs write fluent text, people assume things about them that are not true. Four useful corrections:

It is not a database. A database stores facts and returns exactly what you stored. An LLM does not keep a list of facts. It keeps numbers that shape its predictions. When it says "Phnom Penh is the capital of Cambodia", that is because text continuing "The capital of Cambodia is" almost always continues that way in its training text, not because it looked up a record. This is why it can also state wrong things smoothly. Chapter 9 explores that.

It is not a search engine. A search engine finds existing pages and shows them to you. An LLM produces new text. Some products combine both (search first, then let the model summarize the results), but they are two separate parts.

It is not automatically up to date. A model learns from text up to some date, then training stops. It does not read the news by itself. If a product shows current information, the product fetched it and put it in the prompt for the model.

It is not a person. It does not have a body, a childhood, or goals the way you do. Whether it has anything like understanding is a real open debate, which we treat fairly in the last chapter. Be careful with words like "it wants" or "it knows". They are handy shortcuts, but they can mislead you about what it can and cannot be trusted to do.

5. From Autocomplete to Chatbot

Plain autocomplete continues any text. A chatbot answers you. How do you get from one to the other?

The trick is to format the conversation as one long piece of text and let the model continue it. Imagine this text:

The following is a conversation between a helpful assistant and a user.

User: What is the capital of Cambodia?
Assistant:

A model that has learned how conversations look will predict that the text after Assistant: is an answer, so it continues with something like "The capital of Cambodia is Phnom Penh." That is a chat reply.

Real chat models are trained further so they follow instructions and stay helpful (chapter 5 explains the idea of fine-tuning), and apps wrap your messages in special markers for the roles system, user and assistant (chapter 8). But underneath, the engine is still: predict the next piece, append, repeat.

This explains two things you will notice often:

• The model only "sees" what is in the text you send it. If you did not send it, it does not know it. (Chapter 7.)
• The reply is built one piece at a time, which is why chat answers appear to "type" themselves on the screen.

6. Why One Simple Loop Can Look Clever

If the loop is so simple, why can LLMs write code, translate, or explain a joke? Here is the honest, short version.

To predict the next word of a sentence well, you need to model many things at once. Take this sentence:

Sokha put the keys on the table, then went to the market. Later, she
could not find her

To predict "keys" you must track who "she" is, remember what was mentioned earlier, and know a little about how people lose things. A model that predicts well across billions of examples like this ends up capturing grammar, facts that appear often, common reasoning patterns, and the style of different kinds of writing. Nobody programmed those skills in one by one. They appear because they help with the prediction task.

Two cautions keep this honest:

• Good prediction is not the same as being right. The model picks what is likely, not what is true. That is the root of hallucinations.
• We do not fully understand what happens inside. We know the math that runs and how it is trained. Explaining exactly why a particular answer comes out is still an active research area.

7. How This Series Is Organized

There are 10 chapters. Each one explains one idea with a small program and adds one module to llmlab.

1. What Is an LLM? (this chapter): the loop and the setup.
2. Tokens and Tokenizers: how text becomes pieces, and why pieces decide your bill.
3. Build a Tiny Language Model: count word pairs and generate text.
4. Probabilities, Temperature and Sampling: how a choice is made and what "temperature" does.
5. How a Model Is Trained: learning by adjusting numbers.
6. Embeddings and Attention: how words become numbers that capture meaning.
7. Context Windows and Memory: why a model forgets, and what that costs.
8. Prompts, Roles and Structured Output: steering the model and getting JSON back.
9. Why Models Hallucinate: the cause, and what to do about it.
10. Limits, Cost and Common Questions: the big picture and a talk-ready Q&A.

You can read the chapters in order, and that is what we recommend. Each builds on the toy from the one before.

8. Set Up the llmlab Folder

Everything runs in Node.js, with no packages to install, no API keys, and no internet needed once Node is installed. Check that you have it:

node --version

You should see a version number such as v22.22.2. Any current LTS version works. If you see "command not found", download the LTS installer from nodejs.org, run it, open a new terminal, and try again. If cd and mkdir feel unfamiliar, read Command Line Basics first.

Create the project:

mkdir llmlab
cd llmlab
npm init -y
mkdir src

npm init -y creates a package.json with default answers. It prints the file it wrote, which looks like this (the name is your folder name):

{
  "name": "llmlab",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Now open package.json in your editor and add one line, "type": "module", so that Node understands import and export:

{
  "name": "llmlab",
  "version": "1.0.0",
  "description": "",
  "type": "module",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "license": "ISC"
}

(We also deleted a few lines we do not need. Only "type": "module" matters.) Without it, Node will refuse our files with the error Cannot use import statement outside a module.

When the series is finished the folder will look like this. Do not create these files now; each chapter tells you when to add one.

llmlab/
  package.json
  demo.js           <- we run this to try things
  src/
    corpus.js       <- chapter 1: shared sample text
    autocomplete.js <- chapter 1: hand-written next-word table
    tokenizer.js    <- chapter 2
    rng.js          <- chapter 3
    bigram.js       <- chapter 3
    sampling.js     <- chapter 4
    train.js        <- chapter 5
    embeddings.js   <- chapter 6
    attention.js    <- chapter 6
    context.js      <- chapter 7
    prompt.js       <- chapter 8
    grounding.js    <- chapter 9
    report.js       <- chapter 10
    index.js        <- chapter 10

With ES modules you must write the ./ and the .js extension in every import path. A very common beginner error is leaving out .js.

9. The Shared Corpus

A corpus is a collection of text that a model learns from. Real models read a huge corpus. Ours is tiny: fifteen short lines about a small coffee shop in Phnom Penh, written for this series. Using the same corpus in every chapter lets you compare results from chapter to chapter.

Create src/corpus.js with this exact content:

// Shared sample text for every llmlab chapter. Original text, written for this series.
export const corpus = `
Sokha opens the small coffee shop near the river every morning.
The coffee shop is quiet in the morning and busy at lunch.
Sokha makes iced coffee with milk and iced tea with lime.
The students come to the coffee shop after school.
The students order iced coffee and the students share a table.
Dara is a student and Dara writes code at the table by the window.
Dara writes code for a small app and Dara drinks iced tea.
The app helps the coffee shop take orders on a phone.
Sokha likes the app because the app saves time every morning.
The river is calm in the morning and the coffee shop smells like fresh coffee.
Sokha says that a good coffee shop needs good coffee and a good table.
Dara says that a good app needs good code and a good idea.
The students laugh and the students order more iced coffee.
After lunch the coffee shop is quiet and Sokha cleans the table by the window.
Dara walks home by the river and Dara thinks about the next app.
`;

// Lowercase, strip punctuation, split on whitespace.
export function words(text) {
  return text
    .toLowerCase()
    .replace(/[^a-z\s]/g, " ")
    .split(/\s+/)
    .filter(Boolean);
}

Two exports. corpus is a template string holding the text. words turns any text into a list of lowercase words with punctuation removed. The regular expression /[^a-z\s]/g means "any character that is not a lowercase letter or whitespace", and we replace each with a space.

Try it. Create demo.js in the root of llmlab:

import { corpus, words } from "./src/corpus.js";

const w = words(corpus);
console.log("word count:", w.length);
console.log("first 8 words:", w.slice(0, 8).join(" "));
console.log("unique words:", new Set(w).size);

Run it with node demo.js:

word count: 182
first 8 words: sokha opens the small coffee shop near the
unique words: 67

All three lines are stable, because the corpus never changes. The model we build will see 182 words, but only 67 different words. We call the list of different words the vocabulary. Real models have a vocabulary of tens of thousands of tokens (chapter 2).

10. A Hand-Written Autocomplete Table

Now we write the smallest possible predictNext: a lookup table. For each word we list the words that could come next, most likely first. We type it in by hand.

const table = {
  the: ["coffee", "students", "app", "river"],
  coffee: ["shop", "with", "and"],
  iced: ["coffee", "tea"],
  // ...more words
};

To predict after coffee, look up table["coffee"] and you get ["shop", "with", "and"]. To complete a sentence, repeat:

1. Look up the last word.
2. Choose one of its options.
3. Append it.
4. Stop when the table has no entry or you have enough words.

How do we choose among the options? The most likely one is first, but always taking the first can get stuck in a loop such as "the coffee shop is quiet in the coffee shop is quiet...". So in this chapter we rotate through the options by step number. In chapters 3 and 4 we will replace this with real probabilities and random choice.

Where would a good table come from? Instead of typing it, we could count what follows each word in the corpus. Here is a quick preview for the word the. Save this as count.js and run node count.js.

import { corpus, words } from "./src/corpus.js";

const w = words(corpus);
const counts = new Map();
for (let i = 0; i < w.length - 1; i++) {
  if (w[i] !== "the") continue;
  counts.set(w[i + 1], (counts.get(w[i + 1]) ?? 0) + 1);
}
console.log([...counts].sort((a, b) => b[1] - a[1]));

[
  [ 'coffee', 5 ],
  [ 'students', 5 ],
  [ 'river', 3 ],
  [ 'app', 3 ],
  [ 'morning', 2 ],
  [ 'table', 2 ],
  [ 'window', 2 ],
  [ 'small', 1 ],
  [ 'next', 1 ]
]

Read it as: in our corpus, the word after the was coffee five times, students five times, river three times, and so on. This is the core of a language model, made out of counting. The output is stable because the corpus is fixed. Chapter 3 turns it into a real module for all words.

11. Recipe: The Autocomplete Module

This recipe adds the first module of the toolkit. Create src/autocomplete.js:

// A hard-coded "autocomplete": for some words, a list of likely next words.
// The first word in each list is the most likely one.
export const table = {
  the: ["coffee", "students", "app", "river"],
  coffee: ["shop", "with", "and"],
  shop: ["is", "near", "needs"],
  is: ["quiet", "busy", "calm"],
  quiet: ["in", "and"],
  in: ["the"],
  students: ["order", "come", "laugh"],
  order: ["iced", "more"],
  iced: ["coffee", "tea"],
  and: ["the", "dara"],
  dara: ["writes", "drinks", "says"],
  writes: ["code"],
  code: ["for", "at"],
  with: ["milk", "lime"],
  milk: ["and"],
  tea: ["with", "at"],
  lime: ["and"],
  for: ["a"],
  a: ["small", "good"],
  small: ["app", "coffee"],
  good: ["coffee", "code"],
  at: ["the"],
  app: ["helps", "saves"],
};

// Predict: return the possible next words, or an empty list if unknown.
export function predict(word) {
  return table[word] ?? [];
}

// Predict, append, repeat. `step` rotates through the options so the
// output does not get stuck repeating the same pair of words.
export function complete(start, length) {
  const out = [start];
  for (let step = 0; step < length; step++) {
    const options = predict(out[out.length - 1]);
    if (options.length === 0) break; // the table has no idea, so stop
    out.push(options[step % options.length]);
  }
  return out.join(" ");
}

Three things to notice.

• table is the whole "model": 23 words and their guesses. Everything the program "knows" is in that object.
• table[word] ?? [] means "use the table entry, or an empty list if there is none". The ?? operator only falls back for undefined or null.
• step % options.length is the remainder operator, used to rotate through the options without going out of range. (If this feels unfamiliar, see Modular Arithmetic.)

Use it from demo.js. Add these lines at the end:

import { predict, complete } from "./src/autocomplete.js";

console.log(predict("coffee"));
console.log(predict("banana"));
console.log(complete("the", 6));
console.log(complete("dara", 10));
console.log(complete("students", 12));
console.log(complete("banana", 5));

(Put the new import line at the top of the file with the other one.) Run node demo.js. The last six lines of output are:

[ 'shop', 'with', 'and' ]
[]
the coffee with milk and the students
dara writes code for a small coffee shop near
students order more
banana

All of these lines are stable. Nothing is random, so you should get the same output on any computer. Read them as small experiments:

• predict("banana") returns [] because the table has never seen banana.
• complete("the", 6) starts with the, predicts coffee, then with, and so on. The sentences are not very good, because we typed the table by hand and chose among options in a fixed rotating order.
• complete("students", 12) stops after more, because more has no entry. We asked for twelve words and got three. The loop ended as soon as the table had nothing to say.
• complete("banana", 5) returns only banana. A model that has never seen a word cannot continue it.

That last point is the out-of-vocabulary problem. Chapter 2 shows how real models avoid it by cutting words into smaller pieces.

12. What the Toy Gets Wrong

This toy is useful because it fails in obvious ways. Each failure points to a later chapter.

Keep this table in mind. By chapter 10 you will have built a rough version of each fix, and you will be able to explain what a real model does for each row.

13. Try It Yourself

1. Extend the table. Add entries for sokha, river, morning, and table. Make sure each list contains words that really follow in the corpus. Then run complete("sokha", 10). How long does a sentence get before it stops?
2. Find a loop. Look for a cycle in the table, such as the -> coffee -> shop -> is -> quiet -> in -> the. Change complete to always use options[0] and run complete("the", 14). What happens? (This is why real models do not always take the single most likely word.)
3. Count another word. Copy count.js, change "the" to "coffee", and run it. Which word follows coffee most often in the corpus? Does the answer match the first option in table.coffee?
4. Test on your own phone. Open your phone's keyboard, type "I am going to the", and tap the middle suggestion ten times. Write down the sentence you get. Is it grammatical? Is it sensible? Is it the same if you start over?
5. Prompt, then predict. Write by hand the three words you think a chatbot would add after the text The capital of Cambodia is. Then ask a real chatbot to continue the same text. How close were you?
6. Break the import. Remove the .js from the import path in demo.js and run it. Read the error, then fix it. You will meet this error often.

14. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You now know the whole idea: read the text, predict the next piece, append, repeat. The first thing a real model does with your text, though, is not read it as words at all. It chops it into tokens, and tokens decide what a model can see and what you pay. Continue with Tokens and Tokenizers.