Limits, Cost and Common Questions

Table of Contents

1. Putting It Together
2. Recipe: The report Module
3. Recipe: The llmlab Command-Line Tool
4. Why Cost Grows Faster Than You Expect
5. Latency: Why Answers Stream
6. Small Models, Large Models, and Your Laptop
7. Open Models vs Closed Models
8. What LLMs Are Good At, and What They Are Not
9. Bias, Safety and Privacy
10. Energy and the Environment
11. Does It Think? Does It Understand?
12. What About Jobs?
13. Try It Yourself
14. Quick Review: Common Questions

1. Putting It Together

You have built, in tiny form, almost every idea inside a large language model. Here is the whole pipeline in one picture. Each box is a chapter you have already done:

 "Sokha opens the coffee shop"
              |
              v
   +----------------------+
   | 1. TOKENIZE          |   text -> small pieces -> numbers
   |    (chapter 2)       |   [Sok][ha][ opens][ the][ coffee][ shop]  (example split)
   +----------------------+
              |
              v
   +----------------------+
   | 2. EMBED             |   each token id -> a list of numbers
   |    (chapter 6)       |   (a vector that captures meaning)
   +----------------------+
              |
              v
   +----------------------+
   | 3. TRANSFORMER       |   many layers; attention lets each token
   |    LAYERS            |   look at the others in the context
   |    (chapters 5, 6)   |   (the learned parameters live here)
   +----------------------+
              |
              v
   +----------------------+
   | 4. SCORES -> PROBS   |   one score per possible next token,
   |    (chapters 3, 4)   |   then softmax -> probabilities
   +----------------------+
              |
              v
   +----------------------+
   | 5. SAMPLE            |   temperature, top-k, top-p decide
   |    (chapter 4)       |   which token to pick
   +----------------------+
              |
              v
   +----------------------+
   | 6. APPEND            |   add the token to the context
   |    (chapters 1, 7)   |   (the context window is the limit)
   +----------------------+
              |
              +-----------> repeat from step 1 until done

Three things sit around this loop. Training (chapter 5) is how the numbers inside step 3 got their values. Prompts and roles (chapter 8) decide what text goes in at the start. Grounding and checks (chapter 9) decide what you do with the text that comes out. If you can explain this diagram to a friend, you understand LLMs better than most people who use them every day.

Honest simplifications: real models use sub-word tokens (not whole words), have dozens of transformer layers with many attention "heads", and learn their vectors during training. Your toy used whole words, one table, and hand-made vectors. The loop is the same.

2. Recipe: The report Module

Now let's add the last module to llmlab: a report that shows what a prompt costs and how much of the context window it uses, plus a function that generates text from the toy model at any temperature. It uses wordTokens and estimateCost from Tokens and Tokenizers, nextWordProbs from Build a Tiny Language Model, and the sampling functions from Probabilities, Temperature and Sampling.

Create src/report.js:

import { wordTokens, estimateCost } from "./tokenizer.js";
import { nextWordProbs } from "./bigram.js";
import { softmax, applyTemperature, sampleIndex, greedy } from "./sampling.js";

// Example numbers only. Real prices and window sizes differ per model and change often.
export const EXAMPLE_PRICE_PER_MILLION = 3; // dollars per 1,000,000 input tokens
export const EXAMPLE_CONTEXT_WINDOW = 8000; // tokens

// Real tokenizers split text into sub-word pieces. A common rule of thumb
// for English is about 4 characters per token. This is an estimate.
export function estimateTokens(text) {
  return Math.ceil(text.length / 4);
}

export function usageReport(prompt, {
  pricePerMillion = EXAMPLE_PRICE_PER_MILLION,
  windowSize = EXAMPLE_CONTEXT_WINDOW,
} = {}) {
  const tokens = estimateTokens(prompt);
  return {
    characters: prompt.length,
    wordCount: wordTokens(prompt).length,
    tokens,
    cost: estimateCost(tokens, pricePerMillion),
    windowUsed: tokens / windowSize,
  };
}

// Pick the next word from the bigram model at a given temperature.
// Temperature 0 means "always take the most likely word" (greedy).
export function sampleNext(model, word, temperature, rng) {
  const options = nextWordProbs(model, word);
  if (options.length === 0) return null;
  if (temperature === 0) return options[greedy(options.map((o) => o.p))].word;
  const logits = options.map((o) => Math.log(o.p));
  const probs = softmax(applyTemperature(logits, temperature));
  return options[sampleIndex(probs, rng)].word;
}

export function generateAtTemperature(model, startWord, length, temperature, rng) {
  const out = [startWord];
  let current = startWord;
  for (let i = 0; i < length; i++) {
    const next = sampleNext(model, current, temperature, rng);
    if (next === null) break;
    out.push(next);
    current = next;
  }
  return out.join(" ");
}

export function formatReport(prompt, r, windowSize = EXAMPLE_CONTEXT_WINDOW) {
  return [
    `prompt:        ${JSON.stringify(prompt)}`,
    `characters:    ${r.characters}`,
    `words:         ${r.wordCount}`,
    `tokens (est):  ${r.tokens}`,
    `cost (example): $${r.cost.toFixed(6)}`,
    `window used:   ${(r.windowUsed * 100).toFixed(2)}% of ${windowSize} tokens`,
  ].join("\n");
}

The exports are:

• estimateTokens(text) returns Math.ceil(text.length / 4). This is a rule of thumb for English, not what a real tokenizer returns. The tokenizer of any real model can give a different count, and for languages such as Khmer it is often higher per character, so paste your text into the tokenizer page of the model you use before budgeting.
• usageReport(prompt, { pricePerMillion, windowSize }) returns { characters, wordCount, tokens, cost, windowUsed }. The defaults (3 dollars per million tokens, an 8000 token window) are made-up examples. Put in the real numbers from your provider's page.
• sampleNext(model, word, temperature, rng) picks one next word. Temperature 0 is greedy; anything above that scales the log-probabilities, applies softmax, and samples, exactly as in chapter 4.
• generateAtTemperature(model, startWord, length, temperature, rng) repeats sampleNext and returns the sentence.
• formatReport(prompt, report, windowSize) returns a printable string.

3. Recipe: The llmlab Command-Line Tool

Create src/index.js. It is a small CLI: give it a prompt, and it prints the usage report and a sentence from the toy model.

import { corpus, words } from "./corpus.js";
import { trainBigram } from "./bigram.js";
import { mulberry32 } from "./rng.js";
import { usageReport, formatReport, generateAtTemperature } from "./report.js";

// Usage: node src/index.js "your prompt" [--temp 0.7] [--seed 1]
const args = process.argv.slice(2);
const flag = (name, fallback) => {
  const i = args.indexOf(name);
  return i === -1 ? fallback : Number(args[i + 1]);
};
const prompt = args.find((a, i) => !a.startsWith("--") && !args[i - 1]?.startsWith("--"));
if (!prompt) {
  console.log('Usage: node src/index.js "your prompt" [--temp 0.7] [--seed 1]');
  process.exit(1);
}
const temperature = flag("--temp", 0.7);
const seed = flag("--seed", 1);

console.log(formatReport(prompt, usageReport(prompt)));

const model = trainBigram(words(corpus));
const last = words(prompt).at(-1);
const start = model.has(last) ? last : "the";
console.log(`\ntoy model, temperature ${temperature}, seed ${seed}, start "${start}":`);
console.log(generateAtTemperature(model, start, 10, temperature, mulberry32(seed)));

The tool starts the toy model from the last word of your prompt if the model knows it, and from the if it does not. Run it three ways:

node src/index.js "Explain how the coffee shop app works"
# prompt:        "Explain how the coffee shop app works"
# characters:    37
# words:         7
# tokens (est):  10
# cost (example): $0.000030
# window used:   0.13% of 8000 tokens
#
# toy model, temperature 0.7, seed 1, start "the":
# the river and the small coffee shop needs good app

node src/index.js "Who is Dara" --temp 0
# prompt:        "Who is Dara"
# characters:    11
# words:         3
# tokens (est):  3
# cost (example): $0.000009
# window used:   0.04% of 8000 tokens
#
# toy model, temperature 0, seed 1, start "dara":
# dara writes code and dara writes code and dara writes

node src/index.js "Who is Dara" --temp 1.5 --seed 3
# prompt:        "Who is Dara"
# characters:    11
# words:         3
# tokens (est):  3
# cost (example): $0.000009
# window used:   0.04% of 8000 tokens
#
# toy model, temperature 1.5, seed 3, start "dara":
# dara thinks about the coffee sokha makes iced coffee shop

Everything here is deterministic, because the random generator is seeded (--seed) and the prompt is the same. Change the seed and the sentence changes. At temperature 0 the seed does not matter, because the model always takes the most likely word. Notice that it also falls into a loop ("dara writes code and dara writes code ..."): always taking the single most likely word is predictable, but it can repeat itself. Running node src/index.js with no prompt prints the usage line and exits.

A few things to notice. A 37-character prompt is about 10 tokens, which costs a tiny fraction of a cent at the example price and uses a tiny part of the example window. A single short question is almost free. Cost and limits become real when you send long text, many requests, or long conversations, which is the next section.

4. Why Cost Grows Faster Than You Expect

Hosted models usually charge per token, with one price for input tokens (everything you send) and often a higher price for output tokens (what the model writes). Remember from Context Windows and Memory that a chat does not remember: your app resends the whole history with every new message. So input tokens grow with every turn.

import { estimateCost } from "./src/tokenizer.js";

const PRICE_IN = 3;   // example: dollars per million input tokens
const PRICE_OUT = 15; // example: output tokens usually cost more

const turns = 10;
const userTokens = 100;   // each user message
const replyTokens = 300;  // each model reply
let history = 0;
let inputTotal = 0;
let outputTotal = 0;

for (let turn = 1; turn <= turns; turn++) {
  const sent = history + userTokens;   // the whole history is resent
  inputTotal += sent;
  outputTotal += replyTokens;
  history = sent + replyTokens;
  if (turn === 1 || turn === 5 || turn === 10) console.log("turn", turn, "sends", sent, "input tokens");
}

const cost = estimateCost(inputTotal, PRICE_IN) + estimateCost(outputTotal, PRICE_OUT);
console.log("input tokens total:", inputTotal);
console.log("output tokens total:", outputTotal);
console.log("one conversation: $" + cost.toFixed(4));
console.log("1,000 conversations a day: $" + (cost * 1000).toFixed(2));
// turn 1 sends 100 input tokens
// turn 5 sends 1700 input tokens
// turn 10 sends 3700 input tokens
// input tokens total: 19000
// output tokens total: 3000
// one conversation: $0.1020
// 1,000 conversations a day: $102.00

All the numbers are examples, but the arithmetic is real, and the output is stable. Look at what happened: the user only typed 100 tokens per turn (1,000 in total), yet the app sent 19,000 input tokens, nineteen times more, because every turn re-sends everything before it. The cost of a conversation grows faster than its length.

Ways developers keep this under control:

• Trim or summarise old history (trimHistory from chapter 7).
• Keep prompts short and specific. Do not paste whole documents when a paragraph will do.
• Use a smaller, cheaper model for easy jobs (classify, extract, reformat) and a larger one only for hard jobs.
• Prompt caching. Some providers charge less for the repeated start of a prompt, so keep the stable part (system message, instructions) at the front. Ask your provider how it works.
• Set a maximum output length to cap runaway replies.
• Measure: log token counts per request, since your bill is made of them.

5. Latency: Why Answers Stream

Remember the loop: one token at a time, each needing a full pass through the model. A long answer needs hundreds of passes, so it takes seconds even when each one is fast. Two practical consequences:

• Longer output means a slower answer. Asking for "a short answer" saves both time and money.
• Streaming. Most APIs can send tokens as they are produced, so the user sees text appearing at once instead of staring at a blank screen. The total time is the same; the feeling is much better.

6. Small Models, Large Models, and Your Laptop

A model is mostly a long list of numbers (parameters). Memory needed is simple school arithmetic: number of parameters times bytes per parameter. A technique called quantization stores each number with fewer bits, so the model takes less memory at a small cost in quality.

const bytesPerParam = { "16-bit": 2, "8-bit": 1, "4-bit": 0.5 };

for (const billions of [1, 7, 70]) {
  const row = Object.entries(bytesPerParam).map(
    ([name, bytes]) => `${name}: ${(billions * bytes).toFixed(1)} GB`
  );
  console.log(`${String(billions).padStart(2)}B parameters ->`, row.join(" | "));
}
// 1B parameters -> 16-bit: 2.0 GB | 8-bit: 1.0 GB | 4-bit: 0.5 GB
// 7B parameters -> 16-bit: 14.0 GB | 8-bit: 7.0 GB | 4-bit: 3.5 GB
// 70B parameters -> 16-bit: 140.0 GB | 8-bit: 70.0 GB | 4-bit: 35.0 GB

This is only the memory for the weights themselves. Running a model also needs extra memory for the context and working space, so real needs are higher. Still, it explains a lot:

• A small model (around a billion or a few billion parameters) can run on a normal laptop or even a phone. It is fast, cheap, and private because your text never leaves your machine, but it is weaker at hard reasoning and has less world knowledge.
• A large model (tens or hundreds of billions of parameters) needs expensive server hardware. It is usually stronger on hard tasks, but slower and more costly per request.
• Bigger is not always better for your task. For a narrow job such as sorting messages into categories, a small model with a good prompt can be good enough at a fraction of the cost. Test on your own examples (your evals from the last chapter) before paying for the biggest one.

7. Open Models vs Closed Models

Two broad ways to get a model:

Two points of honesty. First, "open" is used loosely: some models share only their weights (the numbers), some also share the training code, and very few share their training data. The licences differ too, so read the licence before building a product on one. Second, the quality gap between open and closed models changes quickly, and any claim about "which is best" will be out of date soon, so compare them on your own task instead.

8. What LLMs Are Good At, and What They Are Not

Everything above tells you what to expect. A model that predicts text from patterns is good at jobs where patterns and fluent language are the work:

Usually good

• Drafting, rewriting, shortening, translating between common languages, and fixing grammar.
• Summarising and explaining text you give it.
• Brainstorming ideas, names, and examples.
• Turning messy text into structured data (with validation).
• Writing and explaining common code, tests, and regular expressions.
• Answering questions about well-documented topics.
• Acting as a patient tutor you can ask the same question ten times.

Often unreliable

• Exact facts, numbers, dates, and quotations, especially unusual ones.
• Long or exact arithmetic and counting. (Use code.)
• Anything after its training cut-off, unless you provide it.
• Small, local, or rare topics with little written about them.
• Less common languages: quality is often lower for languages with less training text, so test Khmer output with a native speaker before you rely on it.
• Long multi-step plans where one early mistake spoils everything after.
• Knowing what it does not know (chapter 9).

Not a good idea at all

• Making final decisions about people (hiring, grades, medicine, law) with no human responsible.
• Handling secrets such as passwords or keys.
• Being the only check on its own work.

A useful rule of thumb: LLMs are excellent first drafters and weak final authorities. Use them to get started fast, then verify.

9. Bias, Safety and Privacy

Bias. A model learns from human writing, and human writing contains stereotypes and gaps. If some groups, languages, or places are underrepresented, the model is usually worse for them. If the text contains prejudice, the model may repeat it. Makers try to reduce this with careful data choices and tuning (chapter 5), but it is not solved. As a developer: test your app with a variety of names, languages, and situations, and never let it be the only judge of people.

Safety. A helpful model can be asked to help with harmful things, or tricked into ignoring its instructions. Providers add safeguards, but they are not perfect. Prompt injection (chapter 8) is the practical version: text from a web page or a user can contain instructions that the model might follow. Defence in depth means: keep instructions separate from untrusted data, give the model the fewest permissions possible, run code it writes in a sandbox (Docker Sandbox Crash Course), and ask a human before anything irreversible.

Privacy. Whatever you type into a hosted model is sent to someone else's computer. Rules to follow:

• Never paste passwords, API keys, tokens, or private keys. Once sent, you cannot take them back.
• Do not paste other people's private information (IDs, phone numbers, medical or school records) unless you are sure you are allowed to.
• Read the provider's data policy: does it keep your text, and does it use it to train future models? Options and answers differ by provider and plan.
• For sensitive data, consider a local open-weight model, or remove the sensitive details before sending.

Training data and ownership. Models are trained on huge amounts of text, and the question of who owns or may use that text is being debated, argued in courts, and treated differently in different countries. Nobody can give you one final answer yet. If you publish something an AI wrote, you are still responsible for it being correct and not copying others.

10. Energy and the Environment

Training a large model takes a lot of computer time on specialised chips, and every answer afterwards uses some electricity too. That is true, and it is a fair thing to worry about. But exact figures vary widely with the model, the hardware, the data centre, and the way the electricity is made, and numbers quoted online are often old, rough, or missing context, so this chapter does not quote any.

What you can say honestly:

• Training is expensive; one answer is comparatively cheap, but the number of answers is enormous when millions of people use a model every day, so the total matters.
• You influence the cost. Smaller models, shorter prompts, caching, and not asking when a search or a calculator would do the job all use less energy, and they also save money.

11. Does It Think? Does It Understand?

This is the question that starts the biggest arguments, so answer it carefully.

What we know for sure: a language model is a function that turns text into probabilities for the next token. You built a tiny one. It does not have a body, feelings, or a life. Inside a trained model there is nothing but numbers being multiplied.

What is not settled: whether that process, at huge scale, amounts to some kind of understanding. Researchers and philosophers disagree.

• One view: it is only pattern matching. It copies the shape of reasoning without any grasp of meaning, like a parrot with a very large memory.
• Another view: to predict text well about physics, code, or people, a model must build internal representations that capture how things work, and that is a real, if limited, kind of understanding.
• Evidence exists for both views. Models solve problems that need several steps, and they also fail on simple variations that a child would handle, which suggests their abilities are uneven and different from ours.

A balanced thing to tell an audience: "It does something useful that looks like understanding in many tasks, and breaks in ways that a person's understanding usually would not. Whether to call that thinking depends on how you define the word, and scientists have not agreed." For building software, the practical answer matters most: do not assume it understands. Test it on your cases.

12. What About Jobs?

Nobody can predict the future of work exactly, and anyone who sounds certain is guessing. A measured picture:

• Tasks change before whole jobs vanish. Most jobs are bundles of tasks. LLMs are good at some tasks (drafting, summarising, first-pass code) and weak at others (responsibility, trust, hands-on work, knowing the local situation). Jobs that include many automatable tasks will change the most.
• Some work gets cheaper and faster, which can mean fewer people doing it, more of it being done, or both. History offers examples of each, so the outcome is not fixed in advance.
• New work appears: building, testing, checking, and governing AI systems, and doing the parts it cannot.
• People who use these tools well have an advantage over people who do not, in most fields, including programming.
• Fundamentals still matter. To verify what a model writes you need to understand the subject. A programmer who can read and test code is far more useful than one who can only paste it.

13. Try It Yourself

1. Explore the CLI. Run node src/index.js with five different prompts, including one in Khmer. Look at characters, words, and tokens (est). Why is the word count so small for the Khmer prompt (hint: where are the spaces?), and why might the real token count of Khmer text not match the characters-divided-by-4 rule of thumb?
2. Real prices. Look up the price page of any model provider (the numbers change, so read it today). Put the real input and output prices and the real window size into usageReport using its options and compute the cost of your own 1,000-word essay.
3. Add output cost. Extend usageReport to accept outputTokens and outputPricePerMillion, and return totalCost. Update formatReport to print it.
4. Budget a bot. Using section 4's loop, change turns, userTokens and replyTokens. At what number of turns does the input total pass 100,000 tokens? Then add a maxHistoryTokens limit (use trimHistory from chapter 7 as a model) and see how much it saves.
5. Window warning. Make the CLI print WARNING: prompt uses over 80% of the context window when windowUsed is above 0.8. Test with a very long prompt (use "word ".repeat(7000)).

14. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You now understand what is happening inside the tools everyone is talking about: text becomes tokens, tokens become numbers, a trained network turns them into probabilities, one token is chosen, and the loop repeats. You also know the honest limits: it makes things up, it forgets, it costs money, and it needs checking. That is a strong foundation for a talk, a classroom discussion, or your own project.

The next series, Build a Chatbot with an LLM API (planned), moves from the toy to the real thing: calling a hosted model from your own code, streaming answers, managing conversation history, and adding tools. Whenever an AI system you build can run code or touch files, read Docker Sandbox Crash Course first, so the model's actions happen somewhere safe.