Probabilities, Temperature and Sampling

Table of Contents

1. The Missing Step Between Scores and Words
2. Logits: Raw Scores
3. Softmax: Scores to Probabilities
4. Temperature: Sharpen or Flatten
5. Sampling: Rolling the Weighted Die
6. Greedy: Always Take the Top
7. Top-k and Top-p: Cut the Tail
8. Same Prompt at Temperature 0, 0.7 and 1.5
9. Is Temperature 0 Always the Same Answer?
10. Choosing Settings in Real Projects
11. Recipe: The sampling Module
12. Try It Yourself
13. Quick Review: Common Questions

1. The Missing Step Between Scores and Words

In Build a Tiny Language Model our model counted word pairs, divided by the total, and rolled a weighted die. That worked because counting gives us probabilities directly. A real neural network does not count anything at generation time. It ends each step with one raw score for every token it knows, and those scores are not probabilities yet.

So every real LLM has a short pipeline at the very end of each step:

context  ->  model  ->  scores  ->  softmax  ->  probabilities
                                                      |
                              (temperature, top-k, top-p adjust)
                                                      |
                                              pick one token

This chapter builds each stage of that pipeline: softmax (scores to probabilities), temperature (how adventurous the model is), sampling (picking), and the two filters called top-k and top-p. These are the same knobs you will see in the settings of AI tools and in the options of an LLM API later. When someone says "turn the temperature down", this is what they mean.

You need src/bigram.js and src/rng.js from the previous chapter. All code is plain ESM JavaScript; run each demo with node.

2. Logits: Raw Scores

The raw scores a model produces are called logits (say "LOH-jits"). A logit can be any number: positive, negative, large, small. It only means "bigger is more likely". Suppose a tiny model is choosing between three next words and gives these scores:

"coffee" has the highest score, so the model prefers it. But how much does it prefer it? A score of 2.0 versus 1.0, is that "twice as likely"? Scores alone cannot tell us, and they cannot be used to roll a die, because probabilities must be positive and add up to 1. We need a function that converts any list of numbers into valid probabilities.

3. Softmax: Scores to Probabilities

That function is softmax. It does two simple things:

1. Make every score positive by raising e (about 2.718) to the power of the score. In JavaScript that is Math.exp(score). A big score becomes a big positive number, and a negative score becomes a small positive number.
2. Divide each by the total, so the results add up to 1.

export function softmax(logits) {
  const max = Math.max(...logits); // subtract max so exp() cannot overflow
  const exps = logits.map((x) => Math.exp(x - max));
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / sum);
}

The - max line is a safety trick. Math.exp(1000) is too big for a JavaScript number and becomes Infinity, which would break everything. Subtracting the biggest score first makes every exponent 0 or negative, and it does not change the final answer, because the same factor cancels out when you divide. Save this as demo-softmax.js (after creating src/sampling.js from the recipe at the end, or copy softmax into it):

import { softmax } from "./src/sampling.js";

const show = (list) => list.map((x) => x.toFixed(3)).join("  ");

const logits = [2.0, 1.0, 0.1];
const probs = softmax(logits);
console.log(show(probs));
console.log(probs.reduce((a, b) => a + b, 0));
console.log(show(softmax([102.0, 101.0, 100.1])));
console.log(show(softmax([1, 1, 1, 1])));
console.log(show(softmax([1000, 999])));
console.log(show(softmax([-3, 0, 3])));
// 0.659  0.242  0.099
// 1
// 0.659  0.242  0.099
// 0.250  0.250  0.250  0.250
// 0.731  0.269
// 0.002  0.047  0.950

Read the output line by line:

• Line 1: our three scores become 65.9%, 24.2% and 9.9%. "coffee" is preferred, but "tea" and "water" still have a real chance.
• Line 2: the total is 1, as probabilities must be. (Floating point can sometimes print 0.9999999999999999 here; both mean 1.)
• Line 3: we added 100 to every score and the answer did not change. Softmax only cares about the differences between scores.
• Line 4: equal scores give equal probabilities.
• Line 5: scores of 1000 and 999 work fine, no Infinity, thanks to the - max trick.
• Line 6: a gap of 3 between scores makes a huge difference: 95% for the top one.

Everything printed here is stable. There is no randomness yet. The softmax idea is also the heart of the attention step in Embeddings and Attention.

4. Temperature: Sharpen or Flatten

Temperature is a number you divide the logits by before softmax. It is the most famous setting in all of AI, and it is one line of code:

export function applyTemperature(logits, t) {
  if (!(t > 0)) throw new Error("temperature must be > 0 (use greedy for 0)");
  return logits.map((x) => x / t);
}

Think of what dividing does. If t is less than 1, say 0.5, every score doubles, so the gaps between scores grow, and softmax gives the leader an even bigger share. If t is more than 1, scores shrink toward each other, the gaps close, and the probabilities flatten out toward equal. Try it in demo-temperature.js:

import { softmax, applyTemperature } from "./src/sampling.js";

const show = (list) => list.map((x) => x.toFixed(3)).join("  ");
const logits = [2.0, 1.0, 0.1];

for (const t of [0.2, 0.5, 1, 2, 5]) {
  console.log(`t=${String(t).padEnd(3)}`, show(softmax(applyTemperature(logits, t))));
}
try {
  applyTemperature(logits, 0);
} catch (error) {
  console.log(error.message);
}
// t=0.2 0.993  0.007  0.000
// t=0.5 0.864  0.117  0.019
// t=1   0.659  0.242  0.099
// t=2   0.502  0.304  0.194
// t=5   0.400  0.327  0.273
// temperature must be > 0 (use greedy for 0)

Watch the first number in each row, the probability of the top choice:

• At t=0.2 it is 99.3%. The model is almost certain. Low temperature means focused, predictable.
• At t=1 it is 65.9%. This is the model's own opinion, unchanged.
• At t=5 it is 40%, and the three choices are almost equal. High temperature means flat, adventurous, sometimes silly.

The ranking never changes: "coffee" is always first. Temperature does not make the model smarter or dumber. It changes how strictly we obey its ranking. That is the one-sentence answer to "what is temperature?".

You cannot divide by 0, so a temperature of exactly 0 cannot go through this formula. Libraries treat "temperature 0" as a special case that means "always pick the top choice", which is greedy decoding (below). You can think of it as the limit as t gets closer and closer to 0.

5. Sampling: Rolling the Weighted Die

Once you have probabilities, you sample: pick one option at random, with each option's chance equal to its probability. It is the same ruler and dart idea from the last chapter, now as a reusable function:

export function sampleIndex(probs, rng = Math.random) {
  const r = rng();
  let running = 0;
  for (let i = 0; i < probs.length; i++) {
    running += probs[i];
    if (r < running) return i;
  }
  return probs.length - 1; // guards against rounding error
}

It returns the index of the chosen option (0, 1, 2, ...) rather than the word, so it works for any list: words, tokens, anything.

Now connect it to our bigram model. A bigram model gives probabilities, but softmax and temperature expect logits. Luckily you can turn probabilities into logits with a logarithm: Math.log(prob). This is the exact reverse of softmax, so softmax(logits) gives back the original probabilities at temperature 1. Create demo-sampling.js:

import { corpus, words } from "./src/corpus.js";
import { trainBigram, nextWordProbs } from "./src/bigram.js";
import { softmax, applyTemperature, sampleIndex, greedy } from "./src/sampling.js";
import { mulberry32 } from "./src/rng.js";

const model = trainBigram(words(corpus));
const options = nextWordProbs(model, "the");
const probs = options.map((o) => o.prob);
const labels = options.map((o) => o.word);

console.log(labels.join(" "));
console.log(probs.map((p) => p.toFixed(2)).join(" "));
console.log("greedy:", labels[greedy(probs)]);

const rng = mulberry32(1);
const tally = {};
for (let i = 0; i < 10000; i++) {
  const word = labels[sampleIndex(probs, rng)];
  tally[word] = (tally[word] ?? 0) + 1;
}
console.log(tally);

// Same table, three temperatures: the share of the top choice
const logits = probs.map(Math.log);
for (const t of [0.5, 1, 2]) {
  const p = softmax(applyTemperature(logits, t));
  console.log(`t=${t}: top ${p[0].toFixed(2)}, bottom ${p[p.length - 1].toFixed(2)}`);
}
// coffee students app river morning table window next small
// 0.21 0.21 0.13 0.13 0.08 0.08 0.08 0.04 0.04
// greedy: coffee
// {
//   river: 1267,
//   coffee: 2143,
//   app: 1238,
//   small: 390,
//   students: 2052,
//   morning: 828,
//   table: 841,
//   window: 788,
//   next: 453
// }
// t=0.5: top 0.30, bottom 0.01
// t=1: top 0.21, bottom 0.04
// t=2: top 0.16, bottom 0.07

What did we learn?

• After "the" the model has nine options. "coffee" and "students" are tied at 21%.
• We sampled 10,000 times. "coffee" was picked 2,143 times (21.4%) and "students" 2,052 times (20.5%), "small" only 390 times (3.9%). Over many rolls, the counts match the probabilities. That is what sampling means.
• Temperature 1 samples from the model's own distribution. At 0.5 the top choice grows to 30% and the least likely shrinks to 1%. At 2 the gap closes to 16% versus 7%.

Because we used mulberry32(1), the whole output is stable. With Math.random the tallies would differ a little each run (for example, "coffee" might be 2,090 or 2,190), but they would always stay close to the probabilities.

6. Greedy: Always Take the Top

The simplest way to choose has no randomness at all: always pick the option with the highest probability. This is called greedy decoding:

export function greedy(probs) {
  let best = 0;
  for (let i = 1; i < probs.length; i++) {
    if (probs[i] > probs[best]) best = i;
  }
  return best;
}

Greedy always returns the same answer for the same input. That sounds ideal, and for some jobs it is: classifying text, extracting a fixed field, answering a factual lookup. But for writing it has a known weakness, and you will see it in section 8: text from greedy choice often falls into repetitive loops, because the most likely continuation of a phrase tends to lead back to the same phrase.

(When two options tie, greedy returns the first one it saw, which is why we sort the options in a fixed order in nextWordProbs. In the output above, "coffee" wins the tie against "students" only because of that order.)

7. Top-k and Top-p: Cut the Tail

High temperature has a nasty side effect. It makes unlikely words more likely, including the really unlikely ones, the nonsense tail of the list. With a vocabulary of tens of thousands of tokens, there are thousands of bad options, and even a tiny chance each adds up. Two filters fix this by cutting the tail before sampling.

Top-k: keep only the k most likely options, throw the rest away, and scale the survivors so they add up to 1 again.

Top-p (also called nucleus sampling): keep the smallest group of top options whose probabilities add up to at least p, for example 0.9. When the model is sure, that group is tiny (maybe one word). When it is unsure, the group is large. So top-p adapts to the situation, while top-k always keeps a fixed number.

export function topK(probs, k) {
  const keep = new Set(
    probs.map((p, i) => [p, i]).sort((a, b) => b[0] - a[0]).slice(0, k).map(([, i]) => i)
  );
  const kept = probs.map((p, i) => (keep.has(i) ? p : 0));
  const sum = kept.reduce((a, b) => a + b, 0);
  return kept.map((p) => p / sum);
}

export function topP(probs, p) {
  const order = probs.map((x, i) => [x, i]).sort((a, b) => b[0] - a[0]);
  const keep = new Set();
  let total = 0;
  for (const [x, i] of order) {
    keep.add(i);
    total += x;
    if (total >= p) break;
  }
  const kept = probs.map((x, i) => (keep.has(i) ? x : 0));
  const sum = kept.reduce((a, b) => a + b, 0);
  return kept.map((x) => x / sum);
}

Both functions return a list of the same length, with zeros for removed options, so the indexes still line up with your words. Save demo-filters.js:

import { topK, topP } from "./src/sampling.js";

const show = (list) => list.map((x) => x.toFixed(2)).join(" ");
const probs = [0.5, 0.25, 0.15, 0.07, 0.03];
console.log("original", show(probs));
console.log("topK 2  ", show(topK(probs, 2)));
console.log("topP 0.8", show(topP(probs, 0.8)));
console.log("topP 0.9", show(topP(probs, 0.9)));
console.log("topP 0.4", show(topP(probs, 0.4)));
// original 0.50 0.25 0.15 0.07 0.03
// topK 2   0.67 0.33 0.00 0.00 0.00
// topP 0.8 0.56 0.28 0.17 0.00 0.00
// topP 0.9 0.56 0.28 0.17 0.00 0.00
// topP 0.4 1.00 0.00 0.00 0.00 0.00

How to read it:

• topK 2 keeps 0.50 and 0.25, then rescales: 0.50 / 0.75 = 0.67 and 0.25 / 0.75 = 0.33.
• topP 0.8: 0.50 is only 50%, adding 0.25 gives 75%, still under 80%, so 0.15 is added too and the total reaches 90%. The group is the top three.
• topP 0.9 gives the same group, because the top three add up to 90% exactly (floating point arithmetic could in other examples tip such a borderline case either way).
• topP 0.4: the first option alone is already 50%, so the group has one member and the result is 100% on it. Top-p with a small p behaves like greedy.

The output of all five lines is stable. In real systems, you can usually combine temperature with one of these filters, and many APIs apply temperature first, then the filter, then sample. That is the order our generateSampled function uses next. (The exact order and defaults vary between systems, so read the documentation of the one you use.)

8. Same Prompt at Temperature 0, 0.7 and 1.5

Time to see the effect on actual text. In the recipe you will add generateSampled, a version of generate that takes the options temperature, topK and topP, with temperature 0 meaning greedy. Our "prompt" is the start word "the" and we ask for 12 words. Save demo-generate-sampled.js:

import { corpus, words } from "./src/corpus.js";
import { trainBigram } from "./src/bigram.js";
import { generateSampled } from "./src/sampling.js";
import { mulberry32 } from "./src/rng.js";

const model = trainBigram(words(corpus));

for (const temperature of [0, 0.7, 1.5]) {
  console.log(`temperature ${temperature}`);
  for (const seed of [1, 2, 3]) {
    const text = generateSampled(model, "the", 12, { temperature }, mulberry32(seed));
    console.log("  ", text);
  }
}

console.log("topK 2, temperature 1.5");
for (const seed of [1, 2, 3]) {
  console.log("  ", generateSampled(model, "the", 12, { temperature: 1.5, topK: 2 }, mulberry32(seed)));
}
console.log("topP 0.8, temperature 1.5");
for (const seed of [1, 2, 3]) {
  console.log("  ", generateSampled(model, "the", 12, { temperature: 1.5, topP: 0.8 }, mulberry32(seed)));
}
// temperature 0
//    the coffee shop is quiet and dara writes code and dara writes
//    the coffee shop is quiet and dara writes code and dara writes
//    the coffee shop is quiet and dara writes code and dara writes
// temperature 0.7
//    the river and the small coffee shop needs good app saves time
//    the river and dara is calm in the students laugh and a
//    the river and the coffee and the students order iced coffee shop
// temperature 1.5
//    the morning and a table dara drinks iced tea the small app
//    the table by the river is a phone sokha opens the table
//    the table by the coffee sokha makes iced coffee shop needs good
// topK 2, temperature 1.5
//    the students order more iced tea the students come to the coffee
//    the students order iced coffee and the coffee shop after lunch the
//    the students order iced coffee and dara writes code and the coffee
// topP 0.8, temperature 1.5
//    the river and the window dara writes code for a student and
//    the morning and the river is a phone sokha opens the table
//    the morning and the coffee after lunch sokha cleans the river and

Notice:

• Temperature 0 gives the same text for every seed. Greedy ignores the random generator. It also shows the repetition problem: "dara writes code and dara writes" is already looping, and if we asked for 50 words it would keep cycling through the same few words.
• Temperature 0.7 and 1.5 give different text for each seed. The difference between 0.7 and 1.5 is hard to see in three examples, because our toy table is small, but 1.5 jumps around more ("the table by the coffee sokha makes iced coffee shop").
• Top-k and top-p tame the high-temperature text. With topK: 2 all three samples stay on common, sensible paths ("the students order iced coffee"), even though temperature is still 1.5.

All of this output is stable because every run uses a seeded generator and a fixed model. In a real chat app, you get a different result each time you press "regenerate", because the seed is chosen freshly.

What is the "right" temperature? There is no single one. It depends on the job, which we cover in section 10. A value around the middle, such as 0.7, is a common default in many tools, but check the defaults of the tool you use, because they differ.

9. Is Temperature 0 Always the Same Answer?

In our toy, temperature 0 gives the same output every time, guaranteed: it is a pure function with no randomness. Is that also true when you call a big hosted model with temperature 0? Usually the outputs are very similar, but not always identical. Here is the honest picture:

• Temperature 0 removes the random choice. The model should pick the top token each time.
• But the scores themselves can vary slightly. Large models run on many parallel chips, and floating point addition gives slightly different results depending on the order the numbers are added. Order can change with how requests are batched together on the server or which hardware handles yours. Two tokens with almost equal scores can then swap places, and once one word differs, everything after it can differ too, because each new word depends on the earlier ones.
• Providers may also change things behind the scenes: a model update, a different serving setup, or other settings. Some APIs offer a seed option and describe it as "best effort" for repeatable output, not as a promise.

So the practical rule: never build a program that needs a hosted model's answer to be byte-for-byte identical. If your code compares output to a fixed string, or your test expects the exact same sentence every time, expect it to break occasionally. Instead, ask for structured output (see Prompts, Roles and Structured Output), check the properties you care about, and save results if you need repeatability.

Our own llmlab gets exact repeatability because it is small, runs on one machine, does its arithmetic in a fixed order, and we control the seed. Real systems have none of those luxuries.

10. Choosing Settings in Real Projects

Think of temperature and the filters as a dial between reliable and creative. These are rules of thumb, not laws. Test on your own task:

Some practical advice:

1. Change one setting at a time. Temperature and top-p both change the spread, and many guides suggest adjusting one, not both. Check what your provider recommends.
2. Low is not the same as correct. Temperature 0 gives the model's most likely answer, which is not necessarily the true one. A confident wrong answer stays wrong at any temperature. We return to this in Why Models Hallucinate.

11. Recipe: The sampling Module

Add this chapter's tools to llmlab. Create src/sampling.js:

// src/sampling.js
import { nextWordProbs } from "./bigram.js";

// Scores (any numbers) -> probabilities that are positive and sum to 1.
export function softmax(logits) {
  const max = Math.max(...logits); // subtract max so exp() cannot overflow
  const exps = logits.map((x) => Math.exp(x - max));
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / sum);
}

// t < 1 sharpens, t > 1 flattens. t must be above 0.
export function applyTemperature(logits, t) {
  if (!(t > 0)) throw new Error("temperature must be > 0 (use greedy for 0)");
  return logits.map((x) => x / t);
}

// Pick an index using probabilities and a random number in [0, 1).
export function sampleIndex(probs, rng = Math.random) {
  const r = rng();
  let running = 0;
  for (let i = 0; i < probs.length; i++) {
    running += probs[i];
    if (r < running) return i;
  }
  return probs.length - 1; // guards against rounding error
}

// Index of the biggest probability. No randomness.
export function greedy(probs) {
  let best = 0;
  for (let i = 1; i < probs.length; i++) {
    if (probs[i] > probs[best]) best = i;
  }
  return best;
}

// Keep the k most likely options, zero the rest, renormalize.
export function topK(probs, k) {
  const keep = new Set(
    probs.map((p, i) => [p, i]).sort((a, b) => b[0] - a[0]).slice(0, k).map(([, i]) => i)
  );
  const kept = probs.map((p, i) => (keep.has(i) ? p : 0));
  const sum = kept.reduce((a, b) => a + b, 0);
  return kept.map((p) => p / sum);
}

// Keep the smallest group of top options whose total reaches p.
export function topP(probs, p) {
  const order = probs.map((x, i) => [x, i]).sort((a, b) => b[0] - a[0]);
  const keep = new Set();
  let total = 0;
  for (const [x, i] of order) {
    keep.add(i);
    total += x;
    if (total >= p) break;
  }
  const kept = probs.map((x, i) => (keep.has(i) ? x : 0));
  const sum = kept.reduce((a, b) => a + b, 0);
  return kept.map((x) => x / sum);
}

// Bigram generation with all the knobs. temperature 0 means greedy.
export function generateSampled(model, startWord, length, options = {}, rng = Math.random) {
  const { temperature = 1, topK: k, topP: p } = options;
  const out = [startWord];
  let current = startWord;
  for (let i = 1; i < length; i++) {
    const opts = nextWordProbs(model, current);
    if (opts.length === 0) break;
    let idx;
    if (temperature === 0) {
      idx = greedy(opts.map((o) => o.prob));
    } else {
      const logits = opts.map((o) => Math.log(o.prob));
      let probs = softmax(applyTemperature(logits, temperature));
      if (k) probs = topK(probs, k);
      if (p) probs = topP(probs, p);
      idx = sampleIndex(probs, rng);
    }
    current = opts[idx].word;
    out.push(current);
  }
  return out.join(" ");
}

The exported functions are softmax(logits), applyTemperature(logits, t), sampleIndex(probs, rng), greedy(probs), topK(probs, k), topP(probs, p) and generateSampled(model, startWord, length, options, rng). The last one is a bigram-specific helper that wires the pieces together the same way a real decoding loop does: scores, temperature, softmax, filter, sample, append. A later chapter's command line tool reuses it, so keep the file as shown. If you keep an src/index.js, add export * from "./sampling.js";.

12. Try It Yourself

1. Softmax by hand. Compute softmax([0, 0, 0]) and softmax([0, 1]) in your head or on paper, then check with code. Why are the second answer's numbers about 0.27 and 0.73?
2. Find the crossover. For logits [2.0, 1.0, 0.1], loop t from 0.1 to 10 in steps of 0.1 and print the probability of the top choice. At what t is it close to 50%? What happens as t becomes very large?
3. Test the sampler. Sample 100,000 times from [0.7, 0.2, 0.1] with a seeded generator and print the share of each index. Are they close to 70%, 20% and 10%? Try with 10 samples. What changes, and why?
4. Break it on purpose. Call generateSampled with temperature 5 and no filter, then with topK: 3. Compare 5 outputs of each. Which one makes more sense?
5. Try a different start. Use "sokha" and "dara" as start words at temperature 0. Why does the output of greedy generation change with the start word, but not with the seed?

13. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now explain the last step of how an LLM writes: scores become probabilities with softmax, temperature reshapes them, filters trim them, and sampling picks. But so far the scores came from counting word pairs. Where do the numbers inside a real model come from? Continue with How a Model Is Trained, where you watch a tiny model learn its own scores by gradient descent.