Gradient Descent and Loss

Table of Contents

1. Learning Is Reducing Wrongness
2. Loss: One Number for "How Wrong"
3. Walking Downhill in the Fog
4. Gradient Descent on a Bowl
5. The Learning Rate: Too Small, Right, Too Big
6. Fitting a Line to the Study Data
7. Reading the Loss History
8. Why the Line Fit Is Slow
9. Too Much Data per Step: Stochastic and Mini-Batch
10. Local Minima and Why Big Models Still Train
11. The Link to Training a Real Model
12. Recipe: The descent Module
13. Try It Yourself
14. Quick Review: Common Questions

1. Learning Is Reducing Wrongness

Think about learning to throw a ball into a basket. You throw, the ball lands short, and you throw a little harder. It lands too far, so you ease off. After a few throws you stop thinking about it. You never solved an equation. You measured how wrong you were and adjusted a little in the direction that makes you less wrong.

That is the whole idea of training an AI model. It has two parts:

1. A number that says how wrong the model is right now. We call it the loss.
2. A rule that says which way to nudge the model's numbers so the loss gets smaller. We call it gradient descent.

Repeat this thousands or millions of times and the model gets better. In Slopes and Derivatives you built the tool for part 2: the gradient. This chapter uses it. By the end you will have a program that finds the best straight line through real data by itself, and you will be able to explain, in two sentences, what "training" means.

Every demo file below sits next to src/ and runs with node. They use gradient from src/calculus.js (chapter 4) and study from src/data.js (chapter 1).

2. Loss: One Number for "How Wrong"

Here is the study data from chapter 1: hours studied per week and the exam score of ten students. We want a rule that predicts score from hours, score = w * hours + b. The number w says how many points one extra hour adds, and b is the score of someone who studied zero hours. We have to choose w and b. Different choices give different lines, some good, some bad:

hours:    1   2   3   4   5   6   7   8   9   10
real:    48  52  60  63  71  74  80  83  88   93
A:       45  50  55  60  65  70  75  80  85   90    (5h + 40)

To compare lines we need one number per line that says how bad it is. The usual choice is the mean squared error, or MSE:

1. For each student, take the error: prediction minus real score.
2. Square it. This makes every error positive, so a miss of +5 and a miss of -5 do not cancel out. It also punishes big misses much more than small ones: a miss of 10 counts as 100, a miss of 1 counts as 1.
3. Take the average over all students.

A perfect line has MSE 0. The bigger the MSE, the worse the line. src/descent.js has this function ready as mse(predictions, targets). Create the demo and try the three lines from the picture:

// demo-loss.js
import { mse } from "./src/descent.js";
import { study } from "./src/data.js";

const hours = study.map(([h]) => h);
const scores = study.map(([, s]) => s);

// three guesses for the rule  score = w * hours + b
const guesses = [
  { name: "A: 5h + 40", w: 5, b: 40 },
  { name: "B: always 70", w: 0, b: 70 },
  { name: "C: 10h + 0", w: 10, b: 0 },
];

for (const g of guesses) {
  const predictions = hours.map((h) => g.w * h + g.b);
  console.log(g.name, "loss =", mse(predictions, scores).toFixed(3));
}

console.log(mse([1, 2, 3], [1, 2, 3]));
console.log(mse([2, 3, 4], [1, 2, 3]));

// output
A: 5h + 40 loss = 15.100
B: always 70 loss = 209.600
C: 10h + 0 loss = 469.600
0
1

Line A is far better than B or C, and the numbers say so: 15.1 against 209.6 and 469.6. The last two lines are sanity checks. Perfect predictions give 0. Predictions that are off by exactly 1 everywhere give 1, because 1 * 1 = 1 averaged ten times is still 1. All of these outputs are stable; nothing here is random.

Say it out loud: "The loss is one number that says how wrong the model is. Training means making that number small."

Loss is also called a cost or objective. It depends on what we choose to measure. MSE is a good fit for predicting a number like a score. For predicting the next word, models use a different loss, which you will build in the next chapter. The walking-downhill idea below works the same for any loss.

3. Walking Downhill in the Fog

Imagine you are on a hill in thick fog and you want to reach the valley. You cannot see the valley. You can only feel the ground under your feet. So you do this: feel which way the ground slopes down, take a small step that way, and repeat.

Now draw the loss as the height of the ground. The two things we are choosing, w and b, are like your position. A good (w, b) is the bottom of the valley. A bad one is high on the hillside:

loss
 |\                          /
 | *  start (bad guess)    /
 |  \                     /
 |   *                   /
 |    \__*__*__*_______/      <- the bottom: small loss
 +--------------------------------> value of w

You learned in chapter 4 that the gradient is the slope in every direction at once. It points uphill, toward higher loss. So to go downhill, we step the opposite way. For one number x and loss f(x), one step is:

x_new = x - learning_rate * slope_at_x

The learning rate is how big a step you dare to take. For many numbers at once, do the same to each one using its own partial derivative. That is the entire algorithm. It is called gradient descent.

4. Gradient Descent on a Bowl

Start with the simplest hill: a bowl with one number. We choose f(x) = (x - 3)^2 + 1. Its lowest point is at x = 3, where the loss is 1. Pretend we do not know that, start at x = 10, and let the algorithm find it. The function gradientDescent takes f, a start point, a learning rate lr and a number of steps. It returns every step so you can print them:

// demo-bowl.js
import { gradientDescent } from "./src/descent.js";

// a bowl with its lowest point at x = 3, where f = 1
const f = ([x]) => (x - 3) ** 2 + 1;

const path = gradientDescent({ f, start: [10], lr: 0.1, steps: 30 });

for (const p of path) {
  if ([0, 1, 2, 3, 5, 10, 20, 30].includes(p.step)) {
    console.log(p.step, p.xs[0].toFixed(4), p.value.toFixed(4));
  }
}

// output
0 10.0000 50.0000
1 8.6000 32.3600
2 7.4800 21.0704
3 6.5840 13.8451
5 5.2938 6.2613
10 3.7516 1.5649
20 3.0807 1.0065
30 3.0087 1.0001

Read it column by column: step number, current x, current loss. The loss falls from 50 toward 1, and x slides from 10 toward 3. Notice the shape of the progress. Early steps are big (10 to 8.6) and later steps are tiny (3.0807 to 3.0087). That is not a bug. Near the bottom the ground is almost flat, so the slope is tiny, so the step is tiny. The walker slows down by itself as it arrives.

Where did the gradient come from? We passed no grad function, so gradientDescent used the gradient from chapter 4: "nudge each input a tiny bit and measure what the loss does". Everything here is stable output.

Say it out loud: "Gradient descent: measure the slope, step the opposite way, repeat. The steps shrink on their own near the bottom."

5. The Learning Rate: Too Small, Right, Too Big

The learning rate lr is the one setting you must pick yourself. It controls the size of each step. Same bowl, same start, five different values:

// demo-lr.js
import { gradientDescent } from "./src/descent.js";

const f = ([x]) => (x - 3) ** 2 + 1;

for (const lr of [0.001, 0.1, 0.9, 1.0, 1.1]) {
  const path = gradientDescent({ f, start: [10], lr, steps: 30 });
  const last = path[path.length - 1];
  console.log(`lr ${lr}: x = ${last.xs[0].toFixed(4)}, loss = ${last.value.toExponential(3)}`);
}

// output
lr 0.001: x = 9.5920, loss = 4.445e+1
lr 0.1: x = 3.0087, loss = 1.000e+0
lr 0.9: x = 3.0087, loss = 1.000e+0
lr 1: x = 10.0000, loss = 5.000e+1
lr 1.1: x = 1664.6341, loss = 2.761e+6

Three different stories:

• Too small (0.001). After 30 steps x has barely moved, from 10 to 9.59. It is heading the right way, but you would wait a very long time. Like crossing a river one centimeter at a time.
• Good (0.1, and surprisingly 0.9). Both end near 3. With 0.9 the walker actually jumps over the bottom and back each time, but every jump is a bit smaller, so it still lands there.
• Too big (1.0 and 1.1). With 1.0 the walker bounces between 10 and -4 forever and never improves: loss 50 at the end, the same as at the start. With 1.1 each jump overshoots by more than the last, and the loss explodes to millions. This is called divergence.

For this bowl you can even compute the border. Each step multiplies the distance to the bottom by 1 - 2 * lr. For lr = 0.1 that is 0.8, which shrinks. For 0.9 it is -0.8, which flips sides but shrinks. For 1.0 it is -1, so it never shrinks. For 1.1 it is -1.2, which grows. The border for this bowl is lr = 1. Other problems have other borders, and you usually find yours by trying values, often by factors of ten: 0.1, 0.01, 0.001.

Real AI work is the same: the learning rate is one of the first things people tune. Outputs above are stable; they use no randomness.

Say it out loud: "The learning rate is the step size. Too small is slow, too big overshoots and can blow up."

6. Fitting a Line to the Study Data

Now the real task: find w and b so that score = w * hours + b fits the ten students, using the MSE from section 2 as the loss.

The function fitLine(points, { lr, steps }) does it in one go. It starts at w = 0, b = 0 (a flat line at score 0, which is terrible), and at every step it:

1. Goes through all the points and computes the error w*x + b - y.
2. Adds up the partial derivatives of the MSE. For the weight, the slope is 2 * err * x averaged over points. For the bias, it is 2 * err averaged over points. These are the gradients from chapter 4, written out by algebra instead of nudging, because that is faster.
3. Moves w and b a little in the opposite direction.
4. Records w, b and the loss in a history list.

// demo-fit.js
import { fitLine } from "./src/descent.js";
import { study } from "./src/data.js";

const { w, b, history } = fitLine(study, { lr: 0.01, steps: 1000 });

for (const h of history) {
  if ([0, 1, 2, 5, 10, 50, 100, 200, 500, 1000].includes(h.step)) {
    console.log(h.step, h.w.toFixed(3), h.b.toFixed(3), h.loss.toFixed(3));
  }
}
console.log("final:", w.toFixed(3), b.toFixed(3));

// output
0 0.000 0.000 5277.600
1 8.658 1.424 602.370
2 10.493 1.867 384.856
5 10.920 2.463 365.245
10 10.801 3.321 350.250
50 9.903 9.572 250.532
100 8.974 16.042 164.957
200 7.611 25.531 71.897
500 5.743 38.535 7.053
1000 5.096 43.041 1.494
final: 5.096 43.044

The machine began with a useless line (loss 5277.6) and, with no help from us, ended at score ≈ 5.1 * hours + 43. In words: "each extra study hour is worth about 5 points, and someone who studies zero hours gets about 43." Compare with line A from section 2 (5h + 40, loss 15.1): the learned line is a little better, loss 1.49.

The final: line differs by a tiny amount from step 1000 in the table because fitLine applies one more update after recording the last history entry. All numbers here are stable, with no randomness anywhere.

A quick use of the result: what would the line predict for a student who studies 11 hours? We also check it against the textbook formula for the best line, which needs no loop at all:

// demo-fit-more.js
import { fitLine } from "./src/descent.js";
import { study } from "./src/data.js";

// the same line found by the textbook formula (no loop, no learning rate)
const n = study.length;
const mx = study.reduce((s, [x]) => s + x, 0) / n;
const my = study.reduce((s, [, y]) => s + y, 0) / n;
let sxy = 0;
let sxx = 0;
for (const [x, y] of study) {
  sxy += (x - mx) * (y - my);
  sxx += (x - mx) ** 2;
}
const wExact = sxy / sxx;
const bExact = my - wExact * mx;
console.log("formula:", wExact.toFixed(3), bExact.toFixed(3));

const fit = fitLine(study, { lr: 0.01, steps: 5000 });
console.log("5000 steps:", fit.w.toFixed(3), fit.b.toFixed(3));
console.log("predict 11 hours:", (fit.w * 11 + fit.b).toFixed(1));

// output
formula: 5.006 43.667
5000 steps: 5.006 43.667
predict 11 hours: 98.7

After 5000 steps gradient descent lands on exactly the same line as the formula, to three decimals. So the walker really does find the bottom of the valley. For a straight line you do not need descent; the formula is faster. We use descent because it works for models with no formula, which is every neural network. Also, the prediction 98.7 is an extrapolation beyond the data, and a score above 100 is impossible. A line does not know about limits.

7. Reading the Loss History

The history array is the most useful debugging tool in machine learning. Plot or print the loss against the step number and read it like a doctor reads a pulse. Look back at the table in section 6:

• Loss 5277 -> 602 -> 384: huge first steps, since the start was awful.
• Loss 365 -> 350 -> 250: a plateau. Section 8 explains why.
• Loss 71 -> 7 -> 1.5: steady progress, slowing near the end.

A healthy loss curve is a slide that flattens out: steep at first, then slower and slower as it nears the bottom.

Quick rules of thumb: if the loss goes up or turns into NaN, the learning rate is too big. If it falls very slowly and steadily, it is too small, or you just need more steps. If it bounces but trends down, that is normal when each step uses only a few examples (section 9). If it is flat from step one, nothing is learning.

The number to watch is the trend, not any single value. Try the learning rate cliff yourself in exercise 2: for this data, a rate of 0.025 works well but 0.03 makes the loss explode. The best rate is often just below the edge of a cliff like that, which is why people start low and raise it carefully.

8. Why the Line Fit Is Slow

You may wonder why a line fit needed 1000 steps and still was not done, while the bowl needed 30. Two reasons, both worth knowing.

First, the two directions have very different slopes. The weight w multiplies the hours, which go up to 10, so a small change in w moves the loss a lot. The bias b multiplies nothing (just 1), so changing it moves the loss only a little. The valley is a long, narrow trench: steep walls and a nearly flat floor.

A learning rate big enough to move b quickly is too big for w, and makes w bounce. So we have to use a small rate and wait for b to crawl. That is the plateau you saw in the table, where b creeps up from 3 to 9 to 16 to 25 to 38. The fix is to standardize the inputs so every direction has a similar slope. You will do that in Statistics and the Big Picture, and see it make descent much faster.

Second, here is what the gradient actually says at the starting point. We build the loss as a function of [w, b] and ask the chapter 4 gradient for its slope at [0, 0]:

// demo-gradient-check.js
import { gradient } from "./src/calculus.js";
import { study } from "./src/data.js";

// loss as a function of [w, b]
const loss = ([w, b]) =>
  study.reduce((s, [x, y]) => s + (w * x + b - y) ** 2, 0) / study.length;

// at w = 0, b = 0 the loss is high; which way is downhill?
const g = gradient(loss, [0, 0]);
console.log("gradient at [0, 0]:", g.map((v) => v.toFixed(2)));
console.log("one step, lr 0.01 :", [0, 0].map((v, i) => (v - 0.01 * g[i]).toFixed(3)));

// output
gradient at [0, 0]: [ '-865.80', '-142.40' ]
one step, lr 0.01 : [ '8.658', '1.424' ]

Both slopes are negative, so stepping against them raises both numbers. The gradient is a big -865.8 for w and a small -142.4 for b: steep for w, gentle for b, as described. After one step you get w = 8.658, b = 1.424, matching step 1 in section 6. Same machine, whether the derivative is nudged or written by algebra. Stable output.

9. Too Much Data per Step: Stochastic and Mini-Batch

fitLine looks at all ten students before every single step. That is called batch gradient descent. With ten students it is fine. Now imagine training data of a trillion words. Reading all of it just to take one step would take forever.

The trick: a small random sample gives a rough but useful gradient. Like tasting one spoon of soup to judge the whole pot. You will not get the exact answer, but it points about the right way, and it is far cheaper. There are three flavours:

• Batch: use all the examples per step. Exact but slow per step.
• Stochastic (SGD): use just 1 random example. Very noisy, very cheap.
• Mini-batch: use a small group, like 32. The usual choice.

"Stochastic" just means "random". Here is mini-batch descent on the same study data, with a batch of 3 random students per step, using a seeded random generator so you get the same output as printed:

// demo-minibatch.js
import { study } from "./src/data.js";
import { mulberry32 } from "./src/rng.js";
import { mse } from "./src/descent.js";

const rand = mulberry32(42);
const lr = 0.01;
const batchSize = 3;
let w = 0;
let b = 0;

// loss on ALL the data, only so we can watch progress
function lossAll() {
  const predictions = study.map(([x]) => w * x + b);
  return mse(predictions, study.map(([, y]) => y));
}

for (let step = 0; step <= 3000; step++) {
  // pick a few random points instead of all ten
  const batch = [];
  for (let i = 0; i < batchSize; i++) {
    batch.push(study[Math.floor(rand() * study.length)]);
  }

  let dw = 0;
  let db = 0;
  for (const [x, y] of batch) {
    const err = w * x + b - y;
    dw += (2 * err * x) / batchSize;
    db += (2 * err) / batchSize;
  }
  w -= lr * dw;
  b -= lr * db;

  if ([0, 10, 100, 500, 1000, 2000, 3000].includes(step)) {
    console.log(step, w.toFixed(3), b.toFixed(3), lossAll().toFixed(3));
  }
}

// output
0 11.380 1.593 385.817
10 11.041 3.567 349.596
100 8.972 15.774 168.139
500 5.904 38.482 8.123
1000 5.237 43.052 2.276
2000 4.922 43.638 1.709
3000 4.993 43.608 1.428

It reaches a line close to the true one (5.006 and 43.667) even though no single step ever saw all the data. Look at the last rows: the values jiggle around the answer (4.922, then 4.993) instead of settling exactly, because every step sees a different handful of students. That jiggle is the price of cheap steps. In practice people lower the learning rate as training goes on, so the jiggle calms down.

Because we use mulberry32(42), the output is the same every run. This is the real recipe behind training large models: sample a mini-batch, compute the gradient on it, step, repeat. One pass through all the data is called an epoch.

10. Local Minima and Why Big Models Still Train

So far every bowl had one bottom. Real loss surfaces can have several valleys. A local minimum is a valley bottom that is not the deepest one. Gradient descent only feels the ground right under its feet, so it rolls into whichever valley is nearest to where it started:

// demo-local-minima.js
import { gradientDescent } from "./src/descent.js";

// two valleys: a shallow one near x = 0.693, a deeper one near x = -1.443
const f = ([x]) => x ** 4 + x ** 3 - 2 * x ** 2;

for (const start of [-2, -0.5, 0.5, 2]) {
  const path = gradientDescent({ f, start: [start], lr: 0.01, steps: 500 });
  const last = path[path.length - 1];
  console.log(`start ${start}: x = ${last.xs[0].toFixed(3)}, f = ${last.value.toFixed(3)}`);
}

// output
start -2: x = -1.443, f = -2.833
start -0.5: x = -1.443, f = -2.833
start 0.5: x = 0.693, f = -0.397
start 2: x = 0.693, f = -0.397

Starting from -2 or -0.5 the walker finds the deeper valley. Starting from 0.5 or 2 it gets stuck in the shallow one, with a worse loss, and never learns that a better place exists. All four outputs are stable.

That sounds frightening for AI. Why do giant models train well anyway? Here is the honest state of knowledge, simplified:

• A model with billions of numbers has billions of directions. To be stuck in a real local minimum, the loss must go up in every direction at once. In so many directions that is rare. There is almost always some direction that still leads down.
• In practice, the valley bottoms training reaches are often good enough, and noisy mini-batches can shake the walker out of shallow dents. Real training also uses smarter step rules, like momentum, but they are all variations of the same idea.

Be careful with this claim. It is an empirical observation and a research topic, not a guarantee. Nobody can promise a training run finds the best possible model.

11. The Link to Training a Real Model

Put what you built next to How a Model Is Trained. That chapter described training as a loop: guess the next word, measure the error, nudge the numbers. Now you know every part of the loop in math:

• "The model's numbers" are w and b (billions of weights in a real model).
• "How wrong is it?" is the loss: MSE here, cross-entropy for words.
• "Which way should I nudge?" is the gradient of the loss.
• "Look at a few examples at a time" is a mini-batch, and one pass over all the data is an epoch.

What is simplified here, honestly. Our model has two numbers; a real one has billions, and the arithmetic is enormous. We computed the gradient by algebra or by nudging, while real frameworks use automatic differentiation, which you will meet in the neural network chapter. And plain descent is rarely used alone: real training uses improved variants with momentum and carefully scheduled learning rates.

None of this changes the picture you can give a class: a model is a big pile of numbers, the loss says how wrong it is, and training is nudging the numbers downhill, over and over.

12. Recipe: The descent Module

Save this as src/descent.js. It needs gradient from src/calculus.js, which you built in Slopes and Derivatives:

import { gradient } from "./calculus.js";

// Walk downhill on f. Returns every step so you can plot or print it.
export function gradientDescent({ f, grad, start, lr = 0.1, steps = 50 }) {
  const g = grad ?? ((xs) => gradient(f, xs));
  let xs = start.slice();
  const path = [{ step: 0, xs: xs.slice(), value: f(xs) }];
  for (let step = 1; step <= steps; step++) {
    const gr = g(xs);
    xs = xs.map((x, i) => x - lr * gr[i]);
    path.push({ step, xs: xs.slice(), value: f(xs) });
  }
  return path;
}

export const mse = (predictions, targets) =>
  predictions.reduce((s, p, i) => s + (p - targets[i]) ** 2, 0) / predictions.length;

// Fit y = w*x + b by gradient descent on mean squared error.
export function fitLine(points, { lr = 0.01, steps = 1000 } = {}) {
  let w = 0;
  let b = 0;
  const n = points.length;
  const history = [];
  for (let step = 0; step <= steps; step++) {
    let dw = 0;
    let db = 0;
    let loss = 0;
    for (const [x, y] of points) {
      const err = w * x + b - y;
      loss += err * err;
      dw += (2 * err * x) / n;
      db += (2 * err) / n;
    }
    history.push({ step, w, b, loss: loss / n });
    w -= lr * dw;
    b -= lr * db;
  }
  return { w, b, history };
}

gradientDescent({ f, grad, start, lr, steps }) walks downhill on any function f that takes an array of numbers. If you give it a grad function it uses that, otherwise it computes the gradient by nudging. mse(predictions, targets) is the mean squared error. fitLine(points, { lr, steps }) fits y = w*x + b and returns { w, b, history }.

13. Try It Yourself

1. Find the border. In demo-lr.js, try learning rates 0.5, 0.95, 0.99 and 1.01. Which ones converge? Does it match the border lr = 1 that the multiplier 1 - 2 * lr predicts?
2. The line cliff. Call fitLine(study, { lr, steps: 200 }) with lr of 0.025 and 0.03 and print the last loss in history. You should see roughly 7 against an explosion beyond 10^56.
3. A different bowl. Run gradientDescent on f = ([x, y]) => x * x + 10 * y * y from [5, 5] with lr: 0.05 and steps: 100. Then try lr: 0.11. Why does the second blow up even though 0.11 is a small number? (Hint: which of the two directions is steeper?)
4. Your own data. Invent ten pairs of (hours of sleep, mood score) and run fitLine. Does the line have a positive or negative w? Try the formula from demo-fit-more.js on your data and compare.
5. Pass your own gradient. Write grad for f = ([x]) => (x - 3) ** 2 by hand (the answer is [2 * (x - 3)]) and pass it as grad to gradientDescent. You should see the same path as before.
6. Count the epochs. In demo-minibatch.js, how many epochs is 3000 steps of batch size 3 over 10 students? Then try batchSize = 1 and batchSize = 10. How does the jiggle at the end change?

14. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now say what training is: measure the loss, follow the gradient downhill, repeat. But we only trained a model that predicts a number. Language models predict which word comes next, and for that you need probabilities, logarithms and a different loss. That is the next chapter: Probability, Logs and Softmax.