Slopes and Derivatives

Table of Contents

1. Slope: How Steep Is It?
2. A Curve Has a Different Slope Everywhere
3. Nudge and Measure: The Numeric Derivative
4. How Big Should the Nudge Be?
5. A Few Rules, Checked by Code
6. Many Knobs: Partial Derivatives and the Gradient
7. The Chain Rule in Plain Words
8. Flat Slope: Where the Bottom Is
9. The AI Connection: Slope of the Error
10. What Is Simplified Here
11. Recipe: The calculus Module
12. Try It Yourself
13. Quick Review: Common Questions

1. Slope: How Steep Is It?

Picture a road going up a hill. If it climbs 3 metres every 10 metres you walk forward, the road is steep. If it climbs 1 metre every 10 metres, it is gentle. If it never climbs, it is flat. That number, "how much it goes up for each step forward", is the slope:

                          /|
                        /  |  rise = 6
                      /    |
                    /______|
                      run = 3

   slope = rise / run = 6 / 3 = 2

Slope is old school math: for the line y = 2x + 1, the slope is 2. Every time x grows by 1, y grows by 2. Let us write a small helper that measures the slope between two points of any function, and try it on a straight line and on a curve.

// demo-slope.js
// Slope = rise / run = (change in y) / (change in x).
const slope = (f, x1, x2) => (f(x2) - f(x1)) / (x2 - x1);

const line = (x) => 2 * x + 1;
console.log(slope(line, 0, 1));
console.log(slope(line, 10, 14));
// 2
// 2

For a straight line, the slope is the same wherever you measure it: between 0 and 1, or between 10 and 14, the answer is 2. Both lines are exact and stable.

2. A Curve Has a Different Slope Everywhere

Now bend the road. The function y = x * x (square it) makes a bowl:

   y
   9 |                 *
   4 |            *
   1 |       *
   0 |  *--------------------- x
        0   1    2    3

Near x = 0 the curve is nearly flat. Further right it climbs faster and faster. There is no single slope. The steepness depends on where you stand. Measure the slope across different gaps:

// demo-curve.js
const slope = (f, x1, x2) => (f(x2) - f(x1)) / (x2 - x1);
const square = (x) => x * x;

console.log(slope(square, 0, 1));
console.log(slope(square, 1, 2));
console.log(slope(square, 3, 4));

// Squeeze the gap around x = 3.
for (const gap of [1, 0.1, 0.01, 0.001]) {
  console.log(gap, slope(square, 3, 3 + gap).toFixed(4));
}
// 1
// 3
// 7
// 1 7.0000
// 0.1 6.1000
// 0.01 6.0100
// 0.001 6.0010

Three things to see. First, the slope grows as x grows: 1, then 3, then 7. Second, "the slope at x = 3" is a slippery idea, because a slope needs two points. Third, look at the squeezing loop. As the gap gets smaller, the answer settles toward 6. We say: the slope of x * x right at x = 3 is 6.

That settled number is the derivative at x = 3. A derivative is the slope of the curve at one exact point, found by looking at a gap so small it is almost nothing. All lines here are exact and stable.

3. Nudge and Measure: The Numeric Derivative

Here is the whole idea in a sentence: nudge the input a tiny bit and see how much the output moves.

   derivative of f at x  ~  (f(x + h) - f(x - h)) / (2 h)
                              ^^^^^^^^^^^^^^^^
                    nudge up by h, nudge down by h, compare

We look slightly to the right and slightly to the left of x, and take the slope between those two points. Looking on both sides gives a more accurate answer than looking on one side only. h is the tiny nudge, and we use 0.00001 by default. This is the derivative function in the recipe at the end of the chapter: three lines, no algebra.

// demo-derivative.js
import { derivative } from "./src/calculus.js";

const square = (x) => x * x;
console.log(derivative(square, 3));
console.log(derivative(square, 3).toFixed(4));
console.log(derivative(square, -2).toFixed(4));
console.log(derivative(square, 0).toFixed(4));
console.log(derivative(Math.sin, 0).toFixed(4));
// 6.000000000039306
// 6.0000
// -4.0000
// 0.0000
// 1.0000

The first line shows the honest raw result: 6.000000000039306, not exactly 6. A computer stores numbers with limited precision, so a numeric derivative is a very good approximation. That is why we round with toFixed when printing.

Reading the results: at x = 3 the slope is 6. At x = -2 it is -4, negative, meaning the curve is going down there as you move right. At x = 0 it is 0: flat, the very bottom of the bowl. The last line is the slope of the sine curve at 0, which is 1. All lines are stable.

The sign of the derivative is the most useful part. Positive means "the output goes up if I increase x". Negative means "the output goes down if I increase x". Zero means "flat right here".

4. How Big Should the Nudge Be?

Why 0.00001 and not something smaller, like 0.0000000000001? Try several nudge sizes on the slope of x * x at x = 3, whose true answer is 6:

// demo-stepsize.js
import { derivative } from "./src/calculus.js";

const square = (x) => x * x;
for (const h of [1, 0.1, 1e-5, 1e-10, 1e-13]) {
  console.log(String(h).padEnd(7), derivative(square, 3, h));
}
// 1       6
// 0.1     6.000000000000005
// 0.00001 6.000000000039306
// 1e-10   6.000000496442226
// 1e-13   5.995204332975845

Two lessons. A smaller nudge is not always a better one: at 1e-13 the answer is worse than at 0.1, because the two function values are almost the same number and the computer loses the tiny difference between them. And a big nudge is fine for this particular function (even h = 1 is exact for x * x, a happy accident of this shape), but for a more wiggly function a big nudge would miss the curve. So 1e-5 is a practical middle: small enough to be accurate, big enough that the subtraction keeps its digits. It is a rule of thumb, not a law. These outputs are stable on the same kind of computer; the last digits could differ slightly on others.

5. A Few Rules, Checked by Code

Mathematicians have worked out exact rules so you do not have to measure every time. You do not need to memorize them. Here are a few common ones, written as "function becomes its slope-function":

   x * x    ->  2x
   x^3      ->  3 x^2
   5x + 2   ->  5       (a straight line: its slope)
   sin(x)   ->  cos(x)
   e^x      ->  e^x     (it is its own slope)
   ln(x)    ->  1 / x

Let us not trust them. Test each against the nudge-and-measure answer at x = 1.5:

// demo-rules.js
import { derivative } from "./src/calculus.js";

const checks = [
  ["x^2",    (x) => x ** 2,      (x) => 2 * x],
  ["x^3",    (x) => x ** 3,      (x) => 3 * x ** 2],
  ["5x + 2", (x) => 5 * x + 2,   () => 5],
  ["sin x",  (x) => Math.sin(x), (x) => Math.cos(x)],
  ["e^x",    (x) => Math.exp(x), (x) => Math.exp(x)],
  ["ln x",   (x) => Math.log(x), (x) => 1 / x],
];
for (const [name, f, rule] of checks) {
  const x = 1.5;
  console.log(name.padEnd(7), derivative(f, x).toFixed(5), rule(x).toFixed(5));
}
// x^2     3.00000 3.00000
// x^3     6.75000 6.75000
// 5x + 2  5.00000 5.00000
// sin x   0.07074 0.07074
// e^x     4.48169 4.48169
// ln x    0.66667 0.66667

In each row the left number is the measured slope and the right number is the rule's answer. They match to five decimals. This is the habit to keep: when you are unsure about a derivative, measure it. In Build a Neural Network you will use exactly this trick to check hand-written backpropagation. (The ln and e^x curves are the ones from Functions, Growth and Logarithms.)

All lines are stable.

6. Many Knobs: Partial Derivatives and the Gradient

A real model has many numbers to adjust, not one. Imagine a radio with several knobs. You turn one knob a little, keeping the others still, and note how the sound changes. Do that for each knob and you have a list: one slope per knob.

• Each of those slopes is a partial derivative: the slope when you nudge only one input and hold the rest.
• The list of all of them is the gradient.

The gradient is just a vector of slopes. And it has a lovely meaning:

The gradient points uphill, in the direction where the output grows fastest. Walk the opposite way to go downhill.

For a bowl in two inputs, f(x, y) = x*x + y*y, the lowest point is (0, 0):

   y
   |    gradient at (3, 4) is [6, 8]: it points away from the bottom
   |   . (3,4) ----> uphill
   |
   +--------- x     bottom at (0,0)

The gradient(f, xs) function in the recipe takes a function that receives an array and returns a number, and an array of inputs. For each position it nudges that one input up and down:

// demo-gradient.js
import { gradient } from "./src/calculus.js";

// A bowl: f(x, y) = x^2 + y^2. Lowest point at (0, 0).
const bowl = ([x, y]) => x * x + y * y;
console.log(gradient(bowl, [3, 4]).map((g) => g.toFixed(3)));
console.log(gradient(bowl, [-1, 2]).map((g) => g.toFixed(3)));
console.log(gradient(bowl, [0, 0]).map((g) => g.toFixed(3)));

// An uneven bowl: x matters 10x more than y.
const narrow = ([x, y]) => 10 * x * x + y * y;
console.log(gradient(narrow, [1, 1]).map((g) => g.toFixed(3)));

// Three inputs.
const f3 = ([a, b, c]) => a * b + c;
console.log(gradient(f3, [2, 5, 7]).map((g) => g.toFixed(3)));
// [ '6.000', '8.000' ]
// [ '-2.000', '4.000' ]
// [ '0.000', '0.000' ]
// [ '20.000', '2.000' ]
// [ '5.000', '2.000', '1.000' ]

Read them like this:

• At (3, 4) the gradient is [6, 8]: each input's slope, 2x and 2y. Both are positive, so uphill is up-and-right, and downhill is the opposite, toward (0, 0).
• At (-1, 2) the first slope is -2: increasing x here lowers the output, so the bottom is to the right of x = -1.
• At the bottom (0, 0) the gradient is all zeros: nowhere is uphill.
• In the uneven bowl, the first number is 20 and the second 2. The function reacts ten times more strongly to x, and the gradient says so.
• With three inputs the gradient has three numbers. For a*b + c, the slope for a is b = 5, for b is a = 2, and for c is 1.

Every line is stable. This is a gradient, and a gradient is the thing a model uses to decide how to change all of its weights at the same time.

7. The Chain Rule in Plain Words

Often one machine feeds another. You pour x into machine 1 to get u, then pour u into machine 2 to get the final answer. How does the final answer change when you nudge x?

   x ----[ inner: 3x + 1 ]----> u ----[ outer: u * u ]----> out

Think of gears. Turn the first gear a little. The second turns some amount per turn of the first. The third turns some amount per turn of the second. Overall speed is the product of the speeds:

Slope of the whole chain = (slope of inner) times (slope of outer, measured at the point where the inner sent it).

That is the chain rule. Check it with numbers instead of trusting it:

// demo-chain.js
import { derivative } from "./src/calculus.js";

const inner = (x) => 3 * x + 1;   // first machine
const outer = (u) => u * u;       // second machine
const both = (x) => outer(inner(x));

const x = 2;
const slopeInner = derivative(inner, x);
const slopeOuter = derivative(outer, inner(x));
console.log(slopeInner.toFixed(4), slopeOuter.toFixed(4));
console.log((slopeInner * slopeOuter).toFixed(4));
console.log(derivative(both, x).toFixed(4));
// 3.0000 14.0000
// 42.0000
// 42.0000

At x = 2 the inner machine sends out u = 7 with slope 3. The outer machine at u = 7 has slope 14. Multiply: 3 * 14 = 42. And the slope of the whole chain, measured directly, is also 42. The numbers agree on every run.

Why this matters: a neural network is a long chain of small machines (layers). The chain rule lets us get the slope of the final error with respect to a weight in an early layer by multiplying the slopes of the machines in between. Applied layer by layer, backwards from the end, this is called backpropagation, and you will write it in Build a Neural Network.

8. Flat Slope: Where the Bottom Is

Look back at the bowl: at the bottom the slope is zero. That gives a way to find a lowest point: go where the slope is zero. Better, use the slope as a compass. If it is positive, go left (down). If it is negative, go right (down). Take a small step against the slope and the output drops.

// demo-minimum.js
import { derivative } from "./src/calculus.js";

// a bowl shifted: lowest at x = 2
const f = (x) => (x - 2) ** 2 + 1;
for (const x of [0, 1, 1.9, 2, 2.1, 3, 4]) {
  console.log(x, derivative(f, x).toFixed(3));
}

// One step downhill from x = 5, using step = 0.1 * slope.
let x = 5;
console.log(f(x));
x = x - 0.1 * derivative(f, x);
console.log(x.toFixed(3), f(x).toFixed(3));

// Flat is not always a minimum.
const g = (x) => x ** 3;
console.log(derivative(g, 0).toFixed(6));
console.log(g(-0.5), g(0), g(0.5));
// 0 -4.000
// 1 -2.000
// 1.9 -0.200
// 2 0.000
// 2.1 0.200
// 3 2.000
// 4 4.000
// 10
// 4.400 6.760
// 0.000000
// -0.125 0 0.125

Left of x = 2 the slope is negative (the curve is going down toward the bottom). Right of 2 it is positive. Exactly at 2, zero. The slope even gets smaller as you approach, which is a hint that steps should shrink on their own near the bottom.

The step demo is the whole of learning in two lines: from x = 5, where f is 10, we step 0.1 * slope against the slope and land on 4.4, where f is 6.76. Lower. Repeat that and you reach the bottom. That loop is gradient descent, the subject of the next chapter.

The last lines are a warning: x * x * x has slope 0 at x = 0, yet that is not a bottom. The curve goes down before it and up after it, so "flat" only means "a candidate". A flat slope can also be the top of a hill, a local minimum (a dip that is not the lowest dip), or a flat shelf. Real error surfaces have many such places. Chapter 5 comes back to this.

All lines here are stable.

9. The AI Connection: Slope of the Error

Now put it all together with something AI-shaped. Using the study data from src/data.js (hours studied and exam score), suppose a model guesses score = w * hours for some weight w. How wrong is the guess? Average the squared mistakes: that is the loss, and the next chapter explains it fully.

// demo-loss-slope.js
import { study } from "./src/data.js";
import { derivative, gradient } from "./src/calculus.js";

// How wrong is the guess "score = w * hours"? Average squared error.
const loss = (w) =>
  study.reduce((sum, [hours, score]) => sum + (w * hours - score) ** 2, 0) / study.length;

for (const w of [5, 8, 10, 12]) {
  console.log(w, loss(w).toFixed(1), derivative(loss, w).toFixed(1));
}

// Two knobs: score = w * hours + b
const loss2 = ([w, b]) =>
  study.reduce((sum, [h, s]) => sum + (w * h + b - s) ** 2, 0) / study.length;
console.log(gradient(loss2, [0, 0]).map((g) => g.toFixed(1)));
// 5 1911.1 -480.8
// 8 815.2 -249.8
// 10 469.6 -95.8
// 12 432.0 58.2
// [ '-865.8', '-142.4' ]

Each line shows the weight, the loss, and the slope of the loss. At w = 5 the loss is large and the slope is very negative: raising w reduces the error a lot. At w = 10 the slope is still negative, but smaller. At w = 12 the slope has turned positive, meaning we have gone too far. The best weight, where the slope crosses zero, is somewhere between 10 and 12. That is the model telling you which way to turn the knob, and how hard.

The final line has two knobs, w and b, and the gradient has two numbers. Both are negative, so increasing both weights lowers the loss.

This is the core of training: compute the loss, compute its gradient with respect to every weight, step each weight against its gradient, repeat. Here the loss had one or two weights. A real language model's loss has billions, and the process is the same. See How a Model Is Trained for the bigger picture. All lines are stable; they depend only on the study table.

10. What Is Simplified Here

• Real systems rarely use nudge-and-measure. With billions of weights that would mean billions of nudges per step. They use automatic differentiation, which applies the chain rule exactly and in one backward pass. Our numeric version is for understanding, and for checking.
• We only used simple functions. The ideas are the same for the functions inside a network.
• "Walk downhill" is simplified. Real training uses improved versions of the idea with extra tricks, covered at a high level in later chapters.

11. Recipe: The calculus Module

Save this as src/calculus.js in your mathai folder. It has no imports.

// Numeric derivatives: no algebra, just "nudge and measure".
export function derivative(f, x, h = 1e-5) {
  return (f(x + h) - f(x - h)) / (2 * h);
}

// Gradient of f(array) -> number, one partial derivative per input.
export function gradient(f, xs, h = 1e-5) {
  return xs.map((_, i) => {
    const up = xs.slice();
    const down = xs.slice();
    up[i] += h;
    down[i] -= h;
    return (f(up) - f(down)) / (2 * h);
  });
}

Notes on the code:

• derivative is the central difference from section 3.
• gradient loops over the inputs. For input i it makes two copies of the array, up with that one input nudged up and down with it nudged down, then measures the slope. The slice() copies keep the original array untouched.
• The function you pass to gradient must take one array and return one number. That is why the examples destructure with ([x, y]) => ....

12. Try It Yourself

1. Predict first. On paper, find the slope of y = x * x at x = 5 using the rule. Then check with derivative. You should see about 10.
2. Sign check. Pick any x for f(x) = (x - 4) ** 2. Without running code, say whether the slope is positive or negative and which direction is downhill. Then confirm with derivative.
3. Make a new rule. Test the rule that the slope of 1 / x is -1 / (x * x) at a few values of x. How close do the two numbers get at x = 0.01? (Notice how fast things change near zero.)
4. Gradient by hand. For f([a, b]) = 3*a + a*b, work out the two partial derivatives (3 + b and a) and compare with gradient(f, [2, 4]). You should see [7, 2].
5. Chain rule. Change inner to (x) => x + 5 and outer to (u) => u ** 3. Predict the slope of both at x = 1, then check it. (Inner slope is 1, outer slope at u = 6 is 3 * 36.)
6. Walk downhill. Using f(x) = (x - 2) ** 2 + 1, start at x = 10 and repeat x = x - 0.1 * derivative(f, x) twenty times in a loop. Print x each time. Where is it heading (you should see about 2.09 after twenty steps)? What if you change 0.1 to 1.1?

13. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now measure which way is downhill. The next step is to actually walk there, again and again, until the error is small. That is Gradient Descent and Loss.