Build a Neural Network

Table of Contents

1. A Neuron Is a Weighted Vote
2. Activations: Sigmoid and ReLU
3. Why Activations Are Needed
4. The Problem a Line Cannot Solve: XOR
5. The Forward Pass
6. Backpropagation Is the Chain Rule, Layer by Layer
7. Training on XOR
8. Checking Backprop with Nudge and Measure
9. Counting Parameters and Scaling Up
10. What Real Frameworks Automate
11. Recipe: The nn Module
12. Try It Yourself
13. Quick Review: Common Questions

1. A Neuron Is a Weighted Vote

Imagine you decide whether to go to a football match. You listen to three friends. Your best friend's opinion counts a lot. A cousin you rarely see counts a little. One friend always disagrees with you, so you count his opinion as negative. You add up the votes, each one multiplied by how much you trust it, and you add a personal leaning: "I like football, so I start with a small plus." If the total is high enough, you go.

That is a neuron. It takes some numbers in, multiplies each by a weight (how much to trust it), adds them up, adds a bias (the personal leaning), and then squashes the total through an activation function:

inputs    weights
 x1 ----- w1 \
              \
 x2 ----- w2 --->  z = w1*x1 + w2*x2 + b  --->  activation(z)  ---> output
              /
 bias b ----/

You already know the first half. w1*x1 + w2*x2 is the dot product from Dot Product and Similarity, and a whole layer of neurons is the matrix times vector from Matrices and Layers. The only new piece is the activation.

Save this as demo-neuron.js in the root of your mathai folder (next to src). It uses the nn.js module from the Recipe at the end of this chapter, so save that file first.

// demo-neuron.js
import { sigmoid, relu } from "./src/nn.js";

// One neuron: weighted sum + bias, then an activation.
const x = [2, 1];       // inputs
const w = [0.5, -1];    // weights
const b = 0.25;         // bias

const z = w[0] * x[0] + w[1] * x[1] + b;
console.log("weighted sum z:", z);
console.log("sigmoid(z):", sigmoid(z).toFixed(4));
console.log("relu(z):", relu(z));

for (const v of [-4, -2, 0, 2, 4]) {
  console.log(v, "sigmoid", sigmoid(v).toFixed(3), "relu", relu(v));
}
// output
// weighted sum z: 0.25
// sigmoid(z): 0.5622
// relu(z): 0.25
// -4 sigmoid 0.018 relu 0
// -2 sigmoid 0.119 relu 0
// 0 sigmoid 0.500 relu 0
// 2 sigmoid 0.881 relu 2
// 4 sigmoid 0.982 relu 4

By hand: 0.5 * 2 = 1, -1 * 1 = -1, and 1 - 1 + 0.25 = 0.25. All of this output is stable.

2. Activations: Sigmoid and ReLU

An activation function takes the weighted sum z, which can be any number, and bends it. Two common ones:

• Sigmoid: 1 / (1 + e^-z). It squashes any number into the range between 0 and 1. Very negative gives almost 0, zero gives exactly 0.5, very positive gives almost 1. Think of it as a soft yes/no switch, and you can read the result as "how strongly does this neuron say yes".
• ReLU ("rectified linear unit"): max(0, z). Negative numbers become 0, positive numbers pass through unchanged. It is just a gate that blocks negatives.

sigmoid                       relu
 1 |         ______            |        /
   |       /                   |       /
0.5|-----/                     |      /
   |   /                       |     /
 0 |__/_______                 |____/________
      -4  0  4                    0

Sigmoid has a nice slope everywhere, which makes the calculus friendly. ReLU is cheaper to compute, and it is a common default in modern networks. Our toy network uses sigmoid because the 0-to-1 output matches the XOR answers 0 and 1.

One more fact we need soon: the slope of sigmoid. At the far ends the curve is nearly flat, so the slope is tiny. In the middle it is steepest. You can check that the slope equals s * (1 - s), where s is the sigmoid value, using the numeric derivative from Slopes and Derivatives:

// demo-sigmoid-slope.js
import { sigmoid } from "./src/nn.js";
import { derivative } from "./src/calculus.js";

for (const x of [-4, -1, 0, 1, 4]) {
  const s = sigmoid(x);
  console.log(
    `x=${x}: numeric slope ${derivative(sigmoid, x).toFixed(5)}  s*(1-s) ${(s * (1 - s)).toFixed(5)}`
  );
}
// output
// x=-4: numeric slope 0.01766  s*(1-s) 0.01766
// x=-1: numeric slope 0.19661  s*(1-s) 0.19661
// x=0: numeric slope 0.25000  s*(1-s) 0.25000
// x=1: numeric slope 0.19661  s*(1-s) 0.19661
// x=4: numeric slope 0.01766  s*(1-s) 0.01766

Both columns match, so s * (1 - s) is the right slope, and the values are stable. At x = 4 the slope is only 0.018: a neuron whose weighted sum is far from zero learns slowly, which is one reason training can stall.

3. Why Activations Are Needed

Here is the surprising part. Suppose you build a network with no activation at all: layer 1 multiplies by a matrix W1, layer 2 multiplies by a matrix W2. Does the second layer add any power?

No. Think of real life: "double the number, then add the number to itself" is the same as "multiply by 4". Two simple steps collapse into one. The same happens with matrices: W2 (W1 x) equals (W2 W1) x, and W2 W1 is just one new matrix. A hundred linear layers are still one linear layer.

Check it with the matrix functions from chapter 3:

// demo-linear-stack.js
import { matVec, matMul } from "./src/matrix.js";
import { relu } from "./src/nn.js";

const W1 = [[1, -1], [2, 1]];   // 2 inputs -> 2 numbers
const W2 = [[1, 1]];            // 2 numbers -> 1 number
const combined = matMul(W2, W1); // one matrix that does both jobs

for (const x of [[1, 0], [0, 1], [2, 3], [-1, 4]]) {
  const stacked = matVec(W2, matVec(W1, x));
  const single = matVec(combined, x);
  const withRelu = matVec(W2, matVec(W1, x).map(relu));
  console.log(x, "stacked:", stacked, "one matrix:", single, "with relu:", withRelu);
}
console.log("combined matrix:", combined);
// output
// [ 1, 0 ] stacked: [ 3 ] one matrix: [ 3 ] with relu: [ 3 ]
// [ 0, 1 ] stacked: [ 0 ] one matrix: [ 0 ] with relu: [ 1 ]
// [ 2, 3 ] stacked: [ 6 ] one matrix: [ 6 ] with relu: [ 7 ]
// [ -1, 4 ] stacked: [ -3 ] one matrix: [ -3 ] with relu: [ 2 ]
// combined matrix: [ [ 3, 0 ] ]

The stacked and one matrix columns are identical for every input. The two layers collapsed into the single matrix [[3, 0]], which just says "ignore the second input and triple the first". Now look at the with relu column. As soon as we put a bend between the layers, the answers change (for [0, 1] we get 1 instead of 0). The bend means the network can no longer be shrunk into one matrix. That bend is the whole reason deep networks are more than a line. All output here is stable.

4. The Problem a Line Cannot Solve: XOR

XOR ("exclusive or") means "one or the other, but not both". Two switches, and the light is on only if exactly one is on:

 x1  x2 | answer
  0   0 |   0
  0   1 |   1
  1   0 |   1
  1   1 |   0

Draw the four inputs as points on a square. The two points that should say 1 sit on one diagonal, and the two that should say 0 sit on the other:

 x2
  1 |  [1]      (0)
    |
  0 |  (0)      [1]
    +-----------------> x1
       0         1

 [1] = answer 1,  (0) = answer 0

Try to draw one straight line that has both [1] points on one side and both (0) points on the other. You cannot. Every straight line fails. A single neuron, sigmoid(w1*x1 + w2*x2 + b), can only draw one straight line, so it cannot solve XOR.

Do not just trust the picture. Train a single neuron on XOR with the same hand-written update rule and watch it fail. This demo has its own tiny training loop, since nn.js is built for the two-layer network:

// demo-one-neuron.js
import { xor } from "./src/data.js";
import { sigmoid } from "./src/nn.js";
import { mulberry32 } from "./src/rng.js";

function trainNeuron(seed) {
  const rng = mulberry32(seed);
  let w0 = rng() * 2 - 1, w1 = rng() * 2 - 1, b = 0;
  let loss = 0;
  for (let epoch = 0; epoch < 5000; epoch++) {
    loss = 0;
    for (const { x, y } of xor) {
      const out = sigmoid(w0 * x[0] + w1 * x[1] + b);
      loss += (out - y) ** 2;
      const d = 2 * (out - y) * out * (1 - out);
      w0 -= 0.5 * d * x[0];
      w1 -= 0.5 * d * x[1];
      b -= 0.5 * d;
    }
  }
  const outs = xor.map(({ x }) => sigmoid(w0 * x[0] + w1 * x[1] + b));
  return { loss: loss / 4, outs };
}

for (const seed of [1, 2]) {
  const { loss, outs } = trainNeuron(seed);
  console.log(`seed ${seed}: loss ${loss.toFixed(3)}`, outs.map((o) => o.toFixed(2)));
}
// output
// seed 1: loss 0.284 [ '0.53', '0.50', '0.47', '0.43' ]
// seed 2: loss 0.284 [ '0.53', '0.50', '0.47', '0.43' ]

The four outputs are for [0,0], [0,1], [1,0], [1,1] and should be 0, 1, 1, 0. Instead the neuron hedges and answers about 0.5 for everything. The loss stays near 0.28 whatever the seed. The output is stable.

The fix is a hidden layer: a row of neurons between the input and the output. Each hidden neuron draws one line, and the output neuron combines those lines. Two bends are enough to carve out the diagonal pattern. A network with a hidden layer is the smallest thing we can call a neural network.

5. The Forward Pass

Our network from nn.js has this shape:

inputs         hidden layer           output
 x1 --\   /-- h1 (sigmoid) --\
       >-<                     >-- out (sigmoid)
 x2 --/   \-- h2 (sigmoid) --/

Every arrow carries a weight, every neuron has a bias. Running data left to right is the forward pass: hidden numbers first, then the output. In nn.js, makeNet(hidden, seed) creates a network with random starting weights (using the seeded random numbers from chapter 1, so your output matches mine), and forward(net, x) returns both the hidden activations h and the final out.

Before training, the weights are random, so the answers are nonsense. After training they are not. Here we look at both, and at what the two hidden neurons are doing:

// demo-forward.js
import { xor } from "./src/data.js";
import { makeNet, forward, trainEpoch } from "./src/nn.js";

const net = makeNet(2, 1);
console.log("before training:");
for (const { x } of xor) {
  const { h, out } = forward(net, x);
  console.log(x, "hidden", h.map((v) => v.toFixed(2)), "out", out.toFixed(3));
}
for (let epoch = 0; epoch < 4000; epoch++) trainEpoch(net, xor, 0.5);
console.log("after training:");
for (const { x } of xor) {
  const { h, out } = forward(net, x);
  console.log(x, "hidden", h.map((v) => v.toFixed(2)), "out", out.toFixed(3));
}
// output
// before training:
// [ 0, 0 ] hidden [ '0.50', '0.50' ] out 0.562
// [ 0, 1 ] hidden [ '0.27', '0.72' ] out 0.484
// [ 1, 0 ] hidden [ '0.56', '0.51' ] out 0.575
// [ 1, 1 ] hidden [ '0.32', '0.73' ] out 0.495
// after training:
// [ 0, 0 ] hidden [ '0.03', '0.94' ] out 0.020
// [ 0, 1 ] hidden [ '0.00', '0.06' ] out 0.977
// [ 1, 0 ] hidden [ '0.95', '1.00' ] out 0.982
// [ 1, 1 ] hidden [ '0.03', '0.95' ] out 0.018

Before training every output is about 0.5, a shrug. After training, the outputs are 0.020, 0.977, 0.982, 0.018, which is 0, 1, 1, 0 to two decimals. Look at the hidden columns: the network found its own way to re-describe the four inputs so that the two "1" inputs end up looking different from the two "0" inputs. Nobody told it how. Those internal re-descriptions are what the AI world calls learned features, and it is a small cousin of how embeddings are learned. All output is stable, because the seed fixes the starting weights.

6. Backpropagation Is the Chain Rule, Layer by Layer

Training needs the answer to one question for every weight: "if I nudge this weight a little, does the loss go up or down, and by how much?" That is the gradient from Slopes and Derivatives, and the update is the downhill step from Gradient Descent and Loss.

Nudge and measure needs two forward passes per weight, which a network with a billion weights cannot afford. Backpropagation gets every slope in about one extra pass. It is nothing more than the chain rule.

Recall the chain rule in plain words: if a changes b, and b changes c, then the effect of a on c is the effect on b times the effect of b on c. Gears: turning gear A once turns gear B three times, and gear B turns gear C twice, so A turns C six times. You multiply the ratios along the chain.

A weight deep in the network affects the loss through a chain:

weight w2[j] -> output z -> out = sigmoid(z) -> loss = (out - y)^2

weight w1[j] -> hidden h[j] -> output z -> out -> loss

Go backwards along the chain and multiply the slopes as you go. Here is the loss for one example, and how trainEpoch in nn.js walks it:

1. Loss to output. The loss is (out - y)^2, whose slope with respect to out is 2 * (out - y).
2. Through the output sigmoid. out = sigmoid(z) has slope out * (1 - out). Multiply: dOut = 2 * (out - y) * out * (1 - out). This number is "how much the loss changes if the output's weighted sum changes". The code names it dOut.
3. To an output weight. Weight w2[j] multiplies h[j] on its way in, so its slope is dOut * h[j]. The bias b2 just adds, so its slope is dOut.
4. Back through a hidden neuron. The hidden value h[j] reaches the output through weight w2[j], so it receives dOut * w2[j]. Passing back through that neuron's own sigmoid multiplies by h[j] * (1 - h[j]). The code names it dH.
5. To a first-layer weight. w1[j][0] multiplies x[0], so its slope is dH * x[0], and likewise for x[1] and the bias b1[j].

The error signal flows backwards, shrinking or growing at each step, and every weight takes its share. Each line in the middle of trainEpoch is one of those steps:

const dOut = 2 * (out - y) * out * (1 - out);
for (let j = 0; j < h.length; j++) {
  const dH = dOut * net.w2[j] * h[j] * (1 - h[j]);
  net.w2[j] -= lr * dOut * h[j];
  net.w1[j][0] -= lr * dH * x[0];
  net.w1[j][1] -= lr * dH * x[1];
  net.b1[j] -= lr * dH;
}
net.b2 -= lr * dOut;

Each -= is a gradient descent step: new weight = old weight minus learning rate times slope. Look at the order inside the loop: dH is computed first, from the old w2[j], and only then is w2[j] changed. That is correct, because the slope must use the same weights that the forward pass used.

One worry from section 2: if h[j] is close to 0 or 1, then h[j] * (1 - h[j]) is nearly zero and the signal going back almost vanishes. A deep stack of sigmoids makes this worse. It is called the vanishing gradient problem, and a reason ReLU became popular.

7. Training on XOR

One pass over all four examples is an epoch. trainEpoch(net, data, lr) runs one epoch, updates the weights after every example, and returns the average loss. This is the stochastic style from the last chapter, with a "batch" of one. Call it many times and watch the loss fall:

// demo-xor-train.js
import { xor } from "./src/data.js";
import { makeNet, forward, trainEpoch } from "./src/nn.js";

const net = makeNet(4, 1);
for (let epoch = 0; epoch <= 5000; epoch++) {
  const loss = trainEpoch(net, xor, 0.5);
  if ([0, 500, 1000, 2000, 3000, 5000].includes(epoch)) {
    console.log(`epoch ${epoch}: loss ${loss.toFixed(4)}`);
  }
}
for (const { x, y } of xor) {
  console.log(x, "want", y, "got", forward(net, x).out.toFixed(3));
}
// output
// epoch 0: loss 0.2971
// epoch 500: loss 0.0816
// epoch 1000: loss 0.0030
// epoch 2000: loss 0.0009
// epoch 3000: loss 0.0005
// epoch 5000: loss 0.0003
// [ 0, 0 ] want 0 got 0.014
// [ 0, 1 ] want 1 got 0.984
// [ 1, 0 ] want 1 got 0.986
// [ 1, 1 ] want 0 got 0.020

The loss starts near 0.3 (always answering 0.5 would score 0.25), drops fast between epochs 500 and 1000, then creeps toward zero. The outputs are close to the right answers. All output is stable because everything is seeded.

The loss shape is slow start, quick drop, long tail, which many real training runs share.

A caveat: with only 2 hidden neurons, some starting seeds get stuck. You will see that in the next chapter.

8. Checking Backprop with Nudge and Measure

A hand-written backpropagation is easy to get subtly wrong, and the network would still "train", just worse. Professionals test their gradients against nudge and measure. We can too, using derivative from chapter 4. Pick one weight, nudge it up and down, and measure the change in the total loss over the four examples. Then compute the same slope with the backprop formula for that weight (dOut * h[0], summed over the examples) and compare:

// demo-gradcheck.js
import { xor } from "./src/data.js";
import { makeNet, forward } from "./src/nn.js";
import { derivative } from "./src/calculus.js";

const net = makeNet(2, 7);
const lossOf = () =>
  xor.reduce((s, { x, y }) => s + (forward(net, x).out - y) ** 2, 0);

// Nudge one weight and measure how the loss moves.
const numeric = derivative((w) => {
  const old = net.w2[0];
  net.w2[0] = w;
  const l = lossOf();
  net.w2[0] = old;
  return l;
}, net.w2[0]);

// The backprop formula for the same weight, summed over the 4 examples.
let analytic = 0;
for (const { x, y } of xor) {
  const { h, out } = forward(net, x);
  analytic += 2 * (out - y) * out * (1 - out) * h[0];
}
console.log("numeric  :", numeric.toFixed(6));
console.log("backprop :", analytic.toFixed(6));
// output
// numeric  : 0.002204
// backprop : 0.002204

The two numbers agree to six decimals, so the chain-rule steps are right. This is called a gradient check, and it is the best friend of anyone writing backpropagation by hand. The output is stable.

9. Counting Parameters and Scaling Up

The weights and biases are called the parameters of the model. They are the numbers that training changes. Our network with hidden hidden neurons has:

• 2 * hidden first-layer weights,
• hidden first-layer biases,
• hidden output weights,
• 1 output bias.

That is 4 * hidden + 1 in total. Count them in code:

// demo-params.js
import { makeNet } from "./src/nn.js";
const count = (net) => net.w1.flat().length + net.b1.length + net.w2.length + 1;
for (const hidden of [2, 4, 16, 100]) {
  console.log(hidden, "hidden neurons ->", count(makeNet(hidden)), "parameters");
}
// output
// 2 hidden neurons -> 9 parameters
// 4 hidden neurons -> 17 parameters
// 16 hidden neurons -> 65 parameters
// 100 hidden neurons -> 401 parameters

Nine numbers are enough to learn XOR, and nothing limits how big the network gets. A large language model is the same recipe with far more of everything: thousands of inputs instead of 2, wide matrices, dozens of layers, and billions of parameters in total. Each parameter is a weight that was once random and was nudged, again and again, by a gradient like the ones you computed here.

The loop is the same at any size: forward pass, loss, backward pass, nudge the weights, repeat.

So "how AI learns" has no list of rules inside: it is a very large table of numbers, trained by a loop like trainEpoch.

That is why chapter 3 spent time on matrices: each layer's forward pass is a matrix multiply, and many neurons and examples can be computed together as one big matrix times matrix.

10. What Real Frameworks Automate

Our nn.js is a teaching toy. It works for exactly one shape of network, and we derived every gradient by hand. Real libraries change that in several ways, while keeping the same ideas:

• Automatic differentiation. You write only the forward pass. The library records the chain of operations and applies the chain rule backwards for you, for any network shape. No hand-written dH.
• Tensors. Data lives in big multi-dimensional arrays, and the work runs as matrix operations on specialized hardware.
• Better optimizers. We use plain gradient descent. Real training commonly uses variants that adapt the step size for each parameter and keep a running memory of past gradients, and they update on small groups of examples (mini-batches).
• Many more layer types. Attention, normalization, embeddings, all built from the same vectors, matrices, and gradients.

What does not change: a forward pass computes a prediction, a loss measures how wrong it is, the gradient says which way to nudge each weight, and gradient descent nudges them. If you understand nn.js, you understand the loop that every one of these libraries automates. See How a Model Is Trained for the same story told for language models, and Embeddings and Attention for what the layers look like inside a transformer.

Be careful with the claim "the network learned XOR". It means its numbers settled into values that give the right outputs on these four inputs, not that it "understands" XOR. A tiny network also says nothing about how big models behave. It only shows the mechanism.

11. Recipe: The nn Module

Save this as src/nn.js in your mathai folder. It needs src/rng.js from Vectors: Lists of Numbers. It is the network you used above: 2 inputs, a hidden layer of sigmoid neurons, 1 sigmoid output, and hand-written backpropagation for squared error.

import { mulberry32 } from "./rng.js";

export const sigmoid = (x) => 1 / (1 + Math.exp(-x));
export const relu = (x) => Math.max(0, x);

// 2 inputs -> `hidden` sigmoid neurons -> 1 sigmoid output.
export function makeNet(hidden, seed = 1) {
  const rng = mulberry32(seed);
  const r = () => rng() * 2 - 1;
  return {
    w1: Array.from({ length: hidden }, () => [r(), r()]),
    b1: Array(hidden).fill(0),
    w2: Array.from({ length: hidden }, r),
    b2: 0,
  };
}

export function forward(net, x) {
  const h = net.w1.map((row, j) => sigmoid(row[0] * x[0] + row[1] * x[1] + net.b1[j]));
  const out = sigmoid(h.reduce((s, hj, j) => s + hj * net.w2[j], net.b2));
  return { h, out };
}

// One pass over the data with hand-written backpropagation (squared error).
export function trainEpoch(net, data, lr) {
  let loss = 0;
  for (const { x, y } of data) {
    const { h, out } = forward(net, x);
    loss += (out - y) ** 2;
    const dOut = 2 * (out - y) * out * (1 - out);
    for (let j = 0; j < h.length; j++) {
      const dH = dOut * net.w2[j] * h[j] * (1 - h[j]);
      net.w2[j] -= lr * dOut * h[j];
      net.w1[j][0] -= lr * dH * x[0];
      net.w1[j][1] -= lr * dH * x[1];
      net.b1[j] -= lr * dH;
    }
    net.b2 -= lr * dOut;
  }
  return loss / data.length;
}

trainEpoch changes net in place and returns the mean squared error for the epoch.

12. Try It Yourself

1. Change the hidden size in demo-xor-train.js from 4 to 8 and to 3. You should see all three reach a loss below 0.01 by epoch 1000, with slightly different numbers. Size matters less than you might expect for a problem this small.
2. Change the learning rate from 0.5 to 0.05. You should see the loss at epoch 5000 is still about 0.01, much worse than before, so a small step is slow. Then try 20 and see it fail to settle, as in the "too small, right, too big" story from the last chapter.
3. Make a new data array for the AND function ([1,1] gives 1, the other three give 0) and train a net on it. A single neuron can solve AND, so check that the loss gets near 0 even with 1 hidden neuron.
4. Using demo-gradcheck.js as a model, check the slope of a first-layer weight, net.w1[0][0]. The two numbers should agree. Then deliberately break the backprop formula, for example drop the (1 - out) factor, and watch the numbers disagree.
5. In demo-forward.js, print the hidden activations for all four inputs after training, then try to describe in words what each hidden neuron seems to detect. You should find it is hard: learned features are often not tidy, and nobody designed them.

13. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now build and train a neural network and say what every part does. One chapter is left. We will look at the statistics that keep training healthy, how to tell learning from memorizing, and a one-page map of where each piece of math appears in a language model. Continue with Statistics and the Big Picture.