Frameworks like PyTorch hide the maths. Writing a tiny network by hand once makes everything later — including transformers — far less mysterious. We will solve XOR, the classic problem a single straight line cannot solve.

The problem

No single line separates the 1s from the 0s, so linear regression fails. We need a hidden layer.

The code

“`python
import numpy as np
rng = np.random.default_rng(0)

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

W1 = rng.normal(0, 1, (2, 8)); b1 = np.zeros(8) # input -> 8 hidden units
W2 = rng.normal(0, 1, (8, 1)); b2 = np.zeros(1) # hidden -> output
sigmoid = lambda z: 1 / (1 + np.exp(-z))
lr = 0.5

for step in range(5000):
# forward pass
h = np.tanh(X @ W1 + b1)
p = sigmoid(h @ W2 + b2)
loss = -np.mean(y * np.log(p) + (1 – y) * np.log(1 – p))

# backward pass (chain rule, layer by layer)
dz2 = (p – y) / len(X)
dW2 = h.T @ dz2; db2 = dz2.sum(0)
dh = dz2 @ W2.T * (1 – h ** 2) # derivative of tanh
dW1 = X.T @ dh; db1 = dh.sum(0)

# gradient descent
W1 -= lr * dW1; b1 -= lr * db1
W2 -= lr * dW2; b2 -= lr * db2
if step % 1000 == 0:
print(step, round(loss, 4))

print(p.round(3).ravel()) # close to [0, 1, 1, 0]
“`

Line by line

• Weights start random. If they all started equal, every hidden unit would learn the same thing.
• Forward pass: multiply by weights, add bias, squash with tanh, then again with sigmoid to get a probability between 0 and 1.
• Loss: binary cross-entropy — small when the probability given to the right answer is high.
• Backward pass: the chain rule, applied from the output back to the input. p - y is the neat result of combining sigmoid with cross-entropy.
• Update: the same nudge as in linear regression, just for more numbers.

Why the non-linearity is essential

Delete np.tanh and train again: the network collapses into a single linear function and XOR becomes impossible. Stacking layers only adds power because of the bends between them.

The same network in PyTorch

“`python
import torch, torch.nn as nn
Xt, yt = torch.tensor(X, dtype=torch.float32), torch.tensor(y, dtype=torch.float32)
net = nn.Sequential(nn.Linear(2, 8), nn.Tanh(), nn.Linear(8, 1))
opt = torch.optim.SGD(net.parameters(), lr=0.5)
for _ in range(5000):
loss = nn.functional.binary_cross_entropy_with_logits(net(Xt), yt)
opt.zero_grad(); loss.backward(); opt.step()
print(torch.sigmoid(net(Xt)).detach().round(decimals=3).ravel())
“`

loss.backward() does exactly what our four “backward pass” lines did — automatically, for any network shape.