From perceptron to multilayer network
A neuron computes a weighted sum plus bias, then applies a non-linear activation. Stacking layers of neurons gives a universal function approximator: without the non-linearity, any stack of linear layers collapses into a single linear map.
Activations
ReLU (max(0, x)) is the default because it is cheap and avoids saturation; GELU and SiLU are smooth variants used in transformers. Sigmoid and tanh saturate and are now mostly limited to gates and output layers.
Forward and backward
The forward pass computes predictions and the loss. The backward pass computes gradients layer by layer, and the optimiser updates the weights. Every framework automates this with an autograd graph.
import torch, torch.nn as nn
net = nn.Sequential(nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 10))
opt = torch.optim.AdamW(net.parameters(), lr=3e-4)
loss_fn = nn.CrossEntropyLoss()
logits = net(x) # forward
loss = loss_fn(logits, y)
loss.backward() # backward
opt.step(); opt.zero_grad()