The Perceptron and Neural Network Basics

In the 1950s, Frank Rosenblatt looked at the human brain and saw a network of switches. In biology, a neuron receives signals through dendrites, processes them in the cell body, and fires a signal down the axon if a certain threshold is met. The Perceptron is the mathematical equivalent. It takes multiple inputs $x_i$, multiplies each by a weight $w_i$ (representing the strength of that connection), and adds a bias $b$ (the neuron's 'threshold' for firing). The result is the weighted sum $z$, which is then passed through an activation function to determine the final output $y$.

The weighted sum $z$ can be any number, but neurons in the brain usually 'fire' or 'don't fire.' To mimic this, we use activation functions. Without them, a neural network is just a giant linear equation, unable to learn complex patterns. Two modern favorites are ReLU and Tanh. ReLU (Rectified Linear Unit) is simple: if the input is negative, the output is zero; otherwise, it stays the same. Tanh (Hyperbolic Tangent) squashes any input into a range between $-1$ and $1$, making it useful for centering data.

Why do we care about these functions? Real-world data is rarely a straight line. By using functions like Tanh or ReLU, we introduce non-linearity. This allows the network to learn 'bends' and 'curves' in data, such as the difference between a picture of a cat and a dog. The Bias term $b$ is also crucial here; it allows the activation function to shift left or right, giving the neuron more flexibility in when it decides to 'fire' regardless of the inputs.