The model: ŷ = wx + b

The Simplest Useful Model

Linear regression is the foundation of predictive modeling. It is not the most powerful tool - neural networks learn far more complex relationships - but it is the bedrock that everything else builds on. And it remains genuinely useful: housing price prediction, demand forecasting, risk scoring.

The core assumption: the relationship between input features x and output $y$ is approximately linear. Add more square footage to a house and the price rises proportionally. Many real relationships, especially over small ranges, are close enough to linear to make this practical.

The Single-Feature Model

Start with one input and one output. The linear model is:

This is the equation of a line: $y = mx + b$ from middle school, with $m$ renamed to $w$. The symbols w and b are the model's parameters - the numbers we will learn from data.

Concrete example: predict house price from size. Suppose $w = 0.2$ (each extra square foot adds $200, since price is in thousands) and $b = 50$ (base price $50k). For a 1,500 sq ft house:

Why "Weight" and "Bias"?

The term weight captures how influential a feature is. A feature with weight $w = 10$ is 100 times more influential than one with weight $w = 0.1$. The magnitude tells you relative importance; the sign tells you direction (positive means higher value → higher prediction, negative means higher value → lower prediction).

The The bias term ⓘ is what allows a line to not pass through the origin.

Multiple Features

Real problems have many features. For $p$ features $x_1, x_2, \ldots, x_p$:

The sum $w_1 x_1 + \cdots + w_p x_p$ is exactly a dot product $\mathbf{w} \cdot \mathbf{x}$: multiply each pair of matching elements and sum. This notation is compact and computationally efficient.

Three-feature house example (size, bedrooms, distance):

The model is still just a weighted sum - it defines a hyperplane in the $p$-dimensional feature space instead of a line in 2D.

The The dot product as a prediction mechanism ⓘ is the core of every linear model.

For the Full Dataset

To predict all $n$ training examples simultaneously, stack inputs as a matrix:

Here, $\mathbf{X}$ is $n \times p$, $\mathbf{w}$ is $p \times 1$, and $\hat{\mathbf{y}}$ is $n \times 1$ - one prediction per example. This single matrix operation replaces a loop over all training examples.