5. Least Squares

The term least squares describes a frequently used approach to solving overdetermined or inexactly specified systems of equations in an approximate sense. Instead of solving the equations exactly, we seek only to minimize the sum of the squares of the residuals. The least squares criterion has important statistical interpretations. If appropriate probabilistic assumptions about underlying error distributions are made, least squares produces what is known as the maximum-likelihood estimate of the parameters. Even if the probabilistic assumptions are not satisfied, years of experience have shown that least squares produces useful results. The computational techniques for linear least squares problems make use of orthogonal matrix factorizations. 5.1 Models and Curve Fitting A very common source of least squares problems is curve fitting. Let t be the independent variable and let y(t) denote an unknown function of t that we want to approximate. Assume there are m observations, i.e., values of y measured at specified values of t: yi =y ( ti ) , i=1,… ,m. The idea is to model y(t) by a linear combination of n basis functions: y (t) ≈ β1 ϕ1 (t) +⋯+ βn ϕn (t) . The design matrix X is a rectangular matrix of order m by n with elements xi,j = ϕj ( ti ) . The design matrix usually has more rows than columns. In matrix-vector notation, the model is y≈Xβ. The symbol ≈ stands for “is approximately equal to.” We are more precise about this in the next section, but our emphasis is on least squares approximation.