Orthognality

Orthogonality of the four spaces

Def.1 Orthognality:
Two subspaces $V$ and $W$ of a vector space are $O r t h o g o n a l$ if every vector $v v$ is perpendicular to every vector $w w$ in $W W$ .
Orthogonal subspaces

v v^{T} w w = 0

for all

v v

V V

and

w w

W W

The surface of the floor and the wall is perpendicular, but these two spaces are not orthogonal, since we can still find two vectors in each space that are not perpendicular.
Null space and Row Space are orthogonal
If $x x \in N (A)$ , then we have $A x x = 0$ .

[\begin{matrix} r o w_{1} \\ r o w_{2} \\ . . . \\ r o w_{m} \end{matrix}] [\begin{matrix} x \end{matrix}] = [\begin{matrix} r o w_{1} ∙ x \\ r o w_{2} ∙ x \\ . . . \\ r o w_{m} ∙ x \end{matrix}] = [\begin{matrix} 0 \\ 0 \\ . . . \\ 0 \end{matrix}]

Inner product of

x x

and the matrix

A

equals

0

. Similiarly we can prove that Row space and left null space are orthogonal.

Orthognal Complemen

Row space and Null space split $R^{n} R^{n}$ into two orthogonal subspaces. For example for matrix $A = [\begin{matrix} 1 & 2 & 5 \\ 2 & 4 & 10 \end{matrix}]$ , $d i m (C (A^{T})) = 1$ , and $[\begin{matrix} 1 & 2 & 5 \end{matrix}]$ is its basis. Its Null space span a 2 dimensional surface that is perpendicular to the vector $[\begin{matrix} 1 & 2 & 5 \end{matrix}]$

Def.2 Orthogonal Complement:
The Orthogonal complement of a subspace $V V$ contains every vector that is perpendicular to $V V$ ，whichis denoted by $V^{⊥} V^{⊥}$ .

Gilbert Strang-Linear Algebra-Orthogonality

Important:

N (A)

is the orthogonal complement of the Row Space

C (A^{T})

.(in

R^{n}

)

N (A^{T})

is the orthogonal complement of the Comlumn Space

C (A)

（in

R^{m}

)

Projection

Question: How to solve and un-solvable formula $A x = b$
When the matrix $A$ is rectangular, $m > n$ , as there are multitude number on the left side of the formula, there is a large likely that some “bad” data are mixed in, making the formula unsolvable. What linear algebra does is to find the “best solution” in such circumstance. The matrix $A^{T} A$ plays an important role.

A = [\begin{matrix} 1 & 1 \\ 1 & 2 \\ 1 & 5 \end{matrix}], A^{T} A = [\begin{matrix} 1 & 1 & 1 \\ 1 & 2 & 5 \end{matrix}] [\begin{matrix} 1 & 1 \\ 1 & 2 \\ 1 & 5 \end{matrix}]

where

A^{T} A

is invertible. However it is not always invertible. Invertibility requires row vectors of

A

are linear independent. The core of this chapter is to solve

A^{T} A \hat{x} = A^{T} b

to find the best solution.
E.g.

A = a a, b = b b,

find solution of

A x = b

where

a, b a, b

are independent.

The best solution in this case is

A x = p

, where

p

is the projection of

b

A

. Since

a ⊥ e

, then we can get the equation that:

A^{T} e = A^{T} (b - p) = A^{T} (b - A x) = 0

x = \frac{A^{T} b}{A^{T} A}

Projection Matrix

Let us describe the problem in projection matrix, which is $p = P b$ , where $P$ is the projection matrix. Since

x = \frac{A^{T} b}{A^{T} A}

then

A x = A \frac{A^{T} b}{A^{T} A} = p = P b

therefore

P = \frac{A A^{T}}{A^{T} A}

Therefore, when

A x = b

is unsolvable, we can
Firstly, project

b

onto

A

using projection matrix

P

and get

p = P b

;
Secondly, solve

A \hat{x} = p

to find the best solution.

High dimentional projection

In $R^{3} R^{3}$

If row vector

a_{1}, a_{2} a_{1}, a_{2}

make a set of base, then the surface is the row space of the matrix

A = [\begin{matrix} a_{1} a_{1} & a_{2} a_{2} \end{matrix}]

. It is known that the vector

p p

is in the surface, then we have:

p = {\hat{x}}_{1} a_{1} + {\hat{x}}_{2} a_{2} p = {\hat{x}}_{1} a_{1} + {\hat{x}}_{2} a_{2} = A \hat{X} \hat{X}

Set

e e

as the deviation from

b

p

, then

e = b - p = b - e = b - p = b - A \hat{x} \hat{x}

which is orthogonal to

A

and its column space (

e e \in N (A^{T})

). Then we have:

A^{T} (b b - A \hat{X} \hat{X}) = 0

A^{T} b b = A^{T} A \hat{X} \hat{X}

(A^{T} A)^{- 1} A^{T} b b = \hat{X} \hat{X}

p p = A \hat{X} \hat{X} = A (A^{T} A)^{- 1} A^{T} b b

∴ P = A (A^{T} A)^{- 1} A^{T} (p r o j e c t i o n m a t r i x)

Least Square Approximation

Find the best solution for an equation using Projection Matrix. For example,if we want fit 3 point with a straight line:
3 points: $(1, 1), (2, 2), (3, 2)$
Curve equationL $b = C + D t$
In the matrix form we have :

[\begin{matrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{matrix}] [\begin{matrix} C \\ D \end{matrix}] = [\begin{matrix} 1 \\ 2 \\ 2 \end{matrix}]

A x = b

P = A (A^{T} A)^{- 1} A^{T}

. . .

Orthonormal

Def. Orthonormal: if the colomn vectors of a matrix are unit vector that are perpendicular to each other, then they(the vectors) are linear independent and orthonormal.
Def. Orthogonal Matrix if the column vectors of a square matrix are orthonormal, then the square matrix itself is orthonormal matrix.
Feature: the inner product of orthonormal with itsel is identity matrix. Say $Q$ is orthonormal matrix

Q = [\begin{matrix} | & | \\ q_{1} & . . . & q_{n} \\ | & | \end{matrix}]

Q^{T} Q = [\begin{matrix} - - & q_{1}^{T} & - - \\ . . . \\ - - & q_{n}^{T} & - - \end{matrix}] [\begin{matrix} | & | \\ q_{1} & . . . & q_{n} \\ | & | \end{matrix}] = I

*Note:this feature also applies to the situation where the $Q$ is not a square

References

Gilbert, Strang. Introduction to Linear Algebra, 4th Edition[M]. MA 02482 US:Wellsley-Cambridge Press, 2009.
ls317842927. 线性代数学习笔记4 [EB/OL]. http://blog.****.net/ls317842927/article/details/53309746.
MIT-线性代数笔记（上）[EB/OL]. http://www.docin.com/p-1489481927.html.

Gilbert Strang-Linear Algebra-Orthogonality