学习内容源于深度之眼公众号的花书训练营课程
假设 B B B P P P A = P B P − 1 A = PBP^{-1} A = P B P − 1 A A A A A A B B B P P P P P T = P T P = I PP^{T}=P^{T}P=I P P T = P T P = I P T = P − 1 P^T=P^{-1} P T = P − 1 B = P − 1 A P = P T A P
B=P^{-1}AP=P^TAP
B = P − 1 A P = P T A P A A A λ i \lambda_i λ i
我们令单位正交矩阵 P T = [ u 1 , u 2 , ⋯ , u n ] P^T = [\boldsymbol{ u_1,u_2,\cdots,u_n}] P T = [ u 1 , u 2 , ⋯ , u n ] u i \boldsymbol{u_i} u i n n n B B B B = [ u 1 , u 2 , ⋯ , u n ] [ λ 1 λ 2 ⋱ λ n ] [ u 1 T u 2 T ⋮ u n T ] = [ λ 1 u 1 , λ 2 u 2 , ⋯ , λ n u n ] [ u 1 T u 2 T ⋮ u n T ] = λ 1 u 1 u 1 T + λ 2 u 2 u 2 T + ⋯ + λ n u n u n T
\begin{aligned}
B &= [\boldsymbol{ u_1,u_2,\cdots,u_n}]
\begin{bmatrix}
\lambda_1& & & \\
&\lambda_2 & &\\
& & \ddots & \\
& & & \lambda_n
\end{bmatrix}
\begin{bmatrix}
\boldsymbol{u_1}^T \\
\boldsymbol{u_2}^T\\
\vdots \\
\boldsymbol{u_n}^T
\end{bmatrix} \\
& = [\lambda_1\boldsymbol{u_1},\lambda_2\boldsymbol{u_2},\cdots,\lambda_n\boldsymbol{u_n}]
\begin{bmatrix}
\boldsymbol{u_1}^T \\
\boldsymbol{u_2}^T\\
\vdots \\
\boldsymbol{u_n}^T
\end{bmatrix} \\
& = \lambda_1\boldsymbol{u_1}\boldsymbol{u_1}^T + \lambda_2\boldsymbol{u_2}\boldsymbol{u_2}^T + \cdots + \lambda_n\boldsymbol{u_n}\boldsymbol{u_n}^T
\end{aligned}
B = [ u 1 , u 2 , ⋯ , u n ] ⎣ ⎢ ⎢ ⎡ λ 1 λ 2 ⋱ λ n ⎦ ⎥ ⎥ ⎤ ⎣ ⎢ ⎢ ⎢ ⎡ u 1 T u 2 T ⋮ u n T ⎦ ⎥ ⎥ ⎥ ⎤ = [ λ 1 u 1 , λ 2 u 2 , ⋯ , λ n u n ] ⎣ ⎢ ⎢ ⎢ ⎡ u 1 T u 2 T ⋮ u n T ⎦ ⎥ ⎥ ⎥ ⎤ = λ 1 u 1 u 1 T + λ 2 u 2 u 2 T + ⋯ + λ n u n u n T u i u i T \boldsymbol{u_i}\boldsymbol{u_i}^T u i u i T n × n n \times n n × n B B B n n n A A A λ i \lambda_i λ i B B B λ i \lambda_i λ i k k k λ \lambda λ B B B B ^ = λ 1 u 1 u 1 T + λ 2 u 2 u 2 T + ⋯ + λ n u k u k T
\hat{B} = \lambda_1\boldsymbol{u_1}\boldsymbol{u_1}^T + \lambda_2\boldsymbol{u_2}\boldsymbol{u_2}^T + \cdots + \lambda_n\boldsymbol{u_k}\boldsymbol{u_k}^T
B ^ = λ 1 u 1 u 1 T + λ 2 u 2 u 2 T + ⋯ + λ n u k u k T B ^ \hat{B} B ^ B B B η = ∑ i = 1 k λ i ∑ i = 1 n λ i \eta = \frac{\displaystyle \sum_{i=1}^k\lambda_i}{\displaystyle \sum_{i=1}^n\lambda_i} η = i = 1 ∑ n λ i i = 1 ∑ k λ i
矩阵对角化的条件还是蛮苛刻的，必须满足两个条件：方阵以及对称。可见一般情况下一个矩阵 A m × n A_{m\times n} A m × n
对于任意一个矩阵 A m × n A_{m \times n} A m × n ( A T A ) n × n (A^TA)_{n\times n} ( A T A ) n × n
对称性：
( A T A ) T = A T ( A T ) T = A T A (A^TA)^T=A^T(A^T)^T=A^TA ( A T A ) T = A T ( A T ) T = A T A
半正定性：
∀ x ∈ R n , x T ( A T A ) x = ( x T A T ) ( A x ) = ( A x ) T ( A x ) ≥ 0 \forall {\boldsymbol x} \in \R^n,{\boldsymbol x}^T(A^TA){\boldsymbol x} = ({\boldsymbol x}^TA^T)(A{\boldsymbol x})=(A{\boldsymbol x})^T(A\boldsymbol x)\ge0 ∀ x ∈ R n , x T ( A T A ) x = ( x T A T ) ( A x ) = ( A x ) T ( A x ) ≥ 0
同理，可知 ( A A T ) m × m (AA^T)_{m\times m} ( A A T ) m × m A T A = U T D 1 U A A T = V T D 2 V
\begin{aligned}
A^TA = U^TD_1U\\
AA^T = V^TD_2V
\end{aligned}
A T A = U T D 1 U A A T = V T D 2 V U ∈ R n × n , V ∈ R m × m U \in \R^{n\times n},V \in \R^{m\times m} U ∈ R n × n , V ∈ R m × m D 1 ∈ R n × n , D 2 ∈ R m × m D_1\in \R^{n\times n},D_2 \in \R^{m\times m} D 1 ∈ R n × n , D 2 ∈ R m × m D 1 , D 2 D_1,D_2 D 1 , D 2 D 1 , D 2 D_1,D_2 D 1 , D 2 k k k λ 1 , λ 2 , ⋯ , λ k \lambda_1,\lambda_2,\cdots,\lambda_k λ 1 , λ 2 , ⋯ , λ k
则矩阵 A m × n A_{m\times n} A m × n A m × n = V m × m T [ λ 1 1 2 λ 2 1 2 ⋱ λ k 1 2 0 ⋱ ] m × n U n × n
A_{m\times n} = V^T_{m\times m}
\begin{bmatrix}
\lambda_1^{\frac{1}{2}} & & & \\
& \lambda_2^{\frac{1}{2}} & & \\
& & \ddots & \\
& & & \lambda_k^{\frac{1}{2}} \\
& & & & 0 \\
& & & & & \ddots
\end{bmatrix}_{m\times n}
U_{n\times n}
A m × n = V m × m T ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎡ λ 1 2 1 λ 2 2 1 ⋱ λ k 2 1 0 ⋱ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎤ m × n U n × n V T = [ v 1 , v 2 , ⋯ , v m ] , v i ∈ R m . U T = [ u 1 , u 2 , ⋯ , u n ] , u i ∈ R n V^T = [\boldsymbol {v_1,v_2,\cdots,v_m}],{\boldsymbol v_i}\in \R^m.U^T =[\boldsymbol {u_1,u_2,\cdots,u_n}],{\boldsymbol u_i} \in \R^n V T = [ v 1 , v 2 , ⋯ , v m ] , v i ∈ R m . U T = [ u 1 , u 2 , ⋯ , u n ] , u i ∈ R n A m × n = [ v 1 , v 2 , ⋯ , v m ] [ λ 1 1 2 λ 2 1 2 ⋱ λ k 1 2 0 ⋱ ] [ u 1 T u 2 T ⋮ u n T ] = [ λ 1 1 2 v 1 , λ 2 1 2 v 2 , ⋯ , λ k 1 2 v k , 0 , ⋯ , 0 ] [ u 1 T u 2 T ⋮ u n T ] = λ 1 1 2 v 1 u 1 T + λ 2 1 2 v 2 u 2 T + ⋯ + λ k 1 2 v k u k T
\begin{aligned}
A_{m\times n} &=
[\boldsymbol {v_1,v_2,\cdots,v_m}]
\begin{bmatrix}
\lambda_1^{\frac{1}{2}} & & & \\
& \lambda_2^{\frac{1}{2}} & & \\
& & \ddots & \\
& & & \lambda_k^{\frac{1}{2}} \\
& & & & 0 \\
& & & & & \ddots
\end{bmatrix}
\begin{bmatrix}
\boldsymbol{u_1}^T \\
\boldsymbol{u_2}^T \\
\vdots \\
\boldsymbol{u_n}^T \\
\end{bmatrix}\\
&= [\lambda_1^{\frac{1}{2}} {\boldsymbol v_1},\lambda_2^{\frac{1}{2}} {\boldsymbol v_2},\cdots,\lambda_k^{\frac{1}{2}} {\boldsymbol v_k},0,\cdots,0]
\begin{bmatrix}
\boldsymbol{u_1}^T \\
\boldsymbol{u_2}^T \\
\vdots \\
\boldsymbol{u_n}^T \\
\end{bmatrix}\\
&= \lambda_1^{\frac{1}{2}} {\boldsymbol v_1} \boldsymbol{u_1}^T + \lambda_2^{\frac{1}{2}} {\boldsymbol v_2} \boldsymbol{u_2}^T + \cdots +\lambda_k^{\frac{1}{2}} {\boldsymbol v_k} \boldsymbol{u_k}^T
\end{aligned}
A m × n = [ v 1 , v 2 , ⋯ , v m ] ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎡ λ 1 2 1 λ 2 2 1 ⋱ λ k 2 1 0 ⋱ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎤ ⎣ ⎢ ⎢ ⎢ ⎡ u 1 T u 2 T ⋮ u n T ⎦ ⎥ ⎥ ⎥ ⎤ = [ λ 1 2 1 v 1 , λ 2 2 1 v 2 , ⋯ , λ k 2 1 v k , 0 , ⋯ , 0 ] ⎣ ⎢ ⎢ ⎢ ⎡ u 1 T u 2 T ⋮ u n T ⎦ ⎥ ⎥ ⎥ ⎤ = λ 1 2 1 v 1 u 1 T + λ 2 2 1 v 2 u 2 T + ⋯ + λ k 2 1 v k u k T A m × n A_{m\times n} A m × n ( m + n + 1 ) × k (m+n+1)\times k ( m + n + 1 ) × k p p p E r r o r = 1 − ∑ i = 1 p λ i ∑ i = 1 m i n ( m , n ) λ i
Error= 1 -\frac{\displaystyle \sum_{i=1}^p\lambda_i}{\displaystyle \sum_{i=1}^{min(m,n)}\lambda_i}
E r r o r = 1 − i = 1 ∑ m i n ( m , n ) λ i i = 1 ∑ p λ i
背景来源：
已知向量 x 1 , x 2 , ⋯ , x N , x i ∈ R n {\boldsymbol x_1},{\boldsymbol x_2},\cdots,{\boldsymbol x_N},{\boldsymbol x_i} \in\R^n x 1 , x 2 , ⋯ , x N , x i ∈ R n y 1 , y 2 , ⋯ , y N , y i ∈ R 1 {\boldsymbol y_1},{\boldsymbol y_2},\cdots,{\boldsymbol y_N},{\boldsymbol y_i} \in \R^1 y 1 , y 2 , ⋯ , y N , y i ∈ R 1 [ y 1 y 2 ⋮ y N ] = [ x 11 x 12 ⋯ x 1 n x 21 x 22 ⋯ x 2 n ⋮ ⋮ ⋱ ⋮ x N 1 x N 2 ⋯ x N n ] [ a 1 a 2 ⋮ a n ]
\begin{aligned}
\begin{bmatrix}
{\boldsymbol y_1} \\
{\boldsymbol y_2} \\
\vdots \\
{\boldsymbol y_N}
\end{bmatrix}=
\begin{bmatrix}
x_{11} & x_{12} & \cdots &x_{1n} \\
x_{21} & x_{22} & \cdots &x_{2n} \\
\vdots & \vdots & \ddots &\vdots \\
x_{N1} & x_{N2} & \cdots &x_{Nn} \\
\end{bmatrix}
\begin{bmatrix}
{a_1} \\
{a_2} \\
\vdots \\
{a_n}
\end{bmatrix}
\end{aligned}
⎣ ⎢ ⎢ ⎢ ⎡ y 1 y 2 ⋮ y N ⎦ ⎥ ⎥ ⎥ ⎤ = ⎣ ⎢ ⎢ ⎢ ⎡ x 1 1 x 2 1 ⋮ x N 1 x 1 2 x 2 2 ⋮ x N 2 ⋯ ⋯ ⋱ ⋯ x 1 n x 2 n ⋮ x N n ⎦ ⎥ ⎥ ⎥ ⎤ ⎣ ⎢ ⎢ ⎢ ⎡ a 1 a 2 ⋮ a n ⎦ ⎥ ⎥ ⎥ ⎤ a = [ a 1 , a 2 , ⋯ , a n ] T {\boldsymbol a} = [a_1,a_2,\cdots,a_n]^T a = [ a 1 , a 2 , ⋯ , a n ] T X N × n a n × 1 = y N × 1
X_{N \times n} {\boldsymbol a}_{n\times1} = {\boldsymbol y}_{N\times1}
X N × n a n × 1 = y N × 1 N = n N=n N = n X X X a = X − 1 y
{\boldsymbol a} = X^{-1}{\boldsymbol y}
a = X − 1 y X X X
一般情况下 N ≠ n N \neq n N = n ( 5 ) (5) ( 5 ) min a ∣ ∣ X a − y ∣ ∣ 2
\min_a ||X {\boldsymbol a} - {\boldsymbol y}||^2
a min ∣ ∣ X a − y ∣ ∣ 2 J = ∣ ∣ X a − y ∣ ∣ 2 = ( X a − y ) T ( X a − y ) = ( a T X T − y T ) ( X a − y ) = a T X T X a − a T X T y − y T X a + y T y = a T X T X a − 2 y T X a + y T y
\begin{aligned}
J &=||X {\boldsymbol a} - {\boldsymbol y}||^2 \\
&= (X {\boldsymbol a} - {\boldsymbol y})^T(X {\boldsymbol a} - {\boldsymbol y})\\
&= ({\boldsymbol a}^TX^T-{\boldsymbol y^T})(X {\boldsymbol a} - {\boldsymbol y})\\
&= {\boldsymbol a}^TX^TX{\boldsymbol a} - {\boldsymbol a}^TX^T {\boldsymbol y} - {\boldsymbol y}^T X {\boldsymbol a} + {\boldsymbol y}^T{\boldsymbol y} \\
& = {\boldsymbol a}^TX^TX{\boldsymbol a} - 2{\boldsymbol y}^T X {\boldsymbol a} + {\boldsymbol y}^T{\boldsymbol y}
\end{aligned}
J = ∣ ∣ X a − y ∣ ∣ 2 = ( X a − y ) T ( X a − y ) = ( a T X T − y T ) ( X a − y ) = a T X T X a − a T X T y − y T X a + y T y = a T X T X a − 2 y T X a + y T y ∂ J ∂ a = 0 2 X T X a − 2 X T y = 0 X T X a = X T y
\begin{aligned}
\frac{\partial J}{\partial {\boldsymbol a}} &= 0 \\
2 X^TX{\boldsymbol a} -2X^T{\boldsymbol y} &= 0 \\
X^TX{\boldsymbol a} &= X^T{\boldsymbol y} \\
\end{aligned}
∂ a ∂ J 2 X T X a − 2 X T y X T X a = 0 = 0 = X T y X T X X^TX X T X X T X X^TX X T X
当 N > n N>n N > n
此时 ( X T X ) (X^TX) ( X T X ) n × n {n \times n} n × n a = ( X T X ) − 1 X T y
{\boldsymbol a} = (X^TX)^{-1}X^T{\boldsymbol y}
a = ( X T X ) − 1 X T y ( X T X ) − 1 X T (X^TX)^{-1}X^T ( X T X ) − 1 X T X X X ( 8 ) (8) ( 8 )
当 N < n N< n N < n
此时 ( X T X ) (X^TX) ( X T X ) n × n {n \times n} n × n R ( X T X ) ≤ R ( X ) ≤ N R(X^TX)\le R(X) \le N R ( X T X ) ≤ R ( X ) ≤ N R ( X T X ) ≠ n R(X^TX)\neq n R ( X T X ) = n
当遇见第2种情况时，我们该怎么办呢？这里给大家提供一种正则化的方法，即令J = ∣ ∣ X a − y ∣ ∣ 2 + λ ∣ ∣ a ∣ ∣ 2 = ( X a − y ) T ( X a − y ) + λ a T a = ( a T X T − y T ) ( X a − y ) + λ a T a = a T X T X a − a T X T y − y T X a + y T y + λ a T a = a T X T X a − 2 y T X a + y T y + λ a T a
\begin{aligned}
J &=||X {\boldsymbol a} - {\boldsymbol y}||^2 + \lambda|| {\boldsymbol a}||^2\\
&= (X {\boldsymbol a} - {\boldsymbol y})^T(X {\boldsymbol a} - {\boldsymbol y}) + \lambda {\boldsymbol a}^T {\boldsymbol a}\\
&= ({\boldsymbol a}^TX^T-{\boldsymbol y^T})(X {\boldsymbol a} - {\boldsymbol y}) + \lambda {\boldsymbol a}^T {\boldsymbol a} \\
&= {\boldsymbol a}^TX^TX{\boldsymbol a} - {\boldsymbol a}^TX^T {\boldsymbol y} - {\boldsymbol y}^T X {\boldsymbol a} + {\boldsymbol y}^T{\boldsymbol y} + \lambda {\boldsymbol a}^T {\boldsymbol a} \\
& = {\boldsymbol a}^TX^TX{\boldsymbol a} - 2{\boldsymbol y}^T X {\boldsymbol a} + {\boldsymbol y}^T{\boldsymbol y} + \lambda {\boldsymbol a}^T {\boldsymbol a}
\end{aligned}
J = ∣ ∣ X a − y ∣ ∣ 2 + λ ∣ ∣ a ∣ ∣ 2 = ( X a − y ) T ( X a − y ) + λ a T a = ( a T X T − y T ) ( X a − y ) + λ a T a = a T X T X a − a T X T y − y T X a + y T y + λ a T a = a T X T X a − 2 y T X a + y T y + λ a T a ∂ J ∂ a = 0 2 X T X a − 2 X T y + λ a = 0 X T X a + λ a = X T y ( X T X + λ I ) a = X T y
\begin{aligned}
\frac{\partial J}{\partial {\boldsymbol a}} &= 0 \\
2 X^TX{\boldsymbol a} -2X^T{\boldsymbol y} + \lambda {\boldsymbol a}&= 0 \\
X^TX{\boldsymbol a} + \lambda {\boldsymbol a} &= X^T{\boldsymbol y} \\
(X^TX+\lambda I){\boldsymbol a} &= X^T {\boldsymbol y}\\
\end{aligned}
∂ a ∂ J 2 X T X a − 2 X T y + λ a X T X a + λ a ( X T X + λ I ) a = 0 = 0 = X T y = X T y ( X T X + λ I ) (X^TX+\lambda I) ( X T X + λ I )
∀ y ∈ R n , y ≠ 0 \forall {\boldsymbol y} \in \R^n,{\boldsymbol y}\neq {\boldsymbol 0} ∀ y ∈ R n , y = 0 y T ( X T X + λ I ) y = y T X T X y + λ y T y = ( X y ) T ( X y ) + λ y T y = ∣ ∣ X y ∣ ∣ 2 + λ ∣ ∣ y ∣ ∣ 2 > 0
\begin{aligned}
{\boldsymbol y}^T(X^TX+\lambda I){\boldsymbol y} &={\boldsymbol y}^TX^TX{\boldsymbol y} + \lambda {\boldsymbol y}^T{\boldsymbol y}\\
&=(X{\boldsymbol y})^T(X{\boldsymbol y}) + \lambda {\boldsymbol y}^T{\boldsymbol y} \\
&= ||X{\boldsymbol y}||^2+\lambda ||{\boldsymbol y}||^2 > 0
\end{aligned}
y T ( X T X + λ I ) y = y T X T X y + λ y T y = ( X y ) T ( X y ) + λ y T y = ∣ ∣ X y ∣ ∣ 2 + λ ∣ ∣ y ∣ ∣ 2 > 0 ( X T X + λ I ) (X^TX+\lambda I) ( X T X + λ I )
因此有a = ( X T X + λ I ) − 1 X T y
{\boldsymbol a} = (X^TX+\lambda I)^{-1} X^T {\boldsymbol y}
a = ( X T X + λ I ) − 1 X T y
PCA实质上还是一种数据压缩的技术，如上图所示，已知 N N N 2 ∗ N 2*N 2 ∗ N u ⃗ \vec{u} u u ⃗ \vec{u} u N + 2 N+2 N + 2
但是这种用投影点替代真实样本点的方法，必然会带来误差，也就是A和A’之间的距离。因此我们需要找到一个投影方向，使得所有样本点与投影点之间的误差和最小，即最小重构误差。
做PCA前，需要将样本进行中心化操作，这样可以使得误差更小。
首先我们需要知道一个原始点与投影点之间的误差是多少？如图所示，假定 ∣ ∣ u ⃗ ∣ ∣ = 1 ||\vec{u}||=1 ∣ ∣ u ∣ ∣ = 1 x ⃗ \vec{x} x u ⃗ \vec{u} u < x ⃗ , u ⃗ > u ⃗ <\vec{x},\vec{u}>\vec{u} < x , u > u E = x ⃗ − < x ⃗ , u ⃗ > u ⃗ = x − ( x T u ) u
\begin{aligned}
E &= \vec{x} - <\vec{x},\vec{u}>\vec{u}\\
&= x - (x^Tu)u
\end{aligned}
E = x − < x , u > u = x − ( x T u ) u x , u ∈ R n , 且 u T u = 1 x,u\in \R^n,且u^Tu=1 x , u ∈ R n , 且 u T u = 1
我们的目的就是求取误差 J J J u ⃗ \vec{u} u min u ⃗ ∣ ∣ E ∣ ∣ 2 = ∣ ∣ x − ( x T u ) u ∣ ∣ 2 = [ x − ( x T u ) u ] T [ x − ( x T u ) u ] = [ x T − ( x T u ) u T ] [ x − ( x T u ) u ] = x T x − ( x T u ) 2 − ( x T u ) 2 + ( x T u ) 2 = x T x − ( x T u ) 2 = ∣ ∣ x ∣ ∣ 2 − ( x T u ) 2
\begin{aligned}
\min_{\vec{u}}||E||^2
&= ||x - (x^Tu)u||^2 \\
&= [x - (x^Tu)u]^T[x - (x^Tu)u] \\
&= [x^T - (x^Tu)u^T][x - (x^Tu)u]\\
&= x^Tx - (x^Tu)^2 - (x^Tu)^2 + (x^Tu)^2\\
&= x^Tx- (x^Tu)^2\\
&= ||x||^2 - (x^Tu)^2
\end{aligned}
u min ∣ ∣ E ∣ ∣ 2 = ∣ ∣ x − ( x T u ) u ∣ ∣ 2 = [ x − ( x T u ) u ] T [ x − ( x T u ) u ] = [ x T − ( x T u ) u T ] [ x − ( x T u ) u ] = x T x − ( x T u ) 2 − ( x T u ) 2 + ( x T u ) 2 = x T x − ( x T u ) 2 = ∣ ∣ x ∣ ∣ 2 − ( x T u ) 2 max u ⃗ ( x T u ) 2 ⇒ max u ⃗ ( x T u ) ( x T u ) ⇒ max u ⃗ ( u T x ) ( x T u ) ⇒ max u ⃗ u T ( x x T ) u
\max_{\vec{u}} (x^Tu)^2 \Rightarrow \max_{\vec{u}} (x^Tu)(x^Tu) \Rightarrow \max_{\vec{u}} (u^Tx) (x^Tu) \Rightarrow \max_{\vec{u} }u^T(xx^T)u
u max ( x T u ) 2 ⇒ u max ( x T u ) ( x T u ) ⇒ u max ( u T x ) ( x T u ) ⇒ u max u T ( x x T ) u N N N u ⃗ \vec{u} u max u ⃗ u T ∑ i = 1 N ( x i x i T ) u
\max_{\vec{u}} u^T \displaystyle \sum_{i=1}^N (x_ix_i^T)u
u max u T i = 1 ∑ N ( x i x i T ) u X = ∑ i = 1 N ( x i x i T ) X = \displaystyle \sum_{i=1}^N (x_ix_i^T) X = i = 1 ∑ N ( x i x i T ) X X X { max u ⃗ u T X u s t . ∣ ∣ u ∣ ∣ = 1
\left\{
\begin{aligned}
\max_{\vec{u}} u^T Xu \\
st. ||u||=1
\end{aligned}
\right.
⎩ ⎨ ⎧ u max u T X u s t . ∣ ∣ u ∣ ∣ = 1 L ( u , λ ) = u T X u + λ ( 1 − u T u )
\begin{aligned}
L(u,\lambda) = u^TXu + \lambda(1-u^Tu)
\end{aligned}
L ( u , λ ) = u T X u + λ ( 1 − u T u ) u , λ u,\lambda u , λ ∂ L ∂ u = 0 ⇒ X u = λ u ∂ L ∂ λ = 0 ⇒ u T u = 1
\frac{\partial L}{\partial u} = 0 \Rightarrow Xu=\lambda u\\
\frac{\partial L}{\partial \lambda} = 0 \Rightarrow u^Tu=1\\
∂ u ∂ L = 0 ⇒ X u = λ u ∂ λ ∂ L = 0 ⇒ u T u = 1 ( 12 ) (12) ( 1 2 ) X X X X X X n n n
