线性模型

Lei_ZM
2019-09-10

---

---

1. 一元线性回归

求解偏置$b$b和权重$w$w推导思路

1. 由最小二乘法导出损失函数$E(w, b)$E(w,b)

2. 证明损失函数

3. 分别对损失函数$E(w, b)$E(w,b)关于$b$b和$w$w求一阶偏导数

4. 令各自的一阶偏导数等于0解出$b$b和$w$w

1.1. 由最小二乘法导出损失函数$E(w, b)$E(w,b)

$\begin{aligned} E_{(w, b)} &=\sum_{i=1}^{m}\left(y_{i}-f\left(x_{i}\right)\right)^{2} \\ &=\sum_{i=1}^{m}\left(y_{i}-\left(w x_{i}+b\right)\right)^{2} \\ &=\sum_{i=1}^{m}\left(y_{i}-w x_{i}-b\right)^{2} \end{aligned} \tag{西瓜书式3.4}$E(w,b)​​=i=1∑m​(yi​−f(xi​))2=i=1∑m​(yi​−(wxi​+b))2=i=1∑m​(yi​−wxi​−b)2​(西瓜书式3.4)

---

1.2. 证明损失函数

1.2.1. 二元函数判断凹凸性：

设$f(x, y)$f(x,y)在区域$D$D上具有二阶连续偏导数，记$A=f_{x x}^{\prime \prime}(x, y)$A=fxx′′​(x,y)，$B=f_{x y}^{\prime \prime}(x, y)$B=fxy′′​(x,y)，$C=f_{y y}^{\prime \prime}(x, y)$C=fyy′′​(x,y)。则：

1. 在$D$D上恒有$A>0$A>0，且$AC-B^{2}\geq 0$AC−B2≥0时，$f(x, y)$f(x,y)在区域$D$D上是凸函数
2. 在$D$D上恒有$A<0$A<0，且$AC-B^{2}\geq 0$AC−B2≥0时，$f(x, y)$f(x,y)在区域$D$D上是凹函数

1.2.2. 二元凹凸函数求最值：

设$f(x, y)$f(x,y)是在开区域$D$D内具有连续偏导数的凸（或者凹）函数，$(x_{0}, y_{0})\in D$(x0​,y0​)∈D，且$f_{x}^{\prime}(x_{0}, y_{0})=0$fx′​(x0​,y0​)=0，$f_{y}^{\prime}(x_{0}, y_{0})=0$fy′​(x0​,y0​)=0，则$f(x_{0}, y_{0})$f(x0​,y0​)必为$f(x, y)$f(x,y)在$D$D内的最小值（或最大值）。

1.2.3. 证明

证明损失函数$E(w, b)$E(w,b)是关于$w$w和$b$b的凸函数——求$A=f_{xx}^{\prime \prime}(x, y)$A=fxx′′​(x,y)：

$\begin{aligned} \frac{\partial E_{(w, b)}}{\partial w} &=\frac{\partial}{\partial w}\left[\sum_{i=1}^{m}\left(y_{i}-\left(w x_{i}+b\right)\right)^{2}\right] \\ &=\sum_{i=1}^{m} \frac{\partial}{\partial w}\left(y_{i}-w x_{i}-b\right)^{2} \\ &=\sum_{i=1}^{m} 2 \cdot\left(y_{i}-w x_{i}-b\right) \cdot\left(-x_{i}\right) \\ &=2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right) \end{aligned} \tag{西瓜书式3.5}$∂w∂E(w,b)​​​=∂w∂​[i=1∑m​(yi​−(wxi​+b))2]=i=1∑m​∂w∂​(yi​−wxi​−b)2=i=1∑m​2⋅(yi​−wxi​−b)⋅(−xi​)=2(wi=1∑m​xi2​−i=1∑m​(yi​−b)xi​)​(西瓜书式3.5)

故有：

$\begin{aligned} \frac{\partial^{2} E_{(w, b)}}{\partial w^{2}} &=\frac{\partial}{\partial w}\left(\frac{\partial E_{(w, b)}}{\partial w}\right) \\ &=\frac{\partial}{\partial w}\left[2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right)\right] \\ &=\frac{\partial}{\partial w}\left[2 w \sum_{i=1}^{m} x_{i}^{2}\right] \\ &=2 \sum_{i=1}^{m} x_{i}^{2} \end{aligned}$∂w2∂2E(w,b)​​​=∂w∂​(∂w∂E(w,b)​​)=∂w∂​[2(wi=1∑m​xi2​−i=1∑m​(yi​−b)xi​)]=∂w∂​[2wi=1∑m​xi2​]=2i=1∑m​xi2​​

此式即为$A=f_{xx}^{\prime \prime}(x, y)$A=fxx′′​(x,y)。

证明损失函数$E(w, b)$E(w,b)是关于$w$w和$b$b的凸函数——求$B=f_{xy}^{\prime \prime}(x, y)$B=fxy′′​(x,y)：

$\begin{aligned} \frac{\partial^{2} E_{(w, b)}}{\partial w \partial b} &=\frac{\partial}{\partial b}\left(\frac{\partial E_{(w, b)}}{\partial w}\right) \\ &=\frac{\partial}{\partial b}\left[2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right)\right] \\ &=\frac{\partial}{\partial b}\left[-2 \sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right] \\ &=\frac{\partial}{\partial b}\left(-2 \sum_{i=1}^{m} y_{i} x_{i}+2 \sum_{i=1}^{m} b x_{i}\right) \\ &=\frac{\partial}{\partial b}\left(2 \sum_{i=1}^{m} b x_{i}\right) \\ &=2 \sum_{i=1}^{m} x_{i} \end{aligned}$∂w∂b∂2E(w,b)​​​=∂b∂​(∂w∂E(w,b)​​)=∂b∂​[2(wi=1∑m​xi2​−i=1∑m​(yi​−b)xi​)]=∂b∂​[−2i=1∑m​(yi​−b)xi​]=∂b∂​(−2i=1∑m​yi​xi​+2i=1∑m​bxi​)=∂b∂​(2i=1∑m​bxi​)=2i=1∑m​xi​​

此式即为$B=f_{xy}^{\prime \prime}(x, y)$B=fxy′′​(x,y)。

证明损失函数$E(w, b)$E(w,b)是关于$w$w和$b$b的凸函数——求$C=f_{yy}^{\prime \prime}(x, y)$C=fyy′′​(x,y)：

$\begin{aligned} \frac{\partial E_{(w, b)}}{\partial b} &=\frac{\partial}{\partial b}\left[\sum_{i=1}^{m}\left(y_{i}-\left(w x_{i}+b\right)\right)^{2}\right] \\ &=\sum_{i=1}^{m} \frac{\partial}{\partial b}\left(y_{i}-w x_{i}-b\right)^{2} \\ &=\sum_{i=1}^{m} 2 \cdot\left(y_{i}-w x_{i}-b\right) \cdot(-1) \\ &=2\left(m b-\sum_{i=1}^{m}\left(y_{i}-w x_{i}\right)\right) \end{aligned} \tag{西瓜书式3.6}$∂b∂E(w,b)​​​=∂b∂​[i=1∑m​(yi​−(wxi​+b))2]=i=1∑m​∂b∂​(yi​−wxi​−b)2=i=1∑m​2⋅(yi​−wxi​−b)⋅(−1)=2(mb−i=1∑m​(yi​−wxi​))​(西瓜书式3.6)

故有：

$\begin{aligned} \frac{\partial^{2} E_{(w, b)}}{\partial b^{2}} &=\frac{\partial}{\partial b}\left(\frac{\partial E_{(w, b)}}{\partial b}\right) \\ &=\frac{\partial}{\partial b}\left[2\left(m b-\sum_{i=1}^{m}\left(y_{i}-w x_{i}\right)\right)\right] \\ &=\frac{\partial}{\partial b}(2 m b) \\ &=2 m \end{aligned}$∂b2∂2E(w,b)​​​=∂b∂​(∂b∂E(w,b)​​)=∂b∂​[2(mb−i=1∑m​(yi​−wxi​))]=∂b∂​(2mb)=2m​

此式即为$C=f_{yy}^{\prime \prime}(x, y)$C=fyy′′​(x,y)。

综上所述，有：

$\left\{ \begin{aligned} &A=f_{xx}^{\prime \prime}(x, y)=2 \sum_{i=1}^{m} x_{i}^{2} \\ &B=f_{xy}^{\prime \prime}(x, y)=2 \sum_{i=1}^{m} x_{i} \\ &C=f_{yy}^{\prime \prime}(x, y)=2 m \end{aligned} \right.$⎩⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎧​​A=fxx′′​(x,y)=2i=1∑m​xi2​B=fxy′′​(x,y)=2i=1∑m​xi​C=fyy′′​(x,y)=2m​

所以：

$\begin{aligned} A C-B^{2} &=2 m \cdot 2 \sum_{i=1}^{m} x_{i}^{2}-\left(2 \sum_{i=1}^{m} x_{i}\right)^{2} \\ &=4 m \sum_{i=1}^{m} x_{i}^{2}-4\left(\sum_{i=1}^{m} x_{i}\right)^{2} \\ &=4 m \sum_{i=1}^{m} x_{i}^{2}-4 \cdot m \cdot \frac{1}{m} \cdot\left(\sum_{i=1}^{m} x_{i}\right)^{2} \\ &=4 m \sum_{i=1}^{m} x_{i}^{2}-4 m \cdot \bar{x} \cdot \sum_{i=1}^{m} x_{i} \\ &=4 m\left(\sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m} x_{i} \bar{x}\right) \\ &=4 m \sum_{i=1}^{m}\left(x_{i}^{2}-x_{i} \bar{x}\right) \\ &=4 m \sum_{i=1}^{m}\left(x_{i}^{2}-x_{i} \bar{x}-x_{i} \bar{x}+x_{i} \bar{x}\right) \\ &\qquad \sum_{i=1}^{m} x_{i} \bar{x}=\bar{x} \sum_{i=1}^{m} x_{i}=\bar{x} \cdot m \cdot \frac{1}{m} \cdot \sum_{i=1}^{m} x_{i}=m \bar{x}^{2}=\sum_{i=1}^{m} \bar{x}^{2} \\ &=4 m \sum_{i=1}^{m}\left(x_{i}^{2}-x_{i} \bar{x}-x_{i} \bar{x}+\bar{x}^{2}\right) \\ &=4 m \sum_{i=1}^{m}\left(x_{i}-\bar{x}\right)^{2} \end{aligned}$AC−B2​=2m⋅2i=1∑m​xi2​−(2i=1∑m​xi​)2=4mi=1∑m​xi2​−4(i=1∑m​xi​)2=4mi=1∑m​xi2​−4⋅m⋅m1​⋅(i=1∑m​xi​)2=4mi=1∑m​xi2​−4m⋅xˉ⋅i=1∑m​xi​=4m(i=1∑m​xi2​−i=1∑m​xi​xˉ)=4mi=1∑m​(xi2​−xi​xˉ)=4mi=1∑m​(xi2​−xi​xˉ−xi​xˉ+xi​xˉ)i=1∑m​xi​xˉ=xˉi=1∑m​xi​=xˉ⋅m⋅m1​⋅i=1∑m​xi​=mxˉ2=i=1∑m​xˉ2=4mi=1∑m​(xi2​−xi​xˉ−xi​xˉ+xˉ2)=4mi=1∑m​(xi​−xˉ)2​

故有：

$AC-B^{2} = 4 m \sum_{i=1}^{m}\left(x_{i}-\bar{x}\right)^{2} \geq 0$AC−B2=4mi=1∑m​(xi​−xˉ)2≥0

也即损失函数$E(w, b)$E(w,b)是关于$w$w和$b$b的凸函数，得证！

---

1.3. 分别对损失函数$E(w, b)$E(w,b)关于$b$b和$w$w求一阶偏导数

损失函数$E(w, b)$E(w,b)关于$b$b求一阶偏导数：

$\begin{aligned} \frac{\partial E_{(w, b)}}{\partial b} &=\frac{\partial}{\partial b}\left[\sum_{i=1}^{m}\left(y_{i}-\left(w x_{i}+b\right)\right)^{2}\right] \\ &=\sum_{i=1}^{m} \frac{\partial}{\partial b}\left(y_{i}-w x_{i}-b\right)^{2} \\ &=\sum_{i=1}^{m} 2 \cdot\left(y_{i}-w x_{i}-b\right) \cdot(-1) \\ &=2\left(m b-\sum_{i=1}^{m}\left(y_{i}-w x_{i}\right)\right) \end{aligned} \tag{西瓜书式3.6}$∂b∂E(w,b)​​​=∂b∂​[i=1∑m​(yi​−(wxi​+b))2]=i=1∑m​∂b∂​(yi​−wxi​−b)2=i=1∑m​2⋅(yi​−wxi​−b)⋅(−1)=2(mb−i=1∑m​(yi​−wxi​))​(西瓜书式3.6)

损失函数$E(w, b)$E(w,b)关于$w$w求一阶偏导数：

$\begin{aligned} \frac{\partial E_{(w, b)}}{\partial w} &=\frac{\partial}{\partial w}\left[\sum_{i=1}^{m}\left(y_{i}-\left(w x_{i}+b\right)\right)^{2}\right] \\ &=\sum_{i=1}^{m} \frac{\partial}{\partial w}\left(y_{i}-w x_{i}-b\right)^{2} \\ &=\sum_{i=1}^{m} 2 \cdot\left(y_{i}-w x_{i}-b\right) \cdot\left(-x_{i}\right) \\ &=2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right) \end{aligned} \tag{西瓜书式3.5}$∂w∂E(w,b)​​​=∂w∂​[i=1∑m​(yi​−(wxi​+b))2]=i=1∑m​∂w∂​(yi​−wxi​−b)2=i=1∑m​2⋅(yi​−wxi​−b)⋅(−xi​)=2(wi=1∑m​xi2​−i=1∑m​(yi​−b)xi​)​(西瓜书式3.5)

---

1.4. 令各自的一阶偏导数等于0解出$b$b和$w$w

令损失函数$E(w, b)$E(w,b)关于$b$b的一阶偏导数等于0解出$b$b：

$\begin{aligned} \frac{\partial E_{(w, b)}}{\partial b} &=2\left(m b-\sum_{i=1}^{m}\left(y_{i}-w x_{i}\right)\right) =0 \\ &\Rightarrow m b-\sum_{i=1}^{m}\left(y_{i}-w x_{i}\right)=0 \\ & \begin{aligned} \Rightarrow b&=\frac{1}{m}\sum_{i=1}^{m}\left(y_{i}-w x_{i}\right) \\ &=\frac{1}{m}\sum_{i=1}^{m} y_{i} - w \frac{1}{m}\sum_{i=1}^{m} x_{i} \\ &=\bar{y}-w\bar{x} \end{aligned} \end{aligned} \tag{西瓜书式3.8}$∂b∂E(w,b)​​​=2(mb−i=1∑m​(yi​−wxi​))=0⇒mb−i=1∑m​(yi​−wxi​)=0⇒b​=m1​i=1∑m​(yi​−wxi​)=m1​i=1∑m​yi​−wm1​i=1∑m​xi​=yˉ​−wxˉ​​(西瓜书式3.8)

令损失函数$E(w, b)$E(w,b)关于$w$w的一阶偏导数等于0解出$w$w：

$\begin{aligned} \frac{\partial E_{(w, b)}}{\partial w} &=2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right) =0 \\ &\Rightarrow w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}=0 \\ &\Rightarrow w \sum_{i=1}^{m} x_{i}^{2} = \sum_{i=1}^{m}y_{i} x_{i} - \sum_{i=1}^{m} b x_{i} \\ &\qquad b=\bar{y}-w\bar{x} \\ &\Rightarrow w \sum_{i=1}^{m} x_{i}^{2}=\sum_{i=1}^{m} y_{i} x_{i}-\sum_{i=1}^{m}(\bar{y}-w \bar{x}) x_{i} \\ &\Rightarrow w \sum_{i=1}^{m} x_{i}^{2} =\sum_{i=1}^{m} y_{i} x_{i}-\bar{y} \sum_{i=1}^{m} x_{i}+w \bar{x} \sum_{i=1}^{m} x_{i} \\ &\Rightarrow w \sum_{i=1}^{m} x_{i}^{2}-w \bar{x} \sum_{i=1}^{m} x_{i}=\sum_{i=1}^{m} y_{i} x_{i}-\bar{y} \sum_{i=1}^{m} x_{i} \\ &\Rightarrow w\left(\sum_{i=1}^{m} x_{i}^{2}-\bar{x} \sum_{i=1}^{m} x_{i}\right)=\sum_{i=1}^{m} y_{i} x_{i}-\bar{y} \sum_{i=1}^{m} x_{i} \\ &\begin{aligned} \Rightarrow w &= \frac{\sum_{i=1}^{m} y_{i} x_{i}-\bar{y} \sum_{i=1}^{m} x_{i}}{\sum_{i=1}^{m} x_{i}^{2}-\bar{x} \sum_{i=1}^{m} x_{i}} \\ &\qquad \bar{y} \sum_{i=1}^{m} x_{i} = \frac{1}{m}\sum_{i=1}^{m} y_{i} \sum_{i=1}^{m} x_{i} = \bar{x} \sum_{i=1}^{m} y_{i} \\ &\qquad \bar{x}\sum_{i=1}^{m} x_{i} = \frac{1}{m}\sum_{i=1}^{m} x_{i} \sum_{i=1}^{m} x_{i} = \frac{1}{m} \left(\sum_{i=1}^{m} x_{i}\right)^{2} \\ &=\frac{\sum_{i=1}^{m} y_{i} x_{i}-\bar{x} \sum_{i=1}^{m} y_{i}}{\sum_{i=1}^{m} x_{i}^{2}-\frac{1}{m}\left(\sum_{i=1}^{m} x_{i}\right)^{2}} \\ &=\frac{\sum_{i=1}^{m} y_{i}\left(x_{i}-\bar{x}\right)}{\sum_{i=1}^{m} x_{i}^{2}-\frac{1}{m}\left(\sum_{i=1}^{m} x_{i}\right)^{2}}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎‪‫‫‌‬‎‮‌‌‫⁠‌ \end{aligned}‏‌‎‬‎‪‍‭‎ \end{aligned} \tag{西瓜书式3.7}$∂w∂E(w,b)​​​=2(wi=1∑m​xi2​−i=1∑m​(yi​−b)xi​)=0⇒wi=1∑m​xi2​−i=1∑m​(yi​−b)xi​=0⇒wi=1∑m​xi2​=i=1∑m​yi​xi​−i=1∑m​bxi​b=yˉ​−wxˉ⇒wi=1∑m​xi2​=i=1∑m​yi​xi​−i=1∑m​(yˉ​−wxˉ)xi​⇒wi=1∑m​xi2​=i=1∑m​yi​xi​−yˉ​i=1∑m​xi​+wxˉi=1∑m​xi​⇒wi=1∑m​xi2​−wxˉi=1∑m​xi​=i=1∑m​yi​xi​−yˉ​i=1∑m​xi​⇒w(i=1∑m​xi2​−xˉi=1∑m​xi​)=i=1∑m​yi​xi​−yˉ​i=1∑m​xi​⇒w​=∑i=1m​xi2​−xˉ∑i=1m​xi​∑i=1m​yi​xi​−yˉ​∑i=1m​xi​​yˉ​i=1∑m​xi​=m1​i=1∑m​yi​i=1∑m​xi​=xˉi=1∑m​yi​xˉi=1∑m​xi​=m1​i=1∑m​xi​i=1∑m​xi​=m1​(i=1∑m​xi​)2=∑i=1m​xi2​−m1​(∑i=1m​xi​)2∑i=1m​yi​xi​−xˉ∑i=1m​yi​​=∑i=1m​xi2​−m1​(∑i=1m​xi​)2∑i=1m​yi​(xi​−xˉ)​‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎‪‫‫‌‬‎‮‌‌‫⁠‌​‏‌‎‬‎‪‍‭‎​(西瓜书式3.7)

将$w$w向量化，有：

$\begin{aligned} w &=\frac{\sum_{i=1}^{m} y_{i}\left(x_{i}-\bar{x}\right)}{\sum_{i=1}^{m} x_{i}^{2}-\frac{1}{m}\left(\sum_{i=1}^{m} x_{i}\right)^{2}}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎ \\ &\qquad \frac{1}{m}\left(\sum_{i=1}^{m} x_{i}\right)^{2} = \left(\frac{1}{m} \sum_{i=1}^{m} x_{i}\right) \sum_{i=1}^{m} x_{i} = \bar{x} \sum_{i=1}^{m} x_{i} = \sum_{i=1}^{m} x_{i} \bar{x} \\ &=\frac{\sum_{i=1}^{m} \left(y_{i} x_{i}-y_{i} \bar{x}\right)}{\sum_{i=1}^{m} \left(x_{i}^{2}-x_{i} \bar{x}\right)}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎ \\ &=\frac{\sum_{i=1}^{m} \left(y_{i} x_{i}-y_{i} \bar{x}-y_{i} \bar{x}-y_{i} \bar{x}\right)}{\sum_{i=1}^{m} \left(x_{i}^{2}-x_{i} \bar{x}-x_{i} \bar{x}-x_{i} \bar{x}\right)}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎ \\ &\qquad \sum_{i=1}^{m} y_{i} \bar{x}=\bar{x} \sum_{i=1}^{m} y_{i}=\frac{1}{m} \sum_{i=1}^{m} x_{i} \sum_{i=1}^{m} y_{i}=\sum_{i=1}^{m} x_{i} \cdot \frac{1}{m} \cdot \sum_{i=1}^{m} y_{i}=\sum_{i=1}^{m} x_{i} \bar{y} \\ &\qquad \sum_{i=1}^{m} y_{i} \bar{x}=\bar{x} \sum_{i=1}^{m} y_{i}=\bar{x} \cdot m \cdot \frac{1}{m} \cdot \sum_{i=1}^{m} y_{i}=m \bar{x} \bar{y}=\sum_{i=1}^{m} \bar{x} \bar{y} \\ &\qquad \sum_{i=1}^{m} x_{i} \bar{x}=\bar{x} \sum_{i=1}^{m} x_{i}=\bar{x} \cdot m \cdot \frac{1}{m} \cdot \sum_{i=1}^{m} x_{i}=m \bar{x}^{2}=\sum_{i=1}^{m} \bar{x}^{2} \\ &=\frac{\sum_{i=1}^{m} \left(y_{i} x_{i}-y_{i} \bar{x}-x_{i} \bar{y}-\bar{x}\bar{y}\right)}{\sum_{i=1}^{m} \left(x_{i}^{2}-x_{i} \bar{x}-x_{i} \bar{x}-\bar{x}^{2}\right)}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎ \\ &=\frac{\sum_{i=1}^{m} \left(x_{i}-\bar{x}\right)\left(y_{i}-\bar{y}\right)}{\sum_{i=1}^{m} \left(x_{i}-\bar{x}\right)^{2}}‎ \\ &\qquad x=\left(x_{1},x_{2},\cdots, x_{m}\right)^{T} \\ &\qquad y=\left(y_{1},y_{2},\cdots,y_{m}\right)^{T} \\ &\qquad x_{d}=\left(x_{1}-\bar{x},x_{2}-\bar{x},\cdots,x_{m}-\bar{x}\right)^{T} \\ &\qquad y_{d}=\left(y_{1}-\bar{y},y_{2}-\bar{y},\cdots,y_{m}-\bar{y}\right)^{T} \\ &=\frac{x_{d}^{T} y_{d}}{x_{d}^{T} x_{d}} \end{aligned}$w​=∑i=1m​xi2​−m1​(∑i=1m​xi​)2∑i=1m​yi​(xi​−xˉ)​‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎m1​(i=1∑m​xi​)2=(m1​i=1∑m​xi​)i=1∑m​xi​=xˉi=1∑m​xi​=i=1∑m​xi​xˉ=∑i=1m​(xi2​−xi​xˉ)∑i=1m​(yi​xi​−yi​xˉ)​‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎=∑i=1m​(xi2​−xi​xˉ−xi​xˉ−xi​xˉ)∑i=1m​(yi​xi​−yi​xˉ−yi​xˉ−yi​xˉ)​‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎i=1∑m​yi​xˉ=xˉi=1∑m​yi​=m1​i=1∑m​xi​i=1∑m​yi​=i=1∑m​xi​⋅m1​⋅i=1∑m​yi​=i=1∑m​xi​yˉ​i=1∑m​yi​xˉ=xˉi=1∑m​yi​=xˉ⋅m⋅m1​⋅i=1∑m​yi​=mxˉyˉ​=i=1∑m​xˉyˉ​i=1∑m​xi​xˉ=xˉi=1∑m​xi​=xˉ⋅m⋅m1​⋅i=1∑m​xi​=mxˉ2=i=1∑m​xˉ2=∑i=1m​(xi2​−xi​xˉ−xi​xˉ−xˉ2)∑i=1m​(yi​xi​−yi​xˉ−xi​yˉ​−xˉyˉ​)​‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎=∑i=1m​(xi​−xˉ)2∑i=1m​(xi​−xˉ)(yi​−yˉ​)​‎x=(x1​,x2​,⋯,xm​)Ty=(y1​,y2​,⋯,ym​)Txd​=(x1​−xˉ,x2​−xˉ,⋯,xm​−xˉ)Tyd​=(y1​−yˉ​,y2​−yˉ​,⋯,ym​−yˉ​)T=xdT​xd​xdT​yd​​​

---

---

2. 二元线性回归

求解权重$\hat{w}$w^的公式推导推导思路：

1. 由最小二乘法导出损失函数$E_{\hat{w}}$Ew^​

2. 证明损失函数$E_{\hat{w}}$Ew^​是关于$\hat{w}$w^的凸函数

3. 对损失函数$E_{\hat{w}}$Ew^​关于$\hat{w}$w^求一阶偏导数

4. 令各自的一阶偏导数等于0解出$\hat{w}^{*}$w^∗

2.1. 将$w$w和$b$b组合成$\hat{w}$w^

$\begin{aligned} f\left(\boldsymbol{x}_{i}\right) &=\boldsymbol{w}^{T} \boldsymbol{x}_{i}+b \\ &=\left(\begin{array}{cccc} {w_{1}} & {w_{2}} & {\dots} & {w_{d}}\end{array}\right) \left(\begin{array}{c}{x_{i 1}} \\ {x_{i 2}} \\ {\vdots} \\ {x_{i d}}\end{array}\right)+b \\ &=w_{1} x_{i 1}+w_{2} x_{i 2}+\ldots+w_{d} x_{i d}+b \\ &\qquad w_{d+1}=b \\ &=w_{1} x_{i 1}+w_{2} x_{i 2}+\ldots+w_{d} x_{i d}+w_{d+1} \cdot 1 \\ &=\left(\begin{array}{ccccc} {w_{1}} & {w_{2}} & {\dots} & {w_{d}} & {w_{d+1}}\end{array}\right) \left(\begin{array}{c}{x_{i 1}} \\ {x_{i 2}} \\ {\vdots} \\ {x_{i d}} \\ 1\end{array}\right) \\ &=\hat{w}^{T}\hat{x}_{i} \end{aligned}$f(xi​)​=wTxi​+b=(w1​​w2​​…​wd​​)⎝⎜⎜⎜⎛​xi1​xi2​⋮xid​​⎠⎟⎟⎟⎞​+b=w1​xi1​+w2​xi2​+…+wd​xid​+bwd+1​=b=w1​xi1​+w2​xi2​+…+wd​xid​+wd+1​⋅1=(w1​​w2​​…​wd​​wd+1​​)⎝⎜⎜⎜⎜⎜⎛​xi1​xi2​⋮xid​1​⎠⎟⎟⎟⎟⎟⎞​=w^Tx^i​​

---

2.2. 由最小二乘法导出损失函数$E_{\hat{w}}$Ew^​

$\begin{aligned} E_{\hat{\boldsymbol{w}}} &=\sum_{i=1}^{m}\left(y_{i}-f\left(\hat{\boldsymbol{x}}_{i}\right)\right)^{2} \\ &=\sum^{m}\left(y_{i}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{i}\right)^{2} \\ &\qquad \begin{aligned} &\mathbf{X} =\left(\begin{array}{ccccc} {x_{11}} & {x_{12}} & {\dots} & {x_{1 d}} & {1} \\ {x_{21}} & {x_{22}} & {\dots} & {x_{2 d}} & {1} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} & {\vdots} \\ {x_{m 1}} & {x_{m 2}} & {\dots} & {x_{m d}} & {1} \end{array}\right) =\left(\begin{array}{cc} {\boldsymbol{x}_{1}^{\mathrm{T}}} & {1} \\ {\boldsymbol{x}_{2}^{\mathrm{T}}} & {1} \\ {\vdots} & {\vdots} \\ {\boldsymbol{x}_{m}^{\mathrm{T}}} & {1} \end{array}\right) =\left(\begin{array}{c} {\hat{\boldsymbol{x}}_{1}^{T}} \\ {\hat{\boldsymbol{x}}_{2}^{T}} \\ {\vdots} \\ {\hat{\boldsymbol{x}}_{m}^{T}} \end{array}\right) \\ &\boldsymbol{y}=\left(y_{1},y_{2},\cdots,y_{m}\right)^{T} \end{aligned} \\ &=\left(y_{1}-\hat{\boldsymbol{w}}^{T} \hat{x}_{1}\right)^{2} + \left(y_{2}-\hat{\boldsymbol{w}}^{T} \hat{x}_{2}\right)^{2} + \cdots + \left(y_{m}-\hat{\boldsymbol{w}}^{T} \hat{x}_{m}\right)^{2} \\ &=\left(\begin{array}{cccc} {y_{1}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{1}} & {y_{2}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{2}} & {\cdots} & {y_{m}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{m}} \end{array}\right) \left(\begin{array}{c} {y_{1}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{1}} \\ {y_{2}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{2}} \\ {\vdots} \\ {y_{m}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{m}} \end{array}\right) \\ &\qquad \left(\begin{array}{c} {y_{1}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{1}} \\ {y_{2}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{2}} \\ {\vdots} \\ {y_{m}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{m}} \end{array}\right) =\left(\begin{array}{c} {y_{1}} \\ {y_{2}} \\ {\vdots} \\ {y_{m}} \end{array}\right) -\left(\begin{array}{c} {\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{1}} \\ {\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{2}} \\ {\vdots} \\ {\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{m}} \end{array}\right) =\left(\begin{array}{c} {y_{1}} \\ {y_{2}} \\ {\vdots} \\ {y_{m}} \end{array}\right) -\left(\begin{array}{c} {\hat{\boldsymbol{x}}_{1}^{T} \hat{\boldsymbol{w}}} \\ {\hat{\boldsymbol{x}}_{2}^{T} \hat{\boldsymbol{w}}} \\ {\vdots} \\ {\hat{\boldsymbol{x}}_{m}^{T} \hat{\boldsymbol{w}}} \end{array}\right) =\left(\begin{array}{c} {y_{1}} \\ {y_{2}} \\ {\vdots} \\ {y_{m}} \end{array}\right) -\left(\begin{array}{c} {\hat{\boldsymbol{x}}_{1}^{T}} \\ {\hat{\boldsymbol{x}}_{2}^{T}} \\ {\vdots} \\ {\hat{\boldsymbol{x}}_{m}^{T}} \end{array}\right) \hat{\boldsymbol{w}} =\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}} \\ &\qquad \left(\begin{array}{cccc} {y_{1}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{1}} & {y_{2}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{2}} & {\cdots} & {y_{m}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{m}} \end{array}\right) =\left(\begin{array}{c} {y_{1}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{1}} \\ {y_{2}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{2}} \\ {\vdots} \\ {y_{m}-\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{m}} \end{array}\right)^{T} =\left(\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}}\right)^{T} \\ &=\left(\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}}\right)^{T}\left(\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}}\right) \end{aligned}$Ew^​​=i=1∑m​(yi​−f(x^i​))2=∑m​(yi​−w^Tx^i​)2​X=⎝⎜⎜⎜⎛​x11​x21​⋮xm1​​x12​x22​⋮xm2​​……⋱…​x1d​x2d​⋮xmd​​11⋮1​⎠⎟⎟⎟⎞​=⎝⎜⎜⎜⎛​x1T​x2T​⋮xmT​​11⋮1​⎠⎟⎟⎟⎞​=⎝⎜⎜⎜⎛​x^1T​x^2T​⋮x^mT​​⎠⎟⎟⎟⎞​y=(y1​,y2​,⋯,ym​)T​=(y1​−w^Tx^1​)2+(y2​−w^Tx^2​)2+⋯+(ym​−w^Tx^m​)2=(y1​−w^Tx^1​​y2​−w^Tx^2​​⋯​ym​−w^Tx^m​​)⎝⎜⎜⎜⎛​y1​−w^Tx^1​y2​−w^Tx^2​⋮ym​−w^Tx^m​​⎠⎟⎟⎟⎞​⎝⎜⎜⎜⎛​y1​−w^Tx^1​y2​−w^Tx^2​⋮ym​−w^Tx^m​​⎠⎟⎟⎟⎞​=⎝⎜⎜⎜⎛​y1​y2​⋮ym​​⎠⎟⎟⎟⎞​−⎝⎜⎜⎜⎛​w^Tx^1​w^Tx^2​⋮w^Tx^m​​⎠⎟⎟⎟⎞​=⎝⎜⎜⎜⎛​y1​y2​⋮ym​​⎠⎟⎟⎟⎞​−⎝⎜⎜⎜⎛​x^1T​w^x^2T​w^⋮x^mT​w^​⎠⎟⎟⎟⎞​=⎝⎜⎜⎜⎛​y1​y2​⋮ym​​⎠⎟⎟⎟⎞​−⎝⎜⎜⎜⎛​x^1T​x^2T​⋮x^mT​​⎠⎟⎟⎟⎞​w^=y−Xw^(y1​−w^Tx^1​​y2​−w^Tx^2​​⋯​ym​−w^Tx^m​​)=⎝⎜⎜⎜⎛​y1​−w^Tx^1​y2​−w^Tx^2​⋮ym​−w^Tx^m​​⎠⎟⎟⎟⎞​T=(y−Xw^)T=(y−Xw^)T(y−Xw^)​

---

2.3. 证明损失函数$E_{\hat{w}}$Ew^​是关于$\hat{w}$w^的凸函数

凸集定义：

设集合$D\in R^{n}$D∈Rn，如果对任意的$x,y\in D$x,y∈D与任意的$a\in [0,1]$a∈[0,1]，有$ax+(1-a)y\in D$ax+(1−a)y∈D，则称集合$D$D是凸集。

凸集的几何意义：

若两个点属于此集合，则这两点连线上的任意一点均属于此集合。

梯度定义：

设$n$n元函数$f(\boldsymbol{x})$f(x)对自变量$\boldsymbol{x}=\left(x_{1}, x_{2}, \cdots, x_{n}\right)^{T}$x=(x1​,x2​,⋯,xn​)T的各分量$x_{i}$xi​的偏导数$\frac{\partial f(\boldsymbol{x})}{\partial x_{i}} \quad \left(i=1,2,\cdots,n\right)$∂xi​∂f(x)​(i=1,2,⋯,n)都存在，则称函数$f(\boldsymbol{x})$f(x)在$\boldsymbol{x}$x处一阶可导，并称向量

$\nabla f(\boldsymbol{x}) =\left(\begin{array}{c} {\frac{\partial f(\boldsymbol{x})}{\partial x_{1}}} \\ {\frac{\partial f(\boldsymbol{x})}{\partial x_{2}}} \\ {\vdots} \\ {\frac{\partial f(\boldsymbol{x})}{\partial x_{n}}}\end{array}\right)$∇f(x)=⎝⎜⎜⎜⎜⎛​∂x1​∂f(x)​∂x2​∂f(x)​⋮∂xn​∂f(x)​​⎠⎟⎟⎟⎟⎞​

为函数$f(\boldsymbol{x})$f(x)在$\boldsymbol{x}$x处的一阶导数或梯度，记为$\nabla f(\boldsymbol{x})$∇f(x)（列向量）。

Hessian（海塞）矩阵定义：设$n$n元函数$f(\boldsymbol{x})$f(x)对自变量$\boldsymbol{x}=\left(x_{1}, x_{2}, \cdots, x_{n}\right)^{T}$x=(x1​,x2​,⋯,xn​)T的各分量$x_{i}$xi​的二阶偏导数$\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{i} \partial x_{j}} \quad \left(i=1,2,\cdots,n; j=1,2,\cdots,n\right)$∂xi​∂xj​∂2f(x)​(i=1,2,⋯,n;j=1,2,⋯,n)都存在，则称函数$f(\boldsymbol{x})$f(x)在$\boldsymbol{x}$x处二阶可导，并称矩阵

$\nabla^{2} f(\boldsymbol{x}) =\left[\begin{array}{cccc} {\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{1}^{2}}} & {\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{1} \partial x_{2}}} & {\cdots} & {\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{1} \partial x_{n}}} \\ {\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{2} \partial x_{1}}} & {\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{2}^{2}}} & {\cdots} & {\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{2} \partial x_{n}}} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{n} \partial x_{1}}} & {\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{n} \partial x_{2}}} & {\cdots} & {\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{n}^{2}}} \end{array}\right]$∇2f(x)=⎣⎢⎢⎢⎢⎢⎡​∂x12​∂2f(x)​∂x2​∂x1​∂2f(x)​⋮∂xn​∂x1​∂2f(x)​​∂x1​∂x2​∂2f(x)​∂x22​∂2f(x)​⋮∂xn​∂x2​∂2f(x)​​⋯⋯⋱⋯​∂x1​∂xn​∂2f(x)​∂x2​∂xn​∂2f(x)​⋮∂xn2​∂2f(x)​​⎦⎥⎥⎥⎥⎥⎤​

为函数$f(\boldsymbol{x})$f(x)在$\boldsymbol{x}$x处的二阶导数或Hessian（海塞）矩阵，记为$\nabla^{2} f(\boldsymbol{x})$∇2f(x)。若$f(\boldsymbol{x})$f(x)在$\boldsymbol{x}$x各变元的所有二阶偏导数都连续，则$\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{i} \partial x_{j}}=\frac{\partial^{2} f(\boldsymbol{x})}{\partial x_{j} \partial x_{i}}$∂xi​∂xj​∂2f(x)​=∂xj​∂xi​∂2f(x)​，此时$\nabla^{2} f(\boldsymbol{x})$∇2f(x)为对称矩阵。

多元实值函数凹凸性判定定理：

设$D\subset R^{n}$D⊂Rn是非空开凸集，$f:D\subset R^{n} \to R$f:D⊂Rn→R，且$f(\boldsymbol{x})$f(x)在$D$D上二阶连续可微，如果$f(\boldsymbol{x})$f(x)的$Hessian$Hessian矩阵$\nabla^{2} f(\boldsymbol{x})$∇2f(x)在$D$D上是正定的，则$f(\boldsymbol{x})$f(x)是$D$D上的严格凸函数。

凸充分性定理：

若$f:R^{n} \to R$f:Rn→R是凸函数，且$f(\boldsymbol{x})$f(x)一阶连续可微，则$x^{*}$x∗是全局解的充分必要条件是$\nabla f(\boldsymbol{x}^{*})=0$∇f(x∗)=0，其中$\nabla f(\boldsymbol{x})$∇f(x)为$f(\boldsymbol{x})$f(x)关于$\boldsymbol{x}$x的一阶导数（也称梯度）。

---

2.4. 对损失函数$E_{\hat{w}}$Ew^​关于$\hat{w}$w^求一阶偏导数

$\begin{aligned} \frac{\partial E_{\hat{\boldsymbol{w}}}}{\partial \hat{\boldsymbol{w}}} &=\frac{\partial}{\partial \hat{\boldsymbol{w}}}\left[(\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}})^{T}(\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}})\right] \\ &=\frac{\partial}{\partial \hat{\boldsymbol{w}}}\left[\left(\boldsymbol{y}^{T}-\hat{\boldsymbol{w}}^{T} \mathbf{X}^{T}\right)(\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}})\right] \\ &=\frac{\partial}{\partial \hat{\boldsymbol{w}}}\left[\boldsymbol{y}^{T} \boldsymbol{y}-\boldsymbol{y}^{T} \mathbf{X} \hat{\boldsymbol{w}}-\hat{\boldsymbol{w}}^{T} \mathbf{X}^{T} \boldsymbol{y}+\hat{\boldsymbol{w}}^{T} \mathbf{X}^{T} \mathbf{X} \hat{\boldsymbol{w}}\right] \\ &=\frac{\partial}{\partial \hat{\boldsymbol{w}}}\left[-\boldsymbol{y}^{T} \mathbf{X} \hat{\boldsymbol{w}}-\hat{\boldsymbol{w}}^{T} \mathbf{X}^{T} \boldsymbol{y}+\hat{\boldsymbol{w}}^{T} \mathbf{X}^{T} \mathbf{X} \hat{\boldsymbol{w}}\right] \\ &=-\frac{\partial \boldsymbol{y}^{T} \mathbf{X} \hat{\boldsymbol{w}}}{\partial \hat{\boldsymbol{w}}}-\frac{\partial \hat{\boldsymbol{w}}^{T} \mathbf{X}^{T} \boldsymbol{y}}{\partial \hat{\boldsymbol{w}}}+\frac{\partial \hat{\boldsymbol{w}}^{T} \mathbf{X}^{T} \mathbf{X} \hat{\boldsymbol{w}}}{\partial \hat{\boldsymbol{w}}} \\ &\qquad \frac{\partial \boldsymbol{x}^{T} \boldsymbol{a}}{\partial \boldsymbol{x}}=\frac{\partial \boldsymbol{a}^{T} \boldsymbol{x}}{\partial \boldsymbol{x}}=\boldsymbol{a} \\ &\qquad \frac{\partial \boldsymbol{x}^{T} \mathbf{B} \boldsymbol{x}}{\partial \boldsymbol{x}}=\left(\mathbf{B}+\mathbf{B}^{T}\right) \boldsymbol{x} \\ &=-\mathbf{X}^{T} \boldsymbol{y}-\mathbf{X}^{T} \boldsymbol{y}+\left(\mathbf{X}^{T} \mathbf{X}+\mathbf{X}^{T} \mathbf{X}\right) \hat{w} \\ &=2\mathbf{X}^{T}\left(\mathbf{X} \hat{w}-\boldsymbol{y}\right) \end{aligned} \tag{西瓜书式3.10}$∂w^∂Ew^​​​=∂w^∂​[(y−Xw^)T(y−Xw^)]=∂w^∂​[(yT−w^TXT)(y−Xw^)]=∂w^∂​[yTy−yTXw^−w^TXTy+w^TXTXw^]=∂w^∂​[−yTXw^−w^TXTy+w^TXTXw^]=−∂w^∂yTXw^​−∂w^∂w^TXTy​+∂w^∂w^TXTXw^​∂x∂xTa​=∂x∂aTx​=a∂x∂xTBx​=(B+BT)x=−XTy−XTy+(XTX+XTX)w^=2XT(Xw^−y)​(西瓜书式3.10)

所以有：

$\begin{aligned} \frac{\partial^{2} E_{\hat{w}}}{\partial \hat{w} \partial \hat{w}^{T}} &=\frac{\partial}{\partial \hat{w}}\left(\frac{\partial E_{\hat{w}}}{\partial \hat{w}}\right) \\ &=\frac{\partial}{\partial \hat{w}}\left[2 \mathbf{X}^{T}(\mathbf{X} \hat{w}-\boldsymbol{y})\right] \\ &=\frac{\partial}{\partial \hat{w}}\left(2 \mathbf{X}^{T} \mathbf{X} \hat{w}-2 \mathbf{X}^{T} \boldsymbol{y}\right) \\ &=2 \mathbf{X}^{T} \mathbf{X} \hat{w} \end{aligned} \tag{Hessian矩阵}$∂w^∂w^T∂2Ew^​​​=∂w^∂​(∂w^∂Ew^​​)=∂w^∂​[2XT(Xw^−y)]=∂w^∂​(2XTXw^−2XTy)=2XTXw^​(Hessian矩阵)

---

2.5. 令一阶偏导数等于0解出$\hat{w}^{*}$w^∗

$\begin{aligned} &\quad \frac{\partial E_{\hat{w}}}{\partial \hat{w}} =2 \mathbf{X}^{T}(\mathbf{X} \hat{w}-\boldsymbol{y})=0 \\ &\Rightarrow 2 \mathbf{X}^{T} \mathbf{X} \hat{w}-2 \mathbf{X}^{T} \boldsymbol{y}=0 \\ &\Rightarrow 2 \mathbf{X}^{T} \mathbf{X} \hat{w}=2 \mathbf{X}^{T} \boldsymbol{y} \\ &\Rightarrow \hat{w} = \left(\mathbf{X}^{T} \mathbf{X} \right)^{-1} \mathbf{X}^{T} \boldsymbol{y} \end{aligned} \tag{西瓜书式3.11}$​∂w^∂Ew^​​=2XT(Xw^−y)=0⇒2XTXw^−2XTy=0⇒2XTXw^=2XTy⇒w^=(XTX)−1XTy​(西瓜书式3.11)

---

---

3. 广义线性模型

3.1. 指数族分布

指数族（Exponential family）分布是一类分布的总称，该类分布的分布律（或者概率密度函数）的一般形式如下：

$p(y ; \eta)=b(y) \exp \left(\eta^{T} T(y)-a(\eta)\right)$p(y;η)=b(y)exp(ηTT(y)−a(η))

其中，$\eta$η称为该分布的自然参数；$T(y)$T(y)为充分统计量，视具体的分布而定，通常是等于随机变量$y$y本身；$a(\eta)$a(η)为配分函数；$b(y)$b(y)为关于随机变量$y$y的函数，常见的伯努利分布和正态分布均属于指数族分布。

证明伯努利分布属于指数族分布：

已知伯努利分布的分布律为：

$p(y)=\phi^{y}(1-\phi)^{1-y}$p(y)=ϕy(1−ϕ)1−y

其中$y\in\{0,1\}$y∈{0,1}，$\phi$ϕ为$y=1$y=1的概率，即$p(y=1)=\phi$p(y=1)=ϕ，对上式恒等变形可得：

$\begin{aligned} p(y) &=\phi^{y}(1-\phi)^{1-y} \\ &=\exp \left(\ln \left(\phi^{y}(1-\phi)^{1-y}\right)\right) \\ &=\exp \left(\ln \phi^{y}+\ln(1-\phi)^{1-y}\right) \\ &=\exp (y \ln \phi+(1-y) \ln (1-\phi)) \\ &=\exp (y \ln \phi+\ln (1-\phi)-y \ln (1-\phi)) \\ &=\exp (y(\ln \phi-\ln (1-\phi))+\ln (1-\phi)) \\ &=\exp \left(y \ln \left(\frac{\phi}{1-\phi}\right)+\ln (1-\phi)\right) \end{aligned}$p(y)​=ϕy(1−ϕ)1−y=exp(ln(ϕy(1−ϕ)1−y))=exp(lnϕy+ln(1−ϕ)1−y)=exp(ylnϕ+(1−y)ln(1−ϕ))=exp(ylnϕ+ln(1−ϕ)−yln(1−ϕ))=exp(y(lnϕ−ln(1−ϕ))+ln(1−ϕ))=exp(yln(1−ϕϕ​)+ln(1−ϕ))​

对比指数分布的一般形式$p(y;\eta)=b(y)exp\left(\eta^(T)T(y)-a(\eta)\right)$p(y;η)=b(y)exp(η(T)T(y)−a(η))，可知：

所以，伯努利分布的指数族分布对应参数为：

$\begin{aligned} b(y)&=1 \\ \eta&=\ln\left(frac{\phi}{1-\phi}\right) \\ T(y)&=y \\ a(\eta)&=-\ln(1-\phi)=ln(1+exp{\eta}) \end{aligned}$b(y)ηT(y)a(η)​=1=ln(fracϕ1−ϕ)=y=−ln(1−ϕ)=ln(1+expη)​

---

3.2. 广义线性模型的三条假设

1. 在给定$\boldsymbol{x}$x的条件下，假设随机变量$\boldsymbol{y}$y服从某个指数族分布

2. 在给定$\boldsymbol{x}$x的条件下，我们的目标是得到一个模型$h(\boldsymbol{x})$h(x)能预测出$T(\boldsymbol{y})$T(y)的期望值

3. 假设该指数族分布中的自然参数$\eta$η和$\boldsymbol{x}$x呈线性关系，即$\eta=w^{T}x$η=wTx

---

---

4. 对数几率回归

对数几率回归是在对一个二分类问题进行建模，并且假设被建模的随机变量$y$y取值为0或1，因此我们可以很自然地假设$y$y服从伯努利分布。此时，如果我们想要构建一个线性模型来预测在给定$\boldsymbol{x}$x的条件下$y$y的取值的话，可以考虑使用广义线性模型来进行建模。

4.1. 对数几率回归的广义线性模型推导

已知$y$y是服从伯努利分布，而伯努利分布属于指数在发布，所以满足广义线性模型的第一条假设，接着根据广义线性模型的第二条假设我们可以推得模型$h(x)$h(x)的表达式为：

$h(\boldsymbol{x})=E[T(y|\boldsymbol{x})]$h(x)=E[T(y∣x)]

由于伯努利分布的$T(y|\boldsymbol{x})=y|\boldsymbol{x}$T(y∣x)=y∣x，所以：

$h(\boldsymbol{x})=E[y|\boldsymbol{x}]$h(x)=E[y∣x]

又因为$E[y|\boldsymbol{x}]=1\times p(y=1|\boldsymbol{x})+0\times p(y=0|\boldsymbol{x})=p(y=1|\boldsymbol{x})=\phi$E[y∣x]=1×p(y=1∣x)+0×p(y=0∣x)=p(y=1∣x)=ϕ，所以：

$h(\boldsymbol{x})=\phi$h(x)=ϕ

在前面证明伯努利分布属于指数族分布时我们知道：

$\begin{aligned} &\eta=\ln \left(\frac{\phi}{1-\phi}\right) \\ &e^{\eta}=\frac{\phi}{1-\phi} \\ &e^{-\eta}=\frac{1-\phi}{\phi} \\ &e^{-\eta}=\frac{1}{\phi}-1 \\ &1+e^{-\eta}=\frac{1}{\phi} \\ &\frac{1}{1+e^{-\eta}}=\phi &\end{aligned}$​η=ln(1−ϕϕ​)eη=1−ϕϕ​e−η=ϕ1−ϕ​e−η=ϕ1​−11+e−η=ϕ1​1+e−η1​=ϕ​​

将$\phi=\frac{1}{1+e^{-\eta}}$ϕ=1+e−η1​代入$h(\boldsymbol{x})$h(x)的表达式可得：

$h(\boldsymbol{x})=\phi=\frac{1}{1+e^{-\eta}}$h(x)=ϕ=1+e−η1​

根据广义模型的第三条假设：$\eta=w^{T}x$η=wTx，$h(\boldsymbol{x})$h(x)最终可化为：

$h(\boldsymbol{x})=\phi=\frac{1}{1+e^{-w^{T}x}}=p(y=1|\boldsymbol{x}) \tag{西瓜书式3.23}$h(x)=ϕ=1+e−wTx1​=p(y=1∣x)(西瓜书式3.23)

此即为对数几率回归模型。

---

4.2. 极大似然估计法

设总体的概率密度函数（或分布律）为$f(y, w_{1}, w_{2}, \cdots, w_{k})$f(y,w1​,w2​,⋯,wk​)，$y_{1}$y1​，$y_{2}$y2​，…，$y_{m}$ym​，为从该总体中抽出的样本。因为$y_{1}$y1​，$y_{2}$y2​，…，$y_{m}$ym​相互独立且同分布，于是，它们的联合概率密度函数（或联合概率）为：

$L\left(y_{1}, y_{2}, \ldots, y_{m} ; w_{1}, w_{2}, \ldots, w_{k}\right)=\prod_{i=1}^{m} f\left(y_{i}, w_{1}, w_{2}, \ldots, w_{k}\right)$L(y1​,y2​,…,ym​;w1​,w2​,…,wk​)=i=1∏m​f(yi​,w1​,w2​,…,wk​)

其中，$w_{1}$w1​，$w_{2}$w2​，…，$w_{m}$wm​被看作固定但是未知的参数。当我们已经观测到一组样本观测值$y_{1}$y1​，$y_{2}$y2​，…，$y_{m}$ym​时，要去估计未知参数，一种直观的想法就是，哪一组参数使得现在的样本观测值出现的概率最大，哪一组参数可能就是真正的参数，我们就用它作为参数的估计值，这就是所谓的极大似然估计。

极大似然估计的具体方法：

通常记$L\left(y_{1}, y_{2}, \ldots, y_{m} ; w_{1}, w_{2}, \ldots, w_{k}\right)=L\left(w\right)$L(y1​,y2​,…,ym​;w1​,w2​,…,wk​)=L(w)，并称为其似然函数。于是求$w$w的极大似然估计就归结为$L(w)$L(w)的最大值点。由于对数函数是单调递增函数，所以：

$\begin{aligned} \ln L(\boldsymbol{w}) &=\ln \left(\prod_{i=1}^{m} f\left(y_{i}, w_{1}, w_{2}, \ldots, w_{k}\right)\right) \\ &=\sum_{i=1}^{m} \ln f\left(y_{i}, w_{1}, w_{2}, \ldots, w_{k}\right) \end{aligned}$lnL(w)​=ln(i=1∏m​f(yi​,w1​,w2​,…,wk​))=i=1∑m​lnf(yi​,w1​,w2​,…,wk​)​

与$L(w)$L(w)有相同的最大值点，而在许多情况下，求$\ln L(w)$lnL(w)的最大值点比较简单，于是，我们就将求$L(w)$L(w)的最大值点转化为了求$\ln L(w)$lnL(w)的最大值点，通常称$\ln L(w)$lnL(w)为对数似然函数。

对数几率回归的极大似然估计：

已知随机变量$y$y取1和0的概率分别为：

$\begin{aligned} &p(y=1 | \boldsymbol{x})=\frac{e^{\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}+b}}{1+e^{\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}+b}} \\ &p(y=0 | \boldsymbol{x})=\frac{1}{1+e^{\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}+b}} \end{aligned}$​p(y=1∣x)=1+ewTx+bewTx+b​p(y=0∣x)=1+ewTx+b1​​

令$\boldsymbol{\beta}=(w;b)$β=(w;b)，$\hat{\boldsymbol{x}}=(\boldsymbol{x}; 1)$x^=(x;1)，则$w^{T}\boldsymbol{x}+b$wTx+b可简化为$\boldsymbol{\beta}^{T}\hat{x}$βTx^，于是上式可化简为：

$\begin{aligned} &p(y=1 | \boldsymbol{x})=\frac{e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}} \\ &p(y=0 | \boldsymbol{x})=\frac{1}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}} \end{aligned}$​p(y=1∣x)=1+eβTx^eβTx^​p(y=0∣x)=1+eβTx^1​​

记：

$\begin{aligned} &p(y=1 | \boldsymbol{x})=\frac{e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}}=p_{1}(\hat{\boldsymbol{x}};\boldsymbol{\beta}) \\ &p(y=0 | \boldsymbol{x})=\frac{1}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}}=p_{0}(\hat{\boldsymbol{x}};\boldsymbol{\beta}) \end{aligned}$​p(y=1∣x)=1+eβTx^eβTx^​=p1​(x^;β)p(y=0∣x)=1+eβTx^1​=p0​(x^;β)​

于是，使用一个小技巧即可得到随机变量$y$y的分布律表达式：

$p(y | \boldsymbol{x} ; \boldsymbol{w}, b) =y \cdot p_{1}(\hat{\boldsymbol{x}} ; \boldsymbol{\beta})+(1-y) \cdot p_{0}(\hat{\boldsymbol{x}} ; \boldsymbol{\beta}) \tag{西瓜书式3.26}$p(y∣x;w,b)=y⋅p1​(x^;β)+(1−y)⋅p0​(x^;β)(西瓜书式3.26)

或者：

$p(y | \boldsymbol{x} ; \boldsymbol{w}, b) =\left[p_{1}(\hat{\boldsymbol{x}} ; \boldsymbol{\beta})\right]^{y} \left[p_{0}(\hat{\boldsymbol{x}} ; \boldsymbol{\beta})\right]^{1-y}$p(y∣x;w,b)=[p1​(x^;β)]y[p0​(x^;β)]1−y

---

4.3. 对数几率回归的参数估计

根据对数似然函数的定义可知：

$\ln L(\boldsymbol{w}) =\sum_{i=1}^{m} \ln f\left(y_{i}, w_{1}, w_{2}, \ldots, w_{k}\right)$lnL(w)=i=1∑m​lnf(yi​,w1​,w2​,…,wk​)

由于此时的$y$y为离散型，所以将对数似然函数中的概率密度函数换成分布律即可，既有：

$\ell(w,b) :=\ln L(\boldsymbol{w},b) =\sum_{i=1}^{m} \ln f\left(y_{i} | x_{i}; \boldsymbol{w},b\right) \tag{西瓜书式3.25}$ℓ(w,b):=lnL(w,b)=i=1∑m​lnf(yi​∣xi​;w,b)(西瓜书式3.25)

将$p(y | \boldsymbol{x} ; \boldsymbol{w}, b)=y \cdot p_{1}(\hat{\boldsymbol{x}} ; \boldsymbol{\beta})+(1-y) \cdot p_{0}(\hat{\boldsymbol{x}} ; \boldsymbol{\beta})$p(y∣x;w,b)=y⋅p1​(x^;β)+(1−y)⋅p0​(x^;β)代入对数似然函数可得：

$\begin{aligned} \ell(\boldsymbol{\beta}) &=\sum_{i=1}^{m} \ln \left(y_{i} p_{1}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)+\left(1-y_{i}\right) p_{0}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right) \\ &\qquad p_{1}(\hat{\boldsymbol{x}};\boldsymbol{\beta}) = \frac{e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}} \\ &\qquad p_{0}(\hat{\boldsymbol{x}};\boldsymbol{\beta}) = \frac{1}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}} \\ &=\sum_{i=1}^{m} \ln \left(\frac{y_{i} e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}}+\frac{1-y_{i}}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}}\right) \\ &=\sum_{i=1}^{m} \ln \left(\frac{y_{i} e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}+1-y_{i}}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}}\right) \\ &=\sum_{i=1}^{m}\left(\ln \left(y_{i} e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}+1-y_{i}\right)-\ln \left(1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}\right)\right) \\ &\qquad y_{i}\in \{0,1\} \\ &\qquad y_{i}=0 \\ &\qquad \quad \ell(\boldsymbol{\beta})=\sum_{i=1}^{m}\left(\ln \left(0 \cdot e^{\boldsymbol{\beta}^{T} \hat{x}_{i}}+1-0\right)-\ln \left(1+e^{\boldsymbol{\beta}^{T} \hat{x}_{i}}\right)\right)=\sum_{i=1}^{m}\left(\ln 1-\ln \left(1+e^{\boldsymbol{\beta}^{T} \hat{x}_{i}}\right)\right)=\sum_{i=1}^{m}\left(-\ln \left(1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}\right)\right) \\ &\qquad y_{i}=1 \\ &\qquad \quad \ell(\boldsymbol{\beta})=\sum_{i=1}^{m}\left(\ln \left(1 \cdot e^{\boldsymbol{\beta}^{T} \hat{x}_{i}}+1-1\right)-\ln \left(1+e^{\boldsymbol{\beta}^{T} \hat{x}_{i}}\right)\right)=\sum_{i=1}^{m}\left(\ln e^{\boldsymbol{r}_{i}^{T}}-\ln \left(1+e^{\boldsymbol{\beta}^{T} \boldsymbol{z}_{i}}\right)\right)=\sum_{i=1}^{m}\left(\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}-\ln \left(1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}\right)\right) \\ &=\sum_{i=1}^{m}\left(y_{i} \boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}-\ln \left(1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}\right)\right) \end{aligned} \tag{西瓜书式3.27}$ℓ(β)​=i=1∑m​ln(yi​p1​(x^i​;β)+(1−yi​)p0​(x^i​;β))p1​(x^;β)=1+eβTx^eβTx^​p0​(x^;β)=1+eβTx^1​=i=1∑m​ln(1+eβTx^i​yi​eβTx^i​​+1+eβTx^i​1−yi​​)=i=1∑m​ln(1+eβTx^i​yi​eβTx^i​+1−yi​​)=i=1∑m​(ln(yi​eβTx^i​+1−yi​)−ln(1+eβTx^i​))yi​∈{0,1}yi​=0ℓ(β)=i=1∑m​(ln(0⋅eβTx^i​+1−0)−ln(1+eβTx^i​))=i=1∑m​(ln1−ln(1+eβTx^i​))=i=1∑m​(−ln(1+eβTx^i​))yi​=1ℓ(β)=i=1∑m​(ln(1⋅eβTx^i​+1−1)−ln(1+eβTx^i​))=i=1∑m​(lneriT​−ln(1+eβTzi​))=i=1∑m​(βTx^i​−ln(1+eβTx^i​))=i=1∑m​(yi​βTx^i​−ln(1+eβTx^i​))​(西瓜书式3.27)

若$p(y | \boldsymbol{x} ; \boldsymbol{w}, b)=\left[p_{1}(\hat{\boldsymbol{x}} ; \boldsymbol{\beta})\right]^{y}\left[p_{0}(\hat{\boldsymbol{x}} ; \boldsymbol{\beta})\right]^{1-y}$p(y∣x;w,b)=[p1​(x^;β)]y[p0​(x^;β)]1−y，将其代入对数似然函数可得：

$\begin{aligned} \ell(\boldsymbol{\beta}) &=\sum_{i=1}^{m} \ln \left(\left[p_{1}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right]^{y_{i}}\left[p_{0}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right]^{1-y_{i}}\right) \\ &=\sum_{i=1}^{m}\left[\ln \left(\left[p_{1}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right]^{y_{i}}\right)+\ln \left(\left[p_{0}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right]^{1-y_{i}}\right)\right] \\ &=\sum_{i=1}^{m}\left[y_{i} \ln \left(p_{1}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right)+\left(1-y_{i}\right) \ln \left(p_{0}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right)\right] \\ &=\sum_{i=1}^{m}\left\{y_{i}\left[\ln \left(p_{1}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right)-\ln \left(p_{0}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right)\right]+\ln \left(p_{0}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right)\right\} \\ &=\sum_{i=1}^{m}\left[y_{i}\ln \frac{p_{1}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)}{p_{0}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)}+\ln \left(p_{0}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right)\right] \\ &\qquad p_{1}(\hat{\boldsymbol{x}};\boldsymbol{\beta}) = \frac{e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}} \\ &\qquad p_{0}(\hat{\boldsymbol{x}};\boldsymbol{\beta}) = \frac{1}{1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}} \\ &=\sum_{i=1}^{m}\left[y_{i}\ln \left(e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}}\right) + \ln \left(p_{0}\left(\hat{\boldsymbol{x}}_{i} ; \boldsymbol{\beta}\right)\right)\right] \\ &=\sum_{i=1}^{m}\left(y_{i} \boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}-\ln \left(1+e^{\boldsymbol{\beta}^{T} \hat{\boldsymbol{x}}_{i}}\right)\right) \end{aligned}$ℓ(β)​=i=1∑m​ln([p1​(x^i​;β)]yi​[p0​(x^i​;β)]1−yi​)=i=1∑m​[ln([p1​(x^i​;β)]yi​)+ln([p0​(x^i​;β)]1−yi​)]=i=1∑m​[yi​ln(p1​(x^i​;β))+(1−yi​)ln(p0​(x^i​;β))]=i=1∑m​{yi​[ln(p1​(x^i​;β))−ln(p0​(x^i​;β))]+ln(p0​(x^i​;β))}=i=1∑m​[yi​lnp0​(x^i​;β)p1​(x^i​;β)​+ln(p0​(x^i​;β))]p1​(x^;β)=1+eβTx^eβTx^​p0​(x^;β)=1+eβTx^1​=i=1∑m​[yi​ln(eβTx^)+ln(p0​(x^i​;β))]=i=1∑m​(yi​βTx^i​−ln(1+eβTx^i​))​