机器学习（西瓜书）之一元线性回归公式推导

一元线性回归公式推导

1.求解偏置b的公式推导思路：由最小二乘法导出损失函数$E( \omega, b )$E(ω,b)—>证明损失函数$E( \omega, b )$E(ω,b) 是关于$w$w和$b$b的凸函数—>对损失函数$E( \omega, b)$E(ω,b)关于$b$b求一阶偏导数—>令一阶偏导数等于$0$0解出$b$b

由最小二乘法导出损失函数$E( \omega, b)$E(ω,b):

$E_{(w,b)}=\sum^{m}_{i = 1}(y_i-f(x_i))^2=\sum^{m}_{i = 1}(y_i-(\omega x_i+b))^2=\sum^{m}_{i = 1}(y_i-\omega x_i-b)^2 \tag{$ p_{54}.3.4$}$E(w,b)​=i=1∑m​(yi​−f(xi​))2=i=1∑m​(yi​−(ωxi​+b))2=i=1∑m​(yi​−ωxi​−b)2(p54​.3.4)

证明损失函数$E(\omega, b)$E(ω,b)是关于$\omega$ω和$b$b的凸函数—— 求:$A=f_{xx}^{ 〞}(x,y)$A=fxx〞​(x,y)

$\frac{\partial E_{(w,b)}}{\partial \omega} =\frac{\partial }{\partial \omega} [\sum^{m}_{i = 1}(y_i-\omega x_i-b)^2] =\sum^{m}_{i = 1}\frac{\partial }{\partial \omega} (y_i-\omega x_i-b)^2 =\sum^{m}_{i = 1}2·(y_i-\omega x_i-b)·（-x_i） =2(\omega\sum^{m}_{i = 1}x_i ^2-\sum^{m}_{i = 1}(y_i-b)x_i)$∂ω∂E(w,b)​​=∂ω∂​[i=1∑m​(yi​−ωxi​−b)2]=i=1∑m​∂ω∂​(yi​−ωxi​−b)2=i=1∑m​2⋅(yi​−ωxi​−b)⋅（−xi​）=2(ωi=1∑m​xi2​−i=1∑m​(yi​−b)xi​)上式即为即为（3.5)

$A=f_{xx}^{ 〞}(x,y)= \frac{\partial^2E_{(\omega,b)}}{\partial \omega^2} =\frac{\partial }{\partial \omega} (\sum^{m}_{i = 1}\frac{\partial E_{(w,b)}}{\partial \omega} ) =\frac{\partial }{\partial \omega}[2(\omega\sum^{m}_{i = 1}x_i ^2-\sum^{m}_{i = 1}(y_i-b)x_i)] =\frac{\partial }{\partial \omega}[2\omega\sum^{m}_{i = 1}x_i ^2] =2\sum^{m}_{i = 1}x_i ^2$A=fxx〞​(x,y)=∂ω2∂2E(ω,b)​​=∂ω∂​(i=1∑m​∂ω∂E(w,b)​​)=∂ω∂​[2(ωi=1∑m​xi2​−i=1∑m​(yi​−b)xi​)]=∂ω∂​[2ωi=1∑m​xi2​]=2i=1∑m​xi2​

求:$B=f_{xy}^{ 〞}(x,y)$B=fxy〞​(x,y)

$B=f_{xy}^{ 〞}(x,y)= \frac{\partial^2E_{(\omega,b)}}{\partial \omega\partial b} =\frac{\partial }{\partial b} ( \frac{\partial E_{(w,b)}}{\partial \omega} ) =\frac{\partial }{\partial b} [2(\omega\sum^{m}_{i = 1}x_i ^2-\sum^{m}_{i = 1}(y_i-b)x_i)] =\frac{\partial }{\partial b}[-2\sum^{m}_{i = 1}(y_i-b)x_i)] =2\sum^{m}_{i = 1}x_i$B=fxy〞​(x,y)=∂ω∂b∂2E(ω,b)​​=∂b∂​(∂ω∂E(w,b)​​)=∂b∂​[2(ωi=1∑m​xi2​−i=1∑m​(yi​−b)xi​)]=∂b∂​[−2i=1∑m​(yi​−b)xi​)]=2i=1∑m​xi​

求:$C=f_{yy}^{ 〞}(x,y)$C=fyy〞​(x,y)

$\frac{\partial E_{(w,b)}}{\partial b} = \frac{\partial }{\partial b}[\sum^{m}_{i = 1} (y_i-\omega x_i-b)^2] =\sum^{m}_{i = 1}\frac{\partial }{\partial b} (y_i-\omega x_i-b)^2 =\sum^{m}_{i = 1}2·(y_i-\omega x_i-b)·（-1） =2(mb-\sum^{m}_{i = 1}(y_i-\omega x_i))$∂b∂E(w,b)​​=∂b∂​[i=1∑m​(yi​−ωxi​−b)2]=i=1∑m​∂b∂​(yi​−ωxi​−b)2=i=1∑m​2⋅(yi​−ωxi​−b)⋅（−1）=2(mb−i=1∑m​(yi​−ωxi​))上式即为即为（3.6)

所以

也即损失函数$E(\omega, b)$E(ω,b) 是关于$\omega$ω和$b$b的凸函数得证

对损失函数$E(\omega, b)$E(ω,b) 关于b求一阶偏导数：

令一阶偏导数等于 0 解出b:

2.求解权重$\omega,$ω,的公式推导思路:对损失函数$E( \omega, b)$E(ω,b)关于$\omega$ω求一阶偏导数—>令一阶偏导数等于$0$0解出$\omega$ω

令一阶偏导数等于 0 解出$\omega$ω：

3.将$\omega$ω向量化:

数学知识—二元函数判断凹凸性: