机器学习反向传播梯度求导

机器学习反向传播梯度推导

在我的前一篇文章中，已经推导出了单层感知机梯度的计算公式
多层感知机梯度推导

$\frac {\varphi_E} {\varphi_{W_{j_k}}} = (0_k - t_k)0_k(1 - 0_k) W_j^0$φWjk​​​φE​​=(0k​−tk​)0k​(1−0k​)Wj0​
1. 链式法则：
$\frac {\varphi f(x)} {\varphi g(x)} = \frac {\varphi f(x)} {\varphi h(x)} \frac {\varphi h(x)} {\varphi g(x)}$φg(x)φf(x)​=φh(x)φf(x)​φg(x)φh(x)​

$\frac {\varphi_E} {\varphi_{W_{j_k}^1}} = \frac {\varphi_E} {\varphi_{W_{j_k}^2}} \frac {\varphi_{W_{j_k}^2}} {\varphi_{W_{j_k}^1}}$φWjk​1​​φE​​=φWjk​2​​φE​​φWjk​1​​φWjk​2​​​

2. bpnn推导：

$注: \Sigma, \sigma为**函数，同时O_j^J = \sigma(x_j^J)$注:Σ,σ为激活函数，同时OjJ​=σ(xjJ​)

$所以：\frac {\varphi_E} {\varphi_{W_{j_k}^K}} = (O_k^K - t_k)O_k^K(1 - O_k^K) O_j^J$所以：φWjk​K​​φE​​=(OkK​−tk​)OkK​(1−OkK​)OjJ​
$\frac {\varphi_E} {\varphi_{W_{j_k}^K}} = (O_k^K - t_k)\delta_k O_j^J$φWjk​K​​φE​​=(OkK​−tk​)δk​OjJ​
$那么现在的关键就是求出\frac {\varphi_E} {\varphi_{W_{i_j}^J}}以及找出下一层\\权值梯度与上一层权值梯度的关系,依次迭代$那么现在的关键就是求出φWij​J​​φE​​以及找出下一层权值梯度与上一层权值梯度的关系,依次迭代
$\frac {\varphi_E} {\varphi_{W_{i_j}^J}} = \frac {\varphi{{\frac 1 2}\Sigma_{i=0}^m(O_k^K - t_k)^2}} {\varphi_{W_{i_j}^J}}$φWij​J​​φE​​=φWij​J​​φ21​Σi=0m​(OkK​−tk​)2​
$对 {W_{i_j}^J} 的导数有影响的只有O_k^K,所以：$对Wij​J​的导数有影响的只有OkK​,所以：
$\frac {\varphi_E} {\varphi_{W_{i_j}^J}} = \frac {\varphi{{\frac 1 2}(O_k^K - t_k)^2}} {\varphi_{W_{i_j}^J}}$φWij​J​​φE​​=φWij​J​​φ21​(OkK​−tk​)2​
$\frac {\varphi_E} {\varphi_{W_{i_j}^J}} = (O_k^K - t_k){\frac {\varphi{O_k^K }} {\varphi_{W_{i_j}^J}}}$φWij​J​​φE​​=(OkK​−tk​)φWij​J​​φOkK​​
$\frac {\varphi_E} {\varphi_{W_{i_j}^J}} = (O_k^K - t_k) \frac {\varphi{\sigma(x_k^K)}} {\varphi_{W_{i_j}^J}}$φWij​J​​φE​​=(OkK​−tk​)φWij​J​​φσ(xkK​)​
$\frac {\varphi_E} {\varphi_{W_{i_j}^J}} = (O_k^K - t_k)O_k^K(1 - O_k^K) \frac {\varphi_{x_k^K}} {\varphi_{W_{i_j}^J}}$φWij​J​​φE​​=(OkK​−tk​)OkK​(1−OkK​)φWij​J​​φxkK​​​
$使用链式法则： \frac {\varphi_E} {\varphi_{W_{i_j}^J}} = (O_k^K - t_k)O_k^K(1 - O_k^K) \frac {\varphi_{x_k^K}} {\varphi_{O_j^J}} \frac {\varphi_{O_j^J}} {\varphi_{W_{i_j}^J}}$使用链式法则：φWij​J​​φE​​=(OkK​−tk​)OkK​(1−OkK​)φOjJ​​φxkK​​​φWij​J​​φOjJ​​​
$\frac {\varphi_E} {\varphi_{W_{i_j}^J}} = (O_k^K - t_k)\delta_k^K W_{j_k}\frac {\varphi_{O_j^J}} {\varphi_{W_{i_j}^J}}$φWij​J​​φE​​=(OkK​−tk​)δkK​Wjk​​φWij​J​​φOjJ​​​
$\frac {\varphi_E} {\varphi_{W_{i_j}^J}} = (O_k^K - t_k) \delta_k^K W_{j_k} \frac {\varphi_{\sigma(x_j^J)}} {\varphi_{W_{i_j}^J}}$φWij​J​​φE​​=(OkK​−tk​)δkK​Wjk​​φWij​J​​φσ(xjJ​)​​
$类似于上一层推导：\\ \frac {\varphi_E} {\varphi_{W_{i_j}^J}} =(O_k^K - t_k) \delta_k^K W_{j_k} \delta_j^J x_j^0$类似于上一层推导：φWij​J​​φE​​=(OkK​−tk​)δkK​Wjk​​δjJ​xj0​
神经网络计算过程：

1. 通过前向传播计算出训练结果
2. 将训练结果通过反向传播作用于梯度下降