神经网络

Lei_ZM
2019-10-07

---

---

1. 感知机

1.1. 定义

假设输入空间是$\mathcal{X}\subseteq R^{n}$X⊆Rn，输出空间是$\mathcal{y}=\{1，0\}$y={1，0}。输入$\boldsymbol{x}\in\mathcal{X}$x∈X七表示实例的特征向量，对应于输入空间的点；输出$y\in\mathcal{Y}$y∈Y表示实例的类别。由输入空间到输出空间的如下函数：

$f(\boldsymbol{x})=\operatorname{sgn}\left(\boldsymbol{w}^{T} \boldsymbol{x}+b\right)$f(x)=sgn(wTx+b)

称为感知机参数，其中$\boldsymbol{w}$w和$b$b为感知机模型参数，$\operatorname{sgn}$sgn为阶跃函数，即：

$\operatorname{sgn}(z)=\left\{ \begin{array}{ll} {1,} & {z \geqslant 0} \\ {0,} & {z<0}\end{array} \right.$sgn(z)={1,0,​z⩾0z<0​

---

1.2. 感知机的几何解释

线性方程$\boldsymbol{w}^{T} \boldsymbol{x}+b=0$wTx+b=0对应于特征空间（输入空间）$R^{n}$Rn中的一个超平面$S$S，其中$\boldsymbol{w}$w是超平面的法向量，$b$b是超平面的截距。这个超平面将特征空间划分为两个部分。位于两边的点（特征向量）分别被分为正、负两类。因此，超平面$S$S称为分离超平面，如图所示：

---

1.3. 学习策略

假设训练数据集是线性可分的，感知机学习的目标是求得一个能够将训练集正实例点和负实例点完全正确分开的超平面。为了找出这样的超平面$\mathrm{S}$S，即确定感知机模型参数$\boldsymbol{w}$w和$b$b，需要确定一个学习策略，即定义损失函数并将损失函数极小化。损失函数的一个自然选择是误分类点的总数。但是，这样的损失函数不是参数$\boldsymbol{w}$w和$b$b的连续可导函数，不易优化，所以感知机采用的损失函数为误分类点到超平面的总距离。

输入空间$R^{n}$Rn中点$\boldsymbol{x_{0}}$x0​到超平面$\mathrm{S}$S的距离公式为：

$\frac{\left|\boldsymbol{w}^{T} \boldsymbol{x}_{0}+b\right|}{\|\boldsymbol{w}\|}$∥w∥∣∣​wTx0​+b∣∣​​

其中，$\|\boldsymbol{w}\|$∥w∥表示向量$\boldsymbol{w}$w的$L_{2}$L2​范数，也就是模长。

若将$b$b看成哑结点，即将其合进至$\boldsymbol{w}$w可得：

$\frac{\left|\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{0}\right|}{\|\hat{\boldsymbol{w}}\|}$∥w^∥∣∣∣​w^Tx^0​∣∣∣​​

设误分类点集合为$M$M，那么所有误分类点到超平面$\mathrm{S}$S的总距离为：

$\sum_{\hat{x}_{i}\in M} \frac{\left|\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{i}\right|}{\|\hat{\boldsymbol{w}}\|}$x^i​∈M∑​∥w^∥∣∣∣​w^Tx^i​∣∣∣​​

又因为，对于任意误分类点$\hat{x}_{i}\in M$x^i​∈M来说， 都有：

$\left(\hat{y}_{i}-y_{i}\right) \hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{i} > 0$(y^​i​−yi​)w^Tx^i​>0

其中，$\hat{y}_{i}$y^​i​为当期感知机的输出。于是所有误分类点到超平面$\mathrm{S}$S的总距离为：

$\sum_{\hat{\boldsymbol{x}}_{i} \in M} \frac{\left(\hat{y}_{i}-y_{i}\right) \hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{i}}{\|\hat{\boldsymbol{w}}\|}$x^i​∈M∑​∥w^∥(y^​i​−yi​)w^Tx^i​​

由于训练完成后无误分类点，即损失函数值为0，与分母$\|\hat{\boldsymbol{w}}\|$∥w^∥无关，故可舍去，即不考虑$\frac{1}{\|\hat{\boldsymbol{w}}\|}$∥w^∥1​，此时得到感知机的函数为：

$L\left(\hat{\boldsymbol{w}}\right)= \sum_{\hat{\boldsymbol{x}}_{i} \in M} \left(\hat{y}_{i}-y_{i}\right) \hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{i}$L(w^)=x^i​∈M∑​(y^​i​−yi​)w^Tx^i​

显然，损失函数$L\left(\hat{\boldsymbol{w}}\right)$L(w^)是非负的。如果没有误分类点，损失函数值为0。而且误分类点越少，误分类点离超平面越近，损失函数值就越小，在误分类时是参数心的线性函数，在正确分类时是0。因此，给定训练数据集，损失函数$L\left(\hat{\boldsymbol{w}}\right)$L(w^)是$\hat{\boldsymbol{w}}$w^的连续可导函数。

---

1.4. 算法

感知机学习算法是对以下最优化问题的算法，给定训练数据集：

$T=\left\{\left(\hat{\boldsymbol{x}}_{1}, y_{1}\right),\left(\hat{\boldsymbol{x}}_{2}, y_{2}\right), \cdots,\left(\hat{\boldsymbol{x}}_{N}, y_{N}\right)\right\}$T={(x^1​,y1​),(x^2​,y2​),⋯,(x^N​,yN​)}

其中，$\hat{x}_{i}\in R^{n+1}$x^i​∈Rn+1，$y_{i}\in \{0, 1\}$yi​∈{0,1}，求参数$\hat{\boldsymbol{w}}$w^使其为以下损失函数极小化问题的解。

$L\left(\hat{\boldsymbol{w}}\right)= \sum_{\hat{\boldsymbol{x}}_{i} \in M} \left(\hat{y}_{i}-y_{i}\right) \hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{i}$L(w^)=x^i​∈M∑​(y^​i​−yi​)w^Tx^i​

其中，$M$M为误分类点的集合。

感知机学习算法是误分类驱动的，具体采用随机梯度下降法。首先，任意选取一个超平面$\hat{\boldsymbol{w}}^{T}_{0} \hat{\boldsymbol{x}}=0$w^0T​x^=0用梯度下降法不断地极小化损失函数$L\left(\hat{\boldsymbol{w}}\right)$L(w^)，极小化过程中不是一次使$M$M中所有误分类点的梯度下降，而是一次随机选取一个误分类点使其梯度下降。已知损失函数的梯度为：

$\begin{aligned} \nabla L(\hat{\boldsymbol{w}}) =\frac{\partial L(\hat{\boldsymbol{w}})}{\partial \hat{\boldsymbol{w}}} &=\frac{\partial}{\partial \hat{\boldsymbol{w}}}\left[\sum_{\hat{x}_{i} \in M}\left(\hat{y}_{i}-y_{i}\right) \hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{i}\right] \\ &=\sum_{\hat{x}_{i} \in M}\left[\left(\hat{y}_{i}-y_{i}\right) \frac{\partial}{\partial \hat{\boldsymbol{w}}}\left(\hat{\boldsymbol{w}}^{T} \hat{\boldsymbol{x}}_{i} \right)\right] \\ &\qquad \text{矩阵微分公式} \frac{\partial \boldsymbol{x}^{T} \boldsymbol{a}}{\partial \boldsymbol{x}}=\boldsymbol{a} \\ &=\sum_{\hat{\boldsymbol{x}}_{i} \in M}\left(\hat{y}_{i}-y_{i}\right) \hat{\boldsymbol{x}}_{i} \end{aligned}$∇L(w^)=∂w^∂L(w^)​​=∂w^∂​[x^i​∈M∑​(y^​i​−yi​)w^Tx^i​]=x^i​∈M∑​[(y^​i​−yi​)∂w^∂​(w^Tx^i​)]矩阵微分公式∂x∂xTa​=a=x^i​∈M∑​(y^​i​−yi​)x^i​​

那么随机选取一个误分类点$\hat{x}_{i}$x^i​进行梯度下降，可得参数$\hat{\boldsymbol{w}}$w^的更新公式：

$\begin{aligned} \hat{\boldsymbol{w}} &\leftarrow \hat{\boldsymbol{w}}+\Delta\hat{\boldsymbol{w}} \\ &\qquad \Delta\hat{\boldsymbol{w}}=-\eta \nabla L(\hat{\boldsymbol{w}}) \\ &\leftarrow \hat{\boldsymbol{w}}-\eta \nabla L(\hat{\boldsymbol{w}}) \\ &\qquad \text{选取一个误分类点} \\ &\qquad \nabla L(\hat{\boldsymbol{w}}) = \left(\hat{y}_{i}-y_{i}\right) \hat{\boldsymbol{x}}_{i} \\ &\leftarrow \hat{\boldsymbol{w}}-\eta\left(\hat{y}_{i}-y_{i}\right) \hat{\boldsymbol{x}}_{i}=\hat{\boldsymbol{w}}+\eta\left(y_{i}-\hat{y}_{i}\right) \hat{\boldsymbol{x}}_{i} \end{aligned}$w^​←w^+Δw^Δw^=−η∇L(w^)←w^−η∇L(w^)选取一个误分类点∇L(w^)=(y^​i​−yi​)x^i​←w^−η(y^​i​−yi​)x^i​=w^+η(yi​−y^​i​)x^i​​

即有：

$\hat{\boldsymbol{w}}\leftarrow \hat{\boldsymbol{w}}+\Delta\hat{\boldsymbol{w}}=\hat{\boldsymbol{w}}+\eta\left(y_{i}-\hat{y}_{i}\right)$w^←w^+Δw^=w^+η(yi​−y^​i​)
$\Rightarrow \Delta\hat{\boldsymbol{w}} = \eta\left(y_{i}-\hat{y}_{i}\right) \tag{西瓜书式5.2}$⇒Δw^=η(yi​−y^​i​)(西瓜书式5.2)

---

---

2. 神经网络

2.1. 模型结构

单隐层前馈网络模型结构如下：

其中，

• $D=\{(\boldsymbol{x}_{1}, \boldsymbol{y}_{1}), (\boldsymbol{x}_{1}, \boldsymbol{y}_{1}), \cdots, (\boldsymbol{x}_{1}, \boldsymbol{y}_{1})\}$D={(x1​,y1​),(x1​,y1​),⋯,(x1​,y1​)}，$\boldsymbol{x}_{i}\in \mathbb{R}^{d}$xi​∈Rd，$\boldsymbol{y}_{i}\in \mathbb{R}^{l}$yi​∈Rl：训练集

• $d$d：神经元输入个数，输入示例属性描述的个数

• $l$l：输出神经元个数，输出的实值向量维数

• $q$q：隐层神经元的个数

• $\theta_{j}$θj​：输出层第$j$j个神经元的阈值

• $\gamma_{h}$γh​：隐层第$h$h个神经元的阈值

• $v_{ih}$vih​：输入层第$i$i个神经元与隐层第$h$h个神经元之间的连接权重

• $w_{hj}$whj​：隐层第$h$h个神经元与输出层第$j$j个神经元之间的连接权重

• $\alpha_{h}=\sum_{i=1}^{d} v_{ih} x_{i}$αh​=∑i=1d​vih​xi​：隐层第$h$h个神经元接收到的输入

• $\beta_{j}=\sum_{h=1}^{q} w_{hj} b_{h}$βj​=∑h=1q​whj​bh​：输出层第$j$j个神经元接收到的输入

• $b_{h}$bh​：隐层第$h$h个神经元的输出

---

2.2. 标准BP算法

给定一个训练样本$(\boldsymbol{x}_{k}, \boldsymbol{y}_{k})$(xk​,yk​)，假设神经网络模型的输出为$\hat{\boldsymbol{y}}_{k}=(\hat{y}_{1}, \hat{\boldsymbol{y}}_{2}, \cdots, \hat{\boldsymbol{y}}_{l})$y^​k​=(y^​1​,y^​2​,⋯,y^​l​)，即：

$\hat{y}_{j}^{k}=f \left(\beta_{j}-\theta_{j}\right) \tag{西瓜书式5.3}$y^​jk​=f(βj​−θj​)(西瓜书式5.3)

其中 $f$f 为$\mathbf{Sigmoid}$Sigmoid，则网络在一个训练样本$(\boldsymbol{x}_{k}, \boldsymbol{y}_{k})$(xk​,yk​)上的均方误差为：

$E_{k}=\frac{1}{2} \sum_{j=1}^{l}\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2} \tag{西瓜书式5.4}$Ek​=21​j=1∑l​(y^​jk​−yjk​)2(西瓜书式5.4)

如果按照梯度下降法更新模型参数，那么各个参数的更新公式为：

$\begin{aligned} w_{h j} \leftarrow w_{h j}+\Delta w_{h j} &=w_{h j}-\eta \frac{\partial E_{k}}{\partial w_{h j}} \\ \theta_{j} \leftarrow \theta_{j}+\Delta \theta_{j} &=\theta_{j}-\eta \frac{\partial E_{k}}{\partial \theta_{j}} \\ v_{i h} \leftarrow v_{i h}+\Delta v_{i h} &=v_{i h}-\eta \frac{\partial E_{k}}{\partial v_{i h}} \\ \gamma_{h} \leftarrow \gamma_{h}+\Delta \gamma_{h} &=\gamma_{h}-\eta \frac{\partial E_{k}}{\partial \gamma_{h}} \end{aligned}$whj​←whj​+Δwhj​θj​←θj​+Δθj​vih​←vih​+Δvih​γh​←γh​+Δγh​​=whj​−η∂whj​∂Ek​​=θj​−η∂θj​∂Ek​​=vih​−η∂vih​∂Ek​​=γh​−η∂γh​∂Ek​​​

---

2.2.1. 参数$w_{hj}$whj​的更新

已知$E_{k}$Ek​和$w_{hj}$whj​的函数链式关系为：

$\begin{aligned} E_{k}=\frac{1}{2} \sum_{j=1}^{l}&\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2} \\ &\ \downarrow \\ &\begin{aligned} \ \hat{y}_{j}^{k}=f & \left(\beta_{j}-\theta_{j}\right) \\ &\ \downarrow \\ &\ \beta_{j}=\sum_{h=1}^{q} w_{h j} b_{h} \end{aligned} \end{aligned}$Ek​=21​j=1∑l​​(y^​jk​−yjk​)2 ↓ y^​jk​=f​(βj​−θj​) ↓ βj​=h=1∑q​whj​bh​​​

所以：

$\frac{\partial E_{k}}{\partial w_{h j}}=\frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial w_{h j}}$∂whj​∂Ek​​=∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​⋅∂whj​∂βj​​

其中：

$\begin{aligned} &\begin{aligned} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} &=\frac{\partial\left[\frac{1}{2} \sum_{j=1}^{l}\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2}\right]}{\partial \hat{y}_{j}^{k}} \\ &=\frac{1}{2} \times 2 \times \left(\hat{y}_{j}^{k}-y_{j}^{k}\right) \times 1 \\ &=\hat{y}_{j}^{k}-y_{j}^{k} \end{aligned} \\ \\ &\begin{aligned} \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} &=\frac{\partial\left[f\left(\beta_{j}-\theta_{j}\right)\right]}{\partial \beta_{j}} \\ &=f^{\prime}\left(\beta_{j}-\theta_{j}\right) \times 1 \\ &\qquad \text{其中} \ f^{\prime}(x)=f(x)(1-f(x)) \\ &=f\left(\beta_{j}-\theta_{j}\right) \times\left[1-f\left(\beta_{j}-\theta_{j}\right)\right] \\ &\qquad \text{西瓜书式5.3}\ \hat{y}_{j}^{k}=f\left(\beta_{j}-\theta_{j}\right) \\ &=\hat{y}_{j}^{k} \left(1-\hat{y}_{j}^{k}\right) \end{aligned} \\ \\ &\begin{aligned} \frac{\partial \beta_{j}}{\partial w_{h j}} &=\frac{\partial\left(\sum_{h=1}^{q} w_{h j} b_{h}\right)}{\partial w_{h j}} \\ &=b_{h} \end{aligned} \end{aligned}$​∂y^​jk​∂Ek​​​=∂y^​jk​∂[21​∑j=1l​(y^​jk​−yjk​)2]​=21​×2×(y^​jk​−yjk​)×1=y^​jk​−yjk​​∂βj​∂y^​jk​​​=∂βj​∂[f(βj​−θj​)]​=f′(βj​−θj​)×1其中 f′(x)=f(x)(1−f(x))=f(βj​−θj​)×[1−f(βj​−θj​)]西瓜书式5.3 y^​jk​=f(βj​−θj​)=y^​jk​(1−y^​jk​)​∂whj​∂βj​​​=∂whj​∂(∑h=1q​whj​bh​)​=bh​​​

则令$g_j$gj​有：

$\begin{aligned} g_{j} &=-\frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \\ &=-\left(\hat{y}_{j}^{k}-y_{j}^{k}\right) f^{\prime}\left(\beta_{j}-\theta_{j}\right) \\ &=\hat{y}_{j}^{k}\left(1-\hat{y}_{j}^{k}\right)\left(y_{j}^{k}-\hat{y}_{j}^{k}\right) \end{aligned} \tag{西瓜书式5.10}$gj​​=−∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​=−(y^​jk​−yjk​)f′(βj​−θj​)=y^​jk​(1−y^​jk​)(yjk​−y^​jk​)​(西瓜书式5.10)

故有：

$\begin{aligned} \Delta w_{h j} &=-\eta \frac{\partial E_{k}}{\partial w_{h j}} \\ &=-\eta \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial w_{h j}} \\ &=\eta g_{j} b_{h} \end{aligned} \tag{西瓜书式5.11}$Δwhj​​=−η∂whj​∂Ek​​=−η∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​⋅∂whj​∂βj​​=ηgj​bh​​(西瓜书式5.11)

---

2.2.2. 参数$\theta_{j}$θj​的参数更新

已知$E_{k}$Ek​和$\theta_{j}$θj​的函数链式关系为：

$\begin{aligned} E_{k}=\frac{1}{2} \sum_{j=1}^{l}&\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2} \\ &\ \downarrow \\ &\ \hat{y}_{j}^{k}=f\left(\beta_{j}-\theta_{j}\right) \end{aligned}$Ek​=21​j=1∑l​​(y^​jk​−yjk​)2 ↓ y^​jk​=f(βj​−θj​)​

所以：

$\frac{\partial E_{k}}{\partial \theta_{j}}=\frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \theta_{j}}$∂θj​∂Ek​​=∂y^​jk​∂Ek​​⋅∂θj​∂y^​jk​​

其中：

$\begin{aligned} &\begin{aligned} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} &=\frac{\partial\left[\frac{1}{2} \sum_{j=1}^{l}\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2}\right]}{\partial \hat{y}_{j}^{k}} \\ &=\frac{1}{2} \times 2 \times \left(\hat{y}_{j}^{k}-y_{j}^{k}\right) \times 1 \\ &=\hat{y}_{j}^{k}-y_{j}^{k} \end{aligned} \\ \\ &\begin{aligned} \frac{\partial \hat{y}_{j}^{k}}{\partial \theta_{j}} &=\frac{\partial\left[f\left(\beta_{j}-\theta_{j}\right)\right]}{\partial \theta_{j}} \\ &=f^{\prime}\left(\beta_{j}-\theta_{j}\right) \times (-1) \\ &\qquad \text{其中} \ f^{\prime}(x)=f(x)(1-f(x)) \\ &=f\left(\beta_{j}-\theta_{j}\right) \times\left[1-f\left(\beta_{j}-\theta_{j}\right)\right] \times (-1) \\ &\qquad \text{西瓜书式5.3}\ \hat{y}_{j}^{k}=f\left(\beta_{j}-\theta_{j}\right) \\ &=\hat{y}_{j}^{k} \left(1-\hat{y}_{j}^{k}\right) \times (-1) \end{aligned} \\ \end{aligned}$​∂y^​jk​∂Ek​​​=∂y^​jk​∂[21​∑j=1l​(y^​jk​−yjk​)2]​=21​×2×(y^​jk​−yjk​)×1=y^​jk​−yjk​​∂θj​∂y^​jk​​​=∂θj​∂[f(βj​−θj​)]​=f′(βj​−θj​)×(−1)其中 f′(x)=f(x)(1−f(x))=f(βj​−θj​)×[1−f(βj​−θj​)]×(−1)西瓜书式5.3 y^​jk​=f(βj​−θj​)=y^​jk​(1−y^​jk​)×(−1)​​

故有：

$\begin{aligned} \Delta \theta_{j} &=-\eta \frac{\partial E_{k}}{\partial \theta_{j}} \\ &=-\eta \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \theta_{j}} \\ &=-\eta \left(\hat{y}_{j}^{k}-y_{j}^{k}\right) \cdot \hat{y}_{j}^{k} \left(1-\hat{y}_{j}^{k}\right) \times (-1) \\ &=-\eta \left(y_{j}^{k}-\hat{y}_{j}^{k}\right) \cdot \hat{y}_{j}^{k} \left(1-\hat{y}_{j}^{k}\right) \\ &\qquad \text{西瓜书式5.10}\ g_{j}=\hat{y}_{j}^{k}\left(1-\hat{y}_{j}^{k}\right)\left(y_{j}^{k}-\hat{y}_{j}^{k}\right) \\ &=-\eta g_{j} \end{aligned} \tag{西瓜书式5.12}$Δθj​​=−η∂θj​∂Ek​​=−η∂y^​jk​∂Ek​​⋅∂θj​∂y^​jk​​=−η(y^​jk​−yjk​)⋅y^​jk​(1−y^​jk​)×(−1)=−η(yjk​−y^​jk​)⋅y^​jk​(1−y^​jk​)西瓜书式5.10 gj​=y^​jk​(1−y^​jk​)(yjk​−y^​jk​)=−ηgj​​(西瓜书式5.12)

2.2.3. 参数$v_{ih}$vih​的更新

已知$E_{k}$Ek​和$v_{ih}$vih​的函数链式关系为：

$\begin{aligned} E_{k}=\frac{1}{2} \sum_{j=1}^{l}&\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2} \\ &\ \downarrow \\ &\begin{aligned} \ \hat{y}_{j}^{k}=f & \left(\beta_{j}-\theta_{j}\right) \\ &\ \downarrow \\ &\begin{aligned} \ \beta_{j}=\sum_{h=1}^{q} w_{h j} &b_{h} \\ &\downarrow \\ &\begin{aligned} b_{h}=f & \left(\alpha_{h} - \gamma_{h}\right) \\ &\ \downarrow \\ &\ \alpha_{h}=\sum_{i=1}^{d} v_{ih} x_{i} \end{aligned} \end{aligned} \end{aligned} \end{aligned}$Ek​=21​j=1∑l​​(y^​jk​−yjk​)2 ↓ y^​jk​=f​(βj​−θj​) ↓ βj​=h=1∑q​whj​​bh​↓bh​=f​(αh​−γh​) ↓ αh​=i=1∑d​vih​xi​​​​​

所以：

$\frac{\partial E_{k}}{\partial v_{i h}}=\sum_{j=1}^{l} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial b_{h}} \cdot \frac{\partial b_{h}}{\partial \alpha_{h}} \cdot \frac{\partial \alpha_{h}}{\partial v_{i h}}$∂vih​∂Ek​​=j=1∑l​∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​⋅∂bh​∂βj​​⋅∂αh​∂bh​​⋅∂vih​∂αh​​

这里$v_{ih}$vih​存在于每一个$\hat{y}_{j}$y^​j​中，故共有$l$l个函数链关系。

其中：

$\begin{aligned} &\begin{aligned} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} &=\frac{\partial\left[\frac{1}{2} \sum_{j=1}^{l}\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2}\right]}{\partial \hat{y}_{j}^{k}} \\ &=\frac{1}{2} \times 2 \times \left(\hat{y}_{j}^{k}-y_{j}^{k}\right) \times 1 \\ &=\hat{y}_{j}^{k}-y_{j}^{k} \end{aligned} \\ \\ &\begin{aligned} \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} &=\frac{\partial\left[f\left(\beta_{j}-\theta_{j}\right)\right]}{\partial \beta_{j}} \\ &=f^{\prime}\left(\beta_{j}-\theta_{j}\right) \times 1 \\ &\qquad \text{其中} \ f^{\prime}(x)=f(x)(1-f(x)) \\ &=f\left(\beta_{j}-\theta_{j}\right) \times\left[1-f\left(\beta_{j}-\theta_{j}\right)\right] \\ &\qquad \text{西瓜书式5.3}\ \hat{y}_{j}^{k}=f\left(\beta_{j}-\theta_{j}\right) \\ &=\hat{y}_{j}^{k} \left(1-\hat{y}_{j}^{k}\right) \end{aligned} \\ \\ &\begin{aligned} \frac{\partial \beta_{j}}{\partial b_{h}} &=\frac{\partial\left(\sum_{h=1}^{q} w_{h j} b_{h}\right)}{\partial b_{h}} \\ &=w_{h j} \end{aligned} \\ \\ &\begin{aligned} \frac{\partial b_{h}}{\partial \alpha_{h}} &=\frac{\partial\left[f\left(\alpha_{h}-\gamma_{h}\right)\right]}{\partial \alpha_{h}} \\ &=f^{\prime}\left(\alpha_{h}-\gamma_{h}\right) \times 1 \\ &\qquad \text{其中} \ f^{\prime}(x)=f(x)(1-f(x)) \\ &=f\left(\alpha_{h}-\gamma_{h}\right) \times\left[1-f\left(\alpha_{h}-\gamma_{h}\right)\right] \\ &=b_{h}\left(1-b_{h}\right) \end{aligned} \\ \\ &\begin{aligned} \frac{\partial \alpha_{h}}{\partial v_{i h}} &=\frac{\partial\left(\sum_{i=1}^{d} v_{i h} x_{i}\right)}{\partial v_{i h}} \\ &=x_{i} \end{aligned} \\ \end{aligned}$​∂y^​jk​∂Ek​​​=∂y^​jk​∂[21​∑j=1l​(y^​jk​−yjk​)2]​=21​×2×(y^​jk​−yjk​)×1=y^​jk​−yjk​​∂βj​∂y^​jk​​​=∂βj​∂[f(βj​−θj​)]​=f′(βj​−θj​)×1其中 f′(x)=f(x)(1−f(x))=f(βj​−θj​)×[1−f(βj​−θj​)]西瓜书式5.3 y^​jk​=f(βj​−θj​)=y^​jk​(1−y^​jk​)​∂bh​∂βj​​​=∂bh​∂(∑h=1q​whj​bh​)​=whj​​∂αh​∂bh​​​=∂αh​∂[f(αh​−γh​)]​=f′(αh​−γh​)×1其中 f′(x)=f(x)(1−f(x))=f(αh​−γh​)×[1−f(αh​−γh​)]=bh​(1−bh​)​∂vih​∂αh​​​=∂vih​∂(∑i=1d​vih​xi​)​=xi​​​

则令$e_{h}$eh​有：

$\begin{aligned} e_{h} &=-\frac{\partial E_{k}}{\partial \alpha_{h}} \\ &=-\sum_{j=1}^{l} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial b_{h}} \cdot \frac{\partial b_{h}}{\partial \alpha_{h}} \\ &=\sum_{j=1}^{l} \left(-\frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}}\right) \cdot \left(\frac{\partial \beta_{j}}{\partial b_{h}}\right) \cdot \left(\frac{\partial b_{h}}{\partial \alpha_{h}}\right) \\ &=\sum_{j=1}^{l} g_{j} \cdot w_{h j} \cdot b_{h}\left(1-b_{h}\right) \\ &=b_{h}\left(1-b_{h}\right) \sum_{j=1}^{l} w_{h j} g_{j} \end{aligned} \tag{西瓜书式5.15}$eh​​=−∂αh​∂Ek​​=−j=1∑l​∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​⋅∂bh​∂βj​​⋅∂αh​∂bh​​=j=1∑l​(−∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​)⋅(∂bh​∂βj​​)⋅(∂αh​∂bh​​)=j=1∑l​gj​⋅whj​⋅bh​(1−bh​)=bh​(1−bh​)j=1∑l​whj​gj​​(西瓜书式5.15)

所以：

$\begin{aligned} \Delta v_{i h} &=-\eta \frac{\partial E_{k}}{\partial v_{i h}} \\ &=-\eta \sum_{j=1}^{l} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial b_{h}} \cdot \frac{\partial b_{h}}{\partial \alpha_{h}} \cdot \frac{\partial \alpha_{h}}{\partial v_{i h}} \\ &=\eta \left(-\sum_{j=1}^{l} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial b_{h}} \cdot \frac{\partial b_{h}}{\partial \alpha_{h}}\right) \cdot \left(\frac{\partial \alpha_{h}}{\partial v_{i h}}\right) \\ &=\eta e_{h} x_{i} \end{aligned} \tag{西瓜书式5.13}$Δvih​​=−η∂vih​∂Ek​​=−ηj=1∑l​∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​⋅∂bh​∂βj​​⋅∂αh​∂bh​​⋅∂vih​∂αh​​=η(−j=1∑l​∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​⋅∂bh​∂βj​​⋅∂αh​∂bh​​)⋅(∂vih​∂αh​​)=ηeh​xi​​(西瓜书式5.13)

2.2.4. 参数$\gamma_{h}$γh​的更新

已知$E_{k}$Ek​和$\gamma_{h}$γh​的函数链式关系为：

$\begin{aligned} E_{k}=\frac{1}{2} \sum_{j=1}^{l}&\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2} \\ &\ \downarrow \\ &\begin{aligned} \ \hat{y}_{j}^{k}=f & \left(\beta_{j}-\theta_{j}\right) \\ &\ \downarrow \\ &\begin{aligned} \ \beta_{j}=\sum_{h=1}^{q} w_{h j} &b_{h} \\ &\downarrow \\ &b_{h}=f\left(\alpha_{h} - \gamma_{h}\right) \end{aligned} \end{aligned} \end{aligned}$Ek​=21​j=1∑l​​(y^​jk​−yjk​)2 ↓ y^​jk​=f​(βj​−θj​) ↓ βj​=h=1∑q​whj​​bh​↓bh​=f(αh​−γh​)​​​

所以：

$\frac{\partial E_{k}}{\gamma_{h}}=\sum_{j=1}^{l} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial b_{h}} \cdot \frac{\partial b_{h}}{\partial \gamma_{h}}$γh​∂Ek​​=j=1∑l​∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​⋅∂bh​∂βj​​⋅∂γh​∂bh​​

这里$\gamma_{h}$γh​存在于每一个$\hat{y}_{j}$y^​j​中，故共有$l$l个函数链关系。

其中：

$\begin{aligned} &\begin{aligned} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} &=\frac{\partial\left[\frac{1}{2} \sum_{j=1}^{l}\left(\hat{y}_{j}^{k}-y_{j}^{k}\right)^{2}\right]}{\partial \hat{y}_{j}^{k}} \\ &=\frac{1}{2} \times 2 \times \left(\hat{y}_{j}^{k}-y_{j}^{k}\right) \times 1 \\ &=\hat{y}_{j}^{k}-y_{j}^{k} \end{aligned} \\ \\ &\begin{aligned} \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} &=\frac{\partial\left[f\left(\beta_{j}-\theta_{j}\right)\right]}{\partial \beta_{j}} \\ &=f^{\prime}\left(\beta_{j}-\theta_{j}\right) \times 1 \\ &\qquad \text{其中} \ f^{\prime}(x)=f(x)(1-f(x)) \\ &=f\left(\beta_{j}-\theta_{j}\right) \times\left[1-f\left(\beta_{j}-\theta_{j}\right)\right] \\ &\qquad \text{西瓜书式5.3}\ \hat{y}_{j}^{k}=f\left(\beta_{j}-\theta_{j}\right) \\ &=\hat{y}_{j}^{k} \left(1-\hat{y}_{j}^{k}\right) \end{aligned} \\ \\ &\begin{aligned} \frac{\partial \beta_{j}}{\partial b_{h}} &=\frac{\partial\left(\sum_{h=1}^{q} w_{h j} b_{h}\right)}{\partial b_{h}} \\ &=w_{h j} \end{aligned} \\ \\ &\begin{aligned} \frac{\partial b_{h}}{\partial \gamma_{h}} &=\frac{\partial\left[f\left(\alpha_{h}-\gamma_{h}\right)\right]}{\partial \gamma_{h}} \\ &\qquad \text{其中} \ f^{\prime}(x)=f(x)(1-f(x)) \\ &=f^{\prime}\left(\alpha_{h}-\gamma_{h}\right) \times (-1) \\ &=f\left(\alpha_{h}-\gamma_{h}\right) \times\left[1-f\left(\alpha_{h}-\gamma_{h}\right)\right] \times (-1)\\ &=b_{h}\left(1-b_{h}\right) \times (-1) \end{aligned} \\ \end{aligned}$​∂y^​jk​∂Ek​​​=∂y^​jk​∂[21​∑j=1l​(y^​jk​−yjk​)2]​=21​×2×(y^​jk​−yjk​)×1=y^​jk​−yjk​​∂βj​∂y^​jk​​​=∂βj​∂[f(βj​−θj​)]​=f′(βj​−θj​)×1其中 f′(x)=f(x)(1−f(x))=f(βj​−θj​)×[1−f(βj​−θj​)]西瓜书式5.3 y^​jk​=f(βj​−θj​)=y^​jk​(1−y^​jk​)​∂bh​∂βj​​​=∂bh​∂(∑h=1q​whj​bh​)​=whj​​∂γh​∂bh​​​=∂γh​∂[f(αh​−γh​)]​其中 f′(x)=f(x)(1−f(x))=f′(αh​−γh​)×(−1)=f(αh​−γh​)×[1−f(αh​−γh​)]×(−1)=bh​(1−bh​)×(−1)​​

故有：

$\begin{aligned} \Delta \gamma_{h} &=-\eta \frac{\partial E_{k}}{\gamma_{h}} \\ &=-\eta \sum_{j=1}^{l} \frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}} \cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}} \cdot \frac{\partial \beta_{j}}{\partial b_{h}} \cdot \frac{\partial b_{h}}{\partial \gamma_{h}} \\ &=\eta \sum_{j=1}^{l} \left(-\frac{\partial E_{k}}{\partial \hat{y}_{j}^{k}}\cdot \frac{\partial \hat{y}_{j}^{k}}{\partial \beta_{j}}\right) \cdot \left(\frac{\partial \beta_{j}}{\partial b_{h}}\right) \cdot \left(\frac{\partial b_{h}}{\partial \gamma_{h}}\right) \\ &=\eta \sum_{j=1}^{l} g_{j} \cdot w_{hj} \cdot b_{h}\left(1-b_{h}\right) \times (-1) \\ &=-\eta b_{h}\left(1-b_{h}\right) \sum_{j=1}^{l} w_{h j} g_{j} \\ &\qquad \text{西瓜书式5.15} \ e_{h}=b_{h}\left(1-b_{h}\right) \sum_{j=1}^{l} w_{h j} g_{j} \\ &=-\eta e_{h} \end{aligned} \tag{西瓜书式5.14}$Δγh​​=−ηγh​∂Ek​​=−ηj=1∑l​∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​⋅∂bh​∂βj​​⋅∂γh​∂bh​​=ηj=1∑l​(−∂y^​jk​∂Ek​​⋅∂βj​∂y^​jk​​)⋅(∂bh​∂βj​​)⋅(∂γh​∂bh​​)=ηj=1∑l​gj​⋅whj​⋅bh​(1−bh​)×(−1)=−ηbh​(1−bh​)j=1∑l​whj​gj​西瓜书式5.15 eh​=bh​(1−bh​)j=1∑l​whj​gj​=−ηeh​​(西瓜书式5.14)