VGG 16 公式推导

VGG-16共有13层卷积层，5层池化层和3层全连接层，对前两层全连接网络采用dropout和L2正则化防止过拟合，采用批量梯度下降+Momentum以交叉熵为目标损失进行训练优化。

• $n^l$nl—第$l$l层网络节点（卷积核）数目；

• $k_{p,q}^l$kp,ql​—第$l$l层$p$p通道与第$l-1$l−1层$q$q通道对应卷积核；

• $b_p^l$bpl​—第$l$l层$p$p节点（通道）的偏置；

• $W^l$Wl—第$l$l层全连接网络的权重；

• $z^l$zl—第$l$l层未经过**函数的前向输入；

• $a^l$al—第$l$l层经过**函数后的前向输出；

前向传递

第$l$l层卷积操作公式：
$z_{p}^{l}(i,j)=\sum\limits_{q=1}^{{{n}^{l-1}}}{\sum\limits_{u=-1}^{1}{\sum\limits_{v=-1}^{1}{a_{q}^{l-1}(i-u,j-v)k_{p,q}^{l}(u,v)}}}+b_{p}^{l} \\ a_{p}^{l}(i,j)=ReLU\left( z_{p}^{l}(i,j) \right)$zpl​(i,j)=q=1∑nl−1​u=−1∑1​v=−1∑1​aql−1​(i−u,j−v)kp,ql​(u,v)+bpl​apl​(i,j)=ReLU(zpl​(i,j))
第$l$l层最大池化公式：
$z_{p}^{l}(i,j)=\max \left( a_{p}^{l-1}(2i-u,2j-v) \right)u,v\in \left\{ 0,1 \right\}$zpl​(i,j)=max(apl−1​(2i−u,2j−v))u,v∈{0,1}
经过前18层的卷积核池化操作后可获得$7×7×512$7×7×512大小的特征图，需要将其转化为一个25,088维的向量以便作为全连接层的输入，该过程输出为$a^{18}$a18:
$a^{18}=F \left(\left\{z_p^{18}\right\}_{p=1,2,⋯,512}\right)$a18=F({zp18​}p=1,2,⋯,512​)
全连接网络的前两层采用dropout，设为$d$d，第$l$l层节点的连通可用$r^l$rl来表示，其服从伯努利分布：
${{r}^{l}}\sim Bernoulli(d)$rl∼Bernoulli(d)
前向传播为：
${{{\tilde{a}}}^{l}}={{r}^{l}}\odot {{a}^{l}} \\ {{z}^{l+1}}={{W}^{l+1}}{{{\tilde{a}}}^{l}}+{{b}^{l+1}} \\ {{a}^{l+1}}=ReLU({{z}^{l+1}})$a~l=rl⊙alzl+1=Wl+1a~l+bl+1al+1=ReLU(zl+1)
其中，⨀为Hadmard积，即矩阵对应元素相乘。

输出层的**函数为softmax：
$a_{i}^{L}=softmax(z_{i}^{L})=\frac{{{e}^{z_{i}^{L}}}}{\sum\limits_{k=1}^{{{n}^{L}}}{{{e}^{z_{k}^{L}}}}}$aiL​=softmax(ziL​)=k=1∑nL​ezkL​eziL​​
采用交叉熵损失作为损失函数：
$L=-\sum\limits_{i=1}^{{{n}^{L}}}{{{y}_{i}}\log a_{i}^{L}}$L=−i=1∑nL​yi​logaiL​

反向传播

引入中间变量$\delta^l$δl，为第$l$l层的误差，表示损失函数对第l层前向输入$z^l$zl 的梯度，即为$\frac{\partial{L}}{\partial{z^l}}$∂zl∂L​

Softmax函数偏导数计算公式为：

当$i=j$i=j时，
$\frac{\partial }{\partial {{z}_{j}}}\left( \frac{{{e}^{{{z}_{j}}}}}{\sum\nolimits_{k=1}^{n}{{{e}^{{{z}_{k}}}}}} \right)=\frac{{{e}^{{{z}_{j}}}}\sum\nolimits_{k=1}^{n}{{{e}^{{{z}_{k}}}}}-{{\left( {{e}^{{{z}_{j}}}} \right)}^{2}}}{{{\left( \sum\nolimits_{k=1}^{n}{{{e}^{{{z}_{k}}}}} \right)}^{2}}} ={{a}_{j}}\left( 1-{{a}_{j}} \right)$∂zj​∂​(∑k=1n​ezk​ezj​​)=(∑k=1n​ezk​)2ezj​∑k=1n​ezk​−(ezj​)2​=aj​(1−aj​)
当$i\ne j$i​=j时，
$\frac{\partial }{\partial {{z}_{j}}}\left( \frac{{{e}^{{{z}_{i}}}}}{\sum\nolimits_{k=1}^{n}{{{e}^{{{z}_{k}}}}}} \right)=\frac{-{{e}^{{{z}_{i}}}}{{e}^{{{z}_{j}}}}}{{{\left( \sum\nolimits_{k=1}^{n}{{{e}^{{{z}_{k}}}}} \right)}^{2}}}=-{{a}_{i}}{{a}_{j}}$∂zj​∂​(∑k=1n​ezk​ezi​​)=(∑k=1n​ezk​)2−ezi​ezj​​=−ai​aj​
输出层第$j$j个节点的误差为：
$\begin{aligned} \delta _{j}^{L}&=\frac{\partial L}{\partial z_{j}^{L}} \\ &=\sum\limits_{i=1}^{{{n}^{L}}}{\frac{\partial L}{\partial a_{i}^{L}}\frac{\partial a_{i}^{L}}{\partial z_{j}^{L}}} \\ & =\frac{\partial L}{\partial a_{j}^{L}}\frac{\partial a_{j}^{L}}{\partial z_{j}^{L}}+\sum\limits_{i\ne j}{\frac{\partial L}{\partial a_{i}^{L}}\frac{\partial a_{i}^{L}}{\partial z_{j}^{L}}} \\ & =-\frac{{{y}_{j}}}{a_{j}^{L}}a_{j}^{L}(1-a_{j}^{L})+\sum\limits_{i\ne j}{-\frac{{{y}_{i}}}{a_{i}^{L}}(-a_{i}^{L}a_{j}^{L})} \\ & =-{{y}_{j}}(1-a_{j}^{L})+a_{j}^{L}\sum\limits_{i\ne j}{{{y}_{i}}} \\ & =a_{j}^{L}-{{y}_{j}} \end{aligned}$δjL​​=∂zjL​∂L​=i=1∑nL​∂aiL​∂L​∂zjL​∂aiL​​=∂ajL​∂L​∂zjL​∂ajL​​+i​=j∑​∂aiL​∂L​∂zjL​∂aiL​​=−ajL​yj​​ajL​(1−ajL​)+i​=j∑​−aiL​yi​​(−aiL​ajL​)=−yj​(1−ajL​)+ajL​i​=j∑​yi​=ajL​−yj​​
输出层的反向传播误差为：
${{\delta }^{L}}={{a}^{L}}-y$δL=aL−y
第$l$l个隐藏层第$j$j个节点的反向传播误差为：
$\begin{aligned} \delta _{j}^{l}&=\frac{\partial L}{\partial z_{j}^{l}}\\ &=\sum\limits_{i=1}^{{{n}^{l+1}}}{\frac{\partial L}{\partial z_{i}^{l+1}}\frac{\partial z_{i}^{l+1}}{\partial \tilde{a}_{j}^{l}}\frac{\partial \tilde{a}_{j}^{l}}{\partial a_{j}^{l}}\frac{\partial a_{j}^{l}}{\partial z_{j}^{l}}}\\ &=\sum_{i=1}^{n^{l+1}}{\delta^{l+1}_iW^{l+1}_{i,j}r^l_jReLU(z^l_j)'}\\ &={{\left( W_{:,j}^{l+1} \right)}^{T}}{{\delta }^{l+1}}r_{j}^{l}ReLU(z_{j}^{l}{)}' \end{aligned}$δjl​​=∂zjl​∂L​=i=1∑nl+1​∂zil+1​∂L​∂a~jl​∂zil+1​​∂ajl​∂a~jl​​∂zjl​∂ajl​​=i=1∑nl+1​δil+1​Wi,jl+1​rjl​ReLU(zjl​)′=(W:,jl+1​)Tδl+1rjl​ReLU(zjl​)′​
因此第$l$l全连接层的反向传播误差为：
${{\delta }^{l}}={{\left( {{W}^{l+1}} \right)}^{T}}{{\delta }^{l+1}}\odot {{r}^{l}}\odot ReLU({{z}^{l}}{)}'$δl=(Wl+1)Tδl+1⊙rl⊙ReLU(zl)′
由全连接层到池化层的反向传播误差为：
${{\delta }^{18}}={{F}^{-1}}\left( {{({{W}^{19}})}^{T}}{{\delta }^{19}} \right)$δ18=F−1((W19)Tδ19)
其中$\delta^{18}$δ18为$7×7×512$7×7×512的张量。

由第$l+1$l+1池化层的$\delta^{l+1}$δl+1推导第$l$l卷积层的反向传播误差时，对于最大池化，我们需要利用上采样将$\delta^{l+1}$δl+1中每个通道中的元素放到之前做前向传播时最大值的位置处，其他元素为0：
$\begin{aligned} \delta _{p}^{l}&=\frac{\partial L}{\partial z_{p}^{l}} \\ & =\frac{\partial L}{\partial a_{p}^{l}}\frac{\partial a_{p}^{l}}{\partial z_{p}^{l}} \\ & =upsample(\delta _{p}^{l+1})\odot ReLU(z_{p}^{l}{)}' \end{aligned}$δpl​​=∂zpl​∂L​=∂apl​∂L​∂zpl​∂apl​​=upsample(δpl+1​)⊙ReLU(zpl​)′​
因此，由池化层误差计算卷积层反向传播误差公式为：
${{\delta }^{l}}=upsample({{\delta }^{l\text{+1}}})\odot ReLU({{z}^{l}}{)}'$δl=upsample(δl+1)⊙ReLU(zl)′
由第$l+1$l+1卷积层的$\delta^{l+1}$δl+1推导第$l$l卷积层的反向传播误差：
$z_{p}^{l+1}=\sum\limits_{q=1}^{{{n}^{l}}}{a_{q}^{l}*k_{p,q}^{l+1}}+b_{p}^{l+1}$zpl+1​=q=1∑nl​aql​∗kp,ql+1​+bpl+1​

$\delta _{q}^{l}=\frac{\partial L}{\partial z_{q}^{l}}=\sum\limits_{p=1}^{{{n}^{l+1}}}{\frac{\partial L}{\partial z_{p}^{l+1}}\frac{\partial z_{p}^{l+1}}{\partial a_{q}^{l}}\frac{\partial a_{q}^{l}}{\partial z_{q}^{l}}}$δql​=∂zql​∂L​=p=1∑nl+1​∂zpl+1​∂L​∂aql​∂zpl+1​​∂zql​∂aql​​

$\frac{\partial L}{\partial z_{p}^{l+1}}\frac{\partial z_{p}^{l+1}}{\partial a_{q}^{l}}=\delta _{p}^{l+1}*rot180(k_{p,q}^{l+1})$∂zpl+1​∂L​∂aql​∂zpl+1​​=δpl+1​∗rot180(kp,ql+1​)

因此，
$\delta _{q}^{l}=\frac{\partial L}{\partial z_{q}^{l}}=\left[ \sum\limits_{p=1}^{{{n}^{l+1}}}{\delta _{p}^{l+1}*rot180(k_{p,q}^{l+1})} \right]\odot ReLU(z_{q}^{l}{)}'$δql​=∂zql​∂L​=⎣⎡​p=1∑nl+1​δpl+1​∗rot180(kp,ql+1​)⎦⎤​⊙ReLU(zql​)′
当第$l$l层为池化层时，$ReLU(z_{q}^{l}{)}'=1$ReLU(zql​)′=1。

梯度计算

已知全连接层$l$l的反向传播误差$\delta^l$δl，计算梯度$\frac{\partial L}{\partial W^l}$∂Wl∂L​和$\frac{\partial L}{\partial b^l}$∂bl∂L​：
$\frac{\partial L}{\partial {{W}^{l}}}=\frac{\partial L}{\partial {{z}^{l}}}\frac{\partial {{z}^{l}}}{\partial {{W}^{l}}}={{\delta }^{l}}{{\left( {{a}^{l-1}} \right)}^{T}}$∂Wl∂L​=∂zl∂L​∂Wl∂zl​=δl(al−1)T

$\frac{\partial L}{\partial {{b}^{l}}}=\frac{\partial L}{\partial {{z}^{l}}}\frac{\partial {{z}^{l}}}{\partial {{b}^{l}}}={{\delta }^{l}}$∂bl∂L​=∂zl∂L​∂bl∂zl​=δl

当获得平均梯度信息后添加L2正则项，设系数为$\gamma$γ：
$\frac{\partial L}{\partial {{W}^{l}}}=\frac{\partial L}{\partial {{W}^{l}}}+\gamma \left( {{r}^{l}}{{\left( {{r}^{l-1}} \right)}^{T}} \right){{W}^{l}}$∂Wl∂L​=∂Wl∂L​+γ(rl(rl−1)T)Wl
已知卷积层的反向传播误差$\delta^l$δl，计算梯度$\frac{\partial L}{\partial k^l_{p,q}}$∂kp,ql​∂L​和$\frac{\partial L}{\partial b^l_p}$∂bpl​∂L​：
$\frac{\partial L}{\partial k_{p,q}^{l}}=\frac{\partial L}{\partial z_{p}^{l}}\frac{\partial z_{p}^{l}}{\partial k_{p,q}^{l}}=\delta _{p}^{l}*a_{q}^{l-1}$∂kp,ql​∂L​=∂zpl​∂L​∂kp,ql​∂zpl​​=δpl​∗aql−1​

$\frac{\partial L}{\partial b_{p}^{l}}=\frac{\partial L}{\partial z_{p}^{l}}\frac{\partial z_{p}^{l}}{\partial b_{p}^{l}}=\sum\limits_{i=1}^{{{n}^{l}}}{\sum\limits_{j=1}^{{{n}^{l}}}{\delta _{p}^{l}(i,j)}}$∂bpl​∂L​=∂zpl​∂L​∂bpl​∂zpl​​=i=1∑nl​j=1∑nl​δpl​(i,j)

参数更新

当获得一个批次的平均梯度信息后利用批量梯度下降+Momentum方法进行参数更新：
$\begin{aligned} {{v}_{dk}}&=\beta {{v}_{dk}}+(1-\beta )\frac{\partial L}{\partial {{k}^{l}}} \\ {{v}_{db}}&=\beta {{v}_{db}}+(1-\beta )\frac{\partial L}{\partial {{b}^{l}}} \\ {{v}_{dW}}&=\beta {{v}_{dW}}+(1-\beta )\frac{\partial L}{\partial {{W}^{l}}} \end{aligned}$vdk​vdb​vdW​​=βvdk​+(1−β)∂kl∂L​=βvdb​+(1−β)∂bl∂L​=βvdW​+(1−β)∂Wl∂L​​
参数更新：
$\begin{aligned} {{k}^{l}}&={{k}^{l}}-\alpha {{v}_{d{{k}^{l}}}} \\ {{b}^{l}}&={{b}^{l}}-\alpha {{v}_{d{{b}^{l}}}} \\ {{W}^{l}}&={{W}^{l}}-\alpha {{v}_{d{{W}^{l}}}} \end{aligned}$klblWl​=kl−αvdkl​=bl−αvdbl​=Wl−αvdWl​​

[1]. Very Deep Convolutional Networks for Large-Scale Image Recognition

[2]. Dropout: A Simple Way to Prevent Neural Networks from Overfitting

[3]. 卷积神经网络(CNN)反向传播算法