deeplearning.ai - 优化算法 (Optimization Algorithms)

改善深层神经网络：超参数调试、正则化以及优化
吴恩达 Andrew Ng

Mini-batch 梯度下降法

• 把巨大的数据集分成一个一个的小部分

• 5000000 examples, 1000 × 5000, X{1}...X{5000}$X^{\{1\}}...X^{\{5000\}}$ , Y{1}...Y{5000}$Y^{\{1\}}...Y^{\{5000\}}$

• epoch means a single pass through the training set

• Batch gradient descent’s cost decrease on every iteration

• Mini-batch gradient descent may not decrease on every iteration. It trends downwards, but it’s going to be a little bit noisier.

• mini-batch size = m: Batch gradient descent

• mini-batch size = 1: Stochastic gradient descent (随机梯度下降法)
  不会收敛，最终在最小值处波动

• 一般 mini-batch 大小为 64、128、256、512

Exponentially weighted averages (指数加权平均)

• Vt=βVt−1+(1−β)θt$V_t=\beta V_{t-1}+(1-\beta ){\theta }_t$

• Vt$V_t$ approximately average over 11−β$\frac{1}{1-\beta }$ items

• 迭代几次公式，展开递推式

• 初始化 V0=0$V_0=0$

• As for computation and memory efficiency, it’s a good choice.

Bias correction in exponentially weighted average

• during initial phase of estimating, make it more accurate

• Vt1−βt$\frac{V_t}{1-{\beta }^t}$

• oscillations 震荡，波动

Momentum (动量梯度下降法)

• ball rolling down a bowl

• 

• usually β=0.9$\beta =0.9$

RMSprop

root mean square prop

Adam optimization algorithm

Adaptive Moment Estimation

• 结合 Momentum 和 RMSprop

• 适用性广泛

• Hyperparameters:

  α:$\alpha :$ needs to be tuned, β1:0.9${\beta }_1:0.9$ , β2:0.999${\beta }_2:0.999$ , ϵ:10−8$ϵ:{10}^{-8}$

Learning rate decay

随着迭代的增加，渐渐减小学习率

• α=11+decayrate×iterationα0$\alpha =\frac{1}{1+deca{y}_{r}ate\times iteration}{\alpha }_0$

• α=0.95epoch_numα0$\alpha ={0.95}^{epoch\mathrm{\_}num}{\alpha }_0$

• α=kepoch_num√α0$\alpha =\frac{k}{\sqrt{epoch\mathrm{\_}num}}{\alpha }_0$

• discrete staircase

• manually controlling alpha (small model)

The problem of local optima

• saddle point 马鞍点
• 许多低维空间里的直觉在高维空间中并不适用，高维空间极少出现局部最优点
• 平稳区域(plateaus)会降低学习效率