cs224n lecture7 Vanishing Gradients, Fancy RNNs

RNN’s problem

vanishing gradient

解决方案:

  • LSTM
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
  • GRU
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
  • vs
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
  • residual connections
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
  • DenseNet
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
  • HighwayNet
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
  • Bidirectional RNNs
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
    cs224n lecture7 Vanishing Gradients, Fancy RNNs
  • Multi-layer RNNs(stacked RNNs)
    cs224n lecture7 Vanishing Gradients, Fancy RNNs

cs224n lecture7 Vanishing Gradients, Fancy RNNs

exploding gradient

gradient clipping
cs224n lecture7 Vanishing Gradients, Fancy RNNs

In summary

cs224n lecture7 Vanishing Gradients, Fancy RNNs