LSTM及其变体(一)
经典LSTM(1997)
- 1. 三个门的信息流入
-
ft=σ(Wf⋅[ht−1,xt]+bf) (1) -
it=σ(Wi⋅[ht−1,xt]+bi) (2) -
ot=σ(Wo⋅[ht−1,xt]+bo) (3)
- 2. 记忆单元的更新
-
C~t=tanh(WC⋅[ht−1,xt]+bC) (输入整流器) -
Ct=ft∗Ct−1+it∗C~t (记忆遗忘+输入整流)
- 3. 隐单元更新
-
ht=ot∗tanh(Ct) (6)
Bidirectional LSTM
Graves A, Schmidhuber J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures.[J]. Neural Networks the Official Journal of the International Neural Network Society, 2005, 18(5–6):602-610.
首先提出Bi-LSTM并用于帧单元音素分类。
LSTM with peephone connection
将
-
ft=σ(Wf⋅[ht−1,Ct−1,xt]+bf) -
it=σ(Wi⋅[ht−1,Ct−1,xt]+bi) -
ot=σ(Wo⋅[ht−1,Ct,xt]+bo)
GRU
将忘记门和输入门合成了一个单一的 更新门