给定语料库D = { D 1 , D 2 , ⋯  , D N } \mathbb{D}=\left\{\mathcal{D}_{1}, \mathcal{D}_{2}, \cdots, \mathcal{D}_{N}\right\} D = { D 1 , D 2 , ⋯ , D N } D i \mathcal{D_i} D i ( word I 1 i , word I 2 i , ⋯  , word I n i i ) \left(\operatorname{word}_{I_{1}^{i}}, \operatorname{word}_{I_{2}^{i}}, \cdots, \operatorname{word}_{I_{n_{i}}^{i}}\right) ( w o r d I 1 i , w o r d I 2 i , ⋯ , w o r d I n i i ) I j i ∈ { 1 , 2 , ⋯  , V } I_{j}^{i} \in\{1,2, \cdots, V\} I j i ∈ { 1 , 2 , ⋯ , V } n i n_i n i n i n_i n i v = I j i v=I^i_j v = I j i word v \operatorname{word}_v w o r d v V = { w o r d 1 , word 2 , ⋯  , word V } \mathbb{V}=\left\{\mathrm{word}_{1}, \text { word }_{2}, \cdots, \text { word }_{V}\right\} V = { w o r d 1 , word 2 , ⋯ , word V } word v \operatorname{word}_v w o r d v
最简单的表示方式是one-hot编码，对于词汇表的第v个单词word v \operatorname{word}_v w o r d v word v → ( 0 , 0 , ⋯  , 0 , 1 , 0 , ⋯  , 0 ) T \operatorname{word}_{v} \rightarrow(0,0, \cdots, 0,1,0, \cdots, 0)^{T} w o r d v → ( 0 , 0 , ⋯ , 0 , 1 , 0 , ⋯ , 0 ) T ( word i , word j ) \left(\text { word }_{i}, \text { word }_{j}\right) ( word i , word j ) 2 \sqrt 2 2
Bow:Bag of Words：词在文档中不考虑顺序，这称作词袋模型。
CBOW模型(Continuous bag-of-word)：根据上下文来预测一个单词。
在一个单词上下文的CBOW模型中，输入的是前一个单词，输出是后一个单词；输入为输出的上下文，由于只有一个单词作为输入，因此称作一个单词的上下文。
一个单词上下文的CBOW模型如下：
N为隐层的大小，即隐向量h → ∈ R N \overrightarrow{\mathbf{h}} \in \mathbb{R}^{N} h ∈ R N
网络输入x → = ( x 1 , x 2 , ⋯  , x V ) T ∈ R V \overrightarrow{\mathbf{x}}=\left(x_{1}, x_{2}, \cdots, x_{V}\right)^{T} \in \mathbb{R}^{V} x = ( x 1 , x 2 , ⋯ , x V ) T ∈ R V one-hot编码，其中只有一位为1，其他位都为0。
网络输出y → = ( y 1 , y 2 , ⋯  , y V ) T ∈ R V \overrightarrow{\mathbf{y}}=\left(y_{1}, y_{2}, \cdots, y_{V}\right)^{T} \in \mathbb{R}^{V} y = ( y 1 , y 2 , ⋯ , y V ) T ∈ R V
相邻层之间为全连接：
输入层和隐层之间的权重矩阵为W，其尺寸V × N V \times N V × N
隐层和输出层之间的权重矩阵为W ′ \mathbf{W}^{\prime} W ′ N × V N \times V N × V
假设没有**函数，没有偏置项。给定输入x → ∈ R V \overrightarrow{\mathbf{x}} \in \mathbb{R}^{V} x ∈ R V h → ∈ R N \overrightarrow{\mathbf{h}} \in \mathbb{R}^{N} h ∈ R N h → = w T x → \overrightarrow{\mathbf{h}}=\mathbf{w}^{T} \overrightarrow{\mathbf{x}} h = w T x W = [ w → 1 T w → 2 T ⋮ w → V T ] \mathbf{W}=\left[ \begin{array}{c}{\overrightarrow{\mathbf{w}}_{1}^{T}} \\ {\overrightarrow{\mathbf{w}}_{2}^{T}} \\ {\vdots} \\ {\overrightarrow{\mathbf{w}}_{V}^{T}}\end{array}\right] W = ⎣ ⎢ ⎢ ⎢ ⎡ w 1 T w 2 T ⋮ w V T ⎦ ⎥ ⎥ ⎥ ⎤ x → \overrightarrow{\mathbf{x}} x one-hot编码，假设它为此表V \mathbb{V} V word k \operatorname{word}_k w o r d k x 1 = 0 , x 2 = 0 , ⋯  , x k − 1 = 0 , x k = 1 , x k + 1 = 0 , ⋯  , x V = 0 x_{1}=0, x_{2}=0, \cdots, x_{k-1}=0, x_{k}=1, x_{k+1}=0, \cdots, x_{V}=0 x 1 = 0 , x 2 = 0 , ⋯ , x k − 1 = 0 , x k = 1 , x k + 1 = 0 , ⋯ , x V = 0 h → = w → k \overrightarrow{\mathbf{h}}=\overrightarrow{\mathbf{w}}_{k} h = w k w k → \overrightarrow{\mathbf{w}_k} w k V \mathbb{V} V word k \operatorname{word}_k w o r d k word k \operatorname{word}_k w o r d k
给定隐向量h → \overrightarrow{\mathbf{h}} h u → ∈ R v \overrightarrow{\mathbf{u}} \in \mathbb{R}^{v} u ∈ R v u → = W ′ T h → \overrightarrow{\mathbf{u}}=\mathbf{W^{\prime}}^{T} \overrightarrow{\mathbf{h}} u = W ′ T h W ′ = [ w → 1 ′ , w → 2 ′ , ⋯  , w → V ′ ] \mathbf{W}^{\prime}=\left[\overrightarrow{\mathbf{w}}_{1}^{\prime}, \overrightarrow{\mathbf{w}}_{2}^{\prime}, \cdots, \overrightarrow{\mathbf{w}}_{V}^{\prime}\right] W ′ = [ w 1 ′ , w 2 ′ , ⋯ , w V ′ ] u j = w → j ′ ⋅ h → u_{j}=\overrightarrow{\mathbf{w}}_{ j}^{\prime} \cdot \overrightarrow{\mathbf{h}} u j = w j ′ ⋅ h V \mathbb{V} V word j \operatorname{word}_j w o r d j w j → ′ \overrightarrow{\mathbf{w}_j}^{\prime} w j ′ W ′ \mathbf{W}^{\prime} W ′ word j \operatorname{word}_j w o r d j
u → \overrightarrow{\mathbf{u}} u softmax层，则有：y j = p ( word j ∣ x → ) = exp ( u j ) ∑ j ′ = 1 V exp ( u j ′ ) , j = 1 , 2 , ⋯  , V y_{j}=p\left(\text { word }_{j} | \overrightarrow{\mathbf{x}}\right)=\frac{\exp \left(u_{j}\right)}{\sum_{j^{\prime}=1}^{V} \exp \left(u_{j^{\prime}}\right)}, \quad j=1,2, \cdots, V y j = p ( word j ∣ x ) = ∑ j ′ = 1 V exp ( u j ′ ) exp ( u j ) , j = 1 , 2 , ⋯ , V y j y_j y j V \mathbb{V} V word j \operatorname{word}_j w o r d j
假设给定一个单词word I \operatorname{word}_{I} w o r d I word O \operatorname{word}_{O} w o r d O word O \operatorname{word}_{O} w o r d O j ∗ j^{*} j ∗ max W , W ′ p ( word O ∣ word I ) = max W , W ′ y j ∗ = max W , W ′ log exp ( w → j ∗ ′ ⋅ w → I ) ∑ i = 1 V exp ( w → i ′ ⋅ w → I ) = max W , W ′ [ w → j ∗ ′ ⋅ w → I − log ∑ i = 1 V exp ( w → i ′ ⋅ w → I ) ]
\begin{array}{c}{\max _{\mathbf{W}, \mathbf{W}^{\prime}} p\left(\operatorname{word}_{O} | \text { word }_{I}\right)=\max _{\mathbf{W}, \mathbf{W}^{\prime}} y_{j^{*}}=\max _{\mathbf{W}, \mathbf{W}^{\prime}} \log \frac{\exp \left(\overrightarrow{\mathbf{w}}_{j^{*}}^{\prime} \cdot \overrightarrow{\mathbf{w}}_{I}\right)}{\sum_{i=1}^{V} \exp \left(\overrightarrow{\mathbf{w}}_{i}^{\prime} \cdot \overrightarrow{\mathbf{w}}_{I}\right)}} \\ {=\max _{\mathbf{W}, \mathbf{W}^{\prime}}\left[\overrightarrow{\mathbf{w}}_{j^{*}}^{\prime} \cdot \overrightarrow{\mathbf{w}}_{I}-\log \sum_{i=1}^{V} \exp \left(\overrightarrow{\mathbf{w}}_{i}^{\prime} \cdot \overrightarrow{\mathbf{w}}_{I}\right)\right]}\end{array}
max W , W ′ p ( w o r d O ∣ word I ) = max W , W ′ y j ∗ = max W , W ′ log ∑ i = 1 V exp ( w i ′ ⋅ w I ) exp ( w j ∗ ′ ⋅ w I ) = max W , W ′ [ w j ∗ ′ ⋅ w I − log ∑ i = 1 V exp ( w i ′ ⋅ w I ) ] w → I \overrightarrow{\mathbf{w}}_I w I word I \operatorname{word}_I w o r d I
考虑到u j = w → j ′ ⋅ w → I u_{j}=\overrightarrow{\mathbf{w}}_{j}^{\prime} \cdot \overrightarrow{\mathbf{w}}_{I} u j = w j ′ ⋅ w I E = − log p ( word O ∣ word I ) = − [ w → j ∗ ′ ⋅ w → I − log ∑ i = 1 V exp ( w → i ′ ⋅ w → I ) ] = − [ u j ∗ − log ∑ i = 1 V exp ( u i ) ] \begin{array}{c}{E=-\log p\left(\operatorname{word}_{O} | \operatorname{word}_{I}\right)=-\left[\overrightarrow{\mathbf{w}}_{j^{*}}^{\prime} \cdot \overrightarrow{\mathbf{w}}_{I}-\log \sum_{i=1}^{V} \exp \left(\overrightarrow{\mathbf{w}}_{i}^{\prime} \cdot \overrightarrow{\mathbf{w}}_{I}\right)\right]} \\ {=-\left[u_{j^{*}}-\log \sum_{i=1}^{V} \exp \left(u_{i}\right)\right]}\end{array} E = − log p ( w o r d O ∣ w o r d I ) = − [ w j ∗ ′ ⋅ w I − log ∑ i = 1 V exp ( w i ′ ⋅ w I ) ] = − [ u j ∗ − log ∑ i = 1 V exp ( u i ) ] min E \operatorname{min} E m i n E
定义t j = I ( j = j ∗ ) t_j = \mathbb{I}(j=j^{*}) t j = I ( j = j ∗ ) word O \operatorname{word}_O w o r d O e j = ∂ E ∂ u j = y j − t j e_{j}=\frac{\partial E}{\partial u_{j}}=y_{j}-t_{j} e j = ∂ u j ∂ E = y j − t j
当j = j ∗ j=j^* j = j ∗ e j = y j − 1 e_j=y_j-1 e j = y j − 1 y j y_j y j
当j ≠ j ∗ j \neq j^{*} j ̸ = j ∗ e j = y j e_j=y_j e j = y j
根据：u j = w → j ′ ⋅ h → → ∂ u j ∂ w → j ′ = h → u_{j}=\overrightarrow{\mathbf{w}}_{j}^{\prime} \cdot \overrightarrow{\mathbf{h}} \quad \rightarrow \quad \frac{\partial u_{j}}{\partial \overrightarrow{\mathbf{w}}_{j}^{\prime}}=\overrightarrow{\mathbf{h}} u j = w j ′ ⋅ h → ∂ w j ′ ∂ u j = h ∂ E ∂ w → j ′ = ∂ E ∂ u j × ∂ u j ∂ w → j ′ = e j h → \frac{\partial E}{\partial \overrightarrow{\mathbf{w}}_{j}^{\prime}}=\frac{\partial E}{\partial u_{j}} \times \frac{\partial u_{j}}{\partial \overrightarrow{\mathbf{w}}_{j}^{\prime}}=e_{j} \overrightarrow{\mathbf{h}} ∂ w j ′ ∂ E = ∂ u j ∂ E × ∂ w j ′ ∂ u j = e j h w j → ′ \overrightarrow{\mathbf{w}_j}^{\prime} w j ′ w → j ′ ( n e w ) = w → j ′ ( o l d ) − η e j h → \overrightarrow{\mathbf{w}}_{j}^{\prime(n e w)}=\overrightarrow{\mathbf{w}}_{j}^{\prime(o l d)}-\eta e_{j} \overrightarrow{\mathbf{h}} w j ′ ( n e w ) = w j ′ ( o l d ) − η e j h
当估计过量（e j > 0 → y j > t j e_{j}>0 \rightarrow y_{j}>t_{j} e j > 0 → y j > t j w j → ′ \overrightarrow{\mathbf{w}_j}^{\prime} w j ′ h → \overrightarrow{\mathbf{h}} h
当估计不足(e j < 0 → y j < t j e_{j}<0 \rightarrow y_{j}<t_{j} e j < 0 → y j < t j w j → ′ \overrightarrow{\mathbf{w}_j}^{\prime} w j ′ h → \overrightarrow{\mathbf{h}} h
当y j ≃ t j y_{j} \simeq t_{j} y j ≃ t j
定义：E H → = ∂ E ∂ h → = ( ∂ u → ∂ h → ) T ∂ E ∂ u → \overrightarrow{\mathbf{E H}}=\frac{\partial E}{\partial \overrightarrow{\mathbf{h}}}=\left(\frac{\partial \overrightarrow{\mathbf{u}}}{\partial \overrightarrow{\mathbf{h}}}\right)^{T} \frac{\partial E}{\partial \overrightarrow{\mathbf{u}}} E H = ∂ h ∂ E = ( ∂ h ∂ u ) T ∂ u ∂ E u → = W ′ T h → → ( ∂ u → ∂ h → ) T = W ′ \overrightarrow{\mathbf{u}}=\mathbf{W}^{\prime T} \overrightarrow{\mathbf{h}} \quad \rightarrow \quad\left(\frac{\partial \overrightarrow{\mathbf{u}}}{\partial \overrightarrow{\mathbf{h}}}\right)^{T}=\mathbf{W}^{\prime} u = W ′ T h → ( ∂ h ∂ u ) T = W ′ E H → = W ′ e → = ∑ j = 1 V e j w → j ′ \overrightarrow{\mathbf{E H}}=\mathbf{W}^{\prime} \overrightarrow{\mathbf{e}}=\sum_{j=1}^{V} e_{j} \overrightarrow{\mathbf{w}}_{j}^{\prime} E H = W ′ e = j = 1 ∑ V e j w j ′ E H → \overrightarrow{\mathbf{EH}} E H V \mathbb{V} V e j e_j e j
考虑到h → = W T x → \overrightarrow{\mathbf{h}}=\mathbf{W}^{T} \overrightarrow{\mathbf{x}} h = W T x ∂ E ∂ w k , i = ∂ E ∂ h i × ∂ h i ∂ w k , i = E H i × x k \frac{\partial E}{\partial w_{k, i}}=\frac{\partial E}{\partial h_{i}} \times \frac{\partial h_{i}}{\partial w_{k, i}}=E H_{i} \times x_{k} ∂ w k , i ∂ E = ∂ h i ∂ E × ∂ w k , i ∂ h i = E H i × x k x → \overrightarrow{\mathbf{x}} x one-hot编码，所以它只有一个分量非零，因此∂ E ∂ W \frac{\partial E}{\partial \mathbf{W}} ∂ W ∂ E E H → \overrightarrow{\mathbf{EH}} E H w → I ( n e w ) = w → I ( o l d ) − η E H → \overrightarrow{\mathbf{w}}_{I}^{(n e w)}=\overrightarrow{\mathbf{w}}_{I}^{(o l d)}-\eta \overrightarrow{\mathbf{E} \mathbf{H}} w I ( n e w ) = w I ( o l d ) − η E H w → I \overrightarrow{\mathbf{w}}_{I} w I
考虑更新行的第k列，则：w I , k ( n e w ) = w I , k ( o l d ) − η ∑ j = 1 V e j w j , k ′ w_{I, k}^{(n e w)}=w_{I, k}^{(o l d)}-\eta \sum_{j=1}^{V} e_{j} w_{j, k}^{\prime} w I , k ( n e w ) = w I , k ( o l d ) − η j = 1 ∑ V e j w j , k ′ y j ≃ t j y_{j} \simeq t_{j} y j ≃ t j y j y_j y j t j t_j t j
当给定许多训练样本，每个样本由两个单词组成，上述更新不断进行，更新的效果不断积累。
根据单词的共现效果，输出向量与输入向量相互作用并达到平滑。
输出向量w → ′ \overrightarrow{\mathbf{w}}^{\prime} w ′ w → I : w → j ′ ( n e w ) = w → j ′ ( o l d ) − η e j h → \overrightarrow{\mathbf{w}}_{I} : \quad \overrightarrow{\mathbf{w}}_{j}^{\prime(n e w)}=\overrightarrow{\mathbf{w}}_{j}^{\prime(o l d)}-\eta e_{j} \overrightarrow{\mathbf{h}} w I : w j ′ ( n e w ) = w j ′ ( o l d ) − η e j h h → \overrightarrow{\mathbf{h}} h w → I \overrightarrow{\mathbf{w}}_I w I
输入向量w → I \overrightarrow{\mathbf{w}}_I w I w → ′ : w → I ( n e w ) = w → I ( o l d ) − η E H → \overrightarrow{\mathbf{w}}^{\prime} : \quad \overrightarrow{\mathbf{w}}_{I}^{(n e w)}=\overrightarrow{\mathbf{w}}_{I}^{(o l d)}-\eta \overrightarrow{\mathbf{E} \mathbf{H}} w ′ : w I ( n e w ) = w I ( o l d ) − η E H E H → = ∑ j = 1 V e j w → j ′ \overrightarrow{\mathbf{E H}}=\sum_{j=1}^{V} e_{j} \overrightarrow{\mathbf{w}}_{j}^{\prime} E H = ∑ j = 1 V e j w j ′ V \mathbb{V} V e j e_j e j
平衡的速度与效果取决于单词的共现分布，以及学习率。
CBOW模型是根据前几个单词（即上下文）来预测下一个单词，而Skip-Gram模型是根据一个单词来预测前几个单词（即上下文）。
在CBOW模型中：
同一个单词的表达（即输入向量w → I \overrightarrow{\mathbf{w}}_I w I W \mathbf{W} W
同一个单词的输出向量w → O ′ \overrightarrow{\mathbf{w}}_{O}^{\prime} w O ′
在Skip-Gram模型中：
同一个单词的表达（即输入向量w → O ′ \overrightarrow{\mathbf{w}}_{O}^{\prime} w O ′ W ′ \mathbf{W}^{\prime} W ′
同一个单词的输入向量w → I \overrightarrow{\mathbf{w}}_{I} w I
Skip-Gram网络模型如下。其中：
网络输入x → = ( x 1 , x 2 , ⋯  , x V ) T ∈ R V \overrightarrow{\mathbf{x}}=\left(x_{1}, x_{2}, \cdots, x_{V}\right)^{T} \in \mathbb{R}^{V} x = ( x 1 , x 2 , ⋯ , x V ) T ∈ R V
网络输出y → 1 , y → 2 , ⋯  , y → C \overrightarrow{\mathbf{y}}_{1}, \overrightarrow{\mathbf{y}}_{2}, \cdots, \overrightarrow{\mathbf{y}}_{C} y 1 , y 2 , ⋯ , y C y → c = ( y 1 c , y 2 c , ⋯  , y V c ) T ∈ R V \overrightarrow{\mathbf{y}}_{c}=\left(y_{1}^{c}, y_{2}^{c}, \cdots, y_{V}^{c}\right)^{T} \in \mathbb{R}^{V} y c = ( y 1 c , y 2 c , ⋯ , y V c ) T ∈ R V
对于网络中的每个输出y c → \overrightarrow{y_c} y c W ′ W^{\prime} W ′
Skip-Gram网络模型中，设网络第c个输出的第j个分量为u j c = w → j ′ ⋅ h → u_{j}^{c}=\overrightarrow{\mathbf{w}}_{j}^{\prime} \cdot \overrightarrow{\mathbf{h}} u j c = w j ′ ⋅ h y j c = p ( word j c ∣ x → ) = exp ( u j c ) ∑ k = 1 V exp ( u k c ) ; c = 1 , 2 , ⋯  , C ; j = 1 , 2 , ⋯  , V
y_{j}^{c}=p\left(\operatorname{word}_{j}^{c} | \overrightarrow{\mathbf{x}}\right)=\frac{\exp \left(u_{j}^{c}\right)}{\sum_{k=1}^{V} \exp \left(u_{k}^{c}\right)} ; \quad c=1,2, \cdots, C ; \quad j=1,2, \cdots, V
y j c = p ( w o r d j c ∣ x ) = ∑ k = 1 V exp ( u k c ) exp ( u j c ) ; c = 1 , 2 , ⋯ , C ; j = 1 , 2 , ⋯ , V y j c y_{j}^{c} y j c V \mathbb{V} V word j \operatorname{word}_j w o r d j
因为W ′ W^{\prime} W ′ u → c = ( u 1 c , u 2 c , ⋯  , u V c ) T \overrightarrow{\mathbf{u}}_{c}=\left(u_{1}^{c}, u_{2}^{c}, \cdots, u_{V}^{c}\right)^{T} u c = ( u 1 c , u 2 c , ⋯ , u V c ) T
并不是网络每个输出中，得分最高的为预测单词。因为每个输出中，概率分布都相同，即y → 1 = y → 2 = ⋯ = y → C \overrightarrow{\mathbf{y}}_{1}=\overrightarrow{\mathbf{y}}_{2}=\cdots=\overrightarrow{\mathbf{y}}_{C} y 1 = y 2 = ⋯ = y C
Skip-Grame网络的目标是：网络的多个输出之间的联合概率最大。
假设输入为单词word I \operatorname{word}_I w o r d I word O 1 , word O 2 , ⋯  , \operatorname{word}_{O_{1}}, \operatorname{word}_{O_{2}}, \cdots, w o r d O 1 , w o r d O 2 , ⋯ , o C o_{C} o C E = − log p ( word O 1 , word O 3 , ⋯  , word O C ∣ word I ) = − log ∏ c = 1 C exp ( u j c ∗ e ) ∑ k = 1 V exp ( u k c )
E=-\log p\left(\operatorname{word}_{O_{1}}, \operatorname{word}_{O_{3}}, \cdots, \operatorname{word}_{O_{C}} | \text { word }_{I}\right)=-\log \prod_{c=1}^{C} \frac{\exp \left(u_{j_{c}^{*}}^{e}\right)}{\sum_{k=1}^{V} \exp \left(u_{k}^{c}\right)}
E = − log p ( w o r d O 1 , w o r d O 3 , ⋯ , w o r d O C ∣ word I ) = − log c = 1 ∏ C ∑ k = 1 V exp ( u k c ) exp ( u j c ∗ e ) j 1 ∗ , j 2 ∗ , ⋯  , j C ∗ j_{1}^{*}, j_{2}^{*}, \cdots, j_{C}^{*} j 1 ∗ , j 2 ∗ , ⋯ , j C ∗ V \mathbb{V} V u k = u k c = w → k ′ ⋅ h → u_{k}=u_{k}^{c}=\overrightarrow{\mathbf{w}}_{k}^{\prime} \cdot \overrightarrow{\mathbf{h}} u k = u k c = w k ′ ⋅ h E = − ∑ c = 1 C u j c c + C log ∑ k = 1 V exp ( u k )
E=-\sum_{c=1}^{C} u_{j_{c}}^{c}+C \log \sum_{k=1}^{V} \exp \left(u_{k}\right)
E = − c = 1 ∑ C u j c c + C log k = 1 ∑ V exp ( u k )
