名称：A Contextualized Temporal Attention Mechanism for Sequential Recommendation
大多数顺序推荐系统的算法关注于对用户行为的结构，但是却忽略了时间和上下文的信息。作者认为用户过去的行为对现在行为的影响会随着时间和上下文发生变化。
用户空间U ，大小为U；物品空间V ，大小为V。C = { S 1 , S 2 , . . . , S U } \mathbf{C} = \{ \mathbf{S}^1, \mathbf{S}^2,..., \mathbf{S}^U\} C = { S 1 , S 2 , . . . , S U } S u = { ( t 1 u , s 1 u ) , ( t 2 u , s 2 u ) , . . . } \mathbf{S}^u = \{ (t^u_1,s^u_1), (t^u_2, s^u_2), ... \} S u = { ( t 1 u , s 1 u ) , ( t 2 u , s 2 u ) , . . . } t j u t^u_j t j u s j u s^u_j s j u
模型共分为三个部分：content-based attention，temporal kernels和contextualized mixture。
输入 ：原始输入包括窗口大小为L的用户历史记录{ ( t i , s i ) i = 1 L } \{(t_i,s_i)^L_{i=1}\} { ( t i , s i ) i = 1 L } t L + 1 t_{L+1} t L + 1 X = [ s 1 , . . . , s L ] ⋅ E i n p u t ∈ R L × d i n \mathbf{X} = [s_1, ..., s_L] \cdot E_{input} \in R^{L \times d_{in}} X = [ s 1 , . . . , s L ] ⋅ E i n p u t ∈ R L × d i n T = [ t L + 1 − t 1 , . . . , t L + 1 − t L ] ∈ R L × 1 \mathbf{T} = [t_{L+1} - t_1,...,t_{L+1} - t_L] \in R^{L \times 1} T = [ t L + 1 − t 1 , . . . , t L + 1 − t L ] ∈ R L × 1
三层模型可以表示为：α = M α ( X ) → β = M β ( T ) → γ = M γ ( X , β , α ) \mathbf{\alpha} = M^{\alpha}(\mathbf{X}) \rightarrow \mathbf{\beta} = M^{\beta}(\mathbf{T}) \rightarrow \mathbf{\gamma} = M^{\gamma}(\mathbf{X, \beta, \alpha}) α = M α ( X ) → β = M β ( T ) → γ = M γ ( X , β , α ) M α M^{\alpha} M α α ∈ R L × 1 \alpha \in R^{L \times 1} α ∈ R L × 1 M β M^{\beta} M β T 通过K temporal kernels计算每个输入的temporal权重β ∈ R L × K \beta \in R^{L \times K} β ∈ R L × K M γ M^{\gamma} M γ X 中提取context信息，并结合前两个阶段获得的α \alpha α β \beta β γ ∈ R L × 1 \gamma \in R^{L \times 1} γ ∈ R L × 1 x ^ L + 1 = F o u t ( γ T ⋅ X ) ∈ R d o u t × 1 \hat{x}_{L+1} = F^{out} (\gamma ^T \cdot \mathbf{X}) \in R^{d_{out} \times 1} x ^ L + 1 = F o u t ( γ T ⋅ X ) ∈ R d o u t × 1 F o u t F^{out} F o u t
α \alpha α d l d_l d l d h d_h d h d a d_a d a H d l \mathbf{H}^{d_l} H d l H j \mathbf{H}^j H j z i j = A t t e n t i o n ( H j W i Q , H j W i K , H j W i V ) z^j_i = Attention(\mathbf{H}^j W^Q_i, \mathbf{H}^j W^K_i, \mathbf{H}^j W^V_i) z i j = A t t e n t i o n ( H j W i Q , H j W i K , H j W i V ) W i Q , W i K , W i V ∈ R d a × d a / d h W^Q_i, W^K_i, W^V_i \in R^{d_a \times d_a / d_h} W i Q , W i K , W i V ∈ R d a × d a / d h A t t e n t i o n ( Q , K , V ) = s o f t m a x ( Q ⋅ K T d a / d h ) V Attention(\mathbf{Q,K,V}) = softmax(\frac{\mathbf{Q \cdot K^T}}{\sqrt{d_a/d_h}})\mathbf{V} A t t e n t i o n ( Q , K , V ) = s o f t m a x ( d a / d h Q ⋅ K T ) V z i d z^d_i z i d Z j = [ z 1 d , . . . , z d h d ] ⋅ W O Z^j = [z^d_1, ...,z^d_{d_h}] \cdot W^O Z j = [ z 1 d , . . . , z d h d ] ⋅ W O W O ∈ R d a × d a W^O \in R^{d_a \times d_a} W O ∈ R d a × d a H j + 1 = L N ( H j + F j ( Z j ) ) \mathbf{H}^{j+1} = LN(\mathbf{H}^j + F^j(Z^j)) H j + 1 = L N ( H j + F j ( Z j ) ) F j F^j F j
一般attention block到此为止（如BERT4Rec和SASRec）。因为本文中使用self-attention结构是为了对每个输入计算得到基于内容的权重估计，所有需要使用最后一层隐藏层状态H j + 1 H^{j+1} H j + 1 x L T x^T_L x L T ：α = s o f t m a x ( ( H j + 1 W 0 Q ) ( x L W 0 K ) T d i n ) \alpha = softmax(\frac{(\mathbf{H}^{j+1} W^Q_0)(x_L W^K_0)^T}{\sqrt{d_{in}}}) α = s o f t m a x ( d i n ( H j + 1 W 0 Q ) ( x L W 0 K ) T )
β \beta β ϕ ( ⋅ ) : R L → R L \phi (\cdot): R^L \rightarrow R^L ϕ ( ⋅ ) : R L → R L
ϕ ( T ) = a e − T + b \phi(\mathbf{T}) = ae^{-\mathbf{T}} + b ϕ ( T ) = a e − T + b
ϕ ( T ) = − a l o g ( 1 + T ) + b \phi(\mathbf{T}) = -alog(1+\mathbf{T}) + b ϕ ( T ) = − a l o g ( 1 + T ) + b
ϕ ( T ) = − a l T + b \phi(\mathbf{T}) = -al \mathbf{T} + b ϕ ( T ) = − a l T + b
ϕ ( T ) = 1 \phi(\mathbf{T}) = 1 ϕ ( T ) = 1 { ϕ ( ⋅ ) 1 , . . . , ϕ ( ⋅ ) K } \{\phi(\cdot)^1,...,\phi(\cdot)^K\} { ϕ ( ⋅ ) 1 , . . . , ϕ ( ⋅ ) K } T 转为K个temporal权重集合：β = [ ϕ 1 ( T ) , . . . , ϕ K ( T ) ] \beta = [\phi^1(\mathbf{T}),...,\phi^K(\mathbf{T})] β = [ ϕ 1 ( T ) , . . . , ϕ K ( T ) ]
γ \gamma γ C = B i − R N N ( X ) ⊕ C a t t r ∈ R L × d r \mathbf{C} = Bi-RNN(\mathbf{X}) ⊕ C_attr \in R^{L \times d_r} C = B i − R N N ( X ) ⊕ C a t t r ∈ R L × d r C a t t r C_attr C a t t r p i ( k ∣ c i ) p_i(k|c_i) p i ( k ∣ c i ) ϕ k ( ⋅ ) \phi^k(\cdot) ϕ k ( ⋅ ) F γ F^{\gamma} F γ R K R^K R K P ( ⋅ ∣ C ) = s o f t m a x ( F γ ( C ) ) \mathbf{P(\cdot | C)} = softmax(F^{\gamma}(\mathbf{C})) P ( ⋅ ∣ C ) = s o f t m a x ( F γ ( C ) ) β c = β ⋅ P ( ⋅ ∣ C ) γ = s o f t m a x ( α β c )
\mathbf{\beta ^ c = \beta \cdot P(\cdot | C)} \\
\gamma = softmax(\alpha \beta^c)
β c = β ⋅ P ( ⋅ ∣ C ) γ = s o f t m a x ( α β c )
Loss Functions ：
模型训练的损失函数：L N L L = − l o g e r i ∑ j ∈ N s e r j L_{NLL} = -log \frac{e^{r_i}}{\sum_{j \in N_s} e^{r_j}} L N L L = − l o g ∑ j ∈ N s e r j e r i v ∈ V v \in V v ∈ V N s ⊆ V N_s \subseteq V N s ⊆ V
基于排序的指标（优化推荐列表的质量）Bayesian Personalized Ranking（BPR）loss：L B P R = − 1 N s ∑ j ∈ N s l o g σ ( r i − r j ) L_{BPR} = -\frac{1}{N_s}\sum_{j \in N_s} log \sigma(r_i - r_j) L B P R = − N s 1 j ∈ N s ∑ l o g σ ( r i − r j ) σ ( x ) = 1 1 + e ( − x ) \sigma(x) = \frac{1}{1+e^(-x)} σ ( x ) = 1 + e ( − x ) 1
TOP1 Loss：L T O P 1 = 1 N s ∑ j ∈ N s σ ( r j − r i ) + σ ( r j 2 ) L_{TOP1} = \frac{1}{N_s} \sum{j \in N_s} \sigma(r_j - r_i) + \sigma(r^2_j) L T O P 1 = N s 1 ∑ j ∈ N s σ ( r j − r i ) + σ ( r j 2 )
Regularization：每经过一个feed forward层后都使用dropout避免过拟合，dropout率为0.2。
XING：job postings
UserBehavior Alibaba：
