Paper (引用量)

源码实现

复杂度

AutoRegressive

Main Idea

Generating Wikipedia by Summarizing Long Sequences[1] (208)

memory-compressed-attention[2]

compresses key and value + blocked attention

CBAM: Convolutional Block Attention Module[3] (677)

attention-module[4] 

combines the SE attention with a per pixel(local) weight

CCNet: Criss-Cross Attention for Semantic Segmentation[5] (149)

CCNet[6]

each pixel attends to its row and column simultaneously

Efficient Attention: Attention with Linear Complexities[7] (2)

efficient-attention[8]

Softmax(Q)*(Softmax(K^T)*V)

Star-Transformer[9] (24)

fastNLP[10]

uses a relay(global) node and attends to/from that node

Generating Long Sequences with Sparse Transformers[11] (139)

torch-blocksparse[12]

sparse block based attention

GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond[13] (96)

GCNet[14] 

squeeze and excitation with an attention pooling (instead of a GAP)

SCRAM: Spatially Coherent Randomized Attention Maps[15] (1)

-

uses PatchMatch to find close keys

Interlaced Sparse Self-Attention for Semantic Segmentation[16] (13)

IN_PAPER

combination of a short length and then long range(dilated) attention

Permutohedral Attention Module for Efficient Non-Local Neural Networks[17] (2)

Permutohedral_attention_module[18]

uses permutohedral lattice approximation algorithm to approximate the attention output

Large Memory Layers with Product Keys[19] (28)

XLM[20]

search for nearest neighbor keys

Expectation-Maximization Attention Networks for Semantic Segmentation[21] (38)

EMANet[22]

applys expectation maximization to cluster keys into k clusters

Compressive Transformers for Long-Range Sequence Modelling[23] (20)

compressive-transformer-pytorch[24]

compresses distant tokens instead of just stop_grad() ing them, more efficient version of transformerXL

BP-Transformer: Modelling Long-Range Context via Binary Partitioning[25] (8)

BPT[26]

attends to distant tokens coarsely and attends to close tokens in a more fine-grained manner

Axial Attention in Multidimensional Transformers[27] (5)

axial-attention[28]

apply attention on each axis separately

Reformer: The Efficient Transformer[29] (69)

trax[30]

uses LSH to find close keys

Transformer on a Diet[31] (2)

transformer-on-diet[32]

dilated transformer like wavenet

Sparse Sinkhorn Attention[33] (4)

sinkhorn-transformer[34]

uses a cost matrix to limit attention between buckets

SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection[35] (1)

-

learns the q, k connections == dynamically creates a sparse attention matrix

Efficient Content-Based Sparse Attention with Routing Transformers[36] (11)

routing-transformer[37]

computes attention with same-cluster tokens (computed by online k-means)

Longformer: The Long-Document Transformer[38] (15)

longformer[39]

global + blocked attention

Neural Architecture Search for Lightweight Non-Local Networks[40] (2)

AutoNL[41]

computes Q(KV) and also down samples q, k, v both in spatial and channel dimensions

ETC: Encoding Long and Structured Data in Transformers[42] (2)

-

combines global attention (star transformer with multiple global tokens) with local attention

Multi-scale Transformer Language Models[43] (1)

IN_PAPER

UNet like + retina attetion is something close to BP-Transformer

Synthesizer: Rethinking Self-Attention in Transformer Models[44] (5)

-

does not compute pairwise interactions

Jukebox: A Generative Model for Music[45] (9)

jukebox[46]

better attention patterns from Sparse Transformer

GMAT: Global Memory Augmentation for Transformers[47] (0)

gmat[48]

adds global tokens

Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers[49] (0)

google-research[50]

calculate an unbiased stochastic approximation of the attention matrix

Hand-crafted Attention is All You Need? A Study of Attention on Self-supervised Audio Transformer[51] (0)

-

does not compute pairwise interactions and uses fixed mask patters

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention[52] (1)

fast-transformers[53]

uses phi(q)(phi(k)v) and also improves the sequential sampling step

Linformer: Self-Attention with Linear Complexity[54] (3)

linformer-pytorch[55]

project key and value from nd

Real-time Semantic Segmentation with Fast Attention[56] (0)

-

l2_norm(q)*(l2_norm(k)*v)

Fast Transformers with Clustered Attention[57] (0)

fast-transformers[58]

groups queries together with LSH

Big Bird: Transformers for Longer Sequences[59] (0)

-

ETC with random connections