Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization

Keyphrases

• Self-attention 100%
• Memory Complexity 33%
• Linear Transformer 33%
• Sequence Modeling 16%
• Quadratic Complexity 16%
• Training Accuracy 16%
• Hashing Trick 16%
• Linear Hashing 16%
• Attention Connection 16%
• Efficient Transformer 16%
• Quadratic Optimization 16%
• Sparse Attention 16%
• Linear Memory 16%
• Non-autoregressive 16%
• Machine Translation 16%
• Adaptive Momentum 16%

Engineering

• Linear Transformer 100%
• Adaptive Strategy 50%