FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

Authors: Dao Year: 2023 ArXiv/Link: https://arxiv.org/abs/2307.08691

Summary

Further optimization of FlashAttention with improved parallelism and work partitioning strategies, achieving additional speedups.

Key Concepts

• Parallel computation
• Work partitioning
• Memory efficiency
• GPU optimization
• Block-wise tiling

Impact

Continued efficiency improvements for transformer attention computation

Category

Attention Mechanisms

---