EleutherAI ML Scalability & Performance Reading Group

My annotated papers, slides, and meeting recordings for the EleutherAI ML Scalability & Performance research paper reading group.

Sessions:

1. Intro to GPU architecture, CUDA, NCCL, and common ML performance bottlenecks
2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
3. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
4. Ring Attention with Blockwise Transformers for Near-Infinite Context Length
5. Efficient Memory Management for Large Language Model Serving with PagedAttention