NanoGPT (124M) in 5 minutes

Offical Repo for "Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale"

A native PyTorch Library for large model training

open-source multimodal large language model that can hear, talk while thinking. Featuring real-time end-to-end speech input and streaming audio output conversational capabilities.

PyTorch bindings for CUTLASS grouped GEMM.

Advancing LLM with Diverse Coding Capabilities

A framework for the evaluation of autoregressive code generation language models.

Development repository for the Triton language and compiler

Zero Bubble Pipeline Parallelism

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

Triton-based implementation of Sparse Mixture of Experts.

Ring attention implementation with flash attention

The Open English WordNet

Machine Learning Engineering Guides and Tools

All-in-one text de-duplication

An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

distributed trainer for LLMs

Rectified Rotary Position Embeddings

Fast and memory-efficient exact attention

Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning

code samples for the goodreads datasets

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.