Welcome to STATS 305B! Officially, this course is called Applied Statistics II. Unofficially, I call it Models and Algorithms for Discrete Data. We will cover models ranging from generalized linear models to sequential latent variable models, autoregressive models, and transformers. On the algorithm side, we will cover a few techniques for convex optimization, as well as approximate Bayesian inference algorithms like MCMC and variational inference. I think the best way to learn these concepts is to implement them from scratch, so coding will be a big focus of this course. By the end of the course, you'll have a strong grasp of classical techniques as well as modern methods for modeling discrete data.
Instructor: Scott Linderman
TAs: Amber Hu and Michael Salerno
Term: Winter 2024-25
Time: Monday and Wednesday, 1:30-2:50pm
Location: Sequoia Hall, Room 200, Stanford University
Office Hours
- Scott: Wed 10-11am, Wu Tsai Neurosciences Institute, 2nd Floor in the Theory Center
- Michael: Thu, 5-7pm, location TBD
- Amber: Fri 1:30-3:30pm, Sequoia library (Rm 105), starting 1/17
Students should be comfortable with undergraduate probability and statistics as well as multivariate calculus and linear algebra. This course will emphasize implementing models and algorithms, so coding proficiency with Python is required. (HW0: Python Primer will help you get up to speed.)
This course will draw from a few textbooks:
- Agresti, Alan. Categorical Data Analysis, 2nd edition. John Wiley & Sons, 2002. link
- Gelman, Andrew, et al. Bayesian Data Analysis, 3rd edition. Chapman and Hall/CRC, 2013. link
- Bishop, Christopher. Pattern Recognition and Machine Learning. Springer, 2006. link
We will also cover material from research papers.
Please note that this is a tentative schedule. It may change slightly depending on our pace.
| Date | Topic | Slides | Additional Reading |
|---|---|---|---|
| Mon, Jan 6, 2025 | Basics of Probability and Statistics and Contingency Tables HW0 Released |
download | {cite:p}agresti2002categorical Ch. 1-3 |
| Wed, Jan 8, 2025 | Logistic Regression | download | {cite:p}agresti2002categorical Ch. 4-5 |
| Fri, Jan 10, 2025 | HW0 Due | ||
| Mon, Jan 13, 2025 | Exponential Families HW1 Released |
{cite:p}agresti2002categorical Ch. 4-5 |
|
| Wed, Jan 25, 2025 | Generalized Linear Models | {cite:p}agresti2002categorical Ch. 6 |
|
| Mon, Jan 20, 2025 | MLK Day. No class | ||
| Wed, Feb 22, 2025 | L1-regularized GLMs | {cite:p}friedman2010regularization and {cite:p}lee2014proximal |
|
| Fri, Jan 24, 2025 | HW1 Due | ||
| Mon, Jan 27, 2025 | Bayesian Inference HW2 Released |
{cite:p}gelman1995bayesian Ch. 1 |
|
| Wed, Jan 29, 2025 | Markov Chain Monte Carlo | ||
| Mon, Feb 3, 2025 | Variational Inference | ||
| Wed, Feb 5, 2025 | Midterm Exam (in class) | ||
| Mon, Feb 10, 2025 | Mixture Models and EM HW2 Due; HW3 Released |
{cite:p}bishop2006pattern Ch. 9 |
|
| Wed, Feb 12, 2025 | Hidden Markov Models | {cite:p}bishop2006pattern Ch. 13 |
|
| Mon, Feb 17, 2025 | Presidents' Day. No class | ||
| Wed, Feb 19, 2025 | Linear Dynamical Systems | ||
| Fri, Feb 21, 2025 | HW3 Due | ||
| Mon, Feb 24, 2025 | Variational Autoencoders HW4 Released |
{cite:p}kingma2019introduction Ch.1-2 |
|
| Wed, Feb 26, 2025 | Tranformers | {cite:p}turner2023introduction |
|
| Mon, Mar 3, 2025 | State Space Layers (S4, S5, Mamba) | {cite:p}smith2023simplified and {cite:p}gu2023mamba |
|
| Wed, Mar 5, 2025 | Denoising Diffusion Models | {cite:p}turner2024denoising |
|
| Mon, Mar 10, 2025 | Point Processes | ||
| Wed, Mar 12, 2025 | Wrap Up | ||
| Fri, Mar 14, 2025 | HW4 Due |
There will be 5 assignments due roughly every other Friday. They will not be equally weighted. The first one is just a primer to get you up to speed; the last one will be a bit more substantial than the rest.
- Homework 0: Python Primer
- Released Mon, Jan 6, 2025
- Due Fri, Jan 10, 2025 at 11:59pm
We will allow 5 late days to be used as needed throughout the quarter.
-
Midterm Exam: In class on TBD
- You may bring a cheat sheet covering one side of an 8.5x11" piece of paper
-
Final Exam: On TBD in Room TBD
- You may bring a cheat sheet covering both sides of an 8.5x11" piece of paper
Tentatively:
| Assignment | Percentage |
|---|---|
| HW 0 | 5% |
| HW 1-3 | 15% each |
| HW 4 | 20% |
| Midterm | 10% |
| Final | 15% |
| Participation | 5% |