Syllabus
Any sufficiently advanced technology is indistinguishable from magic. Thanks, Arthur C. Clarke (1917-2008)
Welcome to SITP, the online course for a self-guided study which provides an accelerated introduction to deep learning networks and deep learning systems.
All text (every single word) has been written by a human, for a human, as this book is not a reference book, but rather a cognitive helmet of understanding for a human. You are highly encouraged if not expected to read with an LLM, but be aware (that is, to beware) of how much you cognitively offload and outsource your understanding.
- Week 1. Hello Tensor π
- Week 2. I Perceive You π
- Week 3. MetaCircularEvaLuAtor! β©οΈ
- Week 4. GPUs Go Brrr ποΈ
- Week 5. King - Man + Queen = π©
- Week 6. Platoβs Cave π¦π¨πͺπ
- Week 7. AutoNabla (β§ββ¦)
- Week 8. Donβt Think π€
- Week 9. Just Do πββοΈ
- Week 10. Compilers, Theyβre InTense π°
Week 1. Hello Tensor π
- Watch βΆοΈ Lecture 1
- Read π Chapter 1.1 From Certain to Uncertain Knowledge,
- Read π Chapter 1.2 Next Token Prediction is Classification and Compression
- Write Lab 1. Elements of Tensors
Optional Reading
- Read An Example of Statistical Investigation of the Text Eugene Onegin Concerning the Connection of Samples in Chains (Markov 1913)
- Read A Mathematical Theory of Communication (Shannon 1948)
- Read A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence (McCarthy et al. 1955)
Week 2. I Perceive You π
- Watch βΆοΈ Lecture 2
- Read π Chapter 1.3 Parameterizing Classification with Discriminant Functions
- Read π Chapter 1.4 Parameterizing Classification with Probabilistic Models
- Write Lab 2.
Optional Reading
- Read The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain (Rosenblatt 1958)
- Read Statistical Modeling: The Two Cultures (Breiman 2001)
- Read The Epic Story of Maximum Likelihood (Stigler 2008)
Week 3. MetaCircularEvaLuAtor! β©οΈ
- Watch βΆοΈ Lecture 3
- Read π Chapter 2.1 Virtualizing Shapes with Strides
- Read π Chapter 2.2 Accelerating Basic Linear Algebra on CPUs
- Read π Chapter 2.3 From Performance Primitives to Linear Solvers
- Write βοΈ Lab 3. Tensor++ and Matrix Decompositions
Optional Reading
- Read A Set of Level 3 Basic Linear Algebra Subprograms (Dongarra et al. 1990)
- Read Anatomy of High-Performance Matrix Multiplication (Goto and van de Geijn 2008a)
- Read The History of Fortran I, II, and III (Backus 1978)
- Read A History of MATLAB (Moler and Little 2020)
- Read Array Programming with NumPy (Harris et al. 2020)
Week 4. GPUs Go Brrr ποΈ
- Watch βΆοΈ Lecture 4
- Read π Chapter 2.4 Accelerating Matrix Multiplication on GPUs
- Write βοΈ Lab 4. Accelerating Matmul with Tiles and Layouts
Week 5. King - Man + Queen = πΈ
- Watch βΆοΈ Lecture 5
- Read π Chapter 3.1 Learning Static Representations with FFNs
- Read π Chapter 3.2 Iterative Optimization
- Write Lab 5. Embedding Projector
Week 6. Platoβs Cave π¦π¨πͺπ
- Watch βΆοΈ Lecture 6
- Read π Chapter 3.3 Learning Representations with CNNs
- Read π Chapter 3.4 Learning Representations with RNNs
- Read π Chapter 3.5 Learning Representations with GPTs
- Write βοΈ Lab 6. Vision Language Action Model
Week 7. AutoNabla (β§ββ¦)
- Watch βΆοΈ Lecture 7
- Read π Chapter 4.1 Automatic Differentiation
- Read π Chapter 4.2 Gradient Based Optimization
- Write βοΈ Lab 7. Custom Autograd Backwards for VLA