Presenting an early outline of SITP at Toronto School of Foundation Modeling Season 1 (2025)

Preface

The Structure and Interpretation of The AI Curriculum

My alma matter was amongst those which took the SICP approach, and as intended, for someone coming into first year college have taken some elementary computer science courses from high school, it blew my mind (Lambda Calculus!). After graduating college in 2022, I followed my curiosity for diving deeper into the souls of our machine by going on to developing industrial languages and runtimes.“There is only one project, architecture, operating system and languages, compiler, it’s only one project. It’s all together.” – Boris Babayan. Particularly, I hacked on languages with domain specific cloud compilers and runtimes with cloud provisioners, and cloud garbage collectors. At the end of 2022 though, when ChatGPT was released by OpenAI my mind was blown twice more. As someone programming for some time, I could not believe this at all. After two more years of hacking on cloud languages and runtimes, I started my transition from domain specific cloud compilers from GPS to TERRAFORM to domain specific tensor compilers from TORCH to TRITON.It turns out there is effectively zero overlap except that they are both compilers in that they take some source program and produce some target program.

The transition started with a tweet showcasing the beginnings of a tensor library evaluating the forward pass of a feed forward network from Andrej Karpathy’s Neural Networks: Zero to Hero course. While it was illuminating to start implementing each individual torch call that the nets from makemore were making, my knowledge felt quite fragmented as I personally forgot lots of foundation from collegeI had the definition of a gradient and basis in the basement of my subconscious at best. Thank you Professor.West for sharing your juggling, and Professor. Wolczuk for sharing your Ukrainian floor dancing, and I wasn’t sure how to bridge myself to industrial deep learning systems like tinygradHere’s a PR I was able to land in tinygrad making logsumexp numerically stable. It felt like I was contributing to LLVM without having taken a course on compiler construction to learn the basics of parsing, optimization, and lowering., torch, jax, vllm, and sglang.

Shortly after, I decided to take the plunge and started drinking from the firehose of deep learning canon: Hastie et al.,, Murphy, Goodfellow et al.,, you name it. The one thought I could not get out of my head was where is the SICP for software 2.0? While I found two excellent resources on building your own torch-like autograd by Tianqi Chen’s 10-414 Carnegie Mellon and Alexander Rush’s CS5781 at Cornell, I personally would have enjoyed a more unified resource that took me from math, to deep learning, to deep learning systems in a single unbroken sequence of thought, and perhaps others would feel similarly. That is the genesis story for this book, whose central research question is the following: What does the SICP for deep learning look like?

Pur shortly, SITP, is a function that maps learners from the product type of an introductory computer science book such as DCIC (Fisler et al., 2025) and an introductory systems programming book such as TRPL (Klabnick et al., 2026) with experimental modifications (Crichton et al.,) to a frontier model report such as the frontier technical report of The Llama 3 Herd of Models (Meta 2024) and frontier systems book such as Hugging Face’s Ultra Scale Book (Tazi et al., 2025).

$$\begin{array}{rl}\text{SITP}& :\text{DCIC}\times \text{TRPL}\to \text{LLAMA3}\times \text{USPB}\\ \text{SITP}& :=\text{HTDP}(\text{PTDS},\text{LADR},\text{NA},\text{NO},\\ & \text{SLP},\text{DL},\text{RL},\\ & \text{TAOHPC},\text{TAODCP},\text{PMPP})\end{array}$$

where the function $\text{HTDP}(\cdot )$ is the systematic and principled design of a curriculum implemented by chimera-like professors and lecturers whom are usually interested and capable in the intersection of both programming language theory (PLT) and computer science education (CSE). In the same way that we use non-intersective adjectives in the distinctions of fake gun vs gun, non-tight language model vs language model, there is also the teaching professor vs professor“I think of the position as having “twice” the teaching and “half” the research as an assistant professor (of course individual approaches can and do differ!).” – @JoePolitz, September 5, 2024. That is, SITP applies the form of HTDP developed by such teaching professors to the substance of deep learning. What then, is such a form?

The How to Design Programs HTDP text This was the actual text used at my alma matter. Thank you to Professor Ragde for bringing it over, and to Professor Vasiga for bringing it to life. was also seminal in that it was the first computer science text to treat curriculum design as scientific research and an engineering problem (Felleisen et al., 2004), building off the shoulders of the giant that SICP (Abelseon et al., 1985) is.

First, the book discusses explicitly how programs should be constructed. Second, to tame the complexity of programming, it defines a series of teaching languages based on Scheme that represent five distinct knowledge levels through which students pass during their first course. The levels correspond to the complexity of data definitions that the program design guidelines use. Third, the book uses exercises to reinforce the explicit guidelines on program design; few, if any, exercises are designed for the sake of domain knowledge. Finally, the book uses more accessible forms of domain knowledge than SICP. Because of this shift in emphasis, we gave our book the title How to Design Programs (HTDP).

the concreteness fading (Fyfe et al., 2014) of declarative concepts in mathematics (Hestenes et al., 1992) by presenting notions just in time by defining them by their context, and iteratively refining such concepts from the informal to formal ending in HtDP-style IntermezzosWhich present the formal syntax and semantics of a language..

that transfer (Bransford, Shwartz 1999) to procedural skills of computation (Price et al., 2021) in the context of deep learning. Such transfer is possible because the computational discipline of deep learning has lots of structural similarity with mathematics, and is why we have books such as Deisenroth et al., (2020), and subsequent courses follwing such book.

When it comes to the discipline of deep learning, there are two large jumps of transfer a learner must make. The first one is that a learner must learn the mathematics for machine learning, while although elementary, has a non-trivial breadth of probability theory, linear algebra, differential calculus, and if the learner has the time, to deepen such concepts with information theory, measure theory, and analysis. One of the primary texts that seeks to provide a unified treatment in such mathematics for machine learning is (Deisenroth et al., 2020)“For instance, Princeton’s COS302 offered by Ryan Adams, which turns out to be the lab that created created HIPS/autograd (led by Matthew Johnson, Dougal Maclaurin, and David Duvenaud), which inspired PyTorch! See https://soumith.ch/blog/2023-12-17-pytorch-design-origins.md.html#/origins., which provides an excellent two part structure with the foundational mathematics, followed by the applied. The core motivation is that any gaps in mathematics can be filled, preparing a learner for a text like such (Goodfellow et al., 2017) to deepen their understanding in neural network architecture.

More recently, with the advent of scaling large language models (Kaplan et al., 2020) and the slowing of Moore’s Law (Moore 1965), another jump of transfer the learner must make is how to efficiently map their architecture and optimizers to massively parallel hardware at the level of a single graphics processing unit (GPU) and multi GPU. The two canonical texts for this are (Hwu et al., 2026) for systems, and more recently (Tazi et al., 2025)“Alternatively, JAX Scaling Book (link.).

As someone opening up PMPP, it was extremely frustrating to read the appendix that briefly covered the process of deep learning without a proper foundation in the modeling of architecture and optimizers.

When I realized the two large jumps I was making from re-learning foundational mathematics to frontier architecture, and from modeling frontier architecture to accelerating frontier architecture, my “aha” moment was realizing the vast bridges a learner was being asked to cross, and realizing Shriram’s taste of finding some intermediary point to smoothen such a progression. This was the motivation for the explicit Design Recipe in HtDP to go from blank page to a well-designed program, and more recently, property-based testing to go from unit tests to model checking and SAT solvers. To return to the short description, SITP that maps learners from the product type of an introductory computer science book and an introductory systems programming book to a frontier model report such as the frontier technical report and frontier systems book. That is,

$$\text{SITP}:\text{DCIC}\times \text{TRPL}\to \text{LLAMA3}\times \text{USPB}$$

unity/intertwined mathematics and programming

• Evaluation (Benchmarks): Stochastics of probability theory $(\Omega ,\mathcal{F},\mathbb{P})$
• Specification (Architecture): Dimensionality of linear algebra $(V,⟨\cdot ,\cdot ⟩)$
• Implementation (Optimizers): Approximation of differential calculus $\frac{d\mathcal{L}}{dw}$, $\nabla \mathcal{L}$, $\mathbb{J}$, $\mathbb{H}$
• Computation (Systems): Rooflines of processors $\text{max}(compute,communication)$

For instance, the traditional ordering one might approach to the discipline of deep learning is to synthetically deduce definitions starting from what is considered to be elementary (i.e your choice of foundations which you pay lip service to as a working mathematician: ZFC or DTT). The benefits of this approach is that the transfer (Bransford, Shwartz 1999) of concepts is maximized — afterall, studying mathematics is simply generic programming. While this is no doubt the standard for most mathematical texts, this is what was special about the textbook How to Design Programs, an explicitly curriculum designed and engineered for transfer (Felleisen, Findler, Flatt, Krishnamurthi 2004) expositing programming based off the principles of set theory (whatever that may mean).

all in the context of autoregressive language modeling, culminating in the transformers architecture

during a crisis, it’s the Kairos (καιρός). the opportune moment. we are living in the eye of the storm, and it’s hard to predict what comes next.

• one case study: Shampoo/Muon, Muon Kernels (Tri Dao), CuTe Layout
• check afterword. check repo singsys for
  • gpt2 -> dsv3 -> k3
  • hopper, blackwell, rubin, feynman
• this book is a take on such a normative claim.
• not only does SITP cover AI, it uses AI
• https://www.coreauto.com/blog/when-ai-starts-writing-systems-code mlsys 2026 keynote

foo

• primary goal: understanding. but after understanding, automate yourself.
• in some sense, the intelligence revolution is simply an extension of the information revolution, not only in the information theoretical sense where next-token predicion can be viewed as compression, but also in the practical sense where we all first learned Python to automate something.
• everyone must start becoming research engineers. instantiate new patterns of bits or atoms into reality.
  • perhaps thats the next transformer with continual learning (core auto)
  • perhaps thats the next transformer with memory (engram)
  • perhaps thats the next theory of deep learning (learning mechanics)
  • perhaps thats the next deep learning framework (modula)
  • perhaps thats the next gpu (matx)
  • perhaps thats the next human assistant (thinking)
  • perhaps thats the next scientific assistant (periodic labs, isomorphic labs)
  • perhaps thats the next imaging (midjourney, art and medical)
  • perhaps thats the next digital physical (pragmatic)
  • perhaps thats the next education (eureka)
  • perhaps thats the next god with SSI
  • or perhaps it’s curing cancer, or getting to mars
  • these are prompts that are not 1 shot

Jeffrey Zhang
Waterloo, Ontario
August 2026