Course Thesis

Presenting an early outline of SITP at Toronto School of Foundation Modeling Season 1 (2025)

The Structure and Interpretation of The AI Curriculum

My alma matter was amongst those which took the SICP approach, and as intended, for someone coming into first year college have taken some elementary computer science courses from high school, it blew my mind (Lambda Calculus!). After graduating college in 2022, I followed my curiosity for diving deeper into the souls of our machine by going on to developing industrial languages and runtimes.“There is only one project, architecture, operating system and languages, compiler, it’s only one project. It’s all together.” – Boris Babayan. Particularly, I hacked on languages with domain specific cloud compilers and runtimes with cloud provisioners, and cloud garbage collectors. At the end of 2022 though, when ChatGPT was released by OpenAI my mind was blown twice more. As someone programming for some time, I could not believe this at all. After two more years of hacking on cloud languages and runtimes, I started my transition from domain specific cloud compilers from GPS to TERRAFORM to domain specific tensor compilers from TORCH to TRITON.It turns out there is effectively zero overlap except that they are both compilers in that they take some source program and produce some target program.

The transition started with a tweet showcasing the beginnings of a tensor library evaluating the forward pass of a feed forward network from Andrej Karpathy’s Neural Networks: Zero to Hero course. While it was illuminating to start implementing each individual torch call that the nets from makemore were making, my knowledge felt quite fragmented as I personally forgot lots of foundation from collegeI had the definition of a gradient and basis in the basement of my subconscious at best. Thank you Professor.West for sharing your juggling, and Professor. Wolczuk for sharing your Ukrainian floor dancing, and I wasn’t sure how to bridge myself to industrial deep learning systems like tinygradHere’s a PR I was able to land in tinygrad making logsumexp numerically stable. It felt like I was contributing to LLVM without having taken a course on compiler construction to learn the basics of parsing, optimization, and lowering., torch, jax, vllm, and sglang.

Shortly after, I decided to take the plunge and started drinking from the firehose of deep learning canon: Hastie et al., Murphy, Goodfellow et al., you name it. The one thought I could not get out of my head was where is the SICP for software 2.0? While I found two excellent resources on building your own torch-like autograd by Tianqi Chen’s 10-414 Carnegie Mellon and Alexander Rush’s CS5781 at Cornell, I personally would have enjoyed a more unified resource that took me from math, to deep learning, to deep learning systems in a single unbroken sequence of thought, and perhaps others would feel similarly. That is the genesis story for this book, whose central research question is the following: What does the SICP for deep learning look like?

Pur shortly, SITP, is a function that maps learners from the product type of an introductory computer science book such as DCIC (Fisler et al. 2025) and an introductory systems programming book such as TRPL (Klabnik et al. 2026) with experimental modifications (Crichton et al. 2023) to a frontier model report such as the frontier technical report of The Llama 3 Herd of Models (Meta 2024) and frontier systems book such as Hugging Face’s Ultra Scale Book (Tazi et al. 2025).

$$\begin{array}{rl}\text{SITP}& :\text{DCIC}\times \text{TRPL}\to \text{LLAMA3}\times \text{USPB}\\ \text{SITP}& :=\text{HTDP}(\text{PTDS},\text{LADR},\text{NA},\text{NO},\\ & \text{SLP},\text{DL},\text{RL},\\ & \text{TAOHPC},\text{TAODCP},\text{PMPP})\end{array}$$

where the function $\text{HTDP}(\cdot )$ is the systematic and principled design of a curriculum implemented by chimera-like professors and lecturers whom are usually interested and capable in the intersection of both programming language theory (PLT) and computer science education (CSE). In the same way that we use non-intersective adjectives in the distinctions of fake gun vs gun, non-tight language model vs language model, there is also the teaching professor vs professor“I think of the position as having “twice” the teaching and “half” the research as an assistant professor (of course individual approaches can and do differ!).” – @JoePolitz, September 5, 2024. That is, SITP applies the form of HTDP developed by such teaching professors to the substance of deep learning. What then, is such a form?

The How to Design Programs HTDP text This was the actual text used at my alma matter. Thank you to Professor Ragde for bringing it over, and to Professor Vasiga for bringing it to life. was also seminal in that it was the first computer science text to treat curriculum design as scientific research and an engineering problem (Felleisen et al. 2004), building off the shoulders of the giant that SICP (Abelson and Sussman 1996) is.

First, the book discusses explicitly how programs should be constructed. Second, to tame the complexity of programming, it defines a series of teaching languages based on Scheme that represent five distinct knowledge levels through which students pass during their first course. The levels correspond to the complexity of data definitions that the program design guidelines use. Third, the book uses exercises to reinforce the explicit guidelines on program design; few, if any, exercises are designed for the sake of domain knowledge. Finally, the book uses more accessible forms of domain knowledge than SICP. Because of this shift in emphasis, we gave our book the title How to Design Programs (HTDP).

the concreteness fading (Fyfe et al. 2014) of declarative concepts in mathematics (Hestenes et al. 1992) by presenting notions just in time by defining them by their context, and iteratively refining (progressive disclosure) such concepts from the informal to formal ending in HtDP-style IntermezzosWhich present the formal syntax and semantics of a language..

that transfer (Bransford and Schwartz 1999) to procedural skills of computation (Price et al. 2021) in the context of deep learning. Such transfer is possible because the computational discipline of deep learning has lots of structural similarity with mathematics, and is why we have books such as Deisenroth et al. (2020), and subsequent courses follwing such book.

When it comes to the discipline of deep learning, there are two large jumps of transfer a learner must make. The first one is that a learner must learn the mathematics for machine learning, while although elementary, has a non-trivial breadth of probability theory, linear algebra, differential calculus, and if the learner has the time, to deepen such concepts with information theory, measure theory, and analysis. One of the primary texts that seeks to provide a unified treatment in such mathematics for machine learning is (Deisenroth et al. 2020)“For instance, Princeton’s COS302 offered by Ryan Adams, which turns out to be the lab that created created HIPS/autograd (led by Matthew Johnson, Dougal Maclaurin, and David Duvenaud), which inspired PyTorch! See https://soumith.ch/blog/2023-12-17-pytorch-design-origins.md.html#/origins., which provides an excellent two part structure with the foundational mathematics, followed by the applied. The core motivation is that any gaps in mathematics can be filled, preparing a learner for a text like such (Goodfellow et al. 2016) to deepen their understanding in neural network architecture.

More recently, with the advent of scaling large language models (Kaplan et al. 2020) and the slowing of Moore’s Law (Moore 1965), another jump of transfer the learner must make is how to efficiently map their architecture and optimizers to massively parallel hardware at the level of a single graphics processing unit (GPU) and multi GPU. The two canonical texts for this are (Hwu et al. 2022) for systems, and more recently (Tazi et al. 2025)“Alternatively, JAX Scaling Book (link.).

As someone opening up PMPP, it was extremely frustrating to read the appendix that briefly covered the process of deep learning without a proper foundation in the modeling of architecture and optimizers.

When I realized the two large jumps I was making from re-learning foundational mathematics to frontier architecture, and from modeling frontier architecture to accelerating frontier architecture, my “aha” moment was realizing the vast bridges a learner was being asked to cross, and realizing Shriram’s taste of finding some intermediary point to smoothen such a progression. This was the motivation for the explicit Design Recipe in HtDP to go from blank page to a well-designed program, and more recently, property-based testing to go from unit tests to model checking and SAT solvers. To return to the short description, SITP that maps learners from the product type of an introductory computer science book and an introductory systems programming book to a frontier model report such as the frontier technical report and frontier systems book. That is,

$$\text{SITP}:\text{DCIC}\times \text{TRPL}\to \text{LLAMA3}\times \text{USPB}$$

unity/intertwined mathematics and programming

• Evaluation (Benchmarks): Stochastics of probability theory $(\Omega ,\mathcal{F},\mathbb{P})$
• Specification (Architecture): Dimensionality of linear algebra $(V,⟨\cdot ,\cdot ⟩)$
• Implementation (Optimizers): Approximation of differential calculus $\frac{d\mathcal{L}}{dw}$, $\nabla \mathcal{L}$, $\mathbb{J}$, $\mathbb{H}$
• Computation (Systems): Rooflines of processors $\text{max}(compute,communication)$

For instance, the traditional ordering one might approach to the discipline of deep learning is to synthetically deduce definitions starting from what is considered to be elementary (i.e your choice of foundations which you pay lip service to as a working mathematician: ZFC or DTT). The benefits of this approach is that the transfer (Bransford and Schwartz 1999) of concepts is maximized — afterall, studying mathematics is simply generic programming. While this is no doubt the standard for most mathematical texts, this is what was special about the textbook How to Design Programs, an explicitly curriculum designed and engineered for transfer (Felleisen et al. 2004) expositing programming based off the principles of set theory (whatever that may mean).

all in the context of autoregressive language modeling, culminating in the transformers architecture

Jeffrey Zhang
Waterloo, Ontario
August 2026

Course Design Thesis

“Civilization advances by extending the number of important operations which we can perform without thinking of them.” Thanks, Alfred North Whitehead (1861-1947)

> 7:22AM Let me first look at what's in the working directory to scope this.
> Thought for 432s

> 7:30AM The directory is empty and lacks numpy, torch, and pytest,
  so I'll build "nanotorch" — a NumPy-based PyTorch clone with a
  familiar API (Tensor, autograd, nn, optim, data loading, saving/loading), 
  testing against real PyTorch where possible. Starting with environment setup.
> Bash Check CPU count, memory, and Python package tooling
> Machine: 8-core M3 with 8GB RAM, so a workflow can run about 6 agents at once.
  Next I'll create a virtualenv with numpy and pytest, plus real PyTorch to use as the reference in tests.

> Bash Create virtualenv and install numpy and pytest
> Bash Install real PyTorch as a test oracle
> Thought for 2s

> Real PyTorch 2.14.1 is installed as the oracle.
  Before writing the design spec, I'll check a few numpy behaviours the core will depend on.
> Bash Verify numpy reshape/promotion features and torch promotion
> Thought for 607s

> Environment's set with numpy 2.5.4, pytest 9.1.1, and PyTorch 2.14.1 as the oracle.
  Now I'll verify the PyTorch behaviours against the real library before drafting the design contract.
> Bash Probe PyTorch semantics for the design spec
> These results match the spec I had planned. Next I'll write the design contract (docs/DESIGN.md)
  and pyproject.toml, and make the package importable from the venv.
> Write pyproject.toml
> Bash Make repo importable in venv and confirm torch sources exist

> 7:40AM Now the design contract. It's long on purpose: it pins down the core semantics
  and file ownership so parallel agents produce one coherent library.
> Write DESIGN.md
> Thought for 233s

> With the design contract done, I'll now build the core foundation
  first—Tensor, dtypes/type promotion, autograd, views/in-place ops, indexing,
  creation, random, test helpers, and the package skeleton—then subject
  it to five red-team reviews against real PyTorch with fixer rounds until tests pass,
  since everything else depends on this layer.
> Thought for 141s

> 7:50AM Workflow 1 is running in the background: one core implementer,
  then 5 red-team reviewers in parallel, then up to 3 fixer rounds.
  The core is the largest single piece, so this will take a while.
  I'll be notified when it finishes, then review the core myself
  before launching the 9 parallel op/nn families of Workflow 2.

“Everything is vague to a degree you do not realize till you have tried to make it precise.” Thanks, Bertrand Russel (1872-1970)

DESIGN.md

# nanotorch — design and contributor contract

nanotorch is a from-scratch reimplementation of the PyTorch Python API on top of NumPy.
Code written for PyTorch should run unchanged with `import nanotorch as torch` for the
supported subset and produce the same results (up to floating-point rounding and RNG streams).

* **We implement:** tensor semantics (dtypes and type promotion, broadcasting, views,
  in-place ops, indexing), the autograd engine (dynamic graph, higher-order gradients,
  hooks, version counters), every derivative formula, `nn`, `optim`, `utils.data`,
  serialization, printing.
* **NumPy provides:** strided n-d array storage and vectorised kernels (ufuncs, BLAS matmul).

This document is the contract every contributor (human or agent) follows. Section 5 (op
authoring guide) is finalised by the core implementation and must stay in sync with the code.

## 0. Ground rules

1. **Fidelity target: PyTorch 2.x.** The oracle is the real `torch` 2.14.1 installed in
   `.venv`. When in doubt about a behaviour, run torch and match it: names, signatures
   (positional order, keyword names, defaults), output shapes, output dtypes, exception
   *types* (message text should be close but need not be identical).
   torch's own Python sources (`.venv/lib/python3.14/site-packages/torch/...`, e.g.
   `nn/modules/module.py`, `optim/adam.py`, `_tensor_str.py`, `nn/init.py`) are a legitimate
   reference for semantics. Write clean, minimal implementations; when you substantially
   adapt PyTorch code, add a `# Adapted from PyTorch (BSD-3-Clause)` comment.
2. **nanotorch never imports torch.** Only tests import torch (through `pytest.importorskip`).
3. **NumPy (>= 2.1) is the only runtime dependency.** No scipy. Pure-Python fallbacks are
   fine for special functions (implement accurate vectorised approximations).
4. Python >= 3.10 syntax (we run 3.14). Type hints welcome, not required.
5. **Non-goals** (do not implement, do not test): CUDA/MPS execution (CPU only; `device`
   objects for other types can be constructed but allocating on them raises), complex
   dtypes, bfloat16 (placeholders that raise a clear error on use), sparse / quantized /
   named / nested tensors, `torch.compile` / `jit` / `fx` (no-op shims only), distributed,
   multi-process DataLoader workers (run in-process), ONNX, `vmap`, FFT, exact RNG-stream
   parity with PyTorch (same *distributions*, different numbers).
6. Repr strings mirror PyTorch exactly where reasonable, including the `torch.` prefix:
   `torch.float32`, `torch.Size([2, 3])`, `device(type='cpu')`,
   `tensor([1., 2.], requires_grad=True)`, `<MulBackward0 object at 0x...>`.

(i.e Why another deep learning book?)

Learning Outcomes (Tasks) learning probes, observable, signals, channels for feedback interactive learning -> no longer producing code (hieroglyphics)

concept inventory (Hestenes et al. 1992)

Software 2.0, following (Krishnamurthi and Fisler 2020)

• Lectures
• Readings (with concreteness fading following (Fyfe et al. 2014)) (with interactive quizzes following (Crichton and Krishnamurthi 2024)) with SITP
  • Declarative concepts
    • property based testing
    • tradeoff between x and y
    • system design/system invariants
• Labs (with Agentic Coding following (Fisler et al. 2026))
  • Procedural skills
    • write a good design doc/test suite
    • code review
• centered around the Teaching Array Programming Language borscht.Tensor with language levels following the form of (Felleisen et al. 2004) but with the substance of NETSOR and NETSOR$\top$ from (Yang 2019) and (Yang 2020) as well as PTX, CUDA, CuTeDSL, Cutile.

Future Directions

• integrate a trimmed version of such curricula with highschool mathematics, following (Felleisen and Krishnamurthi 2009) and Guzdial
• collect profile data to suggest pedagogic interventions to imporove learner outcomes
• actually encapsulate language levels like racket-like #lang with julia metaprogramming
• map out the semantic space of language features with a Standard Model of APLs following (Krishnamurthi 2008)