,,                                     ,,                      ,,
 .M"""bgd  db                                   `7MM                      db   mm
,MI    "Y                                         MM                           MM
`MMb.    `7MM  `7MMpMMMb.  .P"Ybmmm `7MM  `7MM    MM   ,6"Yb.  `7Mb,od8 `7MM mmMMmm `7M'   `MF'
  `YMMNq.  MM    MM    MM :MI  I8     MM    MM    MM  8)   MM    MM ' "'  MM   MM     VA   ,V
.     `MM  MM    MM    MM  WmmmP"     MM    MM    MM   ,pm9MM    MM       MM   MM      VA ,V
Mb     dM  MM    MM    MM 8M          MM    MM    MM  8M   MM    MM       MM   MM       VVV
P"Ybmmd" .JMML..JMML  JMML.YMMMMMb    `Mbod"YML..JMML.`Moo9^Yo..JMML.   .JMML. `Mbmo    ,V
                          6'     dP                                                    ,V
                          Ybmmmd'                                                   OOb"


 .M"""bgd                   mm
,MI    "Y                   MM
`MMb.  `7M'   `MF',pP"Ybd mmMMmm .gP"Ya `7MMpMMMb.pMMMb.  ,pP"Ybd
  `YMMNq.VA   ,V  8I   `"   MM  ,M'   Yb  MM    MM    MM  8I   `"
.     `MM VA ,V   `YMMMa.   MM  8M""""""  MM    MM    MM  `YMMMa.
Mb     dM  VVV    L.   I8   MM  YM.    ,  MM    MM    MM  L.   I8
P"Ybmmd"   ,V     M9mmmP'   `Mbmo`Mbmmd'.JMML  JMML  JMML.M9mmmP'
          ,V
       OOb"

Singularity Systems: Zero to Hero — The Hacker's Guide to Tensor Compilers (by j4orz)
* these course notes (formalizations of specifications) are complementary to
  the lecture videos (intuitive line by line implementations)

------------------
Course Description
------------------
Singularity Systems: Zero to Hero follows up from Neural Networks: Zero to Hero.
We turn the toy backpropagation engine micrograd into picograd: a deep learning
framework with .compile(model) support.

Singularity Systems bridges the missing gap in the ecosystem for low level system
programmers and performance engineers interested in the internals of deep learning
frameworks. The course provides the software 2.0 equivalent of SICP (interpreters)
and dragon book (compilers), where you will build a PT1 "eager" interpreter
throughout chapters 1-2 and a PT2 "graph" compiler throughout chapters 3-4, line
by line from scratch. Along the way, we compare and contrast the design decisions
and tradeoffs PyTorch has made versus other frameworks such as Jax and Tinygrad.
By the end of Singularity Systems, you will be better equipped to contribute to
production-level deep learning frameworks.

Prereqs
-------
- solid deep learning (sd && llama)
- solid systems programming (C || C++ || Rust || Python*)
  * python is ok (following pure-python systems like PyTorch 2.0 & CUTLASS 4.0)

Syllabus
--------
  0. au197
  1. dfdx(nd)
  2. tiles
  3. pt2
  4. use1az2
  ---appendix
  a. turing

Ch.0 introduces the golden age of systems ml.
Ch.1 implements an interpreter for neural networks (pytorch1)
Ch.2 implements a compiler for a tiling language (triton)
Ch.3 implements a compiler for neural networks (pytorch2)
Ch.4 implements distributed training
** v2 course offering: tenstorrent.

Citation
--------
@article{j4orz2025singsys,
  author  = "j4orz",
  title   = "Singularity Systems: Zero to Hero (The Hacker's Guide to Tensor Compilers)",
  year    = "2025",
  url     = "https://github.com/j4orz/singularitysystems"
}