Browse videos
Building micrograd: neural networks and backpropagation
Study materials
English notes, a quiz, flashcards and a PPT outline are available below the player.
Slides are a PPT outline, not a finished slide deck.
Overview
Andrej Karpathy's first Neural Networks: Zero to Hero lecture, which builds micrograd, a tiny autograd engine, from scratch in Python. It explains backpropagation and neural network training step by step, assuming only basic Python and some high-school calculus.
Why HN discusses it
HN treats it as the best starting point for really understanding backprop and neural nets, because Karpathy explains the why with intuition and builds everything in code.
Before you watch
Basic Python, functions and high-school derivatives are useful. Choose this lecture if you want to understand how gradients move through a computation graph.
A way to study this video
Before running each backward pass, work out one derivative by hand. Compare it with the code and explain how changing a weight changes the loss. Continue to makemore when the scalar examples make sense.
Follow the Andrej Karpathy learning guide · Original Zero to Hero course
Machine learning & deep learning · Python, Ruby & scripting languages
Newest first
In discussion: The Illustrated Transformer
If you have 20 hours to spare I highly recommend this youtube playlist from Andrej Karpathy [VIDEO] It starts with the fundamentals of how backpropagation works then advances to building a few simple models and ends with building a GPT-2 clone. It won't taech you everything about AI models but it gives you a solid foundation for branching out.
In discussion: Deep dive into the small details of micrograd
This blog post analyzes the code with an emphasis on the small details and ideas explored in Andrej Karpathy's awesome micrograd lecture. Link to the original lecture: [VIDEO] I'd love to hear any thougts/reviews about the blog post.
In discussion: The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]
For studying? Mainly watching and re-watching Karpathy's 'Zero To Hero'[1] and Stanford's 'Introduction to Convolutional Neural Networks for Visual Recognition'[2], also a lot of transformers from scratch videos like Umar Jamali's videos[3], and I also study backwards to McCulloch and Pitts. Reading the 30 papers [LINK] and so on. And of course Yannic Kilcher[4], and also listening in on the paper discussions they do…For studying? Mainly watching and re-watching Karpathy's 'Zero To Hero'[1] and Stanford's 'Introduction to Convolutional Neural Networks for Visual Recognition'[2], also a lot of transformers from scratch videos like Umar Jamali's videos[3], and I also study backwards to McCulloch and Pitts. Reading the 30 papers [LINK] and so on. And of course Yannic Kilcher[4], and also listening in on the paper discussions they do on discord. Practicing a lot with just doing backpropagation by hand and making toy models by hand to get intuition for the signal flow, and building all kinds of smallish systems, e.g. how far can you push whisper, small qwen3, and kokoro to control your computer with voice? People think that deepseek/mistral/meta etc are democratizing AI, but its actually Karpathy who teaches us :) so we can understand them and make our own. [1] [VIDEO] [2] [LINK] [3] [LINK] [4] [LINK]
In discussion: How linear regression works intuitively and how it leads to gradient descent
In the same vein, Karpathy's video series "Neural Networks from zero to hero"[0] touches upon a lot of this and intuitions as well. One of the best introductory series (even if you ignore the neural net part of it) and brushes on gradients, differentiation and what it means intuitively. [0] [VIDEO]
In discussion: Understanding Machine Learning: From Theory to Algorithms
From my other comment elsewhere. These resources helped me understand the topics better. If anyone wants to understand fundamentals of machine learning, one of the superb resources I have found is, Stanford's "Probability for computer scientists"[1]. It goes into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. But, this is a primarily a probability course that…From my other comment elsewhere. These resources helped me understand the topics better. If anyone wants to understand fundamentals of machine learning, one of the superb resources I have found is, Stanford's "Probability for computer scientists"[1]. It goes into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. But, this is a primarily a probability course that discusses the fundamentals of machine learning. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familiarity with linear algebra topics) There is a course reader for CS109 [2]. You can download pdf version of this. Caltech's learning from data was really good too, if someone is looking for theoretical understanding of ML topics [3]. There is also book for excellent caltech course[4]. Also, neural networks zero to hero is for understanding how neural networks are built from ground up [5]. [1] [LINK] [2] [LINK] [3] [LINK] [4] [LINK] [5] [VIDEO]
Showing 5 of 20 comments