Andrej Karpathy talks and lectures: where to start

Choose a starting point based on what you want to understand or build. This guide connects seven Karpathy videos in the Talkshelf catalog, from an introduction to language models to coding a neural network and a GPT tokenizer.

Understand language models

Start with Intro to Large Language Models if you want a compact orientation. At about an hour, it is the shorter starting point in this collection.

Continue with Deep Dive into LLMs like ChatGPT for a longer exploration. Allow about three and a half hours, or divide it into several sessions. After watching a section, explain one idea in your own words and write down a question you want the next section to answer.

Build your understanding in code

Begin with building micrograd to work through neural networks and backpropagation. Familiarity with Python and basic derivatives will help. Pause to predict a result, run the example, and compare the result with your prediction.

Next, building makemore introduces a small character-level language model. Follow the intermediate lessons in Karpathy's Neural Networks: Zero to Hero course before moving to Let's build GPT. Those lessons develop the tensor and PyTorch background used in the GPT video.

For each coding session, keep a short record of what you changed and why it changed the result. That record is more useful for your next session than a list of timestamps alone.

Explore tokenization

Let's build the GPT Tokenizer focuses on the component that turns text into tokens. Use it as a focused branch of your learning path when you want to investigate how text is represented for a language model.

Explore computer vision history

CS231n: Introduction and Historical Context is a separate starting point for the history and context of computer vision. This recording is from the 2016 course; its purpose in this guide is background and orientation.

What the HN discussion adds

In a discussion linking to micrograd, bialpio describes the “first video is literally from scratch.” The January 2023 comment also points to the wider Zero to Hero series. It helps explain why micrograd can be a useful entry point for a learner ready to code.

Open each video page for its discussion records and original HN links. Look for comments about prerequisites, practical use and questions the video helped someone answer.

Videos with study materials

Building micrograd currently has a complete English study pack: notes, a quiz, editable flashcards and a PPT outline. The material tags on each video show its available resources. Open the video to see its study options below the player.

Use the notes to revisit the explanation, answer a question before checking it, and return to the relevant part of the video when your answer is uncertain. Add the concepts you want to remember to your flashcard routine.