Browse videos
Building makemore: an introduction to language modeling
Study materials
Materials for this selected video are in preparation. Watch the video and use the public learning guide while they are being prepared.
Slides are a PPT outline, not a finished slide deck.
Overview
Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.
Why HN discusses it
HN recommends it as a long but worthwhile way to demystify how language models work technically.
Before you watch
Comfort with Python and the ideas introduced in micrograd will help. This lesson is for learners ready to move from scalar neural-network examples to character-level language modeling.
A way to study this video
Reproduce the small name-generation experiment, then change one input and predict its effect. Follow the intermediate Zero to Hero lessons before attempting the GPT implementation.
Follow the Andrej Karpathy learning guide · Original Zero to Hero course
Newest first
In discussion: The n-gram Language Model
For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * [LINK] It appears to be a generalized version of the bi-gram project, which he links to here: * [VIDEO] It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects.
In discussion: Show HN: Next-token prediction in JavaScript
You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - [LINK] - Highly recommend this one [VIDEO] - [LINK] - [LINK] 1: ex: [LINK]
In discussion: GPT in 500 Lines of SQL
This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] [VIDEO]
In discussion: Markov Chains are the Original Language Models
I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. [VIDEO]