学习资料
浏览视频
返回视频列表
构建 GPT 分词器：文本如何变成 token
学习资料
这个精选视频的资料正在准备中，可以先观看视频并阅读学习导读。
PPT 资料为文字大纲，尚非完整演示文稿。
视频概览
Andrej Karpathy 从零构建 GPT 分词器，实现字节对编码、编码与解码，并解释许多 LLM 异常行为如何源自分词过程，同时介绍 Unicode 字节编码。
HN 讨论的原因
HN 推荐它作为深入理解分词器的课程，认为从事 LLM 相关工作的观众值得花两小时学习。
观看前准备
具备基础 Python 知识，并理解语言模型如何使用 token 序列会有帮助。这是适合深入了解文本表示的专题课程。
学习建议
尝试编码一句短语、标点和非英语文本。比较 token 边界，解释字符数与 token 数为何不同，再回到模型构建课程。
最新优先
所在讨论： UTF-8 is a brilliant design
Karpathy's "Let's build the GPT Tokenizer" also contains a good introduction to Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32 in the first 20 minutes: [VIDEO]
所在讨论： Why Does Claude Speak Byzantine Music Notation?
Andrej Karpathy goes into quite some depth about how tokenization works here: [VIDEO] tl;dr many of the LLMs use byte-pair encoding to create tokens. You take a set of documents, and then form tokens by repeatedly merging the most common pair of tokens. The initial set of tokens is 256 raw bytes. And the text is typically represented in utf-8. I expect that although the LLMs can understand arbitrarily but cleanly…Andrej Karpathy goes into quite some depth about how tokenization works here: [VIDEO] tl;dr many of the LLMs use byte-pair encoding to create tokens. You take a set of documents, and then form tokens by repeatedly merging the most common pair of tokens. The initial set of tokens is 256 raw bytes. And the text is typically represented in utf-8. I expect that although the LLMs can understand arbitrarily but cleanly offset unicode code points by (eventually) noticing the final byte of each sequence, they would do markedly worse on actually processing and completing on them, because they will not have been reduced to the normal set of tokens. However, if the text is actually output cleanly converted, either in internal thinking tokens or in the beginning of the response, they should do fine. Understanding tokenization is surprisingly useful, even if that video seems awfully long to devote to such a tedious subject. Even Karpathy doesn't like it!
所在讨论： Probably pay attention to tokenizers
I finally understood the weirdness of tokenizers after watching the video Andrej Karpathy made: "Let's build the GPT Tokenizer" ( [VIDEO] ). He goes through why we need them instead of raw byte sequences (too expensive) and how the Byte Pair Encoding algorithm works. Worth spending 2h for the deeper understanding if you deal with LLMs.
所在讨论： Artificial intelligence is losing hype
This is just an artefact of tokenisation though. The model simply isn’t ever shown the letters that make up words, unless they are spelled out explicitly. It sees tokens representing groups of words. This is a little like saying a human isn’t intelligent because they couldn’t answer your question that you asked in an ultrasonic wavelength. If you’d like to learn more this video is a great resource: [VIDEO]
所在讨论： Notebooks Are McDonalds of Code
If there are complaints about code quality in a jupyter notebook, chances are that someone is doing it wrong -- either the author is using it for demonstration purpose, for production or near production environment, or the person looking at others' notebooks is taking them too seriously. Most code in jupyter notebook is bad, but that's fine, because nobody should waste time doing that unless they must reproduce the…If there are complaints about code quality in a jupyter notebook, chances are that someone is doing it wrong -- either the author is using it for demonstration purpose, for production or near production environment, or the person looking at others' notebooks is taking them too seriously. Most code in jupyter notebook is bad, but that's fine, because nobody should waste time doing that unless they must reproduce the results or they are sharing the notebook with others. Notable example of high quality notebook: [VIDEO]
已显示 5 / 8 条评论