Tokens

You might assume that when an AI reads "The sky is blue," it sees four words, just like you do.

It doesn't. The model has no idea what a "word" is. It can't read letters. It doesn't know what spaces mean.

Instead, the model breaks text into small chunks called tokens. Sometimes a token is a whole word. Sometimes it's part of a word. Sometimes it's just a single character.

Think of it like this: you read words, but the AI reads puzzle pieces. It chops text into pieces that are easy for it to work with, even if those pieces look weird to us.

Before the model can learn anything from its training data, all those books, websites, and articles, it first needs to break that text into tokens. Tokenization is always the very first step.

Let's watch tokenization happen in real time.

Type any sentence into the demo on the right. You'll see how the AI chops your words into tokens, and the results might surprise you.

Common words like "the" or "and" usually stay as one piece. But longer or unusual words get split into chunks. Try typing a made-up word like "flurbington" and see what happens, the AI has never seen that word before, so it has to break it into pieces it recognizes.

Notice how each token gets its own color. Some words are one token. Some are two, three, or more.

Here's the puzzle: why not just make every word a token?

Because there are too many words. Think about it, every name, every place, every slang term, every misspelling, every word in every language. You'd need millions of tokens, and the model would barely see most of them during training.

Instead, models use a clever trick: they learn common pieces of words. The word "unhappiness" might become ["un", "happi", "ness"]. The model has never seen "unhappiness" as a whole word, but it knows what "un" means, what "happi" looks like, and what "ness" does.

It's like knowing LEGO pieces. You've never built that exact house before, but you know what each brick does.

Here's something that might blow your mind: the AI never sees text at all. Not words. Not even tokens as letters. It only sees numbers.

Every token has an ID number, like an entry in a dictionary. "Hello" might be token #15496. "The" might be token #464. The sentence "The sky is blue" becomes something like [464, 6766, 318, 4171].

That's it. Those numbers are ALL the model gets. It does math on those numbers, and somehow, beautiful language comes out the other side.

It's like a musician who only sees sheet music as numbers but can still play a gorgeous symphony.

The model's vocabulary is its complete list of tokens, every piece of text it can recognize. Think of it as the AI's dictionary, except instead of definitions, each entry is just a number.

Most modern AI models have vocabularies of about 30,000 to 100,000 tokens. That might sound like a lot, but remember, these aren't just words. They include word pieces, punctuation, numbers, spaces, and tokens from many languages.

Here's what's interesting: the vocabulary is fixed when the model is built. It never changes. The model can't learn new tokens after it's trained. If you invent a brand new word, the model has to spell it out using pieces it already knows.

That's why some words cost more tokens than others, the model handles "the" in one piece, but might need five pieces for your username.

Now you know the AI's secret: it doesn't read words. It chops text into tokens, turns those tokens into numbers, and works with the numbers.

But here's the thing, those numbers are just IDs. Token #15496 doesn't mean anything by itself. The model still needs to learn that "happy" and "joyful" are related, that "bank" can mean a riverbank or a money bank, and that "not good" is the opposite of "good."

How does it learn any of that? It starts knowing nothing and has to figure everything out from scratch. Next lesson: how a model learns from its mistakes.