🎥Video 25 min4 / 8

How does ChatGPT work?

How ChatGPT (and Claude and Every LLM) Works

The Core Idea: Predicting the Next Word

At its heart, a large language model does one thing repeatedly: predict the best next word.

When you type "The capital of France is..." — the model doesn't "know" the answer. Instead, it read billions of texts containing this sentence and knows that "Paris" is the most probable next word in this context.

The Three Steps to Build an LLM

1. Tokenization Text is split into small units called tokens. A token isn't necessarily a full word:

"word" → likely one token
"Tokenization" → might split into ["Token", "ization"]

2. Pre-training The model reads trillions of words from the internet and books. Each time it mispredicts, it adjusts millions of internal parameters to improve. This takes weeks on thousands of GPUs.

3. Fine-tuning (+ RLHF) After pre-training, the model is tuned to be helpful, safe, and honest:

Human-written example conversations
Human ratings of response quality
Reinforcement Learning from Human Feedback (RLHF) to shape behavior

The Transformer: The Beating Heart

The Transformer is the architecture that makes all this possible. Invented in 2017 in the paper "Attention is All You Need."

The key innovation: Attention Mechanism — allows the model to connect distant words in a sentence. For example, in "The book I bought yesterday, I lost it" — the model understands "it" refers to "book" despite the distance.

Context Window

Every LLM has a "short-term memory" called the Context Window — the maximum text it can process in one session.

GPT-4: ~128K tokens
Claude 3.5: ~200K tokens

When a conversation exceeds this limit, the model begins "forgetting" older parts.

What an LLM Does NOT Do

It doesn't remember your conversations between sessions (unless designed to)
It doesn't know events after its training date (Knowledge Cutoff)
It doesn't "think" — it calculates probabilities

python

# Simplified: what happens when you call an LLM API
import anthropic

client = anthropic.Anthropic()

message = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "What is the capital of France?"}
    ]
)

# The model predicts token by token:
# "The" → "capital" → "of" → "France" → "is" → "Paris"
print(message.content[0].text)
# Output: "The capital of France is Paris."

PreviousNext