What is actually inside ChatGPT

Right, I clickbaited you slightly with the ChatGPT in the title. Everything here applies just as much to Claude, Gemini, Perplexity, Grok, DeepSeek and whatever launches next week, because underneath they’re all the same kind of thing built the same way. Almost everybody I know uses one of them now, and almost none of them could tell you what’s inside it. I’ve asked, casually, more times than is probably normal at a dinner table, and the answers land in two buckets. It searches the internet, or it’s a giant database of answers somebody typed in. Both wrong. The real answer is stranger and a lot more boring than either, and once it lands, most of the mystery around this stuff quietly goes away.

I should be upfront about what I am and what I’m not here. I build products on top of these models for a living. I’ve never trained one of these from scratch, and neither has almost anybody you know, because the electricity bill alone rules out most companies on earth. I’ve trained a handful of much smaller machine learning models, which is enough to know the shape of what happens and nowhere near enough to claim I’ve built anything like the tools above. But I’ve spent enough time wiring these things into real systems, and watching them fail in ways that only make sense once you know what they are, to be annoyed at how badly this usually gets explained. You either get a metaphor about brains, which is wrong, or you get the maths, which loses most people by the second line. There’s a middle version that is honest and needs no maths at all.

So here’s the whole claim in one sentence, and then I’ll take it apart.

An AI model is a very large collection of numbers, arranged in a specific pattern, that were adjusted automatically until they got good at predicting things.

That probably sounds like it explains nothing, so here it is as something you can picture. The entire process that produced every one of those tools fits into five short sentences.

Show it a sentence with the last word hidden. Let it guess. Compare the guess to the real word. Nudge the machine very slightly in whatever direction would have made the guess better. Then do it again.

Everything else in this piece is detail hanging off those five sentences. Drawn out, the whole thing looks like this.

Ingredients on the left, the guessing game in the middle, and the thing you actually talk to on the right. Take them one box at a time.

The ingredients are text and electricity

Two things go in. The first is text, and the quantity is the part people underestimate. It isn’t a curated library of good writing, it’s close to everything readable on the internet, plus books, forums, transcripts and decades of people arguing with each other in public, which comes out to trillions of words.

The second ingredient is electricity, or more precisely the computers that burn it. Tens of thousands of specialised machines running in a warehouse for months without stopping. That’s why there are maybe ten organisations on the planet doing this at the top end, and why the news you see from them is increasingly about building data centres rather than building features.

There’s nothing clever in this box, just text and machines. The interesting part is what the middle box does to them.

Training is a guessing game played trillions of times

Back to those five sentences, because the middle box is where all the work happens.

Take “the cat sat on the ___”. Right at the start the numbers are random, so the first guess is nonsense. It might say “photosynthesis” with exactly as much confidence as it says anything else. Then comes the check, which is the easy part, because you covered up the real word yourself and it was “mat”, so you know precisely how wrong the guess was.

The nudge is the clever bit, so slow down here. For every single number in the file, and there are billions of them, the process works out one narrow thing: if I turned this number up a hair, would “mat” have been slightly more likely? Then it moves each number a hair in whichever direction the answer points, and the move is deliberately tiny. One pass changes the machine so little that you couldn’t notice it from a single sentence.

Then it does the same on the next sentence, and the one after that, trillions of times over, at a scale that is genuinely hard to hold in your head. Nobody is supervising it, and it runs for months.

Here’s the part most explanations skip. Nobody ever tells the model that Paris is the capital of France, and nobody types in the rules of English grammar. The machine gets nudged across a billion sentences where the subject and the verb agree, and agreement turns out to be a pattern that makes guesses better, so the numbers drift into a shape that has grammar sitting inside it somewhere. It’s the same story with the France fact, with noticing that a sentence is sarcastic, and with everything else it appears to know. All of it is a side effect of being pushed, over and over, to get better at one boring task.

Nobody wrote the rules inside a model. They wrote the loop that goes looking for them, then ran it for a very long time.

What you get at the end is a file

When the training finishes, the thing that comes out isn’t a program, it’s a file you could copy onto a USB stick. A few companies publish theirs openly, so you can download one this afternoon and look inside it yourself.

There isn’t anything in there a person could read, no instructions and no list of facts, just decimals one after another for billions of rows: 0.0271, then -0.1144, then 0.0038, and on and on until you lose the will. Eight billion of those numbers takes about sixteen gigabytes, roughly four films’ worth of space on your laptop, and that file is the model. Everything it appears to know about medicine, cooking, Tamil poetry and your job is in there, and none of it is written down anywhere you could point at.

The arrangement matters as much as the numbers do. They sit in a fixed layout, called the architecture, and that layout is the one part humans actually designed. Think of it as plumbing. Engineers laid the pipes and decided how many valves go where, then the training set every single valve without a person ever turning one by hand.

Then you make it much bigger and it starts to feel like magic

Everything so far describes a machine that guesses a hidden word. A large language model is that same machine after you scale all three boxes until the result changes character.

You scale up the text, the numbers and the computing power all at once, and the task widens slightly too. Instead of guessing one hidden word at the end of a sentence, it guesses the next bit of text given everything that came before it. That’s the only real change, and it’s why the whole category ended up with a name as unglamorous as “language model”.

So when you type a question into any of these, nothing is being looked up. The model works out what text would most plausibly follow your question, writes a little of it, sticks that onto the end of what it already has, then works out what comes next, and again, and again, until it decides the answer is finished. The reply appearing word by word on your screen isn’t a loading animation, that’s the model writing it a piece at a time, live.

This is also the honest explanation for why these things make things up. A confident wrong answer is not a glitch in the lookup, because there is no lookup. It is what a very good guessing machine produces when the most plausible-sounding thing to say next happens to be false, and from the inside it has no way to tell the difference.

One gap I should name before I stop. What I’ve described is the first stage, which has a name, pretraining, and on its own it doesn’t get you an assistant. It gets you something that continues text and will happily continue it into nonsense. Every model you actually use has been through a second stage afterwards, where people grade its answers and it gets nudged again, this time towards the answers humans preferred. That’s the stage that turns an autocomplete into something behaving like an assistant, and it deserves its own piece.

What to do with this

The next time an AI tool hands you a confident wrong answer, don’t ask what it looked up. Ask what text would plausibly have followed your question, and you’ll usually find the wrong answer sitting right there, looking exactly like a right one.

This is the first piece in a series I’m writing to teach this properly, from the bottom up, without the maths. Next one is about how you give a model access to real documents so it can stop guessing, which has an ugly acronym, RAG, and a very simple idea underneath it. That one has to come second, because it makes no sense until you’ve accepted that the model on its own isn’t searching for anything.

If you want to go one level deeper than this, Grant Sanderson’s visual explanation of how these models work is the best thing on the internet for it. Fair warning, that’s where the maths stops being optional.