I was thinking it would be much more efficient if AI and it’s large language models would store things the way our brains do rather than the way phones and computers do. Devices tend to store things on a byte-by-byte basis. Every character of text, every pixel requires one or more bytes of storage space. Nearly 30 years ago, during a visit to what became archive.org I asked about the size of the whole Internet. Back then, I was told a full backup of the Internet required eleven terabytes. Today it’s probably orders of magnitude bigger. If everything like Grok, Copilot, Gemini or ChatGPT had to replicate that storage, it would be a huge waste. I was thinking if clumps of information could be stored as tokens, then assembled to make up complete memories, essentially the way our brains do, that would be much better. It turns out, I was naively behind the times.

I asked ChatGPT:

Do llms store content from the web on a verbatim or byte-by-byte basis, or do they do something associative or maybe something like vector graphics?

This is the answer I received:

Large language models (LLMs) do not store content from the web verbatim (e.g., byte-by-byte or word-for-word), nor do they store it in formats like vector graphics. Instead, they use a statistical and associative encoding method, typically involving something called token embeddings and neural network weights.

Here’s a breakdown of how it works:
✅ Not Verbatim Storage

LLMs don’t memorize web pages, books, or articles word-for-word (though some limited memorization can happen for very common or repeated content like famous quotes or code snippets).

There’s no database inside the model that you can query for specific documents or exact phrases unless it was extremely frequent or distinctive in training data.

✅ Associative and Statistical Storage

When training, LLMs convert text into tokens (chunks of words or characters).

These tokens are mapped into vectors (numerical representations).

The model learns patterns and relationships between these tokens by adjusting billions of weights in a deep neural network.

Think of it like learning how concepts relate to each other, rather than recording what was seen.

🎨 Vector-Like (But Not Graphics)

The “vector” in LLMs refers to mathematical vectors, not vector graphics.

For example, the word “cat” might be represented as a 768-dimensional vector that places it near “kitten” and far from “truck” in semantic space.

This is called an embedding space.

🧠 Analogies

Here are two analogies that might help:

Brain Analogy: You don’t remember every sentence of a book you’ve read, but you internalize meanings, concepts, and facts — same with LLMs.

Map Analogy: Instead of a photographic map, it’s like a topographic map of meaning — hills of similarity and valleys of difference, not pixel-by-pixel storage.

📍Exception: Overfitting or Copying

In some rare cases (e.g. low-variance code, or repeated boilerplate), models can inadvertently regurgitate text. This is often referred to as memorization and is carefully monitored and minimized.

TL;DR:

LLMs store associative patterns in the form of learned weights and embeddings, not literal or byte-for-byte content. They encode concepts probabilistically and relationally, not as a database or archive.

Would you like a visual or conceptual diagram of how this process works?