· ai-devtools

You could have invented transformers from memory

[code][interactive app][slides][X post][Hacker News][机器之心] Large language models today can recall obscure facts from every corner of the world.


Most recently, I was running late for a friend's board game night, so I asked Astra to remind me the rules to Dune: Imperium. It gladly narrated them to me as I drove over and helped me remember the rules well enough for me to play again.

We're so used to asking models about anything that we take their vast knowledge for granted. How do models remember so much information?

In this post, I'll go through an intuitive way to look at linear maps, MLPs, and self-attention as different kinds of memory stores. From this intuition we can derive some heuristics for how much memory each layer can store. The key property that characterizes a "good" memory store is its ability to recall what it stored.

Earlier coverage

Related coverage

Also worth reading

  1. next.js 16.5.0-canary.50 released
  2. Node.js Brings Debounce, Throttle, and PKCS#12 Parsing Into Core Modules
  3. Daemons can now send traces to Vercel Tracing
  4. FBI flash alert says Artex tool automating cyberattacks at ‘machine speed’
  5. Anthropic now lets you self-host Claude Managed Agents. The catch? 8 hours to act if it flags a safety risk