The Black Box May Have Had the Wrong Instructions
Three expressive Renaissance scholars inspect luminous circuitry inside a black box using magnifiers and an iPhone. Made by Edward Frank Morris
In an essay published on 15 October 2026, researcher Nick Cammarata argues that apparent “polysemanticity” in neural networks may often come from ordinary generalisation, architectural limitations and weak visualisation tools, rather than neurons arbitrarily storing unrelated concepts. He presents examples linking features such as cats and car fronts through shared visual patterns. The essay calls itself a working hypothesis. Its estimates for a Qwen3-32B audit are preliminary, and the argument should be treated as research to examine, not a solved verdict.
Ask ChatGPT to explain the argument, not endorse it
Give ChatGPT the essay and ask for a three-part summary: the earlier interpretation, Cammarata’s alternative, and the evidence offered for each. Then ask it to define “polysemanticity”, “superposition” and “mechanistic interpretability” in plain English. Have it use a cat-and-car example, then ask where the analogy stops being useful. That last question is often where the interesting work begins.
A prompt for reading research critically
Try: “Analyse this essay as a working hypothesis. List its central claims, the examples supporting each claim, what is directly measured, what is inferred, what remains speculative, and what independent replication would strengthen or weaken it. Do not treat the author’s confidence as proof.”
Use AI to build a reading map
Ask ChatGPT to identify the paper’s cited studies and create a reading order, then open the sources yourself. A useful model can help translate terminology and compare arguments. It cannot independently validate a neuron-level experiment unless it has the data and methods, and even then its explanation needs checking. The point of interpretability research is to understand what a model is doing. So the reader ought to practise the same discipline.
Comments