A small Miami company called Subquadratic recently made one of the boldest claims for some time in the field of AI: that it has broken a mathematical bottleneck that has constrained transformer architecture since its beginning.
The announcement was met with a mix of excitement and scepticism – one engineer captured the mood by calling the model either “the biggest breakthrough since the transformer” or “the Theranos of AI”. However, whether or not this company has done precisely what it says, there is a strong case for thinking that the next phase of AI progress may come less from making models larger, than from making them process information more like we do.
Go deeper with GlobalData
Access deeper industry intelligence
Experience unmatched clarity with a single platform that combines unique data, AI, and human expertise.
What the bottleneck actually is
The mechanism at the heart of every mainstream LLM is attention. When a transformer reads text, it compares every token with every other token to work out which words matter to which. That is what makes the models so capable, and also what makes them so expensive: double the input and you roughly quadruple the computation. This quadratic relationship gives Subquadratic its name, and is one of the reasons why LLMs are such notorious consumers of electricity.
Subquadratic’s answer is to swap this “dense” attention for “sparse” attention: attend only to the relationships that matter and ignore the rest. If you are reading a book, you do not weigh every word against every other. The difficulty, and the reason the idea has resisted a decade of attempts, is choosing which relationships to keep without losing the ones that matter. The company claims its mechanism selects them dynamically, differently for each input, in a way that avoids the quadratic compute growth at every stage.
Some caveats are important – one former OpenAI researcher noted that SubQ is “almost surely a sparse attention fine tune” of an existing open model (ironically possibly Deepseek), and Subquadratic has since conceded that it did not train from scratch, which cuts against the grander claim of having built “a new class” of LLM. Nevertheless, a third-party evaluation by the data firm Appen has backed some of the numbers, reporting near-perfect retrieval at six and 12 million tokens and frontier-level coding scores. So while Subquadratic’s strongest claims may be questionable, it does appear to have built something real and useful.
Why the Chinese labs are also the ones to watch
Whether or not Subquadratic has entirely cracked it, its work is part of a broader global move around sparse attention. Late last year, DeepSeek released an experimental architecture built around DeepSeek Sparse Attention. A lightweight “indexer” scores which tokens are worth attending to, and the model does the expensive work only on those; the result is a sharp cut in the cost of long inputs with, by its own account, little loss in quality – enough to roughly halve its pricing. Designs of this kind are now spreading across Chinese open-weight models.
While DeepSeek’s mechanism is also dynamic, Subquadratic’s chief technology officer argues that DeepSeek’s selection step is itself still quadratic, because the model must score the query against the whole context before it can decide what to ignore. Subquadratic’s claim (as yet unverified) is to have made the selection linear too, which is what would let context reliably stretch to twelve million tokens rather than the point where mainstream models, whatever their advertised window, start to lose the thread.
The brain got there first
While biological analogies have sometimes turned out to be blind alleys for AI research, it is worth noting that organic intelligence already seems to rely heavily on a form of sparse attention. The human brain supports an entire conscious life – perception, memory, language, movement – on roughly 20W, about the draw of a dim light bulb.
It’s hard to make a precise like-for-like comparison as there is no organic equivalent of a training-run, but, by contrast, the Encyclopaedia Britannica notes that the data centres supporting an LLM may consume millions of watts of electricity. Much of the gap comes down to selective attention. The brain does not fire every neuron for every piece of processing; only a small fraction are active at any moment. It has always been a sparse machine.
Prediction as the ultimate compression
These considerations are becoming more influential as AI Labs increasingly recruit philosophers and cognitive scientists who think about how minds work, with some of them already proposing hypotheses as to how the brain might be so efficient.
For instance, Anil Seth, professor of cognitive and computational neuroscience at the University of Sussex, and author of book “Being You”, has spent his career developing the idea that perception is not passive reception but active prediction – what he calls a “controlled hallucination” (as opposed to the uncontrolled “hallucinations” that can occur within AI).
The brain generates its best guess about the world and uses incoming data only to correct it; what travels up from the senses is not granular detail but only error against a previous prediction. The energy saving is enormous: the brain mostly processes what it did not anticipate. Sparse attention and predictive processing therefore tackle the same problem – not wasting effort on information you can safely ignore. The context in which the brain is operating (evaluating a physical world in order to move a physical body) is clearly different from that of an LLM (evaluating existing tokens in order to generate new tokens) but the use of context to determine where to best apply attention is common to both.
Follow the money
So where does this leave the future of AI? With an increasing focus on token cost, efficiency is likely to be a key differentiator for LLMs going forward. As Subquadratic argue, if the cost of long contexts falls by an order of magnitude, whole categories of application that were previously too expensive become viable, from analysing entire document sets in a single pass to searching a full codebase without today’s workarounds. This potentially also strengthens the hand of the Chinese who have, as a result of resource constraints, focussed on making compute as affordable as possible. However companies like Subquadratic may help to shift more attention (if you’ll pardon the pun) to innovative efficiency approaches in the West.
The likeliest near-term outcome is not that contemporary transformers are replaced by completely new designs, but that sparse and predictive techniques are quietly absorbed into the mainstream, the way efficient methods usually are. The brain worked out a key principle of efficiency a long time ago. AI appears likely to follow in some regard with potentially significant implications for the cost of compute. In a world of information-overload the concept of sparse attention would appear to be one well worth keeping an eye on.
