In the summer of 2017, AI researchers at Google published a paper titled “Attention Is All You Need.” The paper described a new kind of neural network called the Transformer.
It later turned out that Transformer networks are remarkably good at handling long sequences of data—especially text.
Today, in 2026, the Transformer is the core engine of virtually every mainstream large language model (LLM) on the market. “The entire AI industry is built on transformers,” says Justin Dangel, co-founder and CEO of the AI startup Subquadratic. “They are one of the most important innovations in the history of computer science, and they have already changed the whole world.”
But at the same time, the technology is showing its age. Many of the recent advances in large language models—such as the development of reasoning models and the ability of models to process enormous inputs in one go—are not natural extensions of the core technology. They look more like makeshift patches applied over certain fundamental flaws.
More and more scientists and engineers are starting to ask: How will the technology evolve? What shape will the next generation of AI take? We believe LLMs themselves will not disappear—but how will they be built in the future? Nothing has been settled yet.
A crop of startups is racing into the field, seeking to break new ground at the frontier of existing technology. Some of these contenders are bound to fail, but these upstarts have enormous potential—and, compared with today's industry leaders, they may actually hold the competitive edge.
An Aging Network
First, we need to clarify the existing technical problems. The Transformer's core strength comes from what is known as “dense attention,” a mechanism that encodes the meaning of a piece of text through a series of numbers. During processing, every word in the text—or every part of a word, known as a token—is compared, through multiplication operations, with all the other words or tokens.
Dense attention captures meaning with extremely high precision. But as text grows longer, the computation required to process it climbs steeply. A document of 10,000 English words may require the Transformer to perform 50 million multiplication operations. This is the main reason large language models consume so much energy.
Computing power is expensive. According to OpenAI president Greg Brockman, OpenAI's computing spending this year will reach $50 billion. The International Energy Agency predicts that by 2030, the total electricity consumption of data centers will double.
Worse still, Transformer networks often fall short on the tasks that many new models are being asked to perform. Limited by the way they process text word by word, Transformers are not good at holding large amounts of information simultaneously. In other words, their context window cannot be scaled up very far.
Yet if LLMs are to take on harder tasks, they must be able to take in much larger volumes of data—an entire library of documents, an entire codebase, or the outputs of other large language models.
Reasoning models work by first writing notes to themselves using a chain of thought as a kind of scratchpad, then reading those notes to continue reasoning. This approach also increases the volume of data the model must maintain.
As LLMs grow larger and more powerful, the Transformer bottleneck becomes more conspicuous. What was once their core strength has now become a limitation.
Below are four new approaches aimed at addressing the limitations of Transformer networks. These innovations could thoroughly reshape large language models, making them faster and dramatically more efficient—and perhaps even more intelligent.
Path 01: Redesigning the Attention Mechanism
To make large language models run faster and cost less, one option is to strike directly at the pain point: change the attention mechanism, replacing dense attention with sparse attention.
Sparse attention performs computations on only some of the word pairs in a block of text, rather than all of them. This can sharply reduce the amount of computation an LLM requires.
Over the years, researchers have proposed a great many sparse attention mechanisms, but they have never matched dense attention in capturing meaning. These days, of course, that may no longer be true.
Subquadratic, the startup mentioned above, claims that a sparse attention mechanism it developed is the first to demonstrate performance on par with mainstream top-tier LLMs across a range of tasks, including search and code generation. If true, that claim would be a big deal—though some in the industry remain skeptical.
Subquadratic says its model, SubQ, judges in real time which words are important and which are not for each segment of the input text. It claims tens of thousands of users have signed up for the waiting list to try SubQ, and that the model will soon be open to the general public.
Another startup, Manifest AI, has taken a different tack: rather than reworking how the attention mechanism operates, it replaced the mechanism outright with an entirely new scheme.
The company has developed a mechanism it calls “power retention.” Power retention keeps only the information most relevant to the current task, ensuring that the volume of data the model must maintain does not explode.
Traditional attention mechanisms require an LLM to remember everything within its context window. Sparse attention models such as SubQ, while discarding a great many individual words, still retain an outline of what they have seen. By contrast, power retention gives the model a rolling summary of the contents of its context window—that is, of all the text it can currently see—so that when new information arrives, older and less relevant information is dropped.
The basic principle of power retention has been around for about a decade. Manifest AI claims it has iterated on the technology and built, for the first time, a product comparable to Transformer-based large models. What's more, a Transformer model can be converted into a power-retention model with only a small amount of retraining.
To demonstrate that conversion capability, the team adapted the existing open-source code model StarCoder into PowerCoder, equipped with the power retention mechanism, and also released the Brumby model, which it says matches the performance of certain versions of Alibaba's Qwen (通义千问).
Manifest AI's vision is that when large language models need to handle massive-data tasks, power retention will be the go-to solution. Carles Gelada, the company's co-founder and chief technology officer, has said their approach has a wide range of applications—from analyzing videos lasting several hours to building AI agents capable of carrying out tasks for weeks on end.
Path 02: Building Smaller, More Flexible Models
Liquid AI, a startup incubated at MIT, did not overhaul or abandon the Transformer. Instead, it combined its own liquid neural networks with the Transformer to build what it calls “liquid foundation models,” or LFMs.
Liquid AI's models are far smaller than the vast majority of LLMs and consume less energy as well. The company has already developed models for automakers including Mercedes that fit on small in-car chips; its latest version can even run on a $50, low-power Raspberry Pi development board.
It says businesses with annual revenue below $10 million can use Liquid AI's products free of charge, and the models have been downloaded tens of millions of times, drawing widespread praise.
Liquid neural networks are an extension of convolutional neural networks—another class of neural network that existed before the Transformer—and were inspired by the worm brain. What makes liquid neural networks stand out is a self-adaptive mechanism that lets the model dynamically adjust its own behavior in response to new information, learning as it runs. Transformer-based models cannot do this: once training is complete, their behavioral patterns are fixed.
Liquid AI's early models were relatively basic, but they could already pilot drones and drive vehicles. Now, with liquid foundation models, the company is trying to scale the technology to compete with mainstream large language models. The new models can match the performance of competitors four times their size, including certain versions of Alibaba's Qwen and Google's open-source Gemma.
A typical LLM is assembled by stacking many Transformers. Liquid AI's latest LFM has a hybrid architecture consisting of 20% Transformer and 80% liquid neural network.
The ratio was not calculated by a human, but by another AI system the company built specifically to assist in model design. This “AI that designs AI,” it says, generates a large number of combinations of different neural networks—liquid networks, convolutional networks, Transformers—and screens the combinations to find the best balance between performance and efficiency.
Ramin Hasani, co-founder and CEO of Liquid AI, believes the future of neural networks is full of possibility, and that the Transformer is only a beginning.
“The human brain is a general intelligence system, and it runs on just 20 watts. How does it do that? There is plenty of room for us to innovate further.”
— Ramin Hasani
Path 03: Generating All the Text at Once
Nearly all large language models output content word by word. That suits the way humans speak and write, but it is extremely inefficient for a computer. If an LLM could generate text all at once—spitting out a whole sentence, or even a whole paragraph, in one breath—both speed and cost would improve.
One of the practitioners of this idea is Inception, which is using diffusion technology to build large models.
Diffusion technology is better known as the driving core of the vast majority of image and video generation models. Diffusion large models are trained by first receiving a random grid of pixels (like the static noise on an old television set), then processing all the pixels simultaneously to determine which ones need to be changed, turning the static noise into a high-definition image.
It turns out the same process works for text. Through specific training, the Inception team enabled a diffusion large model to take a random string of words and turn it into a coherent sentence. This kind of diffusion LLM still uses a Transformer to encode meaning, but it outputs an entire block of text at once, accomplishing more at lower cost.
Stefano Ermon, the company's co-founder and CEO, says their innovation is still a large Transformer model, but one that can predict multiple tokens simultaneously. That is why—compared with the overwhelming majority of approaches on the market—the model generates text far faster and more cost-effectively.
The hard part is transferring technology originally designed for image generation to the realm of text.
“With images, you can change a blue pixel to a red pixel by adjusting gradually through intermediate colors. But text doesn't work that way—there is no in-between word between ‘cat' and ‘dog.'”
— Stefano Ermon
Ermon is also a researcher at Stanford University. In 2024, he and two Stanford colleagues found the mathematical method for bridging the text gap, and on that basis trained a diffusion model suited to text, whose performance matched GPT-2, released by OpenAI in 2019, while generating text 10 times faster.
Building on that work, Ermon founded Inception. Now his ambition is aimed at the front rank of the industry. Inception says its latest model, Mercury 2, matches the performance of certain versions of GPT-4 released by OpenAI in 2023, and is likewise 10 times faster.
“We are very bullish on this path; it has the potential to scale. At the end of the day, speed and cost are the core measures of a large model's value—how much intelligence do you get for every dollar you put in?”
— Stefano Ermon
Inception is not the only company betting on diffusion technology. Google is also actively experimenting, and has built a prototype LLM called Diffusion Gemma.
Path 04: Leaving the World of Text Behind
Among the startups exploring innovations around the Transformer's limitations, Pathway's philosophy is the most disruptive. It wants to free large language models from the constraints of language, and to that end it developed the Dragon Hatchling model.
Its most eye-catching achievement to date came in a benchmark pitting models against more than 250,000 extremely difficult sudoku puzzles. Mainstream large models from several leading labs could not solve even one puzzle, yet Dragon Hatchling cracked more than 97% of them.
Pathway wants to make the point that although mainstream LLMs perform well on a great many tasks, they remain powerless against many critical classes of problems—sudoku is just one example.
Zuzanna Stamirowska, the company's co-founder and CEO, says that for large models to produce genuinely original solutions to real-world problems, they must step outside the Transformer framework. Transformers force large models to do all their reasoning through text, but language is not the optimal carrier for certain forms of reasoning.
Pathway's solution is to modify the underlying mathematics of the Transformer, replacing the attention mechanism with a mathematical structure known as “state space.”
Instead of encoding information word by word, state space compresses it into a more abstract representation. On that basis, Dragon Hatchling can still read and write text, while also simulating reasoning patterns that do not depend on sequences of words. This not only improves the model's efficiency but, in theory, also equips it to handle tasks other LLMs cannot.
“Chess and math problems do not exist in the human mind in the form of long sentences. A flash of insight in the brain does not necessarily take linguistic form. We think that if reasoning must be carried out through language, that is itself a limitation. Mainstream LLMs can read a sudoku tutorial and then write code to solve the puzzle—but that is not what we are after. The model we envision is not confined to what is in the books.
The grand ambition of AI is not to solve sudoku; it is to conquer cancer—yet there is no ready-made manual for curing cancer.”
— Zuzanna Stamirowska
Source: These startups are chasing the next big thing in LLMs