Edited by Shan Hui
Over a decade ago, Ilya Sutskever said something like this:
"Models just want to learn."
Dario Amodei recounted Ilya Sutskever's words in an interview. Image source: Dwarkesh Podcast.
As long as the problems of data and compute are properly resolved, models will find their own way to learn.
We all know what happened next. Models grew larger and larger. Training data volumes increased, compute investments skyrocketed, and model capabilities were unlocked layer by layer.
But when researchers focused their attention on recurrent networks and latent spaces, everything changed.
Recurrent models can perform repeated computations within a latent space. The idea is that by running the same set of network layers for a few more iterations, the model gets more computation time, allowing it to "think" a bit longer when encountering tough problems.
However, reality isn't quite so rosy.
Many models are only adept at using the recurrent depth they saw during training. If you force them to compute for a few more iterations, they might start sprinting down the wrong path.
Consequently, some people joke: some models just don't want to learn.
"Thinking More" by "Talking More"
Click on the "Deep Thinking" mode of a large language model, and you will see a long block of text.
It might look like it's full of "nonsense." The model is constantly trying and revising, sometimes even saying, "I need to rethink this problem."
This showcases the model's reasoning process. Although it can't be directly viewed as the complete internal computation process of the model, it does illustrate that the most mature approach for current Transformers to solve difficult problems is to let the model think while it speaks.
This is because a standard Transformer is a bit like a building with a fixed number of floors.
Information enters on the first floor, passes through each layer sequentially, and finally exits from the top floor. Whether it's facing "What is 1+1?" or a complex mathematical proof, the network depth it traverses when generating the next token is essentially the same.
What if one pass isn't enough? Then you just do a few more passes.
The model first generates a token, and then enters the next round of computation carrying the previous context. The longer the output, the more times the same set of networks is invoked, giving the model additional opportunities to break down the problem, check the answer, and revise it.
This method works remarkably well. It enables the Transformer, which originally had a fixed network depth, to flexibly allocate compute based on problem difficulty: say less for simple problems, say more for complex ones, and if all else fails, pit its own several candidate answers against each other.
However, this way of thinking is also somewhat inconvenient.
The internals of a model exist as high-dimensional, continuous hidden states. Because it must "speak" midway, many intermediate computations have to be organized into an outputtable sequence. Even if certain steps have absolutely no need to be spoken aloud, the model still has to generate something before it can continue moving forward.
Thus, another path emerged.
Looping Directly Within Network Layers
The idea behind recurrent models is very straightforward.
Since passing through the network once isn't enough, just go back to certain network layers and pass through them again. With each loop, the model updates its hidden state, reusing the exact same set of parameters.
Chain-of-Thought strings together multiple computations by generating tokens; latent space recurrence, on the other hand, repeatedly updates hidden states within the same set of network layers. Image source: Tilde Research.
The Universal Transformer, Deep Equilibrium Models, and recent recurrent Transformers are all pondering the exact same question: Can a model decide for itself how many times to loop in its head based on problem difficulty?
It actually seems feasible.
A 2025 latent reasoning study scaled a recurrent depth model to 3.5 billion parameters and trained it on 800 billion tokens. During testing, researchers ran the same compute module for a few extra iterations, and the model's performance on certain reasoning tasks genuinely continued to improve, ultimately leveraging the compute equivalent of a 50-billion-parameter model for a single forward pass.
Paper link: https://arxiv.org/abs/2502.05171
As test-time recurrence increases, the model's performance on tasks like GSM8K and HumanEval continues to improve; different tasks saturate at different depths.
But this number of compute iterations isn't a case of "the more, the better."
The recurrent language model Ouro, released in 2025, initially attempted to use 8 recurrent iterations during training, but quickly encountered loss spikes and gradient oscillations. Researchers subsequently reduced the recurrent depth for the main training phase to 4 iterations.
During the testing phase, they pushed the recurrence up to 8 iterations again, but the extra compute failed to yield better answers.
Taking the 1.4B parameter Ouro-Thinking as an example, its score on AIME 2024 gradually rose from 0 in the first iteration to 65 by the fourth; continuing to loop to the eighth iteration, the score dropped back down to 38.67.
In other words, the additional iterations were doing more harm than good.
What's Still Missing
Currently, there are roughly three major hurdles recurrent models need to clear.
First, theoretical computability: Is sharing the same set of parameters and recurrently updating hidden states sufficient to express complex computational processes?
Second, can the model genuinely learn this computational approach?
Third, after training is completed, can the model continue to stably "compute onward" during the inference phase?
There are already quite a few optimistic results regarding the first point; the troubles are mostly concentrated in the latter two.
Repeated invocations lead to error accumulation. As the number of iterations increases, the hidden state might look stable, but in reality, it can no longer support the final result.
The 2026 Parcae research found that recurrent language models are prone to residual explosions and sudden loss spikes.
After redesigning the stabilization mechanism during the recurrent process, the model's validation perplexity dropped by up to 6.3%; when scaled to 1.3 billion parameters, it also outperformed the traditional Transformer baseline under fixed parameter and data quantities.
Paper link: https://arxiv.org/abs/2604.12946
The recurrent state norm of a standard recurrent baseline explodes rapidly, and the training loss subsequently stops decreasing; Parcae and the residual normalization model, which incorporate stabilization mechanisms, remain stable.
This outcome offers a glimmer of hope to the story that "models don't want to learn."
Today's standard Transformer already possesses a whole suite of training gear honed over many years: residual connections, normalization, initialization methods, learning rate strategies, and optimizers—they are all very familiar in practice.
Compared to the long-refined standard Transformer, the training recipe for large-scale recurrent language models is still far from mature.
How architecture, optimizers, loss functions, and intermediate supervision should be paired together likely still requires exploring more handy combinations.
Reference links:
https://blog.tilderesearch.com/blog/one-layer-deeper
© THE END