Beyond Prediction: The Five Realms of 'Memory' in Dynamics Learning

Introduction

From Takens' Embedding Theorem to Koopman operators, from RNNs and Transformers to Neural ODEs and TimeGPT, the history of time series prediction is essentially an evolutionary history of "machine memory." Recently, Professor Zhang Jiang from Jizhi Academy systematically traced the development of dynamics learning, revealing a core question that has spanned half a century: How should a machine represent the past to predict the future? Using "memory" as the main thread, this article connects the development of dynamical systems theory, deep learning, and foundation models, showcasing how artificial intelligence automatically discovers the evolution laws of complex systems from data and gradually moves toward universal prediction capabilities across domains and systems.

Keywords: Dynamics Learning, Time Series Prediction, RNN, Transformer, Diffusion Models, Complex Systems

By Fu Wenxin

Introduction: From "Guesstimation Modeling" to "Data-Driven Discovery," We Use Memory to Teach Machines to Predict a Complex World

Prediction is humanity’s oldest and most cutting-edge intellectual activity. Ancient ancestors observed the stars to predict the seasons; Newton used the law of universal gravitation to predict planetary orbits; modern scientists use supercomputers to predict climate change. The essence of these predictions is the same: deducing an unknown future from an inferred known model.

However, with the explosive development of observation methods (such as satellite remote sensing, EEG recordings, and IoT sensors), we often possess massive amounts of time series data but know very little about the underlying dynamical mechanisms. The problem is thus inverted: Given observational data, can we automatically infer the system's evolution rules?

This is dynamics learning—letting machines automatically discover the rules of system evolution from observational data.

In 2025, this direction reached two landmark moments. Microsoft's Aurora model used a unified framework to simultaneously predict typhoon trajectories, air pollution, and wave heights, publishing its results in Nature. Meanwhile, TimeGPT shoehorned a vast array of time series forecasting tasks into a single pre-trained model, achieving zero-shot prediction.

These stunning models result from half a century of researchers repeatedly asking the same question:

"How exactly should 'the present' be memorized so that 'the future' can naturally grow out of it?"
\n

This question sounds abstract, but it is the hidden thread weaving through the entire evolutionary path. This article calls it the "pursuit of memory"—that is, How do models (or mathematical frameworks) represent past and present states (processing memory), and how do they predict the future (using memory)? This process is summarized into the following five stages, corresponding to the five chapters of this article.

I. Is Memory Useful? — Takens' Theorem: Past Observations Contain a Complete World

1.1 Forward and Inverse Problems

To understand what dynamics learning does, we must first distinguish between two diametrically opposed problems.

The forward problem is the traditional approach: knowing the system's evolution equation and an initial state, find the future trajectory. For example, in astrophysics, knowing a planet's laws of motion, its position, and its speed today allows you to calculate where it will be tomorrow. Mathematically, this is written as:

where is the system's state vector at time t (possibly containing the values of n nodes or variables), and F is the known evolution rule. Substituting s₀ into F gives s₁; substituting again gives s₂—and so on, the entire trajectory unfolds.

The inverse problem is much more common in the real world: F is unknown. We can only observe the data generated by the system's operation, and the task is to learn the hidden F from this data.

The learned is a surrogate model for the original system. With it, we can make predictions (infer the future from the past), run simulations (see how the system evolves under different initial conditions), and even perform optimization and control (find the best intervention strategies).

Here is a crucial mental shift: our ultimate goal is to learn F, but the method employed is to construct a supervised learning problem—predicting the next time step.

Train a model f_θ with parameters θ to predict the next state from the previous one. If the prediction is sufficiently accurate, then f_θ is a good approximation of the true dynamics F.

Figure 1: The forward problem (from right to left) deduces data from a model, whereas the inverse problem (from left to right) deduces a model from data.

Figure 1 illustrates these two directions. Superficially, we are doing prediction, but in reality, we are doing modeling. Learning the dynamical laws is the true objective.

1.2 From One Step to Multiple Steps: Latent Variables and Long-Range Memory

The previous example essentially performs one-step prediction, which rests on the assumption of Markovianity: the next moment depends only on the current moment. However, real-world systems often violate this assumption due to latent variables. The observed variables may be only a part of the entire system state—some dimensions are invisible, and these are often the real causes or more influential variables. This is analogous to us easily observing other people's external behavior but not knowing what they are thinking or how their thoughts are evolving.

These invisible dimensions silently influence the system's evolution, manifesting in the observable data as long-range memory: today's state depends not only on yesterday but also potentially on a week or a month ago. In such cases, one-step prediction is insufficient, and we need to construct a multi-step prediction problem:

The input becomes the history of the past T steps, and the output is a prediction for the future τ steps. This framework better reflects real-world scenarios and is also the arena where architectures like Transformers later shine.

1.3 Takens' Theorem: Reconstructing Phase Space with History

The existence of latent variables seems to make the problem unsolvable—how can we learn the true dynamics if we cannot observe the complete state? Takens' theorem (the embedding theorem) [1] offers a surprising theoretical guarantee: the history of a single variable contains information about other latent variables.

For example, today's temperature, yesterday's temperature, and the day-before-yesterday's temperature—these time-lagged values implicitly contain information about variables like atmospheric pressure and humidity that were not directly measured. Takens' theorem guarantees the sufficiency of this information. As shown in Figure 2, the theorem states that even if you can only observe a single scalar output x(t) of a system, as long as you arrange its time-delay embedding into a vector:

then, provided the observation function is sufficiently "generic" and the embedding dimension d is large enough (specifically, d > 2n, where n is the true dimension of the system), the reconstructed phase space is topologically equivalent to the original phase space. Put simply, even if you only stare at one variable, if you watch it long enough, the memory stored in that variable is sufficient to reconstruct the entire dynamical structure of the system.

Figure 2: Core idea of Takens' theorem: Using only one dimension (e.g., x(t)) of a Lorenz system's time series, a time-delay embedding can establish a diffeomorphic relationship with the attractor of the complete three-dimensional system (x(t), y(t), z(t)).

*Note: A diffeomorphism refers to a bijective map between two smooth manifolds such that both the map and its inverse are smooth (infinitely differentiable), indicating that the two manifolds have the same smooth structure. One can imagine two shapes being kneaded like play-dough into each other (topological homeomorphism), with the entire deformation process being smooth and seamless, without any sharp corners, creases, or tears—much like smoothly deforming a donut into a coffee mug without forcefully creating sharp edges.

This means: Dynamic information is not mysterious; it resides within the temporal correlations of a time series. This idea laid the philosophical groundwork for all subsequent time series prediction methods—the past contains the seeds of the future.

1.4 Chapter Summary

This chapter answers a fundamental question: Is it even theoretically possible to learn dynamical laws solely from observational data?

The answer is yes, but the presence of memory makes things appear tricky: what we observe might be just the tip of the iceberg, with the variables truly driving the system hidden beneath the surface. Today's weather depends not just on yesterday but perhaps also on the atmospheric pressure distribution over the ocean a week ago.

Takens' theorem offers hope in this seemingly desperate spot: You don't need to see all variables; you just need to look at one variable long enough. Time delay itself acts as a mirror, mapping out hidden dimensions. Past memory is the raw material for reconstructing a complete world.

However, Takens' theorem only guarantees that memory is useful. Between being useful and being usable lies an entire methodology. It's like someone telling you, "There's definitely a gold mine in this mountain," but not giving you a map or a shovel.

In the next chapter, we will find the first shovel for extracting memory: the Koopman operator will tell us that a special pair of "glasses" exists, through which the entire nonlinear system looks linear, and predicting a linear system is nothing more than matrix multiplication.

II. How to Extract Memory? — Koopman Operator: Switching Glasses to Make Curves Straight

2.1 Koopman Operator: Making the Nonlinear Linear

Takens tells us we can reconstruct the phase space, but after reconstruction, the problem remains thorny: real-world systems are almost always nonlinear, meaning linear methods cannot be directly applied.

However, Koopman operator theory (proposed in 1931, recently rediscovered) [2] provides a workaround: In function space, nonlinear dynamics can be described by a linear operator. For a deeper dive into the Koopman operator, Professor Lan Yueheng has a more in-depth introduction at Jizhi Academy, which interested readers can further explore at https://pattern.swarma.org/study_group_issue/747.

The Koopman operator offers a brilliant detour: a nonlinear system is difficult to handle in finite-dimensional space, but if you lift it into an infinite-dimensional function space, its evolution can become linear. Specifically, the Koopman operator is defined on the system's observables. Suppose the system state x evolves according to the nonlinear rule x_t+1 = F(x_t). For any observable function g (like "take the square of the temperature value"), the Koopman operator is defined as:

It maps the value of function g at the current state to the value of g at the next time step's state. Even though F is nonlinear in finite-dimensional space, is linear as an operator on the function space. A linear problem is essentially a matrix problem, for which we have a mature set of mathematical tools (like eigenvalue decomposition and spectral analysis). That is its value.

However, the cost of this method is that the function space becomes infinite-dimensional. In practical computation, we must select a finite set of basis functions to approximate the Koopman operator. This is precisely what DMD and its variants do.

2.2 DMD and eDMD: Approximating Koopman from Data

Theoretically, the Koopman operator is infinite-dimensional; practical computation requires a finite-dimensional approximation. Dynamic Mode Decomposition (DMD) is a numerical method for doing just this. As shown in Figure 3, its logic is extremely concise: given time-series data matrices and (which is X shifted one step forward in time), find a linear operator A such that Y ≈ AX. The least-squares solution († denotes the pseudo-inverse) is a finite-dimensional approximation of the Koopman operator [3].

By numerically solving for the eigenvalues and eigenvectors of A, we can reveal the system's dominant dynamic modes—which modes are growing, which are decaying, and what their oscillation frequencies are. For instance, in fluid mechanics, DMD modes exactly correspond to the vortex shedding frequencies in a flow field; in EEG signal analysis, DMD modes correspond to the rhythmic activity of different brain regions.

Figure 3: DMD constructs matrices X and X′ from time series data, learns a linear dynamics operator via A = X′ X†, and then uses x_k+1 = A x_k to predict future states.

Later, Extended DMD (eDMD) introduced nonlinear basis functions ψ(s). Instead of performing DMD directly on the original state, it first lifts the state into a high-dimensional feature space using a set of nonlinear basis functions (like polynomials, radial basis functions, or neural networks). This essentially approximates the Koopman operator using a "kernel trick" approach [4].

2.3 Reservoir Computing: The Miracle of Fixed Random Networks

If Koopman methods approach dynamics from the mathematical path of "dimensionality-raising linearization," Reservoir Computing delivers an answer from the engineering path of "high-dimensional random representations." As shown in Figure 4, its core idea is also simple: use a fixed, randomly initialized recurrent neural network as a "reservoir," and only train the final output layer [5-7].

Because Koopman operator theory is essentially a form of high-dimensional random projection, the intuition is that a sufficiently large random recurrent network with suitable dissipative properties naturally forms a "random feature library." The input signal is expanded into a diverse set of transient response patterns in the high-dimensional space, and a linear combination of these can then approximate any nonlinear function [8].

Figure 4: Basic architecture of reservoir computing: The input signal x_in enters a high-dimensional, randomly connected, and fixed recurrent reservoir (Reservoir). Its internal states x₁, x₂, ..., x_N are linearly combined to produce the output . Only the output layer weights w_i need to be trained.

The reservoir's random recurrent connections provide rich nonlinear transformations and memory capacity for the input signal. As long as the reservoir is large enough and the connections are diverse enough, it can "unfold" the input signal into a high-dimensional space, allowing a simple linear output layer to perform complex prediction tasks. Reservoir computing methods endure because they are extremely fast to train (only the output layer is optimized) and are well-suited for chaotic systems (their random nature shares some characteristics with chaotic systems). On classic chaotic systems like the Lorenz system and the Kuramoto-Sivashinsky equation, reservoir computing can achieve long-term prediction capabilities that surpass many deep models at a fraction of the computational cost [9-10].

2.4 Chapter Summary

Since Takens' theorem guarantees that useful memory information is hidden within historical observations, how do we extract it? The answer is an elegant detour: Don't brute-force the nonlinearity; switch glasses to make the winding path straight.

The core insight of the Koopman operator is exactly this: a nonlinear system twisting and turning in a finite-dimensional space, if observed from a higher-dimensional (even infinite-dimensional) function space, appears to evolve linearly.

DMD and eDMD are the numerical implementations of this theory, approximating the infinite-dimensional Koopman operator with a finite set of basis functions. Reservoir computing arrives at a similar destination from a completely different direction—instead of carefully designing basis functions, it uses a large, randomly initialized recurrent network to provide a sufficiently rich high-dimensional expansion space.

But this path hits a ceiling: the basis functions must be chosen by humans. When a system is sufficiently complex (hundreds of variables, highly nonlinear couplings, multi-scale temporal structures), no one can guarantee which set of basis functions is correct.

In the next chapter, we will introduce how to allow models to learn to decide how to remember on their own.

III. Letting the Model Learn Memory Itself: RNNs, CNNs, and Transformers

With the explosive growth of observation methods, we have more and more data—satellite images, EEG signals, sensor readings. Even the top scientists simply cannot redesign observation functions for every new system in time. This raises a bold question: Can we let the model itself learn how to remember?

This is essentially what backpropagation does. Gradient descent gives the model the ability to "autonomously" learn how to compress information, how to forget noise, and how to extract patterns. The most naive approach is to use a feedforward neural network to directly fit the mapping , using past values as input to directly regress the next moment. With enough neurons, it can approximate any function. However, the problem is that this approach lacks prior knowledge of temporal structure; it treats inputs from t-1 and t-10 equally, and the number of parameters explodes as the historical window length grows.

3.1 RNNs: Learning from Implicit Memory

The emergence of Recurrent Neural Networks (RNNs) gave dynamics learning "memory" for the first time. As shown in Figure 5, unlike feedforward networks, RNNs have recurrent connections between hidden layer units, allowing the network to preserve historical information. For time series prediction, an RNN can recursively update its state like a differential equation [11]:

Figure 5: RNN structure (example): The hidden states h₀ to h₄ in the encoding part are recursively passed along time steps, receiving inputs x₁ to x₄ at each step, and realizing the memory and transmission of sequence information through shared parameters.

The hidden state is like a memory unit that can, in theory, accumulate information from all past time steps. But RNNs have a fundamental structural bottleneck: they are computed sequentially—you must compute h₁ first, then h₂, then h₃. This makes training non-parallelizable and inefficient on long sequences [12].

Another issue is memory decay. During backpropagation in early RNNs, gradients could vanish or explode exponentially. In 1997, Hochreiter and Schmidhuber proposed the LSTM, using gating mechanisms (input gate, forget gate, output gate) to let the network itself decide when to remember and when to forget [13–14]. Its simplified version, the GRU, was proposed later by other researchers [15]. These variants allowed RNNs to remember dependencies over a hundred steps back.

The newest members of the RNN family, such as RWKV and Mamba, are challenging the dominance of Transformers. RWKV uses a form of attention mechanism combined with an RNN structure, designing a model that can perform recursive inference like an RNN (linear complexity) while being trainable in parallel like a Transformer (through prefix sum calculations across the time dimension). This is yet another redefinition of "memory" [16-19].

3.2 CNNs: Treating Time as Space

A Convolutional Neural Network's (CNN) understanding of memory is that local information is more helpful for judging local patterns. At first glance, CNNs seem exclusive to image data.

The operation of a one-dimensional convolution is intuitive: a kernel w of length k slides over the time series, computing a dot product at each step:

By stacking multiple convolutional layers, the receptive field grows exponentially. Each neuron in the L-th layer can theoretically see k^L original time steps—preserving locality while covering long-range dependencies.

This trick works well on periodic data, allowing the periodic patterns to be processed in a higher dimension, as shown in Figure 6. For instance, electrical load data typically has a 24-hour cycle. By "stacking" multiple cycles into a two-dimensional image—with the horizontal axis representing time (hour) and the vertical axis representing the cycle (day)—and then processing it with a 2D CNN, you can very intuitively handle patterns occurring at the same time of day across different cycles (like every evening). This essentially transforms temporal periodicity into spatial periodicity [21].

Figure 6: Two ways CNNs model time series: on the left, a 1D CNN directly processes the raw sequence; on the right, the 1D sequence is reshaped into a 2D array, allowing a 2D CNN to simultaneously capture intra-period patterns and inter-period dependencies.

The choice between CNNs and RNNs is essentially a trade-off between parallelism and memory length. CNNs can be trained fully in parallel (the convolution calculation for each time step is independent), but their receptive field is limited by the number of layers; RNNs can theoretically remember infinitely far back, but must be computed sequentially. This trade-off was not broken until the advent of the Transformer.

3.3 Transformers: Letting All Positions Talk Directly

The Transformer's self-attention mechanism fundamentally changed the way sequence modeling works. It allows any two time steps in a sequence to connect directly [17]:

Each time step can "see" all others; long-range dependencies no longer need to be passed indirectly through intermediate states.

However, Transformers face several issues in time series forecasting.

The first issue is complexity. The computation cost of standard self-attention is quadratic in sequence length. If you feed in a thousand steps, you need millions of attention scores. The Informer paper discovered that these scores exhibit a "long-tail distribution": a very small number of position pairs contribute the vast majority of the attention, while most are close to zero, wasting computational resources.

Informer uses ProbSparse self-attention to reduce the computational load [22]. Through a sampling strategy, it only retains the most important few key-value pairs, reducing the complexity from O(L²) to O(L log L). Combined with a generative decoder that outputs long sequences at once, Informer made long sequence prediction feasible.

Figure 7: Embedding and attention module in the Informer model: The input sequence at time steps t and t + D_x goes through embedding and 1D convolution, then through a multi-head attention mechanism and multiple attention blocks (like Attention Block 2) to capture dependencies in long sequences, enabling efficient long-term time series forecasting.

The second issue is more subtle. When standard Transformers process multivariate time series, the default approach is to concatenate all variables at each time step into a single token. That is, "temperature, humidity, wind speed" at second t and "temperature, humidity, wind speed" at second t+1 attend to each other. Temperature and humidity are repackaged into the same vector every second. The model doesn't know which dimension is temperature and which is humidity; causal relationships between variables (e.g., temperature increasing causing humidity to drop) can only be indirectly transmitted through attention across time steps.

The iTransformer method made a seemingly "heretical" improvement: it inverted the traditional usage of the Transformer [23]. As shown in Figure 8, the original method treats "all variables at each time step" as a token. iTransformer instead treats "the entire time series of each variable" as a token. Thus, attention shifts from the time dimension to the variable dimension. The model directly learns the relationship between the temperature series and the humidity series, rather than the relationship between the 3rd and 4th second. The temporal pattern of each variable is preserved intact in the embedding space and processed by another branch (a feed-forward network). This inversion allows iTransformer to better model inter-variable correlations and intra-variable temporal patterns in multivariate time series, achieving significant improvements on multiple long-sequence prediction benchmarks.

Figure 8: Core architectural difference between a standard Transformer and iTransformer when handling multivariate time series: The traditional Transformer splits tokens by time step, while iTransformer splits them by variable, shifting attention from "temporal dependency" to "variable dependency."

3.4 Chapter Summary

From Koopman to RNNs, CNNs, and Transformers, deep learning has finally transformed memory from a mathematical construct into a learnable object. Gradient descent has endowed models with a new ability: through trial and error, they can autonomously learn what to remember, what to forget, and how to organize it.

For the problem of covering as far back in the past as possible with limited computational resources, these three deep learning methods offer different trade-offs. RNNs chose depth (more time steps, deeper propagation), CNNs chose width (more layers, larger field of view), and Transformers chose direct connections (sacrificing computation for less information attenuation).

The emergence of Informer and iTransformer shows that even within Transformers, optimization is far from over. Informer asked, "How much of the attention is wasted?" iTransformer asked, "On which dimension should attention be applied?" The essence of these questions remains the same: how to make memory more efficient and more precise.

But whether RNN, CNN, or Transformer, they share two common implicit assumptions: first, that time is a discrete grid; second, that the same piece of memory points to only one definite future.

In the next chapter, we will challenge both assumptions simultaneously: diffusion models turn a prediction from a single point into a probability cloud, and Neural ODEs turn time from discrete into continuous.

IV. Letting Memory Embrace Reality: The Uncertainty of Diffusion Models and the Continuity of Neural ODEs

4.1 Diffusion Models: Turning Predictions into Probability Distributions

Deterministic prediction gives only a point estimate, but the real world is full of uncertainty. When a weather station predicts tomorrow's temperature, what we see on our phones is also just a distribution. Diffusion Models introduce the probabilistic generation framework into time series forecasting, outputting a distribution of possible trajectories.

Diffusion models were not born for time series, but they naturally fit conditional generation tasks. For instance, in the classic DDPM (Denoising Diffusion Probabilistic Model), the forward process gradually injects noise into the data, eventually transforming it into pure random noise; the reverse process is a "denoising generation" process—starting from pure noise and step by step restoring the original data. The training objective of DDPM is to predict the noise added at each step, thereby learning the process of generating data from noise [24].

Taking TimeGrad as an example, it combines DDPM with an autoregressive time series model. At each prediction time step, it generates a distribution of predicted values through a step-by-step denoising process, quantifying prediction uncertainty—which is particularly important for risk-sensitive decisions (finance, meteorology) [25].

4.2 Residual Networks: A Method to Increase Frame Rate

Before diving into Neural ODEs, let's reinterpret what residual networks do. The residual network (ResNet) was a breakthrough method that allowed the number of neural network layers to be dramatically increased. The operation performed by each layer of a residual network is [26]:

The network does not learn the next layer's representation from scratch. Instead, it learns how much the current layer should change, i.e., the "residual" , and adds it to the current state. This design allows gradients to flow back losslessly through an identity shortcut, solving the vanishing gradient problem in deep networks and enabling networks hundreds of layers deep.

Rewriting the ResNet recurrence into the following form:

The left side is the amount of change, and the right side is a function of the current state. One might notice this formula looks very much like a differential equation.

Indeed, from a mathematical perspective, the Euler discretization of the continuous differential equation is precisely:

When the step size Δt = 1, it is exactly the form of a ResNet.

This implies: a ResNet is essentially the Euler discretization of some continuous dynamical system. The layer number t corresponds to time, f corresponds to the velocity field, and h_t corresponds to the system state at time t. From this perspective, the now-familiar deep network is not blindly stacking layers of abstraction; it is simulating a continuous-time evolution process.

This naturally leads to the question: If a ResNet is a discretized differential equation, why not solve the continuous equation directly?

4.3 Neural ODEs: Turning "Layers" into "Time"

In 2018, Tian Qi Chen et al. published "Neural Ordinary Differential Equations," the famous Neural ODE method [27]. It quickly garnered a huge number of citations and remains one of the most elegant works fusing mathematics and deep learning. As shown in Figure 9, its core idea is strikingly simple: do away with discrete layers and let the hidden state evolve continuously according to a differential equation parameterized by a neural network:

Here, f_θ is a neural network, and its parameters θ are independent of time t, meaning there is parameter sharing at every moment. Given an initial state h(0) (the network's input) and solving this differential equation, you obtain the state at any given time. The network's output is h(T), where T is the chosen end time. This integration process can be computed using any numerical ODE solver (like Euler's method, Runge-Kutta methods, etc.):

This operation, which seems only to turn discrete layers into continuous integration, solves all three problems left over from ResNets at once:

First, infinite depth, finite parameters. Because the parameters θ of f_θ are shared across all layers (moments), the number of parameters does not grow with depth and can be treated as a constant. If you subdivide the time interval [0, T] arbitrarily fine, it's equivalent to having arbitrarily many layers, without an increase in storage overhead.

Second, depth-adaptivity. The ODE solver (like the Dormand-Prince method with adaptive step size) automatically adjusts the step size based on the rate of dynamic change—taking large steps when changes are smooth (fewer layers) and small, fine-integration steps when changes are drastic (more layers). The network's effective depth changes from a pre-specified hyperparameter to being data-driven.

Third, continuous-time modeling. The discrete concept of a "layer" disappears, replaced by continuous time t. With this modeling approach, you can query the system's state at any moment—not just integer time points, but also intermediate moments like t = 1.2345.

Figure 9: Core difference between residual networks and ODE networks: The residual network on the left maps state step-by-step through a finite number of discrete layer transformations (depth n is finite). The input is x₀, the output is y. The forward computation produces a series of discrete activation values, and the loss is L(y). Backpropagation adjusts each layer's parameters W_i. The ODE network on the right defines a continuous vector field dz/dt = f(z, θ). The state evolves continuously over time, theoretically corresponding to infinite layers. The input is the initial condition z=z₀, the output is the solution at time T, z(T). The forward computation produces a continuous trajectory z(t), and the loss is L(z(T)). Parameters θ are adjusted via the adjoint equation, rather than layer-by-layer backpropagation.

4.4 The Adjoint Method: Replacing Backpropagation with Differential Equations

Traditional backpropagation in deep networks requires storing the activations of all intermediate layers: the deeper the network, the larger the memory footprint. A 1000-layer ResNet requires storing 1000 layers of intermediate results. If Neural ODEs extend the number of layers to infinity, would memory requirement also be infinite?

The Adjoint Sensitivity Method solves this problem. It does not store the intermediate states of the forward pass, but instead calculates the gradients backward by solving another differential equation, the "adjoint equation" [28].

Specifically, define the adjoint state:

that is, the gradient of the loss function L with respect to the hidden state at time t. It can be proven that a(t) satisfies another differential equation:

The training process becomes:

Forward: Use an ODE solver to integrate from h(0) to h(T), and compute the loss .
Backward: Starting from the terminal condition , integrate the adjoint equation backward (from t = T to t = 0) while simultaneously accumulating the parameter gradient .

The entire process does not need to store the intermediate states of the forward computation, and the memory cost becomes constant, not increasing with depth.

Figure 10: The solving process of the adjoint method

This essentially means that backpropagation itself is a dynamical process. The reverse-time evolution of the adjoint variable a(t) and the forward-time evolution of the hidden state form a coupled pair of forward-backward differential equations. The entire training process breaks free from the concept of layers and becomes solving mathematical equations.

4.5 Neural ODEs for Time Series Prediction

Neural ODEs are naturally suited for time series prediction because a time series itself is the trajectory left by a dynamical system over time.

Returning to the core task of our entire article: given an observation sequence {s₀, s₁, …, s_n} at corresponding times {t₀, t₁, …, t_n} (which can be non-uniformly sampled), learn the dynamics that drive the system's evolution. Using a Neural ODE to do this, the steps are remarkably natural:

Construct the equation:
Initial condition: s(t₀) = s₀ (the initial observed state)