ICML 2026 · Oral

A Random Matrix Theory Perspective on the
Consistency of Diffusion Models

Binxu Wang1 · Jacob A. Zavatone-Veth2,4 · Cengiz Pehlevan1,4,3
1Kempner Institute   2Society of Fellows   3SEAS   4Center for Brain Science  — Harvard University

Train two diffusion models on completely different halves of a dataset. Give them the same noise seed. They produce almost the same image — across data splits and across architectures (UNet, DiT). We show this consistency is largely a linear effect — the shared Gaussian statistics already predict the output — and build a random matrix theory that says precisely where and how much independently-trained models will disagree.

Training data p(x₀) non-overlapping Split S₁ · DM₁ Split S₂ · DM₂ same noise seed sampling path DM₁ sampling path DM₂ Gen Split 1 Gen Split 2
Phenomenon reported by Kadkhodaie et al. (2024); Zhang et al. (2024).

1 The observation

Train two diffusion models on disjoint halves of a dataset and sample both from the same noise seed. The outputs are nearly identical — across splits and across architectures. Why should two models that never saw the same image agree? Here is the linear intuition.

Different splits
DM1 DM2
→
Summary statistics

Empirical Gaussian statistics — the first two moments

$\hat\mu,\ \hat\Sigma$
→
Relatively stable
Linear denoiser & sampling map
$$\mathbf{D}^*_{\hat\Sigma}(\mathbf x;\sigma)=\hat\mu+\hat\Sigma\,(\hat\Sigma+\sigma^2 I)^{-1}(\mathbf x-\hat\mu)$$ $$\mathbf{x}^*_{\hat\Sigma}(\mathbf x_{\sigma_T};\sigma)=\hat\mu+(\hat\Sigma+\sigma^2 I)^{\frac12}(\hat\Sigma+\sigma_T^2 I)^{-\frac12}(\mathbf x_{\sigma_T}-\hat\mu)$$

Both the linear denoiser $\mathbf D^*$ and the sampling map $\mathbf x^*$ depend on the data only through $(\hat\mu,\hat\Sigma)$. Across independent splits these first two moments barely change — so the maps barely change, and the generated samples come out consistent. Under high noise and in the generalization regime, deep diffusion models reduce to this same linear / far-field solution, which is why it predicts them.

Explore it yourself. Pick a dataset and a noise seed. Split 1 and Split 2 are independent models trained on disjoint data; Linear theory is the closed-form Gaussian (Wiener-filter) predictor above, using only each split's $(\hat\mu,\hat\Sigma)$. The outer rows show each split's nearest training images — note the samples are not copies of them.

Seed 0
per-seed comparison across splits, architectures, and linear theory

Takeaway. Cross-split and cross-architecture outputs are closer to each other than to the nearest training image — this is generalization, not memorization — and the linear predictor already calls the shot.

▸ What if we break it? — disrupting the first two moments

If consistency really comes from shared $(\hat\mu,\hat\Sigma)$, then forcing the splits to have different moments should destroy it. We stratify the data along a principal component (PC2) into top, bottom, and mid groups — deliberately mismatching the mean and covariance across "splits."

PC2 score densities and sample grids for top, bottom, mid groups
FFHQ32 stratified along PC2: the top / mid / bottom groups have visibly different statistics and different-looking faces.
cross-model consistency MSE matrix
Cross-model consistency (pairwise MSE). Two ordinary i.i.d. splits agree tightly (0.042, bottom-right), while moment-mismatched groups disagree far more (0.15–0.48). Same architecture, same seeds — only the moments changed.

The linear theory is therefore a necessary condition: when Gaussian statistics differ, even deep networks stop being consistent.

2 From memorization to renormalization

Slide the training-set size $n$. With little data, models memorize and reproduce training images. As $n$ grows they generalize, and finite-data shrinkage pulls samples toward the dataset mean — textures and backgrounds smooth out (the "average-image" effect).

Train size n 300
Split 1split 1 generated samples
Split 2split 2 generated samples

3 The mechanism — finite data renormalizes the noise

All finite-sample effects on the average denoiser collapse into a single scalar knob: the renormalized noise scale $\sigma^2 \mapsto \kappa(\sigma^2)$, the unique solution of a Silverstein self-consistency equation set by the data spectrum and the aspect ratio $\gamma=d/n$. Because $\kappa(\sigma^2)\ge\sigma^2$, the model behaves as if the noise were larger — so low-variance modes get over-shrunk toward the mean.

Larger $\gamma$ (less data) → stronger renormalization at small $\sigma^2$.

The mapping bends up most at small noise and large $\gamma$ — exactly the regime of fine image detail with limited data. So high-frequency details need more samples to be reproduced consistently.

Spectrum used here is a representative heavy-tailed natural-image spectrum (power-law). The true per-dataset spectrum can be swapped in.

noise renormalization and overshrinkage figure
Overshrinkage: limited data shrink low-variance eigenmodes harder, pulling generations toward the average image.

4 The theory, in four equations

Renormalized noise (Silverstein)
$$\kappa(\sigma^2)-\sigma^2=\gamma\,\kappa(\sigma^2)\,\operatorname{tr}\!\big[\Sigma(\Sigma+\kappa\,I)^{-1}\big]$$

Finite data act as if the noise were $\kappa(\sigma^2)\!\ge\!\sigma^2$ — an adaptive ridge.

Average denoiser — over-shrinkage
$$\mathbb{E}_{\hat\Sigma}\big[\mathbf D^*_{\hat\Sigma}(\mathbf x;\sigma)\big]\;\asymp\; \mu+\Sigma\big(\Sigma+\kappa(\sigma^2) I\big)^{-1}(\mathbf x-\mu)$$

The population denoiser, but with $\sigma^2$ replaced by the larger $\kappa(\sigma^2)$: low-variance directions are pulled toward the mean.

Cross-split disagreement — a factorized variance law
$$\operatorname{Var}_{\hat\Sigma}\!\big[\mathbf v^\top\mathbf D^*_{\hat\Sigma}(\mathbf x)\big]\;\asymp\; \underbrace{\tfrac{\kappa^2}{\,n-\mathrm{df}_2(\kappa)}}_{\text{scaling }(n)}\; \underbrace{\Diamond(\mathbf v,\kappa,\Sigma)}_{\text{anisotropy}}\; \underbrace{\Diamond(\mathbf x-\mu,\kappa,\Sigma)}_{\text{inhomogeneity}}$$

Disagreement factorizes into which direction (anisotropy), which input/seed (inhomogeneity), and how much data (scaling). Here $\Diamond(\mathbf u,\kappa,\Sigma)=\mathbf u^\top(\Sigma+\kappa I)^{-2}\Sigma\,\mathbf u$.

Whole trajectories — deterministic equivalence for fractional powers
$$\hat\Sigma^{1/2}\;\asymp\;\tfrac{2}{\pi}\!\int_0^\infty \Sigma\big(\Sigma+\kappa(u^2)I\big)^{-1}\,du$$

A new tool that extends deterministic equivalence to $\Sigma^{1/2}$, so the entire sampling map (not just one denoising step) becomes analyzable in closed form.

5 It holds for deep networks

The linear RMT predictions extend qualitatively to real UNet and DiT diffusion models. The same memorization→renormalization transition appears; cross-split deviation shows the predicted anisotropy across eigenmodes; and — using only the population covariance and $n$, with no knowledge of architecture or split — RMT predicts which seeds two models will disagree on (Spearman $r=0.33$).

deep network validation of the RMT theory
Deep-network validation across dataset sizes and eigenmodes (FFHQ, UNet & DiT).
sampling map expectation and variance scaling
Linear-theory predictions (lines) match direct linear-diffusion simulation (points) for the sampling map across dataset sizes.

6 Code & the RMT toolkit

The repository ships rmt_core — a reusable toolkit for the deterministic-equivalence machinery used here: a robust solver for the Silverstein equation $\kappa(z)$, degrees-of-freedom functions, the fractional-power integral identities, and linear-denoiser / Wiener-filter utilities — plus curated example notebooks reproducing the key figures.

View the code on GitHub →

Citation

@inproceedings{wang2026rmtconsistency,
  title     = {A Random Matrix Theory Perspective on the Consistency of Diffusion Models},
  author    = {Wang, Binxu and Zavatone-Veth, Jacob A. and Pehlevan, Cengiz},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026},
  note      = {Oral presentation},
  url       = {https://arxiv.org/abs/2602.02908}
}