Lecture 5 — Hardness of Score Estimation

(a)  Noise forward, structure backOne forward step is Xk+1 = √(1−ηk) Xk + √ηk Zk. The step sizes follow the standard fine-to-coarse schedule — tiny η at low noise, larger later — because added variance kills small scales first: with uniform steps the thin pancake layers would vanish within two ticks. Here the layers melt around k ≈ 15 and the blobs merge around k ≈ 40, reaching ≈ N(0, I) at k = 60. The slider shows the data at time k; by k = T the cloud is indistinguishable from N(0, I). The generate button ties into the same slider: it re-seeds the gold points as fresh pure noise at k = T and runs the reverse chain by pulling k down to 0 itself, using the exact score of the noised distribution at each level — nothing is pretrained, so what you watch is the reverse process. Afterwards the one slider scrubs both clouds through time together: chalk ran forward from the data, gold ran backward from noise, and at k = T they are statistically the same cloud. The generated gold points land on the structure without landing on the faint original samples: fresh draws, not copies — the definition of learning to sample asks exactly that, dTV((S, Y), (S, X)) ≤ ε, which a memorizing generator fails. Theorem 1: with bounded second moment, Lipschitz scores, and fast forward mixing, accurate scores at the required noise levels give a sampler ε-close in total variation; the three error sources are the Gaussian start, discretization, and score error.

data
time k 0
–

(b)  The score is a compassThe score of Dk is sk(x) = ∇ log pk(x) — drawn here as arrows over the density. Why it is all a reverse step needs: localize (only predecessors within ≈√η matter), linearize (log pk is locally linear with slope sk), reverse (adding a linear term to the Gaussian log-density only shifts its center). Proposition 1 makes it exact: E[Xk | Xk+1 = y] = (y + η sk+1(y)) / √(1−η). Drag the gold probe: the arrow shows that recentering, and the circle is the reverse step’s Gaussian spread of scale √η (drawn for an illustrative η = 0.08 — the chain in (a) takes many finer steps, each obeying the same formula). At high noise the field relaxes to the trivial −x; at low noise it points sharply into the structure.

noise k 16
–

(a)  §2.1 — the score is learnable from samplesTraining data manufactured from the forward process itself: noise a data sample to level k and record, at the noisy point, the way back home — the notes’ recorded answer −Zk/√η, drawn here accumulated over the whole noising path so each arrow is steadier; the conditional mean is the same by Proposition 1. Each arrow alone is mostly noise: one noisy point could have come from many places. But its conditional mean at each location is exactly the score — that is Proposition 1 — so averaging the arrows near a point recovers sk there. Squared loss is the right training objective precisely because its minimizer is this conditional mean; the local average shown here is the simplest such fit, and a neural network trained on the same pairs converges to the same field, only sharing strength across locations. Accuracy is needed only where the samples fall, which is all Theorem 1 requires. One more honest wrinkle: the recorded arrows get wilder as the noise shrinks — their variance scales like 1/(1−ak2) — so low-noise levels need far more samples for the same accuracy; real training weights the levels accordingly. The panel opens at a moderate level where the averaging visibly succeeds; slide k down and watch the same procedure struggle. Toggle to the average and compare gold (learned) with the chalk truth beneath; add samples and watch the error shrink.

show
noise k 48
–

(b)  §2.2 — why this must fail on pancakesThe reduction of Theorem 2, drawn as stated. Train the assumed estimator twice — once on N(0, I) samples, once on hCLWE pancake samples (hidden direction w) — and run the notes’ check: on fresh Gaussian points, compare the estimate to the known Gaussian score −x. Both panels use the best estimator Theorem 2 assumes: the exact score of the training distribution. Left, the deviation is identically zero. Right, since sk(x) = −x + ψk(⟨w, x⟩) w, every deviation points along the secret — so any noise level with a visible deviation is a polynomial-time distinguisher for hCLWE, and reading the arrows’ common direction even recovers w. Deviations must appear at some level: if they were small everywhere, Theorem 1 would make one reverse procedure sample close to both distributions, contradicting the test Tw that accepts periodic ⟨x, w⟩ — near 1 on pancakes, near 0 on Gaussians. Slide the noise: at high k the layers have melted and even the right side shows nothing — the leak lives at the lower levels, and root-mean-square error O(1/√log n) there already suffices. The difficulty is computational, not statistical: exhaustive search over w recovers it in exponential time. References: Song, NeurIPS 2024; Bruna, Regev, Song & Tang, STOC 2021.

noise k 8
–