A video model predicts the wrong motion. Can we still change its mind?
In our experiments, editing its internal state restores the observed motion—without retraining. But the same edit stops working deeper in the network.
Watch the explainer below 👇 @ZimingLiu11
4/4 Scope: 2^18 deterministic rules on an 8×8 torus, half held out, every claim checked cell by cell. The tricks are old; the testbed that says why each is needed is new. Real video: future work.
3/4 Causal freezing locks in each frame before generating the next. Same weights, new sampler: Game of Life rollout success rises from 42.2% to 99.9%; billiards, 99.2% to 100%. Billiards can skip ahead using position and velocity; Life must evolve step by step.
2/4 Unseen rules must be inferred from observed history. Pair each situation with its next-frame outcome: KV shift in attention (2.6k params, exact), or a two-frame 3D conv (3.8k, exact). Pairing features lift the same transformer's rollout success from 25.8% to 99.9%.
1/4 Pixel scores hide rule errors. Scored per pixel — the metric of prior work — every baseline looks solved; scored per rollout, all fail: CNN 96.3% → 18.9%, best prior model 64.2%, joint diffusion 0%.
A CNN gets 96.3% of pixels right. Why does it fail most rollouts?
Our cellular automata experiments reveal failures in spatial locality, temporal locality, and temporal stability.
Watch the explainer below 👇 @ZimingLiu11
One tiny bug in activation functions. Big consequences for AI.
Imagine what else we're missing.
That's why we built MetaCircle.
Follow us to uncover the hidden laws of AI. 🔬
There is a "bug" in commonly used activation functions (ReLU/SiLU/GeLU etc.) ---- The mean of post-activations is not zero.
This bug, however small it seems, leads to pathologies of training dynamics, including loss spikes & representation overshooting.
By removing the bug, we get improved performance in LLMs and ViTs.
The science of AI can never be under-estimated.
Why does Adam training suddenly spike?
We mapped its stability across β₁ and β₂, finding a near-linear boundary between spiky and non-spiky runs. The explanation goes beyond curvature: update-scale loss geometry also matters.
Watch the explainer below 👇
@ZimingLiu11
Faster learning isn’t always better.
Activation bias in ReLU, GELU and Swish speeds up early training—but can cause overshooting and hurt final performance.
Reducing it improves late-stage results in vision and language models.
Watch the explainer below 👇
@ZimingLiu11
Can AI predict LLM training before it starts?
Our Meta Model uses past runs to rank configurations and predict loss curves—no backbone fine-tuning.
Across 7 Delphi runs, early observations cut remaining-curve forecast error by 56%.
Watch the explainer below 👇 @ZimingLiu11
4/4 The rarer the context, the larger the gap—roughly a power law in frequency (our paper explains why). Masking low-frequency n-gram reads shrinks the gap. SCONE already works this way: its BPE-style mining gives memory rows only to frequent n-grams.
2/4 (The mechanism of forking) But this splits continuation logits into nearly non-interacting subspaces, one per context. Under replay, each subspace sharpens what it saw and pushes down what it didn't. Nothing pulls it back—overfitting arrives suddenly.
1/4 Why it works: the n-gram table offloads the context signal—table lookup supplies ~3/4 of the final residual-stream norm, and training loss drops. All over-encoding designs induce forking: input, V, or y injection, including a DeepSeek-style Engram model.