←
MetaCircle29 posts
MetaCircle avatar

MetaCircle

@MetaCircle_AI

MetaCircle/元环智能, AI for AI, Science of AI, Mechanistic AutoResearch

◷ Joined July 2026▧ 21 Photos and videos
MetaCircle avatar
MetaCircle@MetaCircle_AI·Nov 4

A video model predicts the wrong motion. Can we still change its mind? In our experiments, editing its internal state restores the observed motion—without retraining. But the same edit stops working deeper in the network. Watch the explainer below 👇
@ZimingLiu11

2:20
MetaCircle avatar
MetaCircle@MetaCircle_AI·Nov 2

4/4 Scope: 2^18 deterministic rules on an 8×8 torus, half held out, every claim checked cell by cell. The tricks are old; the testbed that says why each is needed is new. Real video: future work.

MetaCircle avatar
MetaCircle@MetaCircle_AI·Nov 2

3/4 Causal freezing locks in each frame before generating the next. Same weights, new sampler: Game of Life rollout success rises from 42.2% to 99.9%; billiards, 99.2% to 100%. Billiards can skip ahead using position and velocity; Life must evolve step by step.

Post attachment
MetaCircle avatar
MetaCircle@MetaCircle_AI·Nov 2

2/4 Unseen rules must be inferred from observed history. Pair each situation with its next-frame outcome: KV shift in attention (2.6k params, exact), or a two-frame 3D conv (3.8k, exact). Pairing features lift the same transformer's rollout success from 25.8% to 99.9%.

Post attachment
MetaCircle avatar
MetaCircle@MetaCircle_AI·Nov 2

1/4 Pixel scores hide rule errors. Scored per pixel — the metric of prior work — every baseline looks solved; scored per rollout, all fail: CNN 96.3% → 18.9%, best prior model 64.2%, joint diffusion 0%.

Post attachment
MetaCircle avatar
MetaCircle@MetaCircle_AI·Nov 2

A CNN gets 96.3% of pixels right. Why does it fail most rollouts? Our cellular automata experiments reveal failures in spatial locality, temporal locality, and temporal stability. Watch the explainer below 👇
@ZimingLiu11

3:06
MetaCircle avatar
MetaCircle@MetaCircle_AI·Oct 31

One tiny bug in activation functions. Big consequences for AI. Imagine what else we're missing. That's why we built MetaCircle. Follow us to uncover the hidden laws of AI. 🔬

There is a "bug" in commonly used activation functions (ReLU/SiLU/GeLU etc.) ---- The mean of post-activations is not zero. This bug, however small it seems, leads to pathologies of training dynamics, including loss spikes & representation overshooting. By removing the bug, we get improved performance in LLMs and ViTs. The science of AI can never be under-estimated.

2:04
MetaCircle avatar
MetaCircle@MetaCircle_AI·Oct 31

Why does Adam training suddenly spike? We mapped its stability across β₁ and β₂, finding a near-linear boundary between spiky and non-spiky runs. The explanation goes beyond curvature: update-scale loss geometry also matters. Watch the explainer below 👇 @ZimingLiu11

2:18
MetaCircle avatar
MetaCircle@MetaCircle_AI·Oct 30

Faster learning isn’t always better. Activation bias in ReLU, GELU and Swish speeds up early training—but can cause overshooting and hurt final performance. Reducing it improves late-stage results in vision and language models. Watch the explainer below 👇 @ZimingLiu11

2:04
MetaCircle avatar
MetaCircle@MetaCircle_AI·Oct 28

Can AI predict LLM training before it starts? Our Meta Model uses past runs to rank configurations and predict loss curves—no backbone fine-tuning. Across 7 Delphi runs, early observations cut remaining-curve forecast error by 56%. Watch the explainer below 👇
@ZimingLiu11

2:07
MetaCircle avatar
MetaCircle@MetaCircle_AI·Oct 26

4/4 The rarer the context, the larger the gap—roughly a power law in frequency (our paper explains why). Masking low-frequency n-gram reads shrinks the gap. SCONE already works this way: its BPE-style mining gives memory rows only to frequent n-grams.

Post attachmentPost attachment
MetaCircle avatar
MetaCircle@MetaCircle_AI·Oct 26

3/4 Post-training replays data too, and Forking comes back on small code, math, and RLVR-style corpora. Freezing the n-gram table helps.

Post attachment
MetaCircle avatar
MetaCircle@MetaCircle_AI·Oct 26

2/4 (The mechanism of forking) But this splits continuation logits into nearly non-interacting subspaces, one per context. Under replay, each subspace sharpens what it saw and pushes down what it didn't. Nothing pulls it back—overfitting arrives suddenly.

Post attachment
MetaCircle avatar
MetaCircle@MetaCircle_AI·Oct 26

1/4 Why it works: the n-gram table offloads the context signal—table lookup supplies ~3/4 of the final residual-stream norm, and training loss drops. All over-encoding designs induce forking: input, V, or y injection, including a DeepSeek-style Engram model.

Post attachment