←
Magic36 posts
Magic avatar

Magic

@magicailabs

Aligned superintelligence.

◷ Joined April 2022▧ 9 Photos and videos
Magic avatar
Magic@magicailabs·Sep 8

Frontier pretraining is said to be a big-lab-only game. We don’t have 100k chips yet, so there’s only one way: algorithmic efficiency. Our new recipe matches DeepSeek V4 Pro’s pretrain using 50x less compute – that’s roughly half the FLOPs used for GPT3, or ~$0.5M on GB200. magic.dev/blog/pretraining

Post attachment
stefan avatar
⟳ Magic retweeted
stefan@stefanmatar·Sep 8

Working at Magic feels very different from what I hear from people who reached out to us at other labs. We all sit in the same room. We're a tiny team. Everyone just does whatever is important. Imagine running RL runs for days without compaction. Imagine agents learned from you, across all sessions (and imagine not repeating yourself a million times like a maniac). Man, I'm so excited for the day we get to share the product experiences we are working on.

Magic avatar
Magic@magicailabs·Sep 8

We are likely the smallest team in the world training trillion parameter models. The impact a single person with strong judgement can have has never been higher. If you want to help build aligned superintelligence, consider joining: magic.dev/careers

Magic avatar
Magic@magicailabs·Sep 8

What’s next? 1) Scale RL with long context to teach agents test-time learning. 2) RL against the model’s own latent knowledge of its intent for stronger theoretical alignment properties. 3) Further improvements to pretraining. And then, of course, release the thing!

Magic avatar
Magic@magicailabs·Sep 8

Magic’s goal is to build the best model for SWE and autonomous AI R&D. To balance data trade-offs, we look at “knowledge evals” in addition to our standard out-of-sample suite. We make new knowledge evals for each model generation to avoid overfitting over time.

Post attachment
Magic avatar
Magic@magicailabs·Sep 8

Our pretraining recipe is the multiplicative result of tens of changes across model architecture, optimizer, training objective, and data. And we learned to appreciate that fixing minor bugs is a compute multiplier too. We talk more about our research process in the blog.

Post attachment
Magic avatar
Magic@magicailabs·Sep 8

We scaled this recipe 10x (~$4M) and exceeded all publicly available base models. Pretraining a model this capable would cost >$100M under DeepSeek V4 Pro’s recipe. Of course, we won't stop scaling there. To sanity check post-RL performance, we ran a short math RL run from base.

Post attachment
Magic avatar
Magic@magicailabs·21 Nov 2024

Excited to announce we’re building an Applied Team focused on post-training. Come explore what's possible with our new (and still unreleased) LTM2 models and their 100M token context window. Apply here: magic.dev/careers/5652b448-8…

Magic avatar
Magic@magicailabs·24 Sep 2024

Very excited to welcome @nvidia as Magic's latest investor! With their support, we’re looking forward to scaling long context and inference-time compute.

Magic avatar
Magic@magicailabs·29 Aug 2024

With context solved, we now focus on unbounded inference-time compute as the next (and potentially last) breakthrough we believe is needed to build reliable AGI. Imagine if you could spend $100 and 10 minutes on one task and reliably get a great pull request for an entire feature. That’s our goal. We are 23 people (+ 8000 H100s) working on a single project: co-designing for long context, inference-time compute, and end-to-end RL to automate coding and research. Ben Chess (fmr. OpenAI supercomputing lead) just joined to help us scale and we’re hiring more engineers and researchers across ML, CUDA, infra, security, and more: magic.dev/careers

Magic avatar
Magic@magicailabs·29 Aug 2024

Our LTM (Long Term Memory) mechanism needs >1,000x less compute and memory than Llama 3.1 405B’s attention. Llama 3.1 would need 638 H100s *per user* to store a 100M token KV cache. LTM needs a small fraction of one. SSMs, RNNs, and RAG all exploit weaknesses in evals like Needle In A Haystack, so we made a new eval, HashHop: 1) Incompressible 2) Multi-hop 3) No semantic hints 4) No recency bias

Post attachment
Hersh Desai avatar
⟳ Magic retweeted
Hersh Desai@Hersh_Desai·15 Feb 2024

The era of long context is upon us. The question is whether you want to be 1 of 1000 co-authors on the Gemini paper or 1 of <20 building at Magic.dev

Jeff Dean@JeffDean15 Feb 2024

Needle in a Haystack tests The tech report also details a number of microbenchmark “needle in a haystack” tests (modeled after @GregKamradt’s github.com/gkamradt/LLMTest_…) that probe the model’s ability to retrieve specific information from its context. For text, Gemini 1.5 Pro achieves 100% recall up to 530k tokens, 99.7% up to 1M tokens, and 99.2% accuracy up to 10M tokens.

Post attachment
Hersh Desai avatar
⟳ Magic retweeted
Hersh Desai@Hersh_Desai·15 Feb 2024

I have been continuously in awe of the brilliance, tenacity, and kindness of @EricSteinb and the small but mighty team at Magic.dev. So much so that we've decided to invest $100m! If you're interested in building the future, please do reach out to me or the team!

Post attachment

Magic.dev has trained a groundbreaking model with many millions of tokens of context that performed far better in our evals than anything we've tried before. They're using it to build an advanced AI programmer that can reason over your entire codebase and the transitive closure of your dependency tree. If this sounds like magic... well, you get it. Daniel and I were so impressed, we are investing $100M in the company today. The team is intensely smart and hard-working. Building an AI programmer is both self-evidently valuable and intrinsically self-improving. If this sounds interesting to you, consider joining them!

Post attachment
Eric Steinberger avatar
⟳ Magic retweeted
Eric Steinberger@EricSteinb·15 Feb 2024

I love my team a lot and sometimes it’s stressful but life has never been so fulfilling. If you want to build AGI on a small team of people who care a lot with thousands of GPUs, please apply :) Magic.dev

Magic@magicailabs15 Feb 2024

We've raised $117M from @natfriedman and others to build an AI software engineer. Code generation is both a product and a path to AGI, requiring new algorithms, lots of CUDA, frontier-scale training, RL, and a new UI. We are hiring!

Post attachment