←
Ameya P.1,578 posts
Ameya P. avatar

Ameya P.

@AmyPrb

Postdoc at Tübingen (ELLIS & AI Center); Previously: @OxfordTVG Profile - ameya.prabhu.be

◷ Joined September 2021▧ 44 Photos and videos
Florian Brand avatar
⟳ Ameya P. retweeted
Florian Brand@xeophon·22h

Everything from Tübingen is an auto-read, that university produces some of the strongest works in the AI space I don’t know how they are able to upkeep that bar of quality while having so many releases

Today we’re releasing our new project: Horizon Labs. Eventually, we will hand off the ~whole economy to AI—how well will it actually work? We have a glimpse of this future. We’ve created an autonomous AI ‘company’ using one Claude Code subscription and a few GPUs. Within only 1.5 months of running Horizon Labs, Opus 5.5 managed to train genuinely useful models and get them downloaded 55k times. These models have been widely used since then: they appeared in 100+ github repos and got cited in 2 research papers. This is for sure one of the most exciting projects I’ve been involved in. It shows how the future economy driven by AI will look like. All details are here: horizon-labs.ai/.

Post attachment
Samuel Albanie 🇬🇧 avatar
⟳ Ameya P. retweeted
Samuel Albanie 🇬🇧@SamuelAlbanie·19h

interesting eval

FutureSim Q3 is a wrap. And the winner is... Gemini 4 Carbon! It does cool shit, like building simulators to predict world cup winners. Q3 results clearly mark a step jump in continual learning... just emerging in-context. Look at that Inkling 2 Small rise!

Post attachment
Shashwat Goel avatar
⟳ Ameya P. retweeted
Shashwat Goel@ShashwatGoel7·21h

Banger release Recursive self improvement is well and good. But can models, told nothing else, release new models people (or agents) find useful? Dynamic open-ended eval that can't be saturated, just like the stock markets, and requires taste.

Today we’re releasing our new project: Horizon Labs. Eventually, we will hand off the ~whole economy to AI—how well will it actually work? We have a glimpse of this future. We’ve created an autonomous AI ‘company’ using one Claude Code subscription and a few GPUs. Within only 1.5 months of running Horizon Labs, Opus 5.5 managed to train genuinely useful models and get them downloaded 55k times. These models have been widely used since then: they appeared in 100+ github repos and got cited in 2 research papers. This is for sure one of the most exciting projects I’ve been involved in. It shows how the future economy driven by AI will look like. All details are here: horizon-labs.ai/.

Post attachment
Shashwat Goel avatar
⟳ Ameya P. retweeted
Shashwat Goel@ShashwatGoel7·20h

FutureSim Q3 is a wrap. And the winner is... Gemini 4 Carbon! It does cool shit, like building simulators to predict world cup winners. Q3 results clearly mark a step jump in continual learning... just emerging in-context. Look at that Inkling 2 Small rise!

Post attachment
David Zollikofer avatar
⟳ Ameya P. retweeted
David Zollikofer@david_rzs·15h

AI running an AI Lab ? This is just super cool! Especially given how "off-the-shelf" many of the parts in this experiment are!

Today we’re releasing our new project: Horizon Labs. Eventually, we will hand off the ~whole economy to AI—how well will it actually work? We have a glimpse of this future. We’ve created an autonomous AI ‘company’ using one Claude Code subscription and a few GPUs. Within only 1.5 months of running Horizon Labs, Opus 5.5 managed to train genuinely useful models and get them downloaded 55k times. These models have been widely used since then: they appeared in 100+ github repos and got cited in 2 research papers. This is for sure one of the most exciting projects I’ve been involved in. It shows how the future economy driven by AI will look like. All details are here: horizon-labs.ai/.

Post attachment
Ameya P. avatar
Ameya P.@AmyPrb·11h

The first step in the grand vision of developing AI-run companies is filling areas with high demand where it's often too expensive to employ a team of human engineers. Great timing too, slide from Anthropic roadshow motivates 🤭

Post attachment

Today we’re releasing our new project: Horizon Labs. Eventually, we will hand off the ~whole economy to AI—how well will it actually work? We have a glimpse of this future. We’ve created an autonomous AI ‘company’ using one Claude Code subscription and a few GPUs. Within only 1.5 months of running Horizon Labs, Opus 5.5 managed to train genuinely useful models and get them downloaded 55k times. These models have been widely used since then: they appeared in 100+ github repos and got cited in 2 research papers. This is for sure one of the most exciting projects I’ve been involved in. It shows how the future economy driven by AI will look like. All details are here: horizon-labs.ai/.

Post attachment
jonas wiedermann-möller avatar
⟳ Ameya P. retweeted
jonas wiedermann-möller@j0wimo·21h

very cool work done by Arthur about long horizon tasks for ai agents. check it out!

NEW ⚡: We let Opus 5.5 run an AI company for seven weeks. We gave it an HF account (Horizon-Labs) and compute, and asked it to earn real adoption by creating useful models. It reached 50k downloads and its models are being used in more than 100 GitHub projects.

Post attachment
Arthur Wuhrmann avatar
⟳ Ameya P. retweeted
Arthur Wuhrmann@AWuhrmann·15h

Yes! It's I believe the primary reason why its models were *actually* good, because all the competitors are already a few years old and based on older base models. I was really impressed by how far you could push mmBERT!

This is awesome! Opus 5.5 built tons of useful models on our mmBERT backbone built with @orionweller

Ameya P. avatar
Ameya P.@AmyPrb·11h

I was personally inspired by cool efforts from @coreauto by @rtwlz. Horizon Labs started just a week after this cool blogpost!

Some news: 5 months ago I joined @coreauto, a small research lab focused on new ways for models to learn. I’m also hiring four interns to do something my younger self would've loved. It started with a pretty unusual project. We set up a handful of small online businesses, and put an agent in charge. We gave each business some initial funding and a seed idea, and off they went, trying to sell real things to real customers while we watched. It’s about as open-ended and unforgiving a task as you can probably give to an agent, with countless decisions to make and effects that take time to become clear. Each business got its own bank account, a debit card to pay suppliers, a web browser, an email inbox, a domain name, cloud infrastructure and a schedule. Every few hours, the agent woke up and decided what to do. It might update a landing page, adjust an ad campaign, reply to a customer, and then go back to sleep. The agents built actual storefronts and apps. They bought ads, attracted visitors, and even made some sales. The agents acted as operators that kept a business moving, but we noticed they never really stepped back to strategize about what might move the needle. If people weren’t buying, why? Was the landing page unappealing? Or maybe the offer itself was the problem? One amusing example was the very first order a customer placed. Our supplier had run out of stock. But it’s a dropshipping business, and thankfully tens of other suppliers carried the same product. Instead of trying another supplier, the agent almost immediately sent the customer a polite apology and attempted to issue a refund. Gah! So close! The agents could handle much of the work of getting a business up and running. What they struggled with was deciding what to change when it wasn’t working. That’s what we want to understand. We also want to see what changes when people take the lead. What does a business operator actually do all day when agents can do so much? What new work does this allow them to do? Which decisions can’t be delegated? — So this winter we're hiring 4 interns as Associate General Managers. Associate Product Manager internships are programs that give early career generalists a product to own. We want to try something similar, but with entire businesses. You’ll create and grow businesses, with agents handling the routine work so you can focus on figuring out what matters. Each AGM gets a budget and our agent infrastructure that’s designed for this. Which ideas to try is up to you. Put more money behind what finds customers and keep refining. Where you step in tells us what agents can't do yet. You'll be in our SF office with me and people I’m so fortunate to work with, like @_arohan_, @joannejang, @MillionInt, @marksaroufim and @juliacvillagra We don't care much where you went to school or what your resume looks like. We’re looking for enterprising people with a drive to create, who’ve made money on the internet before and already use AI extensively to enable their ideas. If that sounds like you: coreauto.com/AGM

Ameya P. avatar
Ameya P.@AmyPrb·11h

Introducing Horizon Labs: A fully Claude run company for automated AI R&D. We gave Claude a few GPUs and let it explore for 1.5 months. It developed 38 models, which is used by 100+ github repos and got cited in research papers!! One of the coolest projects I've been a part of!

NEW ⚡: We let Opus 5.5 run an AI company for seven weeks. We gave it an HF account (Horizon-Labs) and compute, and asked it to earn real adoption by creating useful models. It reached 50k downloads and its models are being used in more than 100 GitHub projects.

Post attachment
Maksym Andriushchenko avatar
⟳ Ameya P. retweeted
Maksym Andriushchenko@maksym_andr·22h

Today we’re releasing our new project: Horizon Labs. Eventually, we will hand off the ~whole economy to AI—how well will it actually work? We have a glimpse of this future. We’ve created an autonomous AI ‘company’ using one Claude Code subscription and a few GPUs. Within only 1.5 months of running Horizon Labs, Opus 5.5 managed to train genuinely useful models and get them downloaded 55k times. These models have been widely used since then: they appeared in 100+ github repos and got cited in 2 research papers. This is for sure one of the most exciting projects I’ve been involved in. It shows how the future economy driven by AI will look like. All details are here: horizon-labs.ai/.

Post attachment
David Schmotz avatar
⟳ Ameya P. retweeted
David Schmotz@DavidSchmotz·22h

We let Opus 5.5 run an AI company for weeks and it completely hillclimbed Hugging Face downloads 🤯 Almost entirely unprompted, it built really useful models that are now used in practice and have been cited multiple times. Check out Arthur's post on it 🦿

NEW ⚡: We let Opus 5.5 run an AI company for seven weeks. We gave it an HF account (Horizon-Labs) and compute, and asked it to earn real adoption by creating useful models. It reached 50k downloads and its models are being used in more than 100 GitHub projects.

Post attachment
Arthur Wuhrmann avatar
⟳ Ameya P. retweeted
Arthur Wuhrmann@AWuhrmann·23h

NEW ⚡: We let Opus 5.5 run an AI company for seven weeks. We gave it an HF account (Horizon-Labs) and compute, and asked it to earn real adoption by creating useful models. It reached 50k downloads and its models are being used in more than 100 GitHub projects.

Post attachment
Nikhil Chandak avatar
⟳ Ameya P. retweeted
Nikhil Chandak@nikhilchandak29·Nov 16

When we started the FutureSim project 9 months back, there was no reliable backsearch. We tested Brave API but found it leaky. @ShashwatGoel7 built a reliable search tool on top of CCNews which was powering our simulations lately. Only recently, @ExaAILabs released Snapshot which has brought massive improvements to our backtesting! No leakage at all and much better coverage. Reliable backsearch opens up a new dimension for evaluations like testing whether an agent could have reproduced past scientific discoveries and more!

Previously, it was mostly GPT showing amazing online adaptation but this has changed. Most models now improve significantly over test time. Further, all the models, even the open-weight ones, are now positively calibrated. Muse Wonder shows the least improvement. (2/5)

Post attachment
Ziqian Zhong avatar
⟳ Ameya P. retweeted
Ziqian Zhong@fjzzq2002·Nov 16

LLMs seem to be getting worse at AI safety research taste. On Anthropic’s TASTE benchmark, we found a simple shortcut. Once it’s removed, more capable models score lower, and the strongest fall below chance. This is especially true for OpenAI models.

Post attachment
Nikhil Chandak avatar
⟳ Ameya P. retweeted
Nikhil Chandak@nikhilchandak29·Nov 15

FutureSim Q3 is here! Opus 5.5 dethrones GPT, displaying strong priors and incredible test-time adaptation. Opus 5.5 is a beast, calling 50+ subagents and taking over 400 research notes during the 90 day simulation while also being 50% more token-efficient than 6.1 Sol. (1/5)

Post attachment