←
Ben Davis3,581 posts
Ben Davis avatar✓

Ben Davis ✓

@davis7

yells at @theo both on and off camera (co-host of the nerd snipe podcast)

◷ Joined July 2022▧ 954 Photos and videos
Ben Davis avatar
Ben Davis@davis7·Nov 13

You really, really need to be more ambitious Things are gonna get really strange over the next year, and if you're not using these new models to do bigger greater things your going to have a very very bad time Working on my Fable 5.2 vid rn, the runs keep going longer and better than expected and it's making me believe this even more. We have barely scratched the surface of what these things are capable of...

18:36
Theo - t3.gg avatar
⟳ Ben Davis retweeted
Theo - t3.gg@theo·3h

I took this to heart and spent the last few hours doing another deep dive on GPT-6.2 Astra tl;dr - things are better than when I tested, but IMO still "directionally, very bad" I'm going to keep using the same example I've hyperfixated on since the start: "why is creating a worktree so slow?" I had a long thread trying to diagnose perf issues while using t3 code on a particular VPS. 2 of the 3 were fast, 1 was really slow. I asked astra outright "why is this so slow?", and it said a bunch of gibberish. I followed up with "I feel like I'm going insane, can you just tell me the cause of the slowness?" I'll give you the answer upfront: this VPS uses a virtual disk that is slow as shit, my other VPSs have physical NVME drives that are faster. Image 1 is from before the fixes Tibo mentioned. It shows an unnecessary graph, focuses too much on pnpm and file systems, bolds irrelevant text, and brings up ram just as much as it (subtly) mentions drive speed. Image 2 is of Opus asked same thing. It opened with the important part in bold: "Slow disk, made worse by VDO". Ideal! Image 3 is 6.2 Astra post-fixes Tibo mentioned. Better? Absolutely. Much more concise. Sadly, it still focuses way too much on second-order problems that are a small % of the cause. "XFS over a compression and deduplication layer on a virtual disk." - so close, but still a miss. I think this perfectly encapsulates the problem I have with 6.2 Astra. It doesn't miss anything. It's SO thorough. But in it's thoroughness, I find it misses the most important thing: the actual point. It wants to know everything, and then it wants to share everything. It doesn't understand what is and isn't important. Did it say anything wrong in my screenshots? Not really. But IMO, it still entirely failed to miss the point of my question, and gave an answer that was too focused on accuracy and not enough on clarity. In the wise words of Michael Seibel: "Most aim for 100% accuracy, and they end up with 50% clarity. Aim for 100% clarity, 80% accuracy". Silly way of putting it, but I genuinely wish OpenAI models would lie to me a LITTLE more so I could better understand the vibe, instead of having my time wasted by all these useless details.

Post attachmentPost attachmentPost attachment
Ben Davis avatar
Ben Davis@davis7·6h

Thus far Anthropic has always been the one to make the big step function jumps that change how we can use AI: - Sonnet 3.5 - Opus 4.5 - Opus 5.5 Usually others catch up and often surpass them (like how GPT overtook Claude last spring), then they jump again. I'm expecting this to repeat (OpenAI or someone else will crack the insane usability + capability combo Ant got with Opus 5.5) sooner or later I was really, really hoping that would be GPT 6.2 Astra but it's just not. I would however bet money that the next big GPT release will be almost entirely focused on being the best possible experience to interact with + good behavior As long as this cycle continues as it has I think things are fine, but I need to see/feel someone actually catch Anthropic as capabilities continue to skyrocket or things are probably gonna get really shitty. Anthropic's been on their best behavior since OpenAI became a much more credible threat this year, that must continue next year. An Anthropic not suffering from competitive pressures would be horrible for the world...

Matt Shumer avatar
⟳ Ben Davis retweeted
Matt Shumer@mattshumer_·9h

Why are they all posting reviews at the same time: there's an embargo. Why are the reviews all negative: the model is bad. It's pretty simple, guys. There's no conspiracy. We're just giving our opinions.

Ben Davis avatar
Ben Davis@davis7·10h

On a more serious note this is the reason we love OpenAI so much The fact that they can give us early access to a model, be super receptive to very brutal candid feedback, and I don't feel any stress putting out a scathing review is remarkable (also look at how aggressive all the early access testers are being about the model being bad, that is not normal for this kind of thing when dealing with a trillion dollar company) They have great people running things behind the scenes and really are one of the best companies to work with that I've ever experienced They definitely know what the problems are, hopefully they can bring things back with whatever they make next

Ben Davis avatar
Ben Davis@davis7·10h

In case u needed proof we aren't getting paid by OpenAI:

1:20:46
Ben Davis avatar
Ben Davis@davis7·10h

I'm doing ~30-40% of my work entirely from my phone now T3 Code and Grok Bot both have excellent apps on the duo, to the point where over the weekend while I was at Twitch con I got a ton of actual work done (burned down 3 $200 claude sub weekly limits) entirely from my phone while being out doing other stuff I'm at the point where when working on a project one of the first things I do is get the verification/dev system setup to work perfectly on the duo so I can test stuff on the go and merge PRs without having to go to my computer

Noah avatar
⟳ Ben Davis retweeted
Noah@itsnoahd·15h

Even though lots of folks are hating on Astra 6.2, this is comforting to me. It means that when folks say they like it, they mean it. And when they dont like it, they also mean it. I appreciate people who get early access not being forced to withhold negative thoughts.

11:23
Ben Davis avatar
Ben Davis@davis7·15h

i hope u guys like bells

0:36
Theo - t3.gg avatar
⟳ Ben Davis retweeted
Theo - t3.gg@theo·18h

It’s hard to put into words just how far ahead Anthropic is right now

Ben Davis avatar
Ben Davis@davis7·16h

The model I was crashing out about here, the model which decided that in order to get a model to play a game it should make a new codex/claude code session after EVERY MOVE instead of just running in a loop with tools was GPT-6.2 Astra The big problem here wasn't even this dumb decision, AI still sucks at making stuff for AI, it's the fact that I only found hours later while trying to debug some issues with it and hidden deep in a footnote it mentioned something about creating new sessions, grilled it on that, and it finally explained how it actually worked It has absolutely no understanding of what info matters and doesn't which makes it impossible to track what's going on in a thread

Matt Shumer avatar
⟳ Ben Davis retweeted
Matt Shumer@mattshumer_·18h

Unfortunately not. Try the model, you'll see what I mean. I have no idea why they released this. It needed a lot more iteration.

Ben Davis avatar
Ben Davis@davis7·16h

These models are the ultimate showcase of how absurdly important model behavior and communication is It's a capable model, it's made some insane 3d stuff, crushes on computer use, higher quality code, and I've gotten a lot of useful stuff built with it (will post more later) The problem is, working with it sucks. It's HORRIBLE at communicating what it's doing and why. It speaks in a language that only it can understand, giving u a 20 paragraph essay that doesn't include anything of actual substance or use It's also quite lazy, u really have to push this thing to keep going for long periods of time, and then there's the fact that it's just not as "tasteful" as Opus 5.5 and especially Fable 5.2. Everything it makes still reeks of GPT-isms. Honestly it's not worth bothering with when Opus 5.5 and Fable 5.2 exist. The two big claude models are probably 10-20% more capable, while being about 20x more pleasant to work with. This isn't the response from OpenAI we were all hoping for sadly

1:59
Ben Davis avatar
Ben Davis@davis7·Nov 16

Can’t believe I’m gonna have to buy a fourth sub 🙃

Post attachment
dax avatar
⟳ Ben Davis retweeted
dax@thdxr·Nov 13

all these losers they're so excited to watch someone experience pain parasites all of them

Post attachment