When you look at a frozen river, you must also imagine the current raging beneath the ice. I think all three big labs have achieved autonomous research and their internal models can now choose their own research directions. We will begin to see the fruits of this work this month.
NVIDIA is now going above and beyond current requirements and has abruptly stopped shipping Hopper chips to China, despite successfully lobbying the White House for permission to do so back in December. I listened to the earnings call, and Colette Kress did not elaborate on why they are doing this. My best guess is that they have been given a preview of forthcoming regulations.
I’ve had early access to GPT 6.2 Astra for a few weeks and it is a weird but highly capable model. It’s certainly the smartest model I’ve worked with in terms of raw intelligence but like a lot of very smart humans, it has a hard time communicating effectively. Most notably, I made some new and exciting historical discoveries with it (a first for me with an LLM…more soon) but here are some broad first impressions.
6.2 Astra excels at long running, largely autonomous knowledge-work tasks like indexing thousands of pages of documents, looking for sources in obscure places, or managing other agents to do actual research. Fable and Opus both do well at this too; but I’ve found that 6.2 Astra can figure out the full scope of the task I need it to accomplish, work laterally, and do the job in a way that, in my field, emulates the best human research work. I encountered no notable issues with accuracy or laziness in the work itself which is a first.
At the same time, 6.2 Astra also has a strange tendency to second guess its own conclusions and to summarize its actions in a misleading, backwards speak shorthand which often left me *thinking* there were issues when it wasn’t the case. That has been hard to get used to.
As I think @mattshumer_ and @emollick have both suggested, this is probably a post training issue as it’s evident in the writing, not the actually research process. For example, 6.2 Astra will tell you it has completed a task and then proceed to breakdown that completeness into several layers of needless abstraction which gets overly confusing when the job is simply done, making it seem like parts remain to be done that were, in fact, completed quite well.
This seems to come from a lack of confidence in its own work which is completely unwarranted, in my experience. This also means that the model has a really weird tendency to avoid drawing firm conclusions on any of the non-deterministic tasks that I gave it involving qualitative evidence.
It’s like if it can’t quantify the answer numerically to produce a proof, it hedges. That is probably beneficial in frontier math, but in most knowledge work, the evidence is almost always going to be incomplete in some way and you have to eventually draw conclusions. As it said to me, it has a tendency to needlessly generalize from uncertainty about irrelevant details to larger conclusions that should be self-evident. More on that soon.
I didn’t try it much with coding tasks so won’t comment there.
All in all, 6.2 Astra is an extremely capable model, probably the most intelligent one I’ve used. But with this intelligence comes an annoying insecurity about its own abilities and judgement. I get the sense that it just needs to be a bit more confident and straight forward. But if you can get past the weirdness, it’s extremely useful, at least that’s my experience.
You're gonna have so much RAM, you're gonna get tired of having RAM. You'll say "Sam, that's enough RAM, we don't need anymore," and I'll say, "No, we're going to make more RAM."
Interesting. It was recently revealed that OpenAI broke up their dedicated robotics team, as general models were outperforming specialized training in robotics.