←
Rohan Alexander2,721 posts
Rohan Alexander avatar

Rohan Alexander

@RohanAlexander

I spend too much money on books. And too much time writing them. 🇦🇺. tellingstorieswithdata.com

◷ Joined March 2009▧ 574 Photos and videos
Rohan Alexander avatar
Rohan Alexander@RohanAlexander·Nov 5

2. You need to consistently keep that system updated (because if a submitter is using a better model yours might give bad advice), and you need to consistently judge the judges.

Rohan Alexander avatar
Rohan Alexander@RohanAlexander·Nov 5

What does this all mean? 1. If you want to have AI act as an auto-reviewer then you need to make sure that you have multiple frontier models from different labs and run them multiple times for each review.

Rohan Alexander avatar
Rohan Alexander@RohanAlexander·Nov 5

3. Advice from non-frontier models can actually make frontier model submissions worse. 4. There is still considerable self-bias - the models over-rate their own output.

Rohan Alexander avatar
Rohan Alexander@RohanAlexander·Nov 5

A few other results: 1. Models can differ considerably from run to run - if you're using LLMs to evaluate something you need to run it multiple times 2. Some models still have no sense of self and sometimes name themselves as from another model provider!

Rohan Alexander avatar
Rohan Alexander@RohanAlexander·Nov 5

I look at whether agentic AI can complete a data science project and whether that project is improved by other AI models giving feedback, and I find basically that such loops help weaker models but not stronger models - frontier performance is the frontier.

Rohan Alexander avatar
Rohan Alexander@RohanAlexander·Nov 5

Very happy to have a pre-print of "Iterated creation, grading, and revision of data science projects by language models" available on arXiv.

Chris J. Maddison avatar
⟳ Rohan Alexander retweeted
Chris J. Maddison@cjmaddison·Sep 2

What's the point of the academy, if our ideas cease to have influence and impact?

Rohan Alexander avatar
Rohan Alexander@RohanAlexander·21 Oct 2024

This Friday 25 October 2024, noon (EDT), I'm thrilled to host Jay Alammar, Cohere, on his new book “Hands-On Large Language Models: Language Understanding and Generation”. All welcome.

Post attachment