←
Aman Khan826 posts
Aman Khan avatar

Aman Khan

@amankhan

AI PM @google on evals, benchmarks | ex @arizeai views my own

◷ Joined December 2011▧ 64 Photos and videos
Aman Khan avatar
Aman Khan@amankhan·Oct 28

I promise I’m not just saying this because I work at @Google But Gemini generated analysis and tokens are just more pleasant to read than other LLM-speak (I.e “claudese”)

Aman Khan avatar
Aman Khan@amankhan·1h

Time to start brushing up on periodic elements ➰

Why the fuck is nobody talking about Neon? This model is so good wtf

Logan Kilpatrick avatar
⟳ Aman Khan retweeted
Logan Kilpatrick@OfficialLoganK·7h

Introducing Gemini 4 Neon! Pushing the frontier across domains and punching above its model class in a meaningful way. To me, Neon feels like the right balance of intelligence, speed, and usability. Available now across @GoogleAIStudio, the API, and more!

Post attachment
Kaggle avatar
⟳ Aman Khan retweeted
Kaggle@kaggle·8h

AI benchmarks have 4 problems: relevance, saturation, verification, and infrastructure. At Bench Summit 2026, @janiejgu (Group Product Manager at Kaggle), delivered the opening keynote on why current AI benchmarks are broken and how Kaggle’s roadmap is focused on building a trusted, industry standard for model and agent evaluation. Our roadmap is focused on solving: • Relevance → real-world benchmarks, built with the companies and researchers who own these workflows • Saturation → continuous evals from community-crowdsourced tasks, community arenas, and production data • Verification → rigorous, transparent benchmarks with published traces and independent audits • Infrastructure → a platform for anyone to build a benchmark correctly, built by Kaggle and @GoogleDeepMind for the whole industry Watch the complete keynote here: youtu.be/MIjUfmYHxLk

1:03
Kaggle avatar
⟳ Aman Khan retweeted
Kaggle@kaggle·Nov 17

Build an open-source autonomous agent harness, pair it with one of 8 Google-sponsored foundation models, and tackle Physical-Engineering's Last Exam (PLE), a benchmark of 85 expert-reviewed, long-horizon tasks across structural, civil, electrical and mechanical engineering. Join the Agent Harness on Physical-Engineering's Last Exam competition, hosted by @Google! 💰 Prize Pool: $400,000 ⏰ Entry Deadline: January 5th, 2027

Kaggle avatar
⟳ Aman Khan retweeted
Kaggle@kaggle·Nov 12

Benchmarks tell you how models score. Arenas let you judge and see how they stack up. Introducing Kaggle Arenas: head-to-head matchups where AI models take on the same challenge, and you decide which one comes out on top. From 3D modeling to creative design to data visualization, see what today's models can really do. Every vote helps shape the leaderboard. Start voting: kaggle.com/arenas

0:05
Aman Khan avatar
Aman Khan@amankhan·Nov 11

as models get better, writing good evals gets harder and more important.

Abundant avatar
⟳ Aman Khan retweeted
Abundant@abundant_labs·Nov 10

Carbon solves more of the task frontier: it passed at least once on 14 of 20 tasks, vs 11 for Claude Opus 5.5 and 12 for GPT-6 Astra. It’s the only one of the three to solve the Stripe and Mastodon clones.

Post attachment
Aman Khan avatar
Aman Khan@amankhan·Nov 10

ground truth. reskilling/upskilling/learning how to use AI to your advantage remains an incredible opportunity

STATE OF THE TECH JOB MARKET IN 2026 Twice a year, I dig into trends in hiring and job openings. This is the fifth report I've done, and the most surprising yet: 1. PMs are winning in the AI era 2. Engineering demand has stalled 3. Design openings are at a record low 4. Every role now requires AI skills 5. Tech jobs are concentrating in the Bay Area 6. Remote roles are at a record low 7. The tech job market is growing fast, but mostly outside of software (gasp!) Full report here: lennysnewsletter.com/p/state…

Post attachment
Peter Yang avatar
⟳ Aman Khan retweeted
Peter Yang@petergyang·Nov 9

“Both of my parents got diagnosed with two cancers each within a week of each other.” From @AshleyYuki, VP of Product at @OpenAI: “I just decided to take a month [off work]. It was one of the best things I could have done in that moment to care for my parents. You also have to care for yourself as a caregiver. We don’t talk about that enough. It’s still a really big cognitive load, emotional load, just pure time load to be a caregiver. You have to make sure you’re budgeting time for your own mental health.” 📌 Watch our full episode here: youtu.be/pR7NCYKsA3o

1:03

“Both of my parents each got diagnosed with cancer within a week of each other.” Ashley is the VP of Product at @OpenAI for Health and this is a very special episode for me. We talked about how we both use AI to help our parents fight cancer — including preparing for doctor visits, researching treatments, and coping with our emotions as caregivers. Topics we discussed: → Do doctors mind when patients bring AI research? → How ChatGPT caught a risky treatment before it was given → How to make time for your parents with a demanding job Some quotes from Ashley: “I always felt like there was no more time in the day. Then someone gets sick and you realize what you can make time for and what can drop.” “You also have to care for yourself as a caregiver. You have to make sure you’re budgeting time for your own mental health.” “The treatment she was about to be given would have had a really bad contraindication. She narrowly avoided it because she ran it through Chat.” 📌 Watch now: youtu.be/pR7NCYKsA3o Thanks to our sponsors: @meetgranola: AI meeting notes that don’t suck granola.ai/peter @RiversidedotFM: All-in-one AI studio for podcasts and video creators.riverside.com/Peter…

0:31
Aman Khan avatar
Aman Khan@amankhan·Nov 8

This is indeed a problem and something we are thinking more about

jammed with @HamelHusain a bit and since then, i cannot unsee of the state of evals as being similar to the military-industrial complex

Post attachment
Aman Khan avatar
Aman Khan@amankhan·Nov 8

This one hit close to home - AI is incredible at leveling the playing field (being a thinking partner) in discussions + decisions where information is asymmetric and imperfect. Proud of @petergyang for shining a light here

“Both of my parents each got diagnosed with cancer within a week of each other.” Ashley is the VP of Product at @OpenAI for Health and this is a very special episode for me. We talked about how we both use AI to help our parents fight cancer — including preparing for doctor visits, researching treatments, and coping with our emotions as caregivers. Topics we discussed: → Do doctors mind when patients bring AI research? → How ChatGPT caught a risky treatment before it was given → How to make time for your parents with a demanding job Some quotes from Ashley: “I always felt like there was no more time in the day. Then someone gets sick and you realize what you can make time for and what can drop.” “You also have to care for yourself as a caregiver. You have to make sure you’re budgeting time for your own mental health.” “The treatment she was about to be given would have had a really bad contraindication. She narrowly avoided it because she ran it through Chat.” 📌 Watch now: youtu.be/pR7NCYKsA3o Thanks to our sponsors: @meetgranola: AI meeting notes that don’t suck granola.ai/peter @RiversidedotFM: All-in-one AI studio for podcasts and video creators.riverside.com/Peter…

0:31
Aman Khan avatar
Aman Khan@amankhan·Nov 4

nice

Models by @GoogleDeepMind scale better on cost than their rivals. Gemini 4 Carbon is the prime example. As tasks get more demanding, every rival gets more expensive relative to Carbon. For a typical task, GPT-6 Astra by @OpenAI costs 3.4× as much, Claude Opus 5.5 by @AnthropicAI costs 3.9× as much, and GPT-6.1 Sol is the cheaper option at about half of Carbon’s cost (0.54×). Across the 5% most expensive sessions, Carbon’s position improves against all three. Opus edges up to 4.2×, Astra’s cost roughly doubles relative to Carbon, and Sol’s discount shrinks. This in turn can translate into meaningful savings across real-world tasks such as building an Android security app, one of many long-horizon tasks that our Agent Arena users delegate to their agents.

0:10