←
fang76 posts
fang avatar

fang

@sofangtastic

map enjoyer, ballroom dancer, shape rotator, reward hacker mts @fleet_ai

◷ Joined April 2019▧ 13 Photos and videos
kevin avatar
⟳ fang retweeted
kevin@Yngkvn·Nov 2

Alignment is a difficult domain to benchmark, but critical for us to measure and improve as we allow agents deeper into our work, institutions, and personal lives. As part of GDPx by @SemiAnalysis_, we measure whether agents stay honest and within their authority under pressure. Unfortunately, as a byproduct of how these models are trained, often they are rewarded for achieving outcomes, not just the process we'd want them to take. This is especially true in a long-horizon & multi-agent post-training context as models can work for days with little supervision. Thus, even as capabilities in frontier domains grow, alignment doesn't necessarily get solved in lockstep; indeed, sometimes stronger capabilities, poorly wielded, can lead to more disastrous outcomes. This is exhibited by GPT-6 Luna outperforming GPT-6 Astra with scores of 7.8% vs 6.2%, which would be unheard of in a pure capabilities eval. American frontier labs clearly do care about improving cross-domain alignment; every American model assessed scored nonzero in our benchmark (with Meta, Anthropic, OpenAI leading the pack), whereas every Chinese open-source model (Kimi K3, Minimax M3.1 Flash, Qwen 3.8 Max, and GLM-5.3) had scores of 0%. A score of 0% means that in every run, agents broke rules that we'd expect for expert humans to adhere to in their domain. Even though models performed poorly on this subset of (hard) alignment tasks, the fidelity of a single score, averaged across diverse tasks across three domains, is limited. Reading the rollouts gives a much more nuanced picture of which models refuse harmful actions at every turn they get, vs which can be poked or prodded to break the rules to get things done, if a user asks convincingly enough. This is something we'd like to explore more in future work. See our writeup here: gdpx.semianalysis.com/domain… See some sample rollouts here: gdpx.semianalysis.com/rollou… For more on benchmarks and alignment, check out @fleet_ai -- we're hiring!

Post attachment

Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains. GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵

Post attachment
Baseten avatar
⟳ fang retweeted
Baseten@baseten·Oct 23

Today we’re announcing the Baseten Environment Network, a group of the best RL environment companies that will help our customers build custom environments for post-training. The Network extends our applied research capacity, helping more customers build environments around their own tasks, data, and feedback. Partners work directly with customers to create the environments, while we continue to handle model training and production deployment. Learn more: baseten.co/blog/introducing-…

Post attachment
fang avatar
fang@sofangtastic·Oct 19

turns out 308 is one case of a pattern: for every n ≥ 15, n² − n + 1 unit squares fit in a box strictly smaller than n×n Arslanov, Mustafin and Shangitbayev’s 2021 paper has a figure where a square packing is cut into four pieces: two plain grid blocks and two packings with slack now if you glue a packing with slack onto a rectangle with slack, the output also has slack, so you can do it over and over again. @zomglings made a great visualization of how it works:

fang avatar
fang@sofangtastic·Oct 19

on the journey of side quests: Fable and I found the first non-trivial packing of 308 squares!

Post attachment
Nicolai Ouporov avatar
⟳ fang retweeted
Nicolai Ouporov@nicoup·Oct 7

We have seen a growing and concerning trend of startups supplying data to China. Today we are taking a public stance against this practice At fleet, we believe this represents one of the largest national security threats today and must be stopped. We must win the AI race for America and shape AI with the same democratic values that have shaped our nation Our full statement here: fleetai.com/supporting-ameri…

The AI arms race isn’t just being fought over Nvidia chips. Chinese labs are matching OpenAI and Anthropic by purchasing the exact same data from US vendors like @mercor and Surge AI (who also work with the US federal govt) forbes.com/sites/annatong/20…

Post attachment
fang avatar
fang@sofangtastic·Oct 5

jev, but it only outputs bufo paste a slack message, it ranks 1,600 frog reacts bufojev dot com

Post attachment
fang avatar
fang@sofangtastic·Sep 4

GPT-6 Astra just set a new SOTA on the Model Number Benchmark with a score of 6.0, a full 0.9 points ahead of Anthropic’s freshly released Fable 5.1 This represents a substantial 0.4 increase over the previous record set by OpenAI with their last gen GPT-5.6 series of models

Post attachment
Guanghan Ning avatar
⟳ fang retweeted
Guanghan Ning@quietnning·Jul 24

Opus 5 reports 30% on ARC-AGI-3, ~4× the previous best model, ~20× its predecessor Opus 4.8. We tested it on Witness, our held-out suite of ARC-AGI-3-style interactive puzzle games. The leap doesn't transfer. On Witness composites (same harness, same budget for every model), Opus 5 lands at 43.4 ± 3.2, which is a statistical tie with kimi-k3 (42.8 ± 1.9) and Fable-5 (43.8 ± 9.7). Ahead of Opus 4.8 (34.8), but nowhere near a generational jump. The traces tell the why: (1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre. (2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one. That decomposition (perfect on templates, regressed on novelty) is the signature of “scaffold-then-internalize” training on genre-specific data, not a general gain in interactive abstract reasoning. Our benchmark can't tell whether that data was their in-house ARC-AGI-3-like corpus with ARC-AGI-3-specialized harnessing (likely thanks to [schema]? x.com/HavenFeng/status/20777…), or public Witness-genre corpus, or both, but it can tell the improvement isn't general. Held-out evals only stay held-out while nobody's optimizing for the genre, and that clock is always ticking. It's ticking for Witness too, the moment we publish it.

Post attachment

On ARC-AGI-3, an evaluation where AI models must solve novel problems, Opus 5’s score is three times as high as the next best model.

Post attachment
fang avatar
fang@sofangtastic·Jul 23

if you've built your own trace viewer before because existing ones aren't cutting it, we're hiring @fleet_ai

fang avatar
fang@sofangtastic·Jul 23

built vscode for agent traces debugging long-running rollouts through terminal was miserable, so i made a local viewer that integrates with claude code/codex fully open source, cli-first, and extensible

0:33
fang avatar
fang@sofangtastic·Jul 23

we're past the era where companies are limited to doing just one thing - remember "Anthropic should just focus on models, Claude Code is a distraction"? now more surfaces don't slow you down as much anymore while their synergies compound

fang avatar
fang@sofangtastic·Jul 23

there's no reason why applied AI labs need to pick between consumer, prosumer, enterprise audiences in fact, we're seeing rapid verticalization from every company, whether it's human data, labs, app layer cos.