←
Matt Lehman1,010 posts
Matt Lehman avatar

Matt Lehman

@obsecurus

Principal Investigator | AIxCC Lead Architect (views my own)

◷ Joined May 2010▧ 79 Photos and videos
Matt Lehman avatar
Matt Lehman@obsecurus·23h

Anyone else excited for the new model drops over the next 48 hours?

Matt Lehman avatar
Matt Lehman@obsecurus·Sep 4

In AI if you move the goal posts, own either team, or the refs, these evals aren’t wins or losses, they’re marketing theatre. Effective eval should naturally fall out of any system doing real work and subject to independent verification.

Matt Lehman avatar
Matt Lehman@obsecurus·Aug 27

The ability of an LLM or agentic framework to refactor a code base into a new language, while impressive, does not equate to an effective software or product development lifecycle. The years of toil writing effective tests are key. It would be an interesting experiment for teams to attempt re-writing Bun or NodeJS with all tests removed first. The ability to replicate something that has decades of lessons learned does not mean a mechanism to autonomously create production grade innovative products. Effective tests and evaluations are probably the biggest pain I continue to see and not a void the LLMs seem to fill well.

Matt Lehman avatar
Matt Lehman@obsecurus·Jun 14

They should replace “pioneer day” with “data center day” for grade school. I had to go on a waste management facility field trip.

Matt Lehman avatar
Matt Lehman@obsecurus·Jun 10

Fable 5 applying forced token maxing by converting to a Python strategy over RTK. I've personally found Python generation far less reliable and reliable than Golang, Rust, or TypeScript. I think this is primarily because of the strong typing, but perhaps even moreso the abundance of opinionated community content about the "right" way to implement something that likely went into training.

Post attachment