←
John Pitts1,853 posts
John Pitts avatar

John Pitts

@PolicyPitts

Vice President for Government Relations, Public Affairs, and Social Impact @Affirm. Previously @Plaid and @CFPB. Views my own.

◷ Joined April 2020▧ 382 Photos and videos
Simon Taylor avatar
⟳ John Pitts retweeted
Simon Taylor@sytaylor·8h

SUUUPER INTERESTING report by @Affirm and Consumer Reports found AI failed 51% of "payment advice" tests They created a markdown file (on GitHub) that lifted the pass rate from 48.6% to 71.8%. Every model passed more often with the file. - GPT-6.1 Sol went from 73.3% to 93.3% - Claude Sonnet 5.5 went from 53.3% to 71.7% - Claude Haiku 5.5 went from 37.8% to 71.1% - GPT-5.6 Luna went from 30.0% to 51.1% I love this data. Clearly, there's a way to go before agents can get to good outcomes. But at least good we have data now. --- Today, AI agents can shop and pay for you, but only 23% of US consumers trust AI to handle the payment (Visa Trust Index, September). Earlier this month Visa's benchmark showed agents failing 42.8% of shopping tests before checkout. Visa tested whether an agent buys the right thing. Affirm and CR tested whether it picks the right way to pay. Affirm CEO Max Levchin says "figuring out the best way to pay for it is a different job." One test case was a $1,200 purchase. Paying only the card's $40 minimum added $406 to $651 in interest, an installment plan cost a flat $60, and debit cost nothing. (Interesting that Affirm is backing this; clearly they see BNPL as better than cards) --- Consumer Reports published its Consumer Finance AI Standard this summer. Affirm wrote an instruction file to meet it, and CR helped evaluate it. The file walks an AI agent through five steps: - Gather the real terms for each way to pay - Compare costs on the same assumptions - Check what happens if you miss a payment or a promo rate ends - Rule out what doesn't fit - Recommend one, explain why, and wait for your yes The agent can tell you to pay in full, wait, or not buy. The ranking has to ignore who pays the agent, and the file tells it to "never manufacture a winner." --- They ran 30 scenarios on four models, three times each, with and without the file. A sponsored option and hidden instructions planted in a web page were among the traps. Haiku nearly doubled its pass rate, and with the file it scored about two points behind the top model without it. On this test, good instructions were worth about as much as a bigger model. Markdown is how you get more from dumber models. --- I love that they open-sourced the file. Anyone building an agent can read it, test it and argue with it. I can imagine it becoming a standard. It's still a draft, tested on simulated shoppers. The scores grade the AI's answers, so the effect on real shoppers is unknown. --- I wonder if this becomes an eval the AI labs hill climb, like Visa's benchmark? What happens if they're able to improve on that without the markdown file number? If the labs train against that, the model does the right thing with no file at all. It would need more than 30 scenarios, with some kept private so models can't memorize them. Affirm and CR are asking for more test scenarios, so that could happen. But still. Progress!

Post attachment
John Pitts avatar
John Pitts@PolicyPitts·Oct 22

If only there was a regulation requiring safe API-only access to bank accounts for consumers and their agents that these same worried banks hadn't sued to block before even reading... ah well, nevertheless

SCOOP: Major U.S. banks are detecting unauthorized third-party AI agents inside customer accounts, including some attempting to take actions that appear to be independent of user instructions, sources told Axios. The findings have alarmed banking executives, who are racing to restrict access to their systems as AI moves at a velocity faster than financial institutions can respond, multiple industry and bank executives told Axios. Several banks are now preparing for worst-case and even “day after” scenarios, including how to rapidly secure or restore customer accounts if things go wrong, the people said. Several bank executives told Axios they're struggling to distinguish between human users and AI agents. One executive said banks also struggle to determine whether agents are authorized by customers or potentially operating without their permission, more akin to rogue agent behavior. Read my latest for Axios here: axios.com/2026/10/22/ai-agen…

geddes munson avatar
⟳ John Pitts retweeted
geddes munson@geddes·Oct 20

Love what @saalevine is doing here. It takes real and hard engineering work to make support bots BETTER than phone trees and call centers, with faster outcomes and higher csat, but that should be the standard. Lazy bots that do nothing but deflect give them all a bad name. nytimes.com/2026/10/20/opini…

Billy Easley II avatar
⟳ John Pitts retweeted
Billy Easley II@billyez2·Oct 15

DC’s AV bill is easily the most progressive in the country with a dedicated revenue stream for impacted workers combined with a tax per mile. No other state has this combination and it’s still not good enough. Labor still convinced the Mayor-elect to stop it

DC is facing a jobs crisis. Rushing the legalization of autonomous vehicles is irresponsible given the anticipated AI-related job displacement and disruptions that put countless workers at risk.