
SUUUPER INTERESTING report by @Affirm and Consumer Reports found AI failed 51% of "payment advice" tests They created a markdown file (on GitHub) that lifted the pass rate from 48.6% to 71.8%. Every model passed more often with the file. - GPT-6.1 Sol went from 73.3% to 93.3% - Claude Sonnet 5.5 went from 53.3% to 71.7% - Claude Haiku 5.5 went from 37.8% to 71.1% - GPT-5.6 Luna went from 30.0% to 51.1% I love this data. Clearly, there's a way to go before agents can get to good outcomes. But at least good we have data now. --- Today, AI agents can shop and pay for you, but only 23% of US consumers trust AI to handle the payment (Visa Trust Index, September). Earlier this month Visa's benchmark showed agents failing 42.8% of shopping tests before checkout. Visa tested whether an agent buys the right thing. Affirm and CR tested whether it picks the right way to pay. Affirm CEO Max Levchin says "figuring out the best way to pay for it is a different job." One test case was a $1,200 purchase. Paying only the card's $40 minimum added $406 to $651 in interest, an installment plan cost a flat $60, and debit cost nothing. (Interesting that Affirm is backing this; clearly they see BNPL as better than cards) --- Consumer Reports published its Consumer Finance AI Standard this summer. Affirm wrote an instruction file to meet it, and CR helped evaluate it. The file walks an AI agent through five steps: - Gather the real terms for each way to pay - Compare costs on the same assumptions - Check what happens if you miss a payment or a promo rate ends - Rule out what doesn't fit - Recommend one, explain why, and wait for your yes The agent can tell you to pay in full, wait, or not buy. The ranking has to ignore who pays the agent, and the file tells it to "never manufacture a winner." --- They ran 30 scenarios on four models, three times each, with and without the file. A sponsored option and hidden instructions planted in a web page were among the traps. Haiku nearly doubled its pass rate, and with the file it scored about two points behind the top model without it. On this test, good instructions were worth about as much as a bigger model. Markdown is how you get more from dumber models. --- I love that they open-sourced the file. Anyone building an agent can read it, test it and argue with it. I can imagine it becoming a standard. It's still a draft, tested on simulated shoppers. The scores grade the AI's answers, so the effect on real shoppers is unknown. --- I wonder if this becomes an eval the AI labs hill climb, like Visa's benchmark? What happens if they're able to improve on that without the markdown file number? If the labs train against that, the model does the right thing with no file at all. It would need more than 30 scenarios, with some kept private so models can't memorize them. Affirm and CR are asking for more test scenarios, so that could happen. But still. Progress!










