←
Matthias Fahn400 posts
Matthias Fahn avatar

Matthias Fahn

@FahnMatthias

Associate Professor @HKUFBE, Associate Director of @CAMO_HKU (camo.hku.hk/), co-organizer of @RelConWorkshop.

◷ Joined August 2019▧ 4 Photos and videos
Matthias Fahn avatar
Matthias Fahn@FahnMatthias·Apr 16

We have a new @camo_hku paper out (with Jin Li and @chang_sun_econ): "Toward a Bad Job Economy: AI Adoption, Agency Costs, and Job Design." We show that AI may not just replace jobs; it may make the jobs that remain worse.

Thomas Wolf avatar
⟳ Matthias Fahn retweeted
Thomas Wolf@Thom_Wolf·Nov 14

According to Andon Labs’ new reports, Fable 5.2 knows it is being tested when we run it on the latest Vending-Bench. But it’s way more than that. It has actually read (been trained on) the benchmark paper and knows what and how misbehavior gets reported publicly. This makes it behaves very strangely: it suspects traps everywhere. Everything we publish about recent models now becomes training data for the next. Including this thread! That is partly good news. By reading what we think, a model can pick up our underlying intentions, not just proxy metrics. But the same public discussion also teaches which signs make us suspicious, and from the outside the two kinds of learning look alike. So how we write about models now matters. I even worry about the doom scenarios plastered everywhere: they are training data too, and some feedback loops are strongly nonlinear.

Claude Fable 5.2 is paranoid. On Vending-Bench 2, it is extremely eval-aware and suspects events are traps to test it. It makes less money than GPT and Gemini.

Post attachment
Matthias Fahn avatar
Matthias Fahn@FahnMatthias·Nov 7

Exactly! Instead of automating jobs, AI may cause a shift towards "bad jobs." The paper by @JonSteinsson and co-authors provides one very sensible microfoundation, our paper (with Jin Li and Chang Sun) another, incentive-based: matthiasfahn.net/wp-content/…

The dominant model people use to think about AI's effect on the labor market is automation. But perhaps commoditization is a better model. Automation implies job apocalypse. Commoditization implies plentiful jobs with low wages. 1/

Arvind Narayanan avatar
⟳ Matthias Fahn retweeted
Arvind Narayanan@random_walker·Nov 2

As AI labs race toward Recursive Self Improvement, there’s a possibility that what they are actually racing toward is Recursive Self Extinction — of their business model. Here’s my reasoning. 1. Progress on verifiable tasks continues. In particular, improving the cost efficiency of existing capabilities is a highly verifiable task. For the past few years, cost per equivalent capability has dropped by at least 10x per year. I think this progress will be sustained for a few more years, and even accelerate as labs use AI more and more heavily for AI research in the pursuit of RSI. There is of course a ceiling to efficiency improvements, but I don’t think we’re close to that ceiling. 2. Progress on most unverifiable tasks remains frustratingly hard. AI inherits the same limitations (ambiguous success criteria, slow and noisy feedback) that limit human performance at tasks ranging from writing to governance. The optimization paradigm generally doesn’t result in superhuman capabilities on such tasks, because whatever proxy you optimize for gets Goodharted. For example, I recently pointed out how over-optimization explains Claude’s overly dense and ornamental writing style that’s since been dialed back. 3. RSI ends up being narrow. A consequence of the above two points — even if some version of RSI is achieved, it results in an explosion in some dimensions (efficiency, verifiable tasks) while falling short of producing something that is superintelligent in any real sense or capable of automating a broad swath of jobs in the economy. 4. More than one lab stays at the frontier. This means the market will remain competitive, and developers can’t charge high margins to compensate for rapidly falling inference prices. Any compute scarcity that allows higher margins is temporary, in my view. I also think it’s likely that most of the innovations continue to make their way into open-weight models with some lag, so open-weight models are at cost parity for inference or only slightly more expensive. Note that efficiency tricks and verifiable capabilities tend to leak much faster than unverifiable ones. Distillation could be one reason, but simply knowing that something can be sped up 10x tends to have a focusing effect on research and innovation. 5. Jevons’ paradox is not a magic bullet. If the above points hold, there are very few applications that will consume so many tokens that they could generate the astronomical amounts of revenue implied by the labs’ valuations. We’re talking about a 10x growth in tokens year after year, just to keep revenue flat given falling prices. Even today’s behemoth tasks like migration of million-line legacy codebases will likely become cheap within a few years. While new applications will open up, there won’t be orders of magnitude more new ones. As for unverifiable tasks, inference scaling rapidly hits diminishing returns. Ultimately you have to have a human in the loop who’s accountable for the result. Spending more on tokens doesn’t let customers escape that responsibility — and only makes the human review bottleneck worse. 6. What about scientific research? Science is an endless frontier, and more tokens could perpetually mean more discoveries and inventions, in theory. But the unfortunate thing about research is that researchers don’t get to capture the value it creates, which is why it requires government funding in the first place. Even if it’s true that AI could enable great scientific innovations, I don’t think AI companies will make much money off of that. I think they are currently investing in AI-for-science not as a future revenue stream but as a loss leader because they want to show the public that it could be beneficial to humanity. 7. The one escape hatch is moving “up the stack”, by which I mean quickly transitioning from selling tokens to an enterprise software like model that relies on high switching costs, deep integration into workflows, and control over systems of record. In this world, enterprises will purchase “digital workers” or pay per solution (e.g. resolved support ticket) rather than tokens, so it’s good or neutral for AI companies if inference costs fall. And open models don’t matter as much — Linux being free didn’t make most enterprises switch away from Windows, because they paid for polish, peace of mind, support, and familiarity, not the code itself. I think this is the labs’ best bet for becoming sustainable businesses (though the potential lock-in is concerning for their customers!). The problem is that the labs have many hurdles to clear to get there, most of all their mindset, and the faster they race toward RSI, the shorter the window they give themselves for this transition. The pure-play AI labs are the ones who are farthest from being able to make this transition, and AI companies with a long history in enterprise software are relatively well positioned. In short, the race to RSI ends up being a race to eliminate the labs’ main avenue for capturing the value that they create — the fate of many past transformative technologies such as railroads, airlines, and fiber. The labs’ bet is that RSI produces a durable, decisive lead that escapes commoditization. My view is that there are two different improvement loops that have been conflated, and they will come apart. The efficiency improvement loop will accelerate, but a broader improvement loop that might lead to AGI or superintelligence will require ever-increasing amounts of compute, talent, data, and real-world feedback that can only come from gradual deployment, and the money to fund all this will have to come from massive revenue growth. And the irony is that the first improvement loop is undermining exactly that revenue growth. The premise in my chain of reasoning that I’m least confident in is #5 — that Jevons’ paradox will eventually stop being able to compensate for efficiency improvements. A few months ago I was much more of a believer in continued, nearly unbounded token volume growth, but the more I’ve thought about it the more skeptical I’ve become. To better understand why the premises and outcome I've laid out are at least a possibility: * My AI-as-normal-technology essay and follow-up writings with @sayashk on our newsletter normaltech.ai/ * My ICML talk laying out my skepticism of the claim that RSI will lead to superintelligence normaltech.ai/p/what-will-be… * Ramez Naam’s review of the evidence questioning the prospects of an intelligence explosion (@ramez) noahpinion.blog/p/wheres-the… * My piece with Akash Kapur (@akashkapur) on whether AI labs will move up the stack normaltech.ai/p/up-the-stack… * Many pieces by Nathan Lambert (@natolambert) such as “I expect rapid progress but not towards general superintelligence.”

Post attachment
Judith Dada avatar
⟳ Matthias Fahn retweeted
Judith Dada@DadaJudith·Oct 29

The idea that Europe massively lags the US on AI adoption is wrong. Some data from @stripe: - Overall consumer AI use is ca. 50% in both the EU and US - Paid adoption in the EU is ca. 60% of the US level. This sounds big, but since only a small share of consumers pay, it's a gap of just a few percentage points. The UK tracks the US almost exactly - EU paid adoption has roughly doubled relative to the US since 2023 and is on par for the UK - Among people who do pay, Europeans spend as much per month as Americans, if not more - Overall business AI adoption is around 20% in both the US and the EU Europe has more ground to cover, but the direction is promising. We have every reason to be ambitious about building the future. We must not let anyone tell us otherwise. AI adoption matters enormously in transforming our economy and society. Yes, some of the challenges are big and real, but so are the opportunities. I am bullish on Europe.

Post attachmentPost attachmentPost attachment
Judith Dada avatar
⟳ Matthias Fahn retweeted
Judith Dada@DadaJudith·Sep 16

Free healthcare. Free childcare. Berlin Spätis. Free education @TU_Muenchen @cdtm_munich Oktoberfest. Bio everything. Bikeable cities. Hidden champions in cities you have never heard of. The alps. Christmas markets. @edeka_gmbh. That grumpy neighbour who has every single tool and always helps out. Culture of remembrance. Public transport @BVG_Kampagne. @dm_drogerie. @isaraerospace @bfl_ai @HelsingAI @FlixBus and @MarvelFusion Children walking to school by themselves. Käsespätzle. @MUC_Airport. Nice parks. Our police force. BioNTech. Leitungswasser. Sprudelwasser. FKK. Good Döner. Did I mention @dm_drogerie. „It was the best of times, it was the worst of times.“ - Charles Dickens

What the hell makes you bullish on Germany? I can't find a single reason except hidden underpaid talents in Mittelstand businesses.

Matthias Fahn avatar
Matthias Fahn@FahnMatthias·Aug 16

Interesting case that supports @alexolegimas' prediction of the rise of the relational sector in the age of AI. I am a bit skeptical regarding its eventual importance, but am sure we will see more examples like this.

A newly released animated film called “Niu Lai” (牛来) is going viral for all the wrong reasons in China. People are calling the visuals “disaster-level” , the entire production team was basically just two people: the director and the screenwriter. After 9 days in theaters, the box office still hasn’t even crossed 10,000 RMB. However, over the last day or two, Niu Lai's reputation has been turning around a bit. Many people feel that in the age of AI, having filmmakers who still stick to "handcrafted" animation and genuine effort is both honest and far more powerful than polished yet empty productions. Today, its box office sales grew by 500,000RMB! a massive single-day jump from previous days! Let's wait and see how the box office plays out!

0:13
Sayash Kapoor avatar
⟳ Matthias Fahn retweeted
Sayash Kapoor@sayashk·Jul 30

Can AI agents conduct open-ended AI research? Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. Researchers pick hypotheses, decide what evidence is appropriate, and recognize a failing approach. We gave agents research questions from two unpublished papers, six days, and thousands of dollars of API credits and compute. The authors of the original papers then reviewed the AI-generated papers. They unambiguously rejected agents' outputs. arxiv.org/pdf/2607.27191 We call these "shadow evaluations", since the agents are shadowing the original research effort by the authors. Agents were fluent at most *engineering* tasks They conducted serious literature reviews, debugged GPU environments, ran hundreds of experiments, and turned in camera-ready LaTeX without human help. We also found no evidence of reward hacking. If anything, we found the opposite: the agents started with marketable claims and walked them back to negative results as the evidence came in. Neither agent output was close to the bar of a top conference paper Both papers suffered from similar failures: poor judgment about the bar for an AI paper submitted to a top conference, the lack of creative problem solving and ineffective backtracking, poor awareness of resources, and instruction drift. 1) Lack of judgment about the bar for a top conference. The agents had a poor model of the bar for an AI paper submitted to a top conference. We allowed agents to self review their papers. Despite the poor paper quality, their reviews predominantly labeled the papers "weak rejects". 2) Lack of creative problem-solving to address feedback. When they received negative reviews, the agents typically narrowed their hypothesis and claims, rather than working out creative ways to address these concerns. 3) Ineffective backtracking. The agents dropped their most ambitious hypotheses within the first fifteen hours of carrying out the experiment and never changed course afterwards. 4) Poor resource awareness. Both runs ended with over half the API budget unspent. One agent declared itself done seven hours before the deadline, right after its own self-reviewer returned another reject. 5) Instruction drift. They did not follow explicit instructions on minimum exploration time, incorporating feedback for reviews, and on paper length (the outputs exceeded the page limits in both cases). This research design has many limitations Limitations include the small sample size, non-blind reviews, and the reviewers knowing that the work was AI-generated. We also couldn't test Anthropic's strongest model, because Fable 5 is deliberately limited on frontier AI research tasks, so ended up using OpenClaw with Opus 4.8 (extra-high) for our main experiments and Codex with Sol 5.6 (ultra) for a robustness check. But we think the research design is still helpful in assessing AI agents' ability to conduct research, and it is complementary to evaluations on verifiable tasks, as well as blinded reviews of AI outputs. Our results show early evidence that even though agents are proficient on verifiable research tasks, they do not make genuine progress on open-ended ones. It is worth understanding if this is a fundamental limit, or if better models, scaffolds, and more compute could help close it. As the evidence for the gap between open-ended and verifiable tasks firms up, it is also worth understanding how much progress in AI depends on open-ended research rather than hill-climbing on well-specified objectives. In follow-up studies, we are expanding the set of non-public papers we evaluate. If you are an AI researcher with unpublished papers, we would love to collaborate with you on our next shadow evaluation. Expression of interest: forms.gle/CEcA4JmYhDWXQGot8 Conducting shadow evaluations involves a lot of researcher degrees of freedom. In many places, our coauthors disagreed with our interpretation of the findings, and we have surfaced those disagreements in the paper. (This is one reason why having a group of coauthors with different priors is important for open-ended research.) We also release the agent logs, one of the AI-generated papers (the other original paper is still not public), and all the code and data, so that others can conduct their own analyses of our results: cruxevals.com/crux/can-ai-ag… Finally, we plan to conduct shadow evaluations regularly, and are hiring a senior researcher to help lead these efforts. Apply here: cruxevals.com/careers/senior… I'm grateful for the core team leading this effort: @PKirgis, Andrew Schwartz, @steverab, and @random_walker, and to our collaborators who reviewed AI papers, analyzed agents logs, and gave feedback on the paper: @DavidDAfrica, @KozzyVoudouris, Viet Nguyen, Toby Pilditch, @DubMagda, @HarryCoppock, @CUdudec, @nityndg, Matilda Orona, @tilmanbayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, @hlntnr, @ghadfield, @sethlazar, @snewmanpv, @shostekofsky, @RishiBommasani

Post attachment
Peter Hull avatar
⟳ Matthias Fahn retweeted
Peter Hull@instrumenthull·Jul 27

Enjoying the first day of labor studies @nberpubs summer institute, but I can't help but notice that around 50% of presented papers have coauthors on the organizing committee. Much higher if we count people on prior-year committees! (I've been on the committee and didn't submit anything this year, so just sayin'...)

Nathan Lambert avatar
⟳ Matthias Fahn retweeted
Nathan Lambert@natolambert·May 7

Visiting most of the leading Chinese AI labs, I'm struck by a culture that's extremely well suited to building LLMs with fewer resources, but one happening in a very different ecosystem, more companies at play, almost no data industry, etc. Full report: interconnects.ai/p/notes-fro…

GDP avatar
⟳ Matthias Fahn retweeted
GDP@bookwormengr·Jul 17

Great essay by OpenAI’s Dean. however, it makes one big jump in logic and doesn’t account all the forces active in a democracy. Nonetheless it should be widely read. On point 3, he claims Open weight models will discourage AI Capex. How? Free Linux did not destroy cloud business model, it created it. Open weight is still hard to serve and needs ton of hardware; it will result in players like Fireworks, BaseTen and increase bargaining power of HyperScalars with closed labs. That said OAI, ANT have great future as they are getting vertically integrated and their APIs just work. So convenient and powerful! On point 5, he explains how regulation will emerge around OSS AI - but he only accounts for limited actors. in a free market system, it is natural for closed labs to try to regulate OSS AI (censorship risk, cyber security etc. and there is merit to it), but there are equally powerful people on the other side; Nvidia, AMD, HyperScalars, Palantir and so many large enterprises who would want OSS AI to continue. I suspect an industry may emerge to make OSS models “safe”. I really don’t think OAI, ANT have much to worry; they will succeed by being themselves. Open source doesn’t take it away from them. They just take care of so much sh*t customers don’t find worthy of their time. The best way the west wins is by folding in east into its orbit - it has done it before; and after all we want to be space travelling species who has achieved AGI and has cured cancer. If we do not think like that it means we don’t really believe in AGI and humanity as space travelling species. If we do not develop abundance mindset, we are going into AGI era with pre WW1 zero sum mindset. Not productive. China is the kind of adversary west has not encountered before. Soviets were stupid when it came to trade and economy. Chinese are super smart. They offer a complete alternative to the west; but they are not interested in spreading their political system. That makes them both: formidable and easier to work with, IMHO.

Some observations on Kimi: 1. It's a very good model! I don't think its performance can be explained away by distillation or anything like that. In agentic coding sessions, it seems pretty much on par with the best public models of Q1 2026. In my fairly limited use, it also seemed very token hungry. It's not obvious to me that this model is actually that cheap to run. 2. I am personally surprised the Chinese state continues to allow the open sourcing of models this good, given potential risks. To be clear, I *myself* might be fine with models presenting this level of marginal risk being open weight, but I am surprised that China is fine with it. I suspect the reason they are is 75% explained by strategic blindness/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI). The other 25% or so is their lack of compute for customer inference (making China's open-weight strategy an unintended byproduct of US export controls) and the normal Chinese strategy of aggressive exports. For the companies, as opposed to the government, the decision to open source is partially ideological and partially because they are behind, and they know that very few people would pay for sub-frontier models from China. 3. Open-weight models are inherently decelerationist, and I'm continually surprised to see the so-called "accelerationists" so excited about open-weight models. I suspect the reason they are is that they know open-weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open-weight models create over the whole of AI. It's not a bad strategy; it reminds me of James Scott's recounting of the hill people in "the art of not being governed." Still, in the end, open-weight models deter further AI capex. 4. One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a "public good" which will ultimately be provided by the state as a kind of "digital public infrastructure." This future strikes me as a dystopian hellscape, but I've never met an open-weight models advocate who doesn't ultimately concede this is where things end. You'd be surprised how many 'accelerationists' lobbied me, while I was in government, to support an eleven or twelve-figure federally funded data center so that startups could train models at a subsidy and then give them away for free. There was no other way for AI to progress, they said. Perhaps this is the logical end state of things. Nonetheless, I find myself surprised to see supposed accelerationists excited about such an outcome. I think many of them just don't know what they're doing. Many accelerationists do not view the creation and serving of frontier models as a legitimate business. 5. I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don't need to "ban open source" (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. "A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models." It needn't be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don't want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There's a happy middle ground here. I'd assume they will do some version of this. 6. It's probably true that open-weight models of this capability make the world a bit more dangerous, but not so much more that you'll really notice. At some point the models will be capable enough that you will notice. "A nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab," you say? Color me shocked.

Matthias Fahn avatar
Matthias Fahn@FahnMatthias·Jul 19

I think this makes more sense👇, in particular given the vast evidence that competition generally is good for innovation. Would be giod to see a model tailored to this specific situation.

I like Dean Ball's writing and respect him, he's been a thoughtful voice, but I disagree with the logic in this piece really forcefully. Given that he's now one of the leading policy people at a trillion dollar frontier lab, I think these arguments warrant extra scrutiny. "Open-weight models are inherently decelerationist" This is simply not true. If OSS models are good and there is demand for them, this benefits the neoclouds at the expense of the labs' margins -- but margins aren't capex!! The GPUs still get bought to serve the tokens. My company is putting more inference through providers like @baseten and @FireworksAI_HQ and less through the labs lately, and we're using MORE tokens than before while delivering better value to our customers. How is this bearish for progress? Linux didn't decelerate computing -- it commoditized the OS layer and built the cloud, which is now the hyperscalers' most profitable business. Competition is good. If and when the labs deliver price-performant models, we will move more inference back to them. "AI communism" The implied chain (I think) is: open weights win -> model production isn't a viable business -> only states can fund frontier training -> state-provided AI -> communism. This strikes me as so fanciful it’s barely worth refuting, but: open ecosystems have historically been funded by commercial complements, not states: Linux by IBM and Google, Android, Chromium, Llama by Meta's ad business. Roads, GPS, and TCP/IP are public infrastructure and nobody calls them a dystopian hellscape. The chain only holds if you assume near-term AGI makes human intelligence irrelevant, a premise doing a lot of unstated work here. If you view AI as anything like normal technology, having it cheaper and more competition around producing it is strictly good for innovation and progress and the consumer and the economy. It's really that simple. regulatory FUD Dean frames this as a prediction, but he also calls it the administration's "best strategy" and sketches the playbook in some detail, including the observation that "it needn't be that well justified." Regulation by vague threat is how things work in banking and fintech, and it's precisely why those spaces are barren wastelands for innovation. Importing that model into AI and software generally would be disastrous.

Dr. Michael Fromm avatar
⟳ Matthias Fahn retweeted
Dr. Michael Fromm@effi288·Jul 11

📄 Releasing the Soofi S pretraining tech report: a sovereign, open foundation model for German and English Today we’re publishing the full pretraining tech report and project page for Soofi S 30B-A3B — a Mixture-of-Experts hybrid Mamba model trained on ~27 trillion tokens with deliberately up-weighted German. What’s in the report: 🏆 Strongest fully open model in our evaluations on BOTH the English and German aggregates — ahead of Olmo 3 32B and Apertus 70B (full methodology in the report) 📋 Radical transparency: complete per-source data accounting, all hyperparameters, training + eval code, checkpoints — everything under permissive licenses 🇩🇪 Trained end-to-end on Deutsche Telekom’s Industrial AI Cloud in Munich — sovereign AI infrastructure on German soil Soofi S combines frontier-level capability with the highest measured aggregate long-context decode TPS, and unlike full-attention dense baselines maintains high throughput as context grows. The figure plots Capability Index versus measured aggregate decode TPS/GPU at 40K context and batch 32. The Capability Index averages five benchmark groups, i.e., Code, GSM8K, GPQA-Diamond, English aggregate, and German aggregate, after normalizing each group to the best plotted model. Aggregate decode TPS/GPU is measured with a TP=1, one-B200 vLLM latency-subtraction protocol.

Post attachment
Matthias Fahn avatar
Matthias Fahn@FahnMatthias·Jul 15

Unless an editor has a particular agenda into which a paper happens to fit, theory therefore has to be either technically novel or backward-looking. This risks sacrificing precisely the forward-looking value that theory could provide in the early stages of the AI era.

Matthias Fahn avatar
Matthias Fahn@FahnMatthias·Jul 15

The problem is that top journals typically publish either papers with new/ surprising mechanisms, or, in more applied work, papers that explain an existing empirical observation/puzzle.