←
Ana Marasović2,418 posts
Ana Marasović avatar

Ana Marasović

@anmarasovic

Asst prof @UUtah · Ex @allen_ai @uwnlp @HD_NLP · she/her 🇭🇷

◷ Joined April 2014▧ 215 Photos and videos
Shriram Krishnamurthi (primary: Bluesky) avatar
⟳ Ana Marasović retweeted
Shriram Krishnamurthi (primary: Bluesky)@ShriramKMurthi·Nov 6

I say over and over in PL that the more we prioritize authors over reviewers, the quicker everything will go to hell. Everyone knows this, but I think secretly everyone is terrified that *they* will one day get early-rejected (justly or otherwise), and they just can't bear that.

Minh Pham Dinh avatar
⟳ Ana Marasović retweeted
Minh Pham Dinh@phamdhminh2004·Nov 2

Should a small web agent practice whole tasks, or the skills inside them? 🧩 Skills! With the same amount of data, a 4B agent trained on skill-centric traces beats one trained on task-centric traces on all four benchmarks we tried. 🌐 minhphd.github.io/miniweb 🧵

0:31
Nathan Lambert avatar
⟳ Ana Marasović retweeted
Nathan Lambert@natolambert·Oct 21

What a wild day. We have a lot of work to do to get through this llm transition. But still, feels like the AI world will get crazier and scarier before it gets better. I hope I'm wrong, but my current emotion is something like... "gulp."

Peter Hase avatar
⟳ Ana Marasović retweeted
Peter Hase@peterbhase·Oct 19

I feel quite sad from doing my ICLR reviews today. It appears that the grim reaper of AI has come for scientific publishing. One paper was just desk rejected after I wrote a constructive review for it last week. The next two I had to give 1s. The fourth paper I recommended desk rejection, and the last I would reject for plagiarism, but the truth is that I don’t know if a human could attest to the writing. Please, please try to address this problem, future PCs. The ship has sailed for ICLR 2027, but after this, we will need something to change. Without a healthy peer review system, students are not exposed to unfiltered criticism of their work. This kind of criticism is essential for improving as a scientist, and it is a rising tide that lifts up the whole scientific community. We can’t abandon the system entirely. In order to be incentivized to engage with the system, reviewers either need payment or to be given work that is not complete drudgery to try to read. If reviewers disengage with the system or shirk their labor, as I very much want to do right now, this will lead to a death spiral where authors do not expect to receive fair reviews and reviewers do not expect to review genuine work. I wish there were promising solutions to this problem that did not involve increasing the deadweight loss associated with paper submission, such as requiring videos with submissions or submission fees. But the cost of submission in real terms has gone down, and we will need to artificially raise it in order to even begin to cope with the surge in demand for AI publications. There’s also the “burn it all down” solution, but I find it hard to imagine what that would end in besides reinventing the current system.

Florian Tramèr avatar
⟳ Ana Marasović retweeted
Florian Tramèr@florian_tramer·Oct 19

Some thoughts on reviewing: spylab.ai/blog/reviewing/ First for @satml_conf where we trialed an early reject phase which cut submissions by ~40%. Then more generally about how not everything deserves peer-review, and why we should reward writing things that people actually read.

Post attachment
Tom McGrath avatar
⟳ Ana Marasović retweeted
Tom McGrath@banburismus_·Oct 18

allowing papers like this is a terrible use of reviewers' time - it takes much longer to parse and is strongly correlated with the underlying paper being bad, which means we spend disproportionate amounts of time on bad papers

Kenneth Marino avatar
⟳ Ana Marasović retweeted
Kenneth Marino@Kenneth_Marino·Oct 15

Okay, so someone who was "some PhD student" who made a viral benchmark and someone who worked on product evals at DeepMind for Gemini, I think I might have a unique perspective here. 🧵

META’S HEAD OF AI SLAMS AI BENCHMARKS "it’s some PhD student who made these benchmarks" "you’re kind of outsourcing all of your product thinking to this viral benchmark" "so clearly that’s wrong" I think there is something to this. the whole industry developed so rapidly that there was hardly any time to establish proper benchmarking. companies used what was available, competitors then had to catch up on the same benchmarks, and some of them eventually became industry standards apparently there is quite a lot of randomness and chance in how we are measuring AI progress today

0:16
Ana Marasović avatar
Ana Marasović@anmarasovic·Oct 6

I too am at #COLM2026! (first conf since july 25, oops) I'm not presenting anything, but we just released a benchmark (👇), and I'd love to chat anything AI safety (CoT monitoring esp), interpretability, simulating users, and agents

FWIW this is the most ambitious benchmark I've co-created! 𝗞𝗡𝗢𝗪𝗦 jointly & reliably evaluates agents on: 1) live web search, 2) use of common web productivity tools, 3) visual/spatial understanding; programmatically where possible and by LLM judges where needed, more 👇

Alex Gill avatar
⟳ Ana Marasović retweeted
Alex Gill@alex_gill_nlp·Oct 2

A useful computer-use agent doesn't just find information. It turns it into something you can share. Introducing KNOWS: agents research the live web, then build a real Google Doc, Sheet or Slide deck in the browser.

0:56