Should I use Macro F1 or Accuracy? Why not Kappa? Why do some use this, and others that? What's actually evaluated here? 😵💫
Happy to share the final version of this paper on multi-class classification evaluation:
direct.mit.edu/tacl/article/…#machinelearning#nlproc#ml
This is a concerning trend.
Are people trying to "boost" their CV with a paper or is it AI startups trying to show "AI science"?
What's the deal with this? There's no material reward.
Another detail that might be interesting to folks:
"The fraction of desk rejected papers at TMLR used to be about 6% in 2023 but is now at about 53%."
(Important caveat that our policies and stances towards desk rejection have shifted, not just submission quality.)
TMLR has faced a deluge of submissions, necessitating stricter desk rejection policies due to limited reviewer capacity
Co-EiC Nihar Shah reached out to authors of 10 papers slated for desk reject. Could they answer questions about their *own* submission?
medium.com/@TmlrOrg/asking-a…
I don't get it. When it comes to code, AI should be way smarter than me. Yet there are dozens of examples where I ask the LLM to optimize something, it completely fails, I suggest "What if we do this?" and it says "Wow, great idea. That will work." And of course, I'm skeptical--maybe it's being sycophantic--but then we implement my idea and it's legitimately much more performant!
I am not some hotshot software engineer; I don't understand how I'm able to easily find improvements that frontier LLMs overlook. And this happens all the time!
What am I bringing to the table that it doesn't already have?
hey, what's with those reviewers who give
excitement 4 (this paper is exciting)
soundness 4 (this paper is strong)
overall score 2.5 (maybe reject)
i never understand those reviewers, like, how do they think?
Years ago you only needed papers when applying for job as post-doc or faculty. Now secondary-school students are using LLMs to write papers, to help them get into a good UG university... Inevitable result is wave of AI slop, which (as below) "authors" may not even understand.
We are seeing a new trend in submissions to @arxiv (and presumably to conferences and journals): Authors submitting papers whose contents they likely do not understand. 1/
I'm fascinating by how miserly AI progress is.
A lot of us assumed that once AI was as capable as it is now, we'd see insane progress in the things we actually cared about like number of useful projects completed. But that hasn't really happened.
It amazes me that the machines have literally proved a millennium prize problem, one of the hardest challenges in math, and yet somehow the result is still almost useless to us.
It's just a mountain of crap we humans have to dig through in hopes of salvaging crumbs of insights.
AI gives answers but in the least useful way possible.
Received three reviews. The first is short, helpful, and positive, human written. The other two are long lists with super picky minor details, no major issue outlined, and very negative. Is this situation normal now in the academic review system?
Evaluating Embeddings Dynamically in 200+ Languages with Automatically Induced Minimal Pairs.
Just accepted at EMNLP ("main").
Fine work by Andrianos Michail and others.
#EMNLP2026#nlprocarxiv.org/abs/2607.00171
LLM review weirdness...
Just renaming an uploaded pdf from "paper.pdf" to "paper_final_draft_pdf_ready_for_review.pdf" boosts average scores
(gpt-5.6-terra)
i think that exploding paper submissions could also be mitigated somewhat by stopping the conditioning of phd completion on "n papers at top conferences." there need to be better ways. maybe back to a writing a coherent monograph? or accepting *good* papers regardless of venue?