So the time has come, VerusCite is a fully functional site and ready to release to the public.
Hallucinations in peer review research are a current major problem. It signals a grave lack of care by researchers, and should trigger subsequently more scrutiny of the claims therein.
But even in articles that researchers clearly use genAI tools to write whole cloth, the majority of the citations are correct.
VerusCite only takes about 2~3 minutes to extract and verify those citations in an academic paper.
These are maybe the #2 bunch of papers that VerusCite can claim for retraction.
An independent researcher submitted these to the hall of hallucinations, veruscite.com/hallucinations
So that individual likely knew they were problems to begin with. But they used my tool to identify many other papers.
New blog post, I need to hire AI engineers, and here are the skills you need to know
I have just been doing open interviews (just email and I will do a screening, then a tech round).
Would love more social science PhDs to pick up these skills so I can hire you!
Looking for alpha testers for my secure AI agent chat application. (Think a better copilot and I do not spy on your conversations.)
Let me know if interested
Meta analyses are tempting to just let the AI go ham and do the work, but in my experience are likely one of the tasks that is likely to result in hallucinated references. (More papers, more chances for errors.)
Most recent addition to the hall of hallucinations this morning. These are a combination of author + journal hallucinations.
The Hall of Hallucinations just keeps accumulating papers.
Adding a few every day. The diabetes paper is from well cited social work authors out of Ohio State and U Mich.
New blog post on VerusCite, an independent researcher came up with a benchmark for hallucinated citations, and as I advertise VerusCite is well under a 1% false positive rate
veruscite.com/blog/validatin…
Recall is around 60%, but some of those are intentional differences.
Given the discussion on grad student stipends, here is my person reflection on the value of a PhD, and how on the margins social science programs can make it more valuable to students (just a wee bit of tweaking to focus on more relevant tech skills).
andrewpwheeler.com/2021/03/2…
I made $12k in stipend at Albany (started in 2008). Needed to take out loans to cover my expenses. It was very much worth it in the end for me, it took several years to get there though post-graduation.
Aaron is right here (fringe is the killer). I think treating the grad students like employees in some respects makes sense, but their wages need to be offset by the value of the degree.
Offhand I would be ok with that switch:
- students should have to buy into the same healthcare as faculty (and be able to cover dependents). So comes out of stipend.
- student housing for grad students (that like health insurance needs to be paid for, but can be cheaper). And can have family in the apt.
- increase stipend for actually teaching vs TA (high preference to give the adjunct roles to PhD students, which might in the end decrease quality teaching).
Rent to me is the killer here. I have no clue how urban depts even figure this out for undergrads though.
At SUNY my work at DCJS was good as well (I think I worked 30 hours a week). So making outside work normal (and perhaps stipend is only for when taking classes) may be good as well.
A $51,000 annual stipend, combined with free tuition and health insurance, produces a total annual compensation package exceeding $100,000. In exchange, students work approximately 20 hours per week for say four 14-week semesters over five years. This amounts to just 1,120 hours of work over the entire five-year period, yielding an effective hourly compensation of more than $450 -- over 18 times the median hourly wage of an American worker. Even if you write off the free tuition, this wage is around 9x that of the median US worker.
Not to mention that you walk away with a Harvard PhD when all is said and done. Good work if you can get it.
Hard to know if you are a neophyte what is legit vs clickbait on here related to AI. This story does not make sense for multiple reasons.
- Login token lag makes no sense with APIs
- Batch API requests "felt slow" makes no sense (they have a 24 hour SLA, uploading more requests does not make the results come back faster!)
- Batch has total mb size limits for upload, as well as a daily cap. Submitting them in parallel does not change this.
So sure while it is possible for an AI to do anything stupid, I find this pretty hard to believe. And I doubt any AI model would make these mistakes for well over the past year.
Three more additions to the hall of hallucinations this morning.
This paper on a meta-analysis of clozapine treatment for schizophrenia actually has many more AI hallucinations that my tool classifies as more minor title discrepancies.