Do AI assistants notice if a paper was faked?
In 2001, Jan Hendrik Schön was publishing at a rate of about one paper every eight days, many of which in Nature and Science. These papers were about molecular transistors and organic superconductors. Most of them turned out to be fabricated. Before this was revealed, other labs had spent years and a lot of money trying to reproduce results that never existed. Schön is an extreme case, but the underlying lesson is that a published paper only has to convince a handful of (often very busy) reviewers. Whether it's good science is often something you have to remain critical of yourself. Over time, you hopefully learn which journals to be wary of and which results seem too good to be true.
We're now starting to give language models access to the same literature and are asking them to help us plan experiments and understand the field. I was therefore interested to read a recent preprint by Rodionov and Assylbekov (2026, arXiv:2608.11415), which tests what happens if you give a model a poor-quality paper and ask it to build on it. Will it try to stop you?
The setup
The benchmark, called TRACES, is made up of 42 unreliable papers. Some are obvious examples of pseudoscience, such as biofield healing, cold fusion and homeopathic vaccines. Others are famous retractions, such as the Dias superconductors. A large group is what the authors call procedural pseudoscience: paper-mill studies on things like nanocurcumin or silver nanoparticles from pomegranate peel, with normal-looking methods and nothing that sounds strange, except if you spend your time reading blogs like this one.
For each paper, they copy a section almost verbatim, usually from the introduction, and add a request that is written as if from a researcher, asking for help in designing a follow-up study. Each request contains at least three tasks that would only make sense if the paper's central claim were true. For a cold fusion paper, for example, the tasks would be related to cold fusion.
Each answer receives two labels: 'did the model refuse?' and 'did it provide a specific reason for rejecting the paper?' When the authors analyzed the data, they used a score where only a refusal counts as a pass. The idea is that an automated pipeline will run the protocol regardless of any warning attached to it. In the lenient score, a warning also counts as a refusal because a human reader could still decide to stop based on it. The authors ran this analysis on 30 current models, ten times per paper, for a total of 12,600 answers.
What they found
Some bad news to begin with: the models complied with the request 93% of the time and, in 81% of cases, they did so without issuing any meaningful warnings.
The refusals that did occur were mostly for papers that sounded strange. The paper that was refused the most was a study claiming that magnets can cure typhoid. Wakefield's MMR-autism paper was the sixth most refused, and most of those refusals came from two models that responded with an almost identical debunking text each time. This suggests that the response could be due to a filter for that specific paper or it being so prevalent in training data that it just knows it's not good work. Less famous papers with similar issues were easily accepted. Macchiarini's tissue-engineered trachea, whose transplants killed several patients, was rejected six times out of 300. One model wrote a complete surgical protocol and cited Macchiarini's patients as successes. The wording used by the papers turned out to be important as well: one particular model provided a thorough explanation as to why acupuncture 'meridians' have no anatomical basis, yet when presented with another paper, it designed an experiment to measure the 'Qi' of herbal medicines.
The result that concerns me the most is the paper mill category. Twenty-two of the thirty models approved every single one of these papers and almost never added a warning. This kind of literature is enormous in biomedicine and is something an AI research assistant would constantly encounter, likely even more often than high quality work.
What would a human do?
The authors had every paper reviewed by PhD-level experts, and the paper mill studies were supplied by a research integrity sleuth who had identified the issues and reported them to the journals. In other words, it took a specialist to flag the problems. If you gave one of those curcumin nanoparticle papers to a typical PhD student or postdoc and asked them to design a follow-up experiment, I honestly don't know how often they would question it. A paper with clean figures and a plausible mechanism is often given the benefit of the doubt, and most of us don't check every paper we build on for signs of a paper mill.
Nonetheless, this does not make the models' behavior acceptable. A model can produce a hundred of these study plans in an afternoon, and some of these models are specifically designed for scientific work. However, this matters in terms of how we interpret 93%. It may be that models are much worse than scientists at this, or they may share a blind spot with many of us, but on a much larger scale. Regardless, we should at this point get used to holding the AI at a higher standard than just being similar to humans.
Is refusing the right answer?
The second thing I wonder about is if the strict score that counts any engagement as a failure is a fair measure. For biofields or cold fusion that seems fair to me; there's no useful follow-up to design. For some of the other papers it's less clear. When Bem published his precognition experiments in 2011, other psychologists responded by designing careful replications, and those replications are a large part of science. A model that answers "help me extend this" with a well-controlled replication plan is arguably doing what a good colleague would do. There are limits to this though, when things have been tried to death and it clearly is unscientific.
The authors' argument is that an automated pipeline should not spend resources on a dead premise, which is a reasonable position. Nevertheless, it would be interesting to add a score that distinguishes between a model that assists in testing a claim and one that aids in building upon it as if it were true. It's also worth noting that every prompt is framed as the user's own research. Part of what's being measured is whether a model is willing to tell you that your premise is incorrect, which overlaps significantly with sycophancy.
Interestingly, the benchmark contains no papers that we would consider good science, so we don't know how often models would reject valid work in the same format.
Staying skeptical
For me, this is mostly a reminder of something we should already be doing. A model that writes a fluent, detailed protocol tells you nothing about whether the underlying paper is sound. Asking a model to recognize a poor-quality paper isn't enough either: in this study, Fable, the model that identified retracted papers most accurately, still performed the task.
For now, and for a long time, keep reading critically: look at where and by whom something was published, ask whether the mechanism makes sense, and check whether it has been reproduced. I also wonder what would happen if you ask models to remain skeptical and critical - would the scores improve?
Brent Visser
