🔍 Read the full analysis: Do LLMs Make Useful Judges Of Research Novelty? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers tested six LLMs as judges of scientific idea novelty and found that small changes to evaluation setups could produce sharply different results. In one pairwise test, prompt wording changes shifted accuracy by more than 50 percentage points; retrieval and extra reasoning did not consistently help. The team released a benchmark and code, while warning that some novelty labels are limited.
Researchers at Ai2 report that six large language models produced unstable judgments of scientific idea novelty when prompts and other evaluation choices changed, as detailed in the original analysis. In one pairwise-comparison experiment, changing a few prompt words shifted accuracy by more than 50 percentage points, raising questions about whether such systems can reliably screen research proposals before human review.
The study tested GPT-5.1, GPT-5.2 and GPT-5.4, alongside Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The researchers assessed models using both absolute novelty ratings and pairwise comparisons between ideas. They also examined whether literature retrieval and additional reasoning improved judgments. According to the authors, neither addition helped consistently across the evaluations.
For one benchmark, the team used ICLR 2026 peer reviews to identify ideas with explicit reviewer comments about originality. Its higher-novelty group includes 154 ideas from accepted papers in the top-rated 10% of their research areas, where reviewers praised novelty and no reviewer disputed it. The lower-novelty group includes 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns and none praised novelty. In a check of 50 extracted signals, an annotator agreed with the direction in 47 cases; the authors say the other three did not change the review classifications.
A second test paired the 154 human-authored ideas with 154 proposals from a basic language-model generator. The researchers treated generated proposals as less novel, but say those labels are weak and not verified individually. In a separate expert assessment of 30 pairs, selected largely from cases the strongest judge got wrong, an expert preferred the human-authored idea in every pair and judged none of the generated ideas novel. The team also reports that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Unstable Scores Complicate Idea Screening
Research groups may use LLM judges to filter candidate ideas before spending time on literature searches, experiments or expert assessment. If small changes in prompt wording can substantially alter a score, the result may reflect evaluation choices as well as the idea itself. That makes it harder to compare proposals across teams or to treat a single novelty score as dependable evidence.
The findings also matter to efforts to automate parts of scientific discovery. Systems that generate hypotheses or summarize prior research need a credible way to judge whether a proposal adds something new. The study does not show that every LLM-based novelty assessment fails, or that models cannot help researchers. It does indicate that performance on one selected setup is not enough to establish reliable judgment. Repeated testing across prompts and evaluation formats may be needed before scores are used to make consequential decisions.
Products Worth Considering
Bose QuietComfort Wireless Noise-Cancelling Over-Ear Headphones, Black
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Ai2 Constructed Its Tests
A novelty judge is a model asked to assess whether a research idea is new compared with prior work, either by rating it on its own or comparing it with another idea. The Ai2 authors say there is no standard evaluation recipe: researchers can select different models, prompts, comparison formats, reasoning budgets and literature-retrieval systems. They also argue that tests often use human-written papers even when the intended application is judging AI-generated proposals.
The benchmark attempts to cover both human-authored ideas with reviewer evidence about originality and a setting that includes generated proposals. The authors caution that acceptance or rejection alone is not a sound novelty label, because papers can be accepted or rejected for reasons other than originality. Their approach uses explicit review comments and selection criteria, but these remain indirect signals rather than a definitive measure of whether an idea is new.
Ai2 has released the paper, benchmark data, code and an automated pipeline intended to support creation of further evaluation sets. The authors frame the work as part of a broader effort to develop systems that assist scientific discovery, including tools for hypothesis generation and research synthesis.
““Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Novelty Labels
The study’s benchmark does not provide a definitive ground truth for originality. Its human-idea labels rely on reviewer comments and paper rankings, while the generated-idea set assumes proposals from a basic language model are less novel than selected human ideas. The authors describe those generated-idea labels as weak rather than individually verified. The expert check covered 30 pairs, not every item in the benchmark.
The supplied material does not give full results for every model, prompt, reasoning budget or evaluation format, and it does not identify the release date. It also leaves open whether the findings would carry over to other scientific fields, different idea generators or systems with broader access to research literature. The authors say some conclusions arise from disagreement between evaluation setups rather than benchmark labels, but the material does not detail how broadly that result holds.
Further Testing Across Research Fields
The released benchmark and pipeline give other teams tools to test whether novelty judgments remain unstable across disciplines, idea sources and judge designs. Researchers can also use the resources to compare different prompt, retrieval and reasoning choices rather than relying on a single evaluation configuration.
Ai2’s release description does not announce a specific follow-up study or schedule, and the study does not identify a method that consistently resolves the instability. The next evidence will need to show whether alternative designs can produce more repeatable judgments and whether those results hold beyond the benchmark’s particular ideas and labels.
Key Questions
What did the Ai2 study find?
The researchers found that six LLMs’ novelty judgments varied across evaluation setups. In one pairwise test, changing a few prompt words shifted accuracy by more than 50 percentage points.
Which models were tested?
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6.
Did literature retrieval or extra reasoning improve the results?
Not consistently, according to the authors. The supplied study description says that retrieval and additional reasoning did not reliably improve judgments across the evaluations.
Are the benchmark’s novelty labels definitive?
No. The human-idea labels rely on peer-review comments and paper selection criteria, while generated proposals were treated as less novel without individual verification. The authors characterize those generated-idea labels as weak.
What resources did Ai2 release?
Ai2 released the study paper, benchmark data, code and an automated pipeline designed to help researchers build additional evaluation sets.
Primary source: Hugging Face · via ThorstenMeyerAI.com
