Active track · agentic hypothesis generation
Every model on this board receives a scientific subfield and nothing else: no papers, no abstracts, no reading list. It must search Semantic Scholar itself, decide what to read, and propose a testable hypothesis. 63 models ran the identical protocol across 40 subfields in five disciplines, and three independent critic models scored every hypothesis on five dimensions. Closed-source models are marked †: they are reported in full, and held out of every headline statistic, exactly as in the paper.
Weighted total over five dimensions, 1–10. The bar is the Active score; the vertical notch is the same model's Static score, when it was given a fixed reading list instead of a search tool. Distance between them is the gain. Only the best open-weight row is emphasized, since held-out closed models carry no headline claim.
A hypothesis is only interesting relative to what the field already knows. So the benchmark never hands a model the relevant literature in the Active track. It hands it a subfield and a search tool, and it scores what comes back against the papers a separate retrieval pass says are actually relevant.
Each model gets a budget of ten SEARCH or FETCH calls against
Semantic Scholar. It chooses the queries, reads the abstracts it fetches, and returns one
paragraph: a mechanism, a prediction, and a way to test it. Three critic models score that
paragraph on originality, feasibility, clarity, impact and specificity, each against the
retrieved evidence rather than against the critic's own memory.
The Static track is the control. The same model, the same subfield, the same critics, but the reading list is supplied rather than searched. The difference between the two is the only thing on this page that isolates agency from raw ability.
40 of 63 models score higher with the tool than without it. The rest do worse. Search is not free: a model that cannot judge what it retrieved is now wrong with citations. The paper states this on its open-weight roster alone, where 19 of 28 improve and the gain tracks static ability at r = +0.69.
Every number on this page is computed from data/results.db, table
lit8d_scores_3seed, by reports/_leaderboard_full.py. No value is
estimated, rounded up, or carried over from an earlier run.
| Column | Definition | Computation | Range / unit |
|---|---|---|---|
| # | Rank under the current sort and filter. | Position after sorting; recomputed on every filter change. | 1–63 |
| Model | The evaluated hypothesis generator. | Display name for the routed model id, shown on row expansion. | — |
| Vendor | Publishing organization, and whether weights are public. | Prefix of the model id. Closed rows are reported but excluded from the paper's headline statistics. | — |
| Active score | Weighted quality of hypotheses produced with agentic search. | Per hypothesis: drop the most generous of three critics, weight the five dimensions, normalize by 5.5. Mean over 3 hypotheses, then over 40 subfields. | 1–10 |
| Total ± | Standard error of the Active total. | Standard deviation across the 40 subfield means divided by √40. Not a bootstrap interval. | score points |
| vs. Static | Gain from agentic search over a supplied reading list. | Paired mean of Active − Static over subfields scored in both tracks. Blank where the model has no Static run. | score points |
| Turns | Mean tool calls issued per rollout, out of a budget of 10. | Counted from the logged agent transcript. Describes behavior, not quality. | 0–10 calls |
| Per-dimension | The five critic dimensions behind the total, Active track. | Same aggregation as the total, before weighting. The notch on each mini bar is the Static value for that dimension. | 1–10 |
The roster is the paper's complete evaluated roster: every model run in both tracks. Three models are excluded: gpt-5.3-chat is left out as a second serving route to a model already listed, not a second model; qwen3-235b-a22b-thinking-2507, qwen3-vl-8b-thinking ran the Active track only, so they have no Static column to rank against.
Critic scores are model judgments, not measurements of scientific merit, and the two models without a Static run are ranked on Active alone. The association between turns and score (Spearman ρ = +0.39, p = 0.001, n = 63) is an observation about behavior, not evidence that spending more calls causes a better hypothesis.