Grid v2.0.0
Methodological transparency
No scoring is perfect. Here are the 12 trade-offs we assume, documented and argued.
Version 1.0 — Last updated: 18/04/2026
Publi-Score is the quality score for scientific publications. Before confronting our methodology with the public — researchers, journalists, epidemiologists — we identified and documented its real limitations.
These 12 trade-offs are not flaws: they are deliberate design choices, assumed and argued. Whenever an empirical validation is planned, it is explicitly indicated.
The 12 assumed trade-offs
Click on a trade-off to read the objection and our response.
What we verify
7 empirical validation protocols to turn our hypotheses into data.
| # | Objective | Protocol |
|---|---|---|
| 1 | Reproducibility of AI scoring | 20 publications × 5 runs (temperature 0). Target: within-publication gap < 2 pts, correlation with a panel of 3 experts r > 0.85. |
| 2 | Calibration of the weights | Panel of 5 to 10 experts (epidemiologists, biostatisticians). Compare 3 grids (current, equal weights, expert weights). Spearman correlation of the rankings. |
| 3 | Gap between quick and full score | Comparison of the quick score (metadata only) with the full score (full text) on 52 publications scored in both modes (November 2026, grid v2.0.0). Result: the quick score is conservative. It is on average 9 points below the full score, and its tier is equal to or lower than the full-score tier for 51 publications out of 52 (98%). One publication in two drops a tier in quick mode; only one goes up. Mean gap 11 points, maximum gap 34 points. |
| 4 | Catalogue selection bias | Tier × conclusion distribution across the corpus. Target: at most 60% for any single conclusion, at least 2 publications per tier (A-E). Result (November 2026, 164 publications graded A to E): the most frequent conclusion is positive at 55.5%, and every tier holds at least 2 publications. Targets met; tier A, with 2 publications, sits at the minimum and will be re-measured after each new scoring round. |
| 5 | Robustness of min() vs alternatives | 12 test publications × 4 variants: min(A,B), mean(A,B), weighted(A×0.6, B×0.4), geometric(√AB). Result: a single tier change (C→B), exactly the expected whitewashing case (conflicts of interest masked by the prestige of a major journal). min() validated. |
| 6 | Sensitivity of AI scoring to the prompt | 3 prompt variants (concise, detailed, with examples) × 10 publications. Measure the variance between prompts. |
| 7 | Effect of the h-index on early-career researchers | Identify publications in the corpus by researchers with less than 5 years of career. Compare the score with and without the h-index (2 pts). If the gap exceeds 2 pts, consider a further reduction. |
To understand the scoring rules in their entirety:
See the full methodology