Scoring modes
How Publi-Score calculates the score depending on available data. Empirical data on 18 articles.
Comparison table
| ⚡ Quick | 🔬 Partial AI (abstract) | 📝 Full manual | 🤖 Full AI (PDF) | |
|---|---|---|---|---|
| Input | PMID/DOI | PMID only | PMID + PDF | PMID + PDF |
| Criteria coverage | ~53/100 | ~85/100 | 100/100 | 100/100 |
| Integrity coverage | ~80% | ~80% | 100% | 100% |
| Duration | ~2–5 sec | ~30–60 sec | ~10–30 min | ~30–60 sec |
| Objectivity | ✅ Auto | ✅ LLM | ⚠️ Human bias | ✅ LLM |
| Published in catalogue | ✅ | ✅ | ❌ | ✅ |
| Account required | No | Yes (free) | No | Yes (free) |
| Quota | Unlimited | Shared with Full AI (same quota) | Unlimited | 5/month (free) |
What the quick mode covers (and doesn't cover)
✓ What it covers
- • §2.3 Bibliometric impact — 100% (citations, h-index)
- • §2.6 Freshness — ~69% (publication date)
- • §2.1 Level of evidence — ~57% (study type, randomisation)
- • Retractions — 100% (PubMed API + Retraction Watch)
- • Alert signals — 100% (predatory journals, EoC)
Total: ~53% of criteria · ~80% integrity
⚠ What quick mode doesn't cover
- • Real ITT analysis (intention to treat)
- • Compliance with pre-registered protocol
- • Raw data sharing
- • Clinical benefit/risk ratio
- • §2.7 Reporting quality — 0% (requires PDF)
~47% of criteria not evaluable without PDF
The Partial AI mode without PDF
Partial AI (abstract) mode uses only the abstract and article metadata — without PDF. AI evaluates all criteria accessible from these sources, covering ~85% of the total score.
~85%
criteria covered
+3 pts
∆ average vs. full with PDF: 3 pts (measured on 18 corpus articles)
1/12
Only 1 tier change in 12 articles (TOGETHER: B→A)
0/3 pts
criterion not evaluable without PDF
Why ~85% and not 100%?
- • §2.7 Reporting quality (3 pts) — evaluates the clarity of results, tables and figures. Inaccessible without the full PDF: always 0 pt.
- • §2.4 Reproducibility & transparency — some sub-criteria (code sharing, raw data) are partially inferable from the abstract, but without certainty. AI scores them conservatively.
This mode produces a detailed score with per-criterion justifications — it is partial, not degraded. It remains significantly more reliable than quick mode (~53%) and activates automatically when the PDF is not open access.
Activation: automatic fallback if OA PDF unavailable. Measured on 18 articles, COVID-19 and Vaccination clusters — Publi-Score calibration corpus.
Empirical data
Measured on 18 articles, COVID-19 and Vaccination clusters — Publi-Score calibration corpus.
4 critical overestimation cases in quick mode
These 4 articles are among the most viewed in the corpus. Quick mode assigns them tier A or B, while Full AI mode reveals tier D.
| Article | ⚡ Quick | 🔬 Partial AI (abstract) | 🤖 Full AI (PDF) | Gap Q→F | Main reason |
|---|---|---|---|---|---|
| Polack/Pfizer — NEJM 2020 | A | E | D | −51 pts | Major industrial COI + short editorial delay not captured |
| Voysey/AZ — Lancet 2021 | B | D | D | −38 pts | AstraZeneca COI + adaptive design + data not shared |
| Hammond/Paxlovid — NEJM 2022 | B | E | D | −37 pts | Industry-only trial + raw data unavailable |
| Molnupiravir — NEJM 2022 | B | D | D | −33 pts | Merck/Ridgeback trial — non-public data |
Why quick mode overestimates: it normalises on criteria accessible via APIs. §2.3 (bibliometric impact) weighs ~14 pts and is maximal for NEJM/Lancet articles — which are often industrial trials (Pfizer, AZ, Merck) whose strong COI is only visible on in-depth analysis.
Which mode to choose?
| Context | Recommended mode |
|---|---|
| First exploration, monitoring | ⚡ Quick |
| Clinical decision, citation, teaching | 🤖 Full AI (PDF) |
| PDF unavailable (NEJM, Lancet…) | 🔬 Partial AI (abstract) |
| Personal learning, methodological exploration | 📝 Full manual |