A question we hear often: “Is the quick score reliable?”
The honest answer: it depends. Reliable for what purpose? What decision is at the end of the chain?
This article explains exactly what the quick score covers, what it doesn't, and — crucially — in which contexts the gap between the two modes can be decisive.
What the quick score does in 5 seconds
When you enter a PMID or DOI into Publi-Score, the quick score is calculated automatically by querying public APIs: PubMed, Crossref, OpenAlex, Semantic Scholar, ClinicalTrials.gov, PROSPERO.
In 5 seconds, it knows:
- The declared study type (RCT, meta-analysis, cohort, case series…)
- The declared sample size
- Whether the study was pre-registered on ClinicalTrials.gov or PROSPERO
- The submission-to-acceptance delay (peer review signal) via Crossref
- Bibliometric impact: journal impact factor, citations, author h-index
- Open access status and available version
- Recency of the study
- Integrity signals: retraction, predatory journal, erratum, expression of concern
What the quick score cannot do: read the PDF. It only has access to metadata and the abstract.
The 47 missing points — and why they matter
The quick score covers ~53 points out of 100 evaluable criteria. The remaining ~47 require the PDF.
§2.1 Level of evidence — ~12 missing points
The declared study type is not always the actualstudy type. An article may declare itself an “RCT” in PubMed metadata while the PDF reveals sham randomization. In quick mode, the article is scored on what it claims to be, not what it is.
§2.2 Methodological quality — ~10 missing points
A priori power calculation, effective intention-to-treat analysis, missing data handling — these criteria can only be evaluated by reading the Methods section. An article may declare “intention-to-treat analysis” in its abstract while excluding patients on page 7 of the PDF.
§2.4 Reproducibility & transparency — ~14 missing points
The most impacted section. Pre-registration existence is detectable via APIs. But compliance with the pre-registered plan — does the article do what it said it would do? are primary outcomes the same? is there outcome switching? — is invisible without the PDF.
§2.5 Clinical relevance — ~7 missing points
NNT, NNH, absolute vs relative differences, external applicability: these require reading the results and discussion. The abstract often summarizes them using the most favorable metrics.
The integrity coefficient: where the quick score already excels
The integrity coefficient is covered at ~80% in quick mode. This is its strength.
✓ Automatically detected
- PubMed retraction → immediate X score
- Expression of Concern
- Verified predatory journal
- Submission-to-acceptance < 10 days
⚠ Requires PDF
- COI unverifiable without the PDF (declared or not in the article body)
- Ghostwriting
- Internally inconsistent data
Two situations where the gap is decisive
Situation 1 — The article looked good, but wasn't
A vitamin D supplementation trial, IF 8 journal, N = 2,400, pre-registered. Quick score: C (62/100). The PDF reveals undisclosed outcome switching (primary endpoint changed after enrollment). Full AI score: E (31/100). Gap: 31 points.
Situation 2 — The article looked mediocre, but was rigorous
A cognitive bias study, pre-registered, published in a low-impact journal. Quick score: D (44/100) — penalized by low IF. The PDF reveals: rigorous ITT analysis, data on OSF, full CONSORT compliance. Full AI score: B (79/100). Gap: 35 points in the other direction.
When to use quick vs full
| Context | Mode |
|---|---|
| Sorting 20 articles, topic monitoring | ⚡ Quick |
| Clinical decision — adapting practice | 🤖 Full AI (PDF) |
| Citation in a paper, teaching | 🤖 Full AI (PDF) |
| Fact-checking a media claim | ⚡ then 🤖 if suspicious |
| Suspected retracted article | ⚡ Quick (retraction 100%) |
Summary
| ⚡ Quick score | 🤖 Full AI (PDF) score | |
|---|---|---|
| Coverage | ~53% of criteria | 100% |
| Retractions | ✅ 100% | ✅ 100% |
| Outcome switching | ❌ Invisible | ✅ Visible |
| Time | ~5 seconds | ~45 seconds |
| Quota | Unlimited | 5/month (free) |
| Best for | Sorting, monitoring | Decision, citation |
The quick score is a powerful first-pass tool. The full score is a decision tool. They don't replace each other — they complement each other. Every quick score in the catalogue is accompanied by a “Metadata only” badge and a visible completeness bar.
Test both modes on an article →
Full methodology documentation: publi-score.ai/methodology