You paste a PMID into Publi-Score. The score is calculated. But looking at the mode badge, you see: 🔬 Partial AI (abstract).
The article is in the New England Journal of Medicine or the Lancet — behind a paywall. Publi-Score did not have access to the PDF. The AI worked solely on the abstract and public metadata.
Legitimate question: is this score worth anything?
Short answer: yes — with a precise, documented, and visible limitation. This article explains exactly what the AI can and cannot evaluate without the PDF.
What the AI reads without the PDF
Even without access to the PDF, Publi-Score queries several public sources before calling the AI:
- PubMed — full abstract, MeSH terms, retraction status, declared study type
- Crossref — DOI, submission and acceptance dates, licence
- OpenAlex — citations, impact factor, quartile, authors' h-index
- ClinicalTrials.gov / PROSPERO — existence and date of pre-registration
- Semantic Scholar — open access PDF available or not, influential citations
- Retraction Watch — retraction signal independent of PubMed
The AI (Claude Haiku for extraction, Claude Sonnet for scoring) receives the full abstract and all this metadata. It evaluates each criterion in the grid with what it has — explicitly noting what it cannot conclude.
24 of 26 subcriteria evaluated
Out of the 26 subcriteria in the Publi-Score grid, Partial AI mode evaluates 24. A structured abstract contains far more information than it appears — methodology, effect size, confidence intervals, design, authors' conclusions — and AI Sonnet extracts these elements with per-subcriterion justifications.
✓ §2.1 Level of evidence — ~90% covered
Study type, randomisation, presence of a control group, sample size — these elements are often in the abstract or retrievable via ClinicalTrials.gov.
✓ §2.3 Bibliometric impact — 100% covered
Citations, impact factor, authors' h-index — fully retrieved via public APIs. No PDF needed.
✓ §2.6 Freshness & currency — ~95% covered
Publication date, submission-to-acceptance delay — available via Crossref.
✓ §2.2 Methodological quality — covered
Statistical analysis, bias management, ITT, double-blinding, power calculation — AI Sonnet extracts these elements from the structured abstract with justifications, staying conservative when information is missing.
✓ §2.5 Clinical relevance — covered
Benefit/risk ratio, practical recommendations, adverse effects — extracted from the conclusion and summarised discussion.
✓ §2.7 Reporting quality — covered
Compliance with reporting guidelines (CONSORT, PRISMA…) and spin detection — inferable from the structure and language of the abstract.
The 2 subcriteria only the PDF reveals
Two subcriteria remain impossible to evaluate without the PDF — they are factual elements that are never in the abstract:
§2.4 Raw data sharing
Data sharing requires examining the "Data Availability" section of the PDF and verifying the existence of the referenced repository (Zenodo, Dryad, OSF…). The abstract almost never mentions this information.
§2.4 Code sharing
Same for analysis source code — GitHub/GitLab availability, licence, version used. Factual information not present in the abstract.
These 2 subcriteria stay at zero in Partial AI mode, with no penalty for the article — it is an absence of information, not a failure.
60% Fidelity — real data on representative corpus
Our reference corpus comprises 50 thematic articles (COVID, Mental Health, Nutrition, Cardio, Onco, Infectious, Meta-epistemology, Genetics — balanced A/B/C/D/E tier distribution reflecting ordinary medical science). 28 of them have a measurable Full AI (PDF) reference — the other 22 are paywalled articles without PMC deposit or with too short an abstract. On this paired sub-corpus (n=28), we compared each Partial AI score to the Full AI score of the same article. Measurement:
12.8 pts
average absolute ∆ between Partial AI and Full AI
12/28
articles at the exact tier (A/B/C/D/E identical)
24/28
articles within one tier of difference
How to read: the Partial AI score gives the exact tier in 43% of cases, the close tier (±1) in 86% of cases. Concordance = average of both tiers = (43% + 86%) / 2 = 65%.Coverage = 24/26 evaluable subcriteria = 92%.Fidelity = concordance × coverage = 65% × 92% = 60%.
Example — Moncrieff et al., Molecular Psychiatry 2022
This umbrella review challenging the serotonin hypothesis of depression scores:
- Full AI mode (PDF): 47/100 → tier C
- Partial AI mode (abstract): 35/100 → tier D
One tier gap (12 pts) — typical for umbrella reviews where the abstract summarises dozens of meta-analyses without methodological details. Partial mode remains very useful to guide reading (close tier) — for clinical decision or precise citation, the PDF remains the reference.
Measurement of 2026-04-28, distribution corpus n=50, paired sub-corpus n=28. Average signed Δ = +11.8 pts, average absolute Δ = 12.8 pts.
Partial AI ≠ degraded score
The confusion comes from the word "partial". It describes the coverage of the analysis, not the quality of the score.
In Partial AI mode, the AI produces detailed justifications per criterion, exactly as in Full AI mode: for each sub-criterion, it explains why it awarded a given score, citing the elements from the abstract or metadata that guided its decision. This is not an approximate overall rating — it is a structured analysis with full traceability.
Comparison of the two AI modes
| 🔬 Partial AI (abstract) | 🤖 Full AI (PDF) | |
|---|---|---|
| Input | Abstract + metadata | Full PDF + metadata |
| Subcriteria coverage | 92% (24/26) | 100% (26/26) |
| Fidelity (concordance × coverage) | 60% (65% × 24/26) | 100% (reference) |
| Per-criterion justifications | ✅ Yes | ✅ Yes |
| §2.4 Data/code sharing | 2 subcriteria NA | Evaluated |
| Activation | Automatic if PDF unavailable | Open access PDF required |
| Time | ~30–60 sec | ~30–60 sec |
| Published in catalogue | ✅ Yes | ✅ Yes |
When to use Partial AI mode?
This mode activates automatically when Publi-Score does not find an open access PDF. You have nothing to choose — but you can interpret it:
- Close tier in 86% of cases: the result is reliable to guide reading or bibliographic triage.
- If you have access to the PDF through your institution: you can upload the PDF to switch to Full AI mode — the score will be recalculated on 100% of criteria.
- For a clinical decision or citation: prefer Full AI mode if the PDF is accessible. The average gap of 12.8 pts is to be interpreted case by case, and raw data/code sharing (§2.4) is only verifiable with the PDF.
Partial AI mode is a complete score on what is accessible — not an incomplete score on the article. The limitation is explicit, documented, and visible in every criterion justification. That transparency is what distinguishes it from an opaque rating.
Analyse an article (with or without PDF) →
Understand all scoring modes: publi-score.ai/methodology/modes-scoring