About Proven Effects
Proven Effects separates what is proven from what is merely promising. It continuously watches the world's research streams for evidence about interventions in the problem areas its members track, codes every study for methodological rigor and integrity, verifies its own work, and keeps living summaries of what works — with every claim one click from its source documents.
How a study becomes evidence — an interactive walkthrough
What this platform is for
Practitioners, researchers, and funders face the same problem: for any pressing issue there are hundreds of claimed solutions, and the loudest are rarely the best-evidenced. Proven Effects exists to answer three questions continuously, so you don't have to re-run a literature review every quarter:
- What actually works for this problem, by the weight of rigorous evidence — not publicity?
- What is coming — which promising interventions are one good trial away from proof, and which preprints should be watched (but not yet trusted)?
- What does the evidence really say — is that "proven" study preregistered, followed up, and honestly reported, or does it carry red flags?
How it works — the methodology
1 · Continuous monitoring
Crawlers monitor a diverse, international evidence stream — peer-reviewed publications (PubMed), registered clinical trials (ClinicalTrials.gov), working papers, reports, and gray literature in the Crossref index, preprint servers (Europe PMC), and news and press releases including government and agency reports (GDELT) — for evidence about interventions in problem areas our team defines, translating non-English studies where needed. Schedules are adaptive: daily while a problem is hot, relaxing to weekly, monthly, and quarterly as results dry up, with renewal required after 24 months.
2 · Rigor coding — two independent frameworks
Each relevant study is coded on the Maryland Scientific Methods Scale (SMS), the five-point rating of methodological rigor used across evidence-based policy:
| Level | Design |
|---|---|
| 5 | Random assignment to treatment and control (randomized controlled trial) |
| 4 | Controlled or matched comparison — strong quasi-experiment (matching, difference-in-differences, regression discontinuity, instrumental variables) |
| 3 | Treated vs. untreated comparison without demonstrated equivalence |
| 2 | Before-and-after comparison in the treated group only |
| 1 | Correlation between treatment and outcome; no comparison group |
Every study also carries a second, independent coding on Jerry Ratcliffe's evidence hierarchy (EH 1 anecdote … 4 quasi-experiment, 5 RCT, 6 systematic review/meta-analysis), and each treatment shows its status under both frameworks side by side. Under the hierarchy's logic a single positive level-6 synthesis is itself proof; where the frameworks disagree, the treatment is flagged so you can weigh the codings yourself. EH codings marked * were derived from the SMS for studies coded before the hierarchy was added.
Outcomes are coded positive / negative / mixed / null-effect / unclear, and each treatment's evidence score is the sum of (level × outcome weight) across its coded studies. Preliminary sources — preprints, working papers, registrations, press — are scored but visibly flagged PRELIMINARY / IN PROGRESS, and any status that depends on them carries a PRELIM. EVIDENCE caveat.
3 · Classification — the Convergent Evidence Standard
PROVEN — validated by at least three Level 4 or 5 studies with positive results EMERGING — one or two Level 4/5 studies with positive results NOVEL — proposed or promising, but not yet tested at Level 4 rigor
The headline classification is the Convergent Evidence Standard, which considers both frameworks together: PROVEN requires at least three positive studies rated strong by both frameworks (SMS 4–5 and EH 4–5), or a positive EH-6 synthesis corroborated by at least one such study; EMERGING is one or two both-strong positives, a positive synthesis alone, or three-plus positives strong under one framework only; everything else is NOVEL. Agreement between independent standards is a harder test than either alone — a convergent PROVEN is the platform's strongest claim.
4 · Integrity appraisal — beyond the nominal level
Not all studies are created equal, even at the same design level. Two randomized trials can both earn "SMS 5" and differ enormously in trustworthiness: one preregistered its primary endpoint, followed participants for two years, and reported a null result honestly; the other switched endpoints midstream, tested a dozen subgroups, and reported the one with p = .048. The nominal level cannot see that difference — so every study here also receives an integrity appraisal.
The instrument is the recognized standard in evidence synthesis, used by Cochrane reviews worldwide:
- RoB 2 — the revised Cochrane Risk of Bias tool for randomized trials (Sterne JAC et al., BMJ 2019;366:l4898; tool & guidance). Domains: the randomization process; deviations from intended interventions; missing outcome data; measurement of the outcome; selection of the reported result.
- ROBINS-I — Risk of Bias in Non-randomized Studies of Interventions (Sterne JAC et al., BMJ 2016;355:i4919). Domains: confounding; selection of participants; classification of interventions; deviations; missing data; measurement of outcomes; selection of the reported result.
Alongside the domain judgments, the appraiser checks a fixed catalog of red flags drawn from the questionable-research-practices literature (see Ioannidis, PLoS Med 2005): no preregistration or registry ID; a primary endpoint that is unclear or switched; a sample too small for the claimed effect; no or only short-term follow-up; many outcomes or subgroups without correction (p-hacking risk); effects claimed only in subgroups or secondary outcomes; p-values clustering just below .05; causal language beyond what the design supports; unaccounted attrition; undisclosed or conflicted funding; no blinding where feasible; surrogate endpoints. Strengths are credited the same way: preregistration, prespecified endpoints, intention-to-treat analysis, blinded assessment, 12-month-plus follow-up, adequate power, independent replication, and transparently reported null results.
Each study carries one of three overall judgments, in RoB 2's own vocabulary: INTEGRITY: LOW CONCERN INTEGRITY: SOME CONCERNS INTEGRITY: HIGH CONCERN — "high" when any domain is judged high-risk or three or more red flags accumulate; "low" only when no domain exceeds low and no substantive flags appear. The full domain judgments, flags, and strengths are on every study card in the repository.
Limits, honestly stated: appraisals are drafted by the AI from each study's abstract or summary — a screening judgment, not a full-text Cochrane assessment. Where the abstract is silent the domain is marked "no information" rather than penalized. Appraisals are generated in overnight batches, are provisional until a reviewer validates them, and can be disputed with the ⚑ report button like any other coding.
5 · Verification & learning
Machine codings are drafted by an AI classifier and are provisional until validated. Every 5th relevant study is sampled for QA: a blinded red-team agent independently re-codes it from the source material (never seeing the stored coding), an adjudicator compares them field by field, and disagreements queue for human element-by-element review. Clean passes auto-resolve — except every 4th, held so the auditor itself stays audited — and hidden known-wrong calibration chips measure reviewer vigilance. Validated corrections repair the records AND distill into lessons injected into every future coding prompt; users' ⚑ problem reports feed the same loop. The platform stops making the mistakes it is corrected on.
What you can use it for
- Decide — ground a program or funding choice in the ranked, appraised evidence for a problem, starting from its Problems page.
- Replicate — each proven treatment's What Works page distills common elements, replication guidance, and warnings, with the source documents one click away.
- Watch — follow treatments to be alerted when their status changes; daily digests summarize crawler activity.
- Contribute — upload studies for evaluation, report miscodings (validated reports correct the record and teach the classifier), and propose new problem areas to monitor.
- Audit — every number traces to source: repository → study → coding → appraisal → original document.
Access is by invitation — request to join from the landing page, or log in if you already have it.
The insights and data on this platform are developed with the assistance of AI. Humans are in the loop at critical points — validation, QA review, administration — but all users must verify observations against the original research before operational use. AI hallucinates.