Evidlake
Menu

Hierarchy of Evidence

Also known as: levels of evidence, evidence ladder, GRADE.

A ranking of kinds of evidence by how well their design guards against error: reviews and randomised experiments above observational studies, anecdotes at the bottom.

Updated Last reviewed

Synthetic dataExamples on this page use synthetic data created for illustration.

Not adviceThis is educational material. It is not investment, business or other professional advice.

Definition

A hierarchy of evidence ranks kinds of evidence by how well their design guards against error, with systematic reviews and randomised experiments above observational studies, and expert opinion or anecdote at the bottom. It is a guide to caution, not a verdict on any single study.

In plain terms

The idea comes from medicine. The GRADE approach starts randomised trials high and observational studies low, then moves the rating down for risk of bias, inconsistent results, indirectness, imprecision or publication bias, and up for a very large effect. The same logic carries over to other fields: a replicated controlled experiment is stronger than a single one, which is stronger than an observational comparison, which is stronger than a dashboard of descriptive numbers, which is stronger than opinion.

It is a ladder, not a law. A careful observational study can be better than a flawed experiment. What the ladder gives is a first question: how could this kind of evidence mislead us?

Example

The claim: “Product pages with a video sell more.”

EvidenceRungHow it could mislead
A randomised test: video added to a random half of 40 pagesExperimentSmall sample; short period
Comparison of all pages with and without videoObservationalPages with video may already be the popular ones (Confounding)
A dashboard showing average sales for each groupDescriptiveShows a difference, says nothing about why
A seller says “video works”OpinionExperience of one seller, probably no comparison

The first deserves most weight, the last least; none of them is worthless.

Common misunderstandings

  • “A higher rung is always right.” A well-run lower study can beat a badly run higher one; the ladder is a starting point.
  • “Evidence levels are only for medicine.” Any field with causal claims can use the same questions.
  • “Expert opinion is worthless.” It is weak evidence for causal claims, and valuable for forming hypotheses and judging plausibility.
  • Confounding

    A third factor that influences both the supposed cause and the outcome, creating or hiding an association between them.

  • Corroboration

    Confirming a claim with evidence independent of the original source. Reports that trace back to the same origin count as one.

  • Reproducibility

    A result is reproducible when someone else, with the same data and code, gets the same result. It needs the data version, code, settings and environment to be recorded.

References

  1. Gordon H. Guyatt et al.. GRADE: An Emerging Consensus on Rating Quality of Evidence and Strength of Recommendations . BMJ, 2008. DOI: 10.1136/bmj.39489.470347.AD
  2. Patricia B. Burns, Rod J. Rohrich, Kevin C. Chung. The Levels of Evidence and Their Role in Evidence-Based Medicine . Plastic and Reconstructive Surgery, 2011. DOI: 10.1097/PRS.0b013e318219c171