Evidlake
Menu

Confirmation Bias

Also known as: myside bias, positive test strategy.

Looking for, noticing and remembering evidence that supports what we already believe, and discounting what contradicts it.

Updated Last reviewed

Synthetic dataExamples on this page use synthetic data created for illustration.

Not adviceThis is educational material. It is not investment, business or other professional advice.

Definition

Confirmation bias is the tendency to look for, notice and remember evidence that supports what we already believe, and to discount evidence that contradicts it. It operates in what we search for, how we read results and which results we keep.

In plain terms

Peter Wason’s classic 1960 task showed it in miniature: asked to discover a hidden rule, people proposed examples that fit their favourite guess and rarely tried examples that could prove it wrong. Nickerson’s review of the literature found the same pattern in many guises.

In analysis it appears as choosing the source, window or metric that favours the hypothesis, stopping when the answer looks right, and filing contrary results as “noise”. Defences: write down, beforehand, what result would change your mind; deliberately look for the case that would refute you; ask someone to argue the opposite; and keep the counter-evidence in the record (Pre-registration).

Example

A simplified version of Wason’s task. The hidden rule is “any three numbers in increasing order”. The tester suspects “numbers going up by 2”.

TestFits the suspected rule?Fits the hidden rule?AnswerDoes it teach anything?
2, 4, 6YesYesYesLittle: confirms what was expected
8, 10, 12YesYesYesLittle
1, 2, 3NoYesYesA lot: the suspected rule is too narrow
6, 4, 2NoNoNoSome: decreasing fails

Testing only the first two kinds, the guess would “survive” for ever and still be wrong.

Common misunderstandings

  • “Only stubborn people do it.” It is a general feature of reasoning, found in careful people too.
  • “Intelligent people are immune.” Intelligence can supply better arguments for what one already believes.
  • “Testing examples that fit a hypothesis is always wrong.” It is useful, but it cannot by itself show the hypothesis is false.
  • Pre-registration

    Writing down, before seeing results, the question, the data and exactly how it will be analysed, and keeping that record unchanged.

  • Framing

    Choosing which aspects of a situation to emphasise so as to suggest a particular problem, cause or remedy. The same facts can be framed in different ways.

  • Hindsight Bias

    Once an outcome is known, believing it was predictable and that you knew it all along. It makes the past look more certain than it was.

References

  1. Raymond S. Nickerson. Confirmation Bias: A Ubiquitous Phenomenon in Many Guises . Review of General Psychology, 1998. DOI: 10.1037/1089-2680.2.2.175
  2. P. C. Wason. On the Failure to Eliminate Hypotheses in a Conceptual Task . Quarterly Journal of Experimental Psychology, 1960. DOI: 10.1080/17470216008416717