Correlation vs. Causation in Practice: Four Questions Before You Believe a Pattern

A dashboard shows that product pages with a video sell more than pages without one. The conclusion is tempting: add videos, sell more. But this is a correlation, a fact about how two things moved together in the past, and the conclusion is a causal claim about what would happen if you intervened. The gap between the two is where money and effort get wasted. This article gives four questions that close the gap, or at least show how wide it is.

The claim

Suppose 200 pages have a video and average 48.8 sales a week; 800 pages have none and average 17.8. The ratio is 2.7. Is that the effect of video?

Question 1: Could it run the other way?

Maybe sales lead to videos, not videos to sales. Sellers whose products are already selling have more money and more reason to invest in better pages. Direction is the first thing to check, and timing is the first tool: did the sales rise before or after the video was added? A comparison of the same pages before and after is far more informative than a comparison of different pages.

Reverse causation is common in business data: products with more reviews sell more (or products that sell more collect more reviews); pages with more traffic rank higher (or rank higher because of traffic).

Question 2: Is a third factor behind both?

A common cause (Confounding) can produce a perfect association with no causal link at all. Here the third factor is likely to be how popular the product already was. Suppose 200 of the 1,000 products are popular. Sellers added a video to 150 of them and to only 50 of the 800 others. If a popular product sells 60 a week and an unpopular one 15, whether or not it has a video:

Within each column the video makes no difference at all (60 vs 60, 15 vs 15). Yet the overall averages differ by a factor of 2.7, purely because videos were given mostly to the popular products. The popularity drives both the videos and the sales.

The standard remedy is to compare like with like: look at popular products with and without video, and at others with and without. This works for factors you have measured. It cannot help with factors you have not (the seller’s skill, marketing, a product’s novelty), which is why design beats adjustment.

Question 3: Could it be chance?

With enough comparisons, some will look impressive by luck. Test 20 unrelated features against sales at the usual 5% level and the chance that at least one looks “significant” is about 64% (Multiple Comparisons Problem). If an analyst tried many features, windows and groups before finding this one, its “significance” means little. Ask how many things were tried, and whether the pattern appears again in data that were not used to find it (Holdout Set).

Small groups are especially prone to this. An average of 4.6 stars from 12 ratings can easily be a fluke; the same average from 3,400 ratings is not (Confidence Interval).

Question 4: Who is in the data?

The products you can see are not a random sample (Selection Bias). If your data come from a best-seller list, you have only the winners; the products that tried videos and failed to sell are not on the list (Survivorship Bias). Any comparison made inside a selected group can be distorted by the way the group was selected.

What would convince you?

After the four questions, the useful move is to ask what evidence would settle it. In rough order of strength (Hierarchy of Evidence):

1. A randomised test. Add a video to a random half of comparable pages and leave the rest alone. Randomisation breaks the link between popularity and video, so a difference in sales afterwards is the effect of the video.
2. A natural experiment. Something changed for reasons unrelated to the outcome: for example, the platform enabled videos for some product groups first. Compare the groups on either side of the change.
3. Before-and-after on the same pages, with a comparison group that did not change, to separate the video from the season.
4. Adjusted observational comparison, stating plainly which factors were and were not measured.

Each step down the list is cheaper and more open to doubt.

A way to write it up

Instead of “videos increase sales by 170%”, write what the evidence supports. For example: “Pages with video sold 2.7 times as much, but videos were added mostly to products that already sold well. Within products of similar popularity, the difference was 0. We have not tested whether adding video to a page changes its sales.”

That sentence is longer, and it will not mislead anyone into a costly mistake. It also tells the reader what experiment to run next, which is how a correlation stops being a belief and becomes a question that can be answered.

The short version

• Direction: which came first, and can you check with timing?
• Third factor: what else could drive both? Compare like with like.
• Chance: how many things were tried? Does it hold on new data?
• Selection: who is in the data, and how did they get there?
• Then: what experiment would settle it? Say what you have and have not shown.