Data analysis is critical for decision making, but if the numbers are misinterpreted, decisions based on those conclusions could lead companies or individuals astray. That’s why it’s important to understand basic statistics, survey design and common ways data interpretations go wrong — regardless of your role.

I’m a marketing director for an events company headquartered in London. Last week, a colleague announced a “finding” from a new piece of data analysis done by the head of research. She said that our customers want to meet people from the same sector at an event, not those from other sectors. For 10 years, the post-show surveys have revealed the majority of attendees want to meet a group we call VIPs. This new finding could change the way events are organized: reorganizing the exhibition floor into sector clusters or cutting funding for VIPs.

Looking at the analysis, the problem jumped off the page. The analyst had drawn conclusions from very small sector groups and presented them with equal weight to the preferences of the very large groups — we’re looking at a difference of 15-44 people in a sector to thousands of people in a sector. Once analysis for the sector samples with more than 1,000 people and the VIPs were shown, the results were consistent with those of the past.

How could that happen? It’s much more common than you might think. In a 2022 study of 3,914 participants, researchers from the UNSW Business School in Australia and The Hong Kong Polytechnic University found that people ignore the sample size in their judgments and decisions and tend to be unduly confident in conclusions drawn from studies with as few as three participants.

“It is especially appalling to think many important businesses and public policy decisions might have been made based on unreliable results from small samples,” Dr. Siran Zhan, co-author of the study, wrote on the university blog.

Sample size, and particularly relative sample size as described above, is important. If you’re reviewing the results of a customer survey, keep that in mind. Smaller sample sizes almost guarantee alarmingly significant results because the impact of an outlier (a number much smaller or much larger than the mean of the other responses) will be bigger and skew the results. The closer the sample size is to your population, the more reliable the results will be. You should see the absolute number for each group, not just a percentage of the total, along with a numerical total.

Let’s look at more common ways data can mislead and how to determine when data is good enough for decision making.

Finding the survey or source data is always a good idea because misleading statistics are everywhere — in the news, ad campaigns and even scientific literature. For internal data, be sure that all results are included. A common pitfall in processing data is “cherry picking,” which means including only the results that support the desired conclusion.

Data presentations can also be rife with misleading information. Charts, graphs and other data visualizations can make results easier to understand, but only if they’re presented with accuracy. Look for the omission of the baseline or an axis that starts above zero. Intervals can be chosen to make differences look far greater than they are — is the y-axis marked at every counting number? Are the intervals uneven? When you see a visualization that looks different than most others you’ve encountered, look again to understand why. If it doesn’t make sense, it may be misleading.

If you feel like you need to brush up on your knowledge of statistics, you can take a free online course from the Massachusetts Institute of Technology (MIT): Fundamentals of Statistics. It starts Aug. 26 and runs through Dec. 17. Go to edx.org to enroll, along with 160,917 others — now that’s a number I’m happy to see.

Leslie Meredith has been writing about technology for more than a decade. As a mom of four, value, usefulness, and online safety take priority. Have a question? Email Leslie at asklesliemeredith@gmail.com.

Newsletter