Why Your Sample Size Is Probably Too Small

Many studies on usability, online A/B testing, market research and countless other areas draw conclusions when, frankly, they should not be. Why not? Their sample size is too low.

A lack of research subject recruiting leaves them with a paltry number of people that make the conclusions unreliable. Small sample sizes skew data by making one-time or limited occurrences seem more common than they actually are. Similarly, relatively common occurrences may not show up at all during the study.

The numbers behind this phenomenon are kind of complicated, but often a small sample size in a study can cause results that are almost as bad, if not worse, than not running a study at all.

Usability Misconceptions

• Many studies ignore the “confidence rate,” or how likely the statistical phenomena observed are to be true
• For studies that intend to find out the failure rate of a particular product, packaging or website with strict pass/fail requirements, sample size becomes even more important
• When looking at pass/fail tests that observe a “0” failure rate of failure and want 95% confidence:

○      A study of 30 (n=30) that finds zero failures indicates that the failure rate is still around 11.6%

○      With n=100, the failure rate is still 3.6%

○      When n=1000, the true failure rate is 0.36%

The Bare Minimum

• Many studies should want to assert:

○      A confidence rate of 95%

○      A small margin of error of +/- 5%

○      A moderate standard of deviation of 0.5, where responses are generally not split dramatically from one another (think of a standard bell curve)

• To satisfy all these requirements, this study will need at least 385 people
• For studies that want to make demographic/segment observations under these conditions, each representative segment needs to have at least 385 people, and stratified sampling should be used to obtain equitable results

Despite these statistical assertions, many studies think that 100 or even 30 people is an acceptable number. Studies that want to draw segment conclusions — e.g. “people from Milwaukee liked our product better than those in Chicago” — will need a much higher number than they expect of each demographic.

If finding this many people and keeping the sample size relatively random seems daunting, remember that companies like CFR Inc. have your back. We help studies recruit people from all over the country in enormously varied segment groups using reliable, eager subjects.

Visit our research subject recruiting page to find out more about we can help you.