Most of the time, performing poor market research can be worse than not performing market research at all. Why? Because we often base our business decisions on the results of market research data, and flawed data can lead us down the wrong path.
Collecting a misrepresentative sample is one of the most common sources of such a problem, but luckily, these four tests can help you decide whether or not your sample was viable:
The Eyeball Test
The quickest and least scientific approach is to simply take a look at your data trends. For sample elements with a ranged score, such as asking someone to rate something from one to ten, the responses should be somewhat “normally distributed” according to Gaussian ideals. In other words, your responses rates when graphed out should resemble a bell curve. Bell curves are often weighted towards a preferred response, stretching or squashing it accordingly, but erratic shapes that do not resemble bell curves just may indicate a misrepresentative sample.
Alternatively, having explicit expected values can allow you to “eyeball” your responses in comparison. If you wanted a representative sample of the population of Houston, for instance, you could compare a graph of the typical demographics of the city to a scaled graph of your sample. You will likely see some natural deviations, but extremes are a good indicator of sampling issues.
Mean and Variance
Another simple test — often dubbed a “back of the envelope” calculation — is to compare your mean, medians and standard deviations. Under a perfectly normal distribution of ideal random samples, the sample data’s mean and median are the same. 68% of the data will be within one standard deviation of the mean, 95% within two standard deviations and 99% within three standard deviations.
Knowing expected values once again helps, especially among populations with skewed trends. For example, a population within Boca Raton, Florida will likely have older skewed age demographics compared to the rest of the country, so always consult your expected trends before scrutinizing data.
Market Research Sampling Size
One place to begin when considering sample viability is to ensure that you have enough participants in your sample to start with. The smaller the sample, the more likely minor trends will skew the data and major trends could go unobserved.
So how do you know if your sample size is too small. The needed sample size will change according to the size of the actual population as well as your intended confidence level and margin of error. You can use free sample size tools like this one from the Australian Bureau of Statistics to get a rough calculation. Remember to account for your expected response rate!
Consult a Market Research Agency
CFR, Inc. would love to answer your questions about sampling, your data and statistical tests you can use to determine viability. Simply drop us a line with your research question, and we will be more than happy to answer!
