A research project manager we know took a consumer economics class from a well-meaning but misguided professor. In his attempt to make research studies more accessible to students, he would be lenient to a fault. Even still, most of his teaching tendencies were not objectively harmful — except one.
“Make sure your research is able to answer your research question by finding enough people that your hypothesis relates to,” he said one day. “To do this, make sure your study can appeal to whomever it is you’re are trying to answer a research question about.” His examples revealed the folly: pretend you have a sports-themed survey if you want to get a large portion of male participants, and so on.
What makes this bit of advice so painful is not just its recommendation to cater to a specific bias, but rather the notion that a study can only yield high-quality data if it prioritizes a specific sampling population. This technique is known as “population specification,” and it can lead to a host of sampling errors, bias and other data collection issues.
Why Population Specification Can Be Bad for Field Research Data Collection
Population specification is actually a standard procedure in many forms of field research. Some studies may be trying to answer questions about opinions among university students exclusively, for example.
The problem arises when you are trying to describe a trend in a population relative to another when they are the only ones your field research focuses on. For instance, a study trying to determine how many university students ride the bus to class would naturally specify a population of university students. You are simply trying to measure trends and preferences among that population relative only to itself. However, a study that tries to determine whether university students prefer clothing within a certain price range must always include a population with which to compare them to.
Preferably, the study would be able to designate multiple demographics to show a range of responses, such as including graduate students, undergrads, post-grads, working professionals, high-school-only individuals and so on. By having a range, the study can make a relative assessment by saying, “Post-grads felt this way compared to university students.” Otherwise, you could be identifying something as unique when it is not. Polling just university students to see if they like ice cream would likely yield positive responses all around, but one could not conclude that “University students have a predisposition to liking ice cream” because so might the entire U.S. population!
Ensure that your study has a representative sample of your chosen population and not just a cherry-picked one. Oh, and definitely do not try to cater to one population with a study design since that can create an overwhelming response bias.
What's the worst advice you've heard? Let us know in the comments. Also be sure to check out some of our video on some of the other mistakes we've seen researchers make--and how to fix them.
