Please be an educated consumer and know that there are two kinds of research: scientific and un-scientific. You cannot make generalizations from studies if they were not done scientifically! Statisticians have known this a long time - remember the headline “Dewey Beats Truman” in the 1948 presidential election!

Scientific research follows the rules of statistics and randomness and includes a detailed methodology statement explaining how these rules were followed. It is especially important when selecting a random sample or a stratified random sample of people to be surveyed. If these rules are followed, then you can use the survey results of a sample to accurately predict the behavior of all your association members or other organizations, i.e. you can generalize. The person conducting the scientific research will be able to tell you the level of confidence – for example, “you can predict within plus-or-minus 2.5 percentage points at the 95% level of confidence.” They can explain how many people you need to survey to achieve different levels of confidence and they can also explain non-response bias.

Unscientific research or open-ended research does not rely on these methods and should never be used to predict the behavior of all of your association members or of associations in general. This type of research includes such things as interviews, focus groups, and any survey of a sample of people not done scientifically. When reporting the results of these projects, be careful to say something like “The majority of the respondents to this survey say that . . .” You cannot generalize about all members or all associations. There should be a statement at the beginning of the final report saying the survey was not done scientifically and generalizations cannot be made.

In a nutshell, with a scientific survey you can predict behavior with precision. You can say that all members do this . . ., or most associations do that . . . 

With unscientific research you cannot predict behavior with precision. You cannot say that all members do this, or most associations do that.

Bottom line is you want your researcher to have taken at least one statistics course in college or grad school!

My company has done scientific surveys for over 100 national associations. And yes, we have advanced degrees in research! For more info call 703-772-5263 or e-mail AllegianceResearch@gmail.com