We are constantly bombarded with polls, surveys and studies on every topic imaginable.  Examples: "Percentage of Americans who attend Church on a regular basis..."; "Number of overweight Americans..."; "Percentage of Europeans supporting..."; "Favorite Bart chalkboard declaration on The Simpsons" (actual USA Today poll, 2/7/03).  We hope that you are skeptical in accepting the published numbers.  Who responded?  Why?  How many responded?  How were the questions phrased?  Who conducted the survey?  These are all questions that should come to mind.

The Problem with Surveys;
How Methods and Biases Contribute to Inaccuracies

Surveys usually provide flawed results due to the many sources of errors in sampling.  This occurs daily in reported statistics and happens even when those conducting the survey are not biased.  We hope that when you see survey results published on anything, you take the time to read the footnotes describing the survey sample and methods used.  Have you ever read a survey footnote like this?...  "Based on responses of 1347 of our readers."  Do you lower your head in disgust?  We've seen this in popular magazines, newspapers and TV programs.  Let's take a national newspaper as an example.  USA TODAY has an average daily circulation of of 2.3 million copies.  It has millions of online readers and millions more who subscribe to their e-mail newsletters.  For the sake of simplicity, let's ignore the millions of online readers and look just at readers of the print edition.  A mere 1347 respondents out of 2.3 million readers is 0.005%.  Clearly, that doesn't represent the broad opinion across the country, or even of USA TODAY readers.  STRIKE 1.  Then, look at the nature of the respondents.  They are people with strong opinions on the subject.  It doesn't matter what question the survey poses - only those with a strong opinion will be motivated to take the time to respond.  And, the "Don't Care" and "No Opinion" votes are never submitted.  STRIKE 2.   What about phone surveys?  Most people are too busy for them and hang up.  What is the profile of a person who remains on the line to answer the questions?  Maybe they have nothing better to do.  Maybe they are unemployed with lots of free time.  Maybe they have a personal interest in the survey (e.g. Someone on welfare in a poll of Democratic vs. Republican presidential candidates).  People answering polls often do not represent the population being studied... period.  STRIKE 3.  Then, look at how the question is phrased.  This has a great impact on the results.  For example, consider the question "Are you a good parent who reads to your children every night?"  Or, "Do you attend Church on a regular basis?"  Are you going to say you're not a 'good parent'?  What if you read to your children most nights, but not every night?  What if you attend Church often, just not when your favorite football team is playing or there isn't yard work to do?  Do you still answer that you go 'on a regular basis'?  What if you answer 'yes' even though you never go to Church because that is the 'morally correct' answer and you're wary of who may see your response?  A not so distant survey of public opinion about President Bill Clinton produced widely varying results with different wording.  Adding one phrase ("AS A PERSON") changed the results by 20 percent.  We hope you can see that the wording of the question can easily introduce errors.  STRIKE 4.

Survey Results Are Presented As Though They
Are Representative Of The Population...  But, Are They?

Are you tired of surveys that claim to accurately present broad public opinion?  NEWSWEEK boldly reported that Al Gore was comfortably ahead of George W. Bush just three weeks before the 2000 election.  The margin was 47% to 39%, favoring Gore.  What went wrong?!  Isn't Dubya our President?  For starters, the poll (conducted Sept. 7-8 by Princeton Survey Research Associates) only included 756 registered voters, with only 595 admitting they were likely to vote in the election.  Is this a broad view of the American public?  It appears that Democrats had more free time to answer the pollster's questions.  NEWSWEEK claimed a margin of error of +/- 4 percent.  Even with that margin of error, Gore should have won - but he didn't.

Flaws in surveys, as described above, occur even when there is no bias or 'expected' results.  However, many of the polling organizations have hidden agendas, especially when it comes to religious surveys.  For example, everyone's heard of the Gallup Poll, but few know Gallup was founded and operated by Christian evangelical George Gallup.  This is a widely accepted and quoted polling organization.  However, it is run by a man who openly admits:

"The most important purpose of polls is to explore people's response to God
and indicate ways to strengthen that response.  That is a form of ministry."
- George Gallup, interview by The Business Journal of New Jersey

Do you see the conflict of interest in Gallup polls?  Don't you think his goal of strengthening the religious response could affect his polling methods and results?  Is it any wonder that the Gallup International Millennium Survey reported that "almost two-thirds of global respondents say God is very important in their personal life"?!  Or that 47% of Americans attend Church at least once a week?  If half the entire population went to Church every week, most of them on Sunday, wouldn't we need bigger churches?  Wouldn't there be massive traffic jams?  Wouldn't the NFL switch their games to another day?

As you read polls on various topics, consider the source.  Why should you believe a survey on tobacco funded by the Philip Morris?  What is the value of a poll on abortion conducted or funded by the Catholic Church?  Should you believe a poll on the existence of alien life conducted by Conspiracy-Net.com?  Is it fact that people born in November (Scorpio) are more sexy because a survey says so on Astrologycom.com?  Does a poll on Creationism vs. Evolution based on program viewers of Pat Robertson, Jerry Falwell or Jim Bakker mean anything?.  STRIKE 5.

The next time you see a survey, first read the footnotes.  Look at the sample size and polling method.  Look at how the questions are phrased.  Look at who conducted and funded the survey.  Be skeptical!  And, by all means, if this background information is not provided, do yourself a favor: Don't bother reading it; just toss it in the trash...

"Survey Says!"