Chapter 6: Fair Response Choices – Equal Weight and Coherence
By Alan F. Kay, PhD
© 2004, (fair use with attribution and copy to author)
July 13, 2004
In Chapter 2, we saw how complex and unusable the full response could be to a single either-or question of the type “Favor A or B?” A very wise, hypothetical respondent was willing to explain to the interviewer under precisely which conditions s/he would favor each of the four choices: only A, only B, both, and neither. One very clear conclusion, confirmed again in is that if “neither” and “both” are not offered as acceptable answers, the media poll report could be misleading and spinful, i.e., sinful.
In Chapter 5, we saw how even more unfair is the choiceless choice question, “Favor A? yes or no?”, and how you can spot the more subtle spin of the sponsor-favored choice A, beefed up to be stronger and more compelling than alternatives B, C, etc. We explained the method by which poll designers can be almost certain that the choices offered, A, B, C, etc., are fairly presented, with minimal imbalance and minimal bias. All of the different techniques for keeping the various multiple choices offered in a survey question as fair, unbiased, and balanced as possible, provide choices of what is summarized with the phrase “equal weight”.
Combinations of Two Statements/Items in a Response Choice
So far, we have considered choices A and B that are each a single item or a single statement. However, A and B each may be two (or more) different and, at least somewhat, unrelated statements and items. A question from a Hart-Teeter survey, conducted Feb. 20-24, 1997, for the Council on Excellence in Government, is a good example.
First, who or what is Hart-Teeter? Democrat Peter Hart and Republican Bob Teeter are a prominent polling team, whose poll findings often appear in the Wall Street Journal and elsewhere. Teeter was the Republican pollster for the ATI polling team in its early days, until he resigned to become a top advisor to the successful campaign of George H.W. Bush for president in 1988. Here was their question.
“Please tell me which one of the following statements about people’s attitudes toward the federal government comes closer to your own point of view.
(A) It’s a good thing for America that many people are skeptical about the federal government, and we need to guard against government getting too big and powerful.
Or
(B) It’s a bad thing for America that many people are skeptical about the federal government, and we need to appreciate what government does and try to improve it.”
Which would you choose, A or B? How about if we rematched the two parts of each substantive choice this way:
(A’) It’s a good thing for America that many people are skeptical about the federal government, and we need to appreciate what government does and try to improve it.
(B’) It’s a bad thing for America that many people are skeptical about the federal government, and we need to guard against government getting too big and powerful.
It happens that chapters 12 and 13 of Locating Consensus for Democracy get to the bottom of why people mistrust the federal government, and why the two political parties and most of Washington do not want to understand the reasons for the public’s mistrust. Although the full explanation, examined in the next chapter, is a little more complex, reduced to two alternatives (A”) and (B”), these reasons are as follows:
(A”) People mistrust the federal government because the government is getting too big and powerful.
(B”) People mistrust the federal government because it favors special interests and does not operate in the interests of all the people.
Extensive research showed that the overwhelming majority of the public (>75%) agree with B,” a smaller majority agree with A” (implying over 25% agree with both.) Republicans and conservatives lean more to A” than others do.
Did Peter Hart, Bob Teeter or perhaps the poll sponsors think that, with their A and B version of this question, good Democrats would agree with A and good Republicans agree with B? ATI results showed that many people, including Republicans and Democrats, would agree as much with A’ as with A and agree as much with B’ as with B, so that conjecture does not stand up. In my opinion if this question were tested with any choices other than A” and B”, the findings would be junk.
Cleaning Up Bad Either-Or Questions
You might imagine that either-or questions begging for “both” or “neither” responses are rare. They are not. They are very common. Here is a group of them that appeared in a very high-profile poll that was given to a random sample of people both before and after they attended the National Issues Convention in Austin, Texas, in January 1996. Some of these also appeared in a Washington Post poll shortly thereafter:.
Here are some things people think are important. Tell me which one of the two statements comes closer to your feelings on the subject. [Questions are numbered as in the Convention Report]
- Some people think that America’s interests should always come first.
Others think that people should consider the interests of people in other countries too. - Some people think that we should all respect authority.
Others think that we should all do what we think best, regardless of authority. - Some people think that people should be able to do whatever they want with their property.
Others think that sometimes the community should be able to tell people what they can and can’t do with their property. - Some people think that no one should be too rich or too poor.
Others think that everyone should be able to earn and keep as much or as little as he or she can. - Some people think that people should make more sacrifices for the future.
Others think that people should just do the best they can in the present. - Some people think that the most important thing in foreign affairs is to get other countries to do what we want.
Others think that the most important thing in foreign affairs is to act morally.
| S3. | Some people say that there has been a breakdown in the traditional family. Others say that families are just finding new forms. |
| S4. | Some people think that the biggest problem for the American family is economic pressure. Others think that it is the breakdown of traditional American values. |
| S5. | Some people think that government has become too involved with the family through social welfare and family planning programs. Others think that government has not done enough for the family through child care, family planning and education. |
In all these questions, if “both,” “neither,” and “it depends” were read by the interviewers as offered choices along with the question actually asked — “Which do you agree with more?” — these additional new choices, collectively, would generally score better than the two, together, actually offered.
Finally, offering a wide range of response choices in a single question (called a battery in Chapter 4) is a good way to clean up a mess like this.
Combinations of Multiple Statements/Items in One Response Choice
Everything heretofore said to the contrary not withstanding, there can be legitimate reasons for either-or questions. We illustrate this point with Q8 and Q9 of ATI-21, April 1993. The DKs are 2% and 3%, low enough to indicate that the respondents had no problem understanding these rather lengthy response choices.
Q8. Now, I would like to read you two statements to see which is best.
[After reading statements ask] Please tell me which is closer to the way you think about things]First, President Bill Clinton supports the idea of maintaining a strong and mobile military, but he has proposed cutting defense spending by $126 billion over five years, which amounts to an 8% cut averaged over the next five years. The president would use these savings to reduce the budget deficit and redirect funds to address America’s economic needs
38% prefer.
OR
Second, Gen. Colin Powell has proposed cuts in defense spending that are less than Clinton’s. Powell has proposed cuts of $60 billion over five years, a 4% average cut per year. Powell warns against larger cuts because he thinks they would limit America’s ability to respond to crises abroad.
60% prefer.
Q9. Now, I would like to read you two other statements. Please tell me which statement you agree with more. . . . .
Statement 1. Some people say the United States should not rush major cuts in defense spending just because the Cold War is over. Many foreign leaders are more unpredictable and dangerous than ever before. The United States needs to maintain its current military strength to deal with the danger of international terrorism, drug trafficking, and weapons of mass destruction falling in the hands of third world dictators. And major cuts in defense spending will seriously damage large manufacturers and cost millions of industrial jobs at a time when the economy is in bad shape.
56% agree
Statement 2. Other people say the end of the Cold War offers a unique opportunity to cut defense spending. Half of our defense budget went to defend against the Soviet threat, which no longer exists, and we no longer need the conventional forces that were used to defend Europe and Japan. We can use these savings to rebuild our economy and convert defense industries to civilian industries, including worker training and unemployment compensation for workers laid off by defense contractors.
41% agree
Question Q8 is comprised of fairly exact, short descriptions of views on military policy, one by then-President Bill Clinton, widely discredited on military matters as lacking experience, and the second by Gen. Colin Powell, then chairman of the Joint Chiefs of Staff, and, following the first Gulf war, the most respected U.S. general since Dwight Eisenhower. The question is asked as an either-or question, “Which is best?” Yes, there are multiple items in each view, but they all belong to the two leaders with little overlap. It is a fair, accurate and useful question.
The small DK size of 2% (as compared to some large number like 30%) attests to the public’s willingness to agree with one side or the other when the sides are clearly defined and politically significant as they are in both questions, Q8 and Q9.
Question Q9 takes out the authority figures from Q8 and states what was a good presentation of their underlying views, the “philosophy” (really “rationale”) behind the positions of Clinton and Powell. When the authorities are specifically mentioned in Q8 the preference is for Powell is a little better than three to two. It is remarkable that when the philosophies of the two leaders are given without mentioning their names the ratio of support for Powell’s ideas compared to Clinton’s ideas was still better than four to three, only a small drop from when the authority figures themselves have been removed from the choice.
If it were true that some different combination of the facts taken from Statements 1 and 2 of Q9 would find a much higher support than the 56% support of Statement 1, we could not learn that from the results of Q8 and Q9. It could be learned systematically by testing the effect of each new idea in the two statements, a tedious but reliable method. I think it unlikely that we would find any support much above 56%, but it is certainly possible, even though unlikely, that there was some not-readily-determined consensus buried away in these findings.
Sometimes multipart choices are useful for understanding public opinion when each such choice is related in the minds of the public to a coherent idea, or a well-known unifying name, or associated by the public with a well-known authority figure. I have called this “coherence.” Multipart choices otherwise will uncover little useful data and should be avoided.