Chapter 9: Asking the Right Questions and Asking the Questions Right
By Alan F. Kay, PhD
© 2004, (fair use with attribution and copy to author)
Aug. 9, 2004
Is the Federal Government Too Big or Too Small?
An aspect of government that leaders most dispute is its proper size. (In modern times, leaders do not debate. They dispute.) The Republicans, when out of power, squealed at how bloated government had become. They became the party of small government, and still talk about making government smaller even as Republican leaders build up the federal government larger than ever. The cowed Democrats, whether in or out of power, are quick to apologize when their proposals seem to increase the power or size of government.
And yet, those who are familiar with successful government programs like Treasury Direct or Social Security Benefit Distribution generally would like such programs to have adequate staff and resources to keep up their good service at low cost.
Anyone believing that government is intrinsically unable to do a good job, of course, would want to shrink it down indefinitely. Anyone in irresolvable doubt about a political program might opt to cut it and at least save the taxpayers expense. But that doesn’t fly for the public. The people think entirely differently.
The Public Goes for the Right Size
The public has a simple and commendable approach to sizing any government activity. If a poll asks whether an ongoing government activity whose purpose is approved by most people should be (1) cut, (2) enlarged, or (3) the right size to do the job, the public goes overwhelmingly for “the right size.” In their efforts to distinguish themselves from the other party in closely fought campaigns, both the Democrats and the Republicans since about 1980 have become more ideologically polarized and less willing to consider new ideas. The public is much less polarized.
Poll questions have to be the right size to do their job, too. To put a complicated proposal into language that the average person can understand is a challenge that wordsmiths love. From the first draft of a new public-interest polling questionnaire to the final draft, polls typically go through 20 revisions, as the various people involved, the pollsters, the issue experts, the analysts, the sponsor(s), want to pass on question wording.
The Art of Question Design
The question cannot be too wordy. Every word in it can remain only if it serves a required function. Ideas and phrases in the question should be in a logical, understandable order. Long sentences may have to be broken up. With rare exceptions, technical and academic words must be scrapped when a more common word will do. Common short words, particularly verb/nouns like “run,” when used with specific prepositions, adverbs, etc., have multiple meanings that are well understood and easily distinguished in context by ordinary people. Such short words can be very useful. People who have the common touch, who can cleverly mimic the speech of many different random persons, are helpful in finding good language to express the required ideas. Notice how much these recommendations parallel the question design features that, if all present, lead to small DKs. (See Chapter 5.)
Commonly understood metaphors and colloquialisms are okay if they really fill the need. Some think that such phrases are good because everybody is familiar with them. Usually, ordinary English can be formulated to achieve something equivalent to a colloquialism but clearer and more accurate. Experts, as they get comfortable in their professional language, have trouble with the ordinary language of the public, rich though it may be, and tend to use even common words with meaning somewhat different from what the public expects.
An example of expert bias that also illustrates the value of equal weight choices came in a nuclear weapons survey that asked:
(1) Should the goal of the United States be to
(a) eliminate all nuclear weapons,
(b) greatly reduce the number of weapons,
(c) reduce the number by a little,
(d) maintain current levels, or
(e) build new, more advanced nuclear weapons?”
The question appears biased. First, because the “no change” option is not the middle choice. But beyond that, choice (e), to balance its weight with the weight of (a), either (e) should be replaced with something like (e’) or (a) be replaced with something closer to, say (a’) :
(e’) increase the number of nuclear weapons
(a’) eliminate unnecessary and/or unsafe nuclear weapons.
Another question asked in this survey was,
(2) “Do you think the United States should spend more or less money to maintain and update its nuclear weapons, or do you think the United States is spending about the right amount?”
The desirable features of updated nuclear weapons are described by the Pentagon. They include such things as increased bunker penetration, less weight and greater accuracy, which translate to a capability of killing more people. The “more” response for question (2) was chosen by 23%. If more informative language had been used rather than the bland and positive word update, support for “more” would have been reduced. Why is this? In the nuclear expert community a word like update is well-understood to mean what the Pentagon means by it, but not so with the public, when under the constraints of the poll interview, people have to reply off the top of their head. People think of buying updated items this way: if item is broken, rusty, etc., or a new one more reliable, easier to use, safer, or better design, then go for it. In a quick reply, people have little chance to think about the deadly aspects of an activity that few have ever thought about, namely buying nuclear weapons. If the deadly effects of nuclear weapons were included in the description, the 23% support for spending “more” on nuclear weapons would drop sharply.
Where do the people want to go with nuclear weapons? Chapter 4 shows that in December 1987, Americans opted for total elimination by 56% to 41%. By June 1991, the margin had increased further reaching 60% to 38%.
Good Practices in Reading Questions by Interviewers
Words which have multiple meanings or even sound too close to such words, and can be confusing in the context of the question, must be ruled out, since some people may misinterpret them. Interviewers need to know what words are key to the purpose of the question and have to be clearly heard by all respondents to avoid erroneous responses. Putting such words in CAPS in the interviewer’s text is good practice. Words which may be hard to pronounce, like unusual names, must be clarified for the interviewer with a “sounds like” or re-spelled lettering.
Interviewers must be well trained and have an opportunity to rehearse the more difficult new questions. The goal is, “All questions, including preambles and transition statements be understood when read to any respondent (in all their infinite varieties!) by an average interviewer.”
Finally, it is best to run the questionnaire through a pre-test with 20 or so interviews. Random respondents may find question wording difficulties that emerge and need correcting before the full survey interview schedule begins.
A rule of good survey design is that the interviewer cannot speak too long without allowing the respondent to speak. As an interviewer is speaking, people’s thoughts often drop out and become less recallable if opportunities for them to speak are delayed even a few seconds. The statements in a very long question can be broken up into two or more smaller statements by means of a short question, provided of course that the response statistics to the short question appear in the survey report. One way to do this is having the interviewer jump in with:
“Do you understand what I’ve said?” or “Are you with me?”
The answer is recorded and if it is “no”, the interviewer follows up with,
“Let me repeat,” or, “Let me repeat that. Stop me if I say something not clear to you.”
All of this gives the correct impression that the survey sponsor really wants to hear correctly what the respondent’s answers are. Taking a poll seriously in this manner makes the respondent take it more seriously too, and keeps him/her responding thoughtfully longer and more accurately than s/he otherwise might do.
Often a policy proposal question is long because the policy has a number of features or items. This presents another opportunity to shorten proposal questions with break points after each item. For example, if you were surveying people on whether they approve of embodying the Ten Commandments into a Constitutional amendment, it is best to ask each separately. One question becomes 10, and the findings are much more useful than a single up-or-down vote on approval of all 10. Maybe a president who is faith-based oriented could get a bill through Congress some day with nine out of the Ten Commandments passed. Pragmatic advisors would tell the president to take it and go for the tenth at a later time.
Long and Short Questions Confirming Findings
The following are a set of questions, some very, very long, that had to be as long as they were in order to make them intelligible to the respondents. How can we tell that they are not too long? As we have seen often before, the DKs are a key clue, and here they are small — between 2% and 5%. The only question with a large DK, 14%, is B6, length, surprise, merely 14 words with a 29 word preamble. The longest question is the 143 word A3, with a 150 word preamble and a DK of only 4%.
These questions were asked in a survey (ATI-25, ’94, some also asked earlier) on means for funding the United Nations when members do not pay their dues. The first three A1-A3 were asked of a random A sample of 519 respondents. The next three B4-B6 were asked of a different B sample of 517. The next question E7 was asked of both samples, A plus B totaling 1036 respondents. A good exercise for a wordsmith as s/he looks at these questions is this. Think about whether and how these long questions might be shortened without watering down their meaning. After looking at these questions, we will examine other clues as evidence that they are not too long. Each question set starts with a pre-amble:
Pre-amble for A1-A3. Collecting dues from sovereign nations is tricky and may never work very well. There are ways that the United Nations could become less dependent on dues. Before we look at these we should realize that many of the big issues of today were not thought about when the United Nations was founded 50 years ago, for example: environmental pollution, depletion of natural resources, over-population, over-crowded cities, mass migration, drug trafficking, global finance, global manufacturing, and global marketing. Individual nations alone cannot regulate these activities that cross national borders. Increasingly, issues are thrown into the lap of the United Nations which it was never designed to handle. The United Nations can debate these forever, but it will not be able to do all that the world wants done unless the United Nations can raise more money. Let’s look now at different ways the United Nations could be funded without depending so much on dues from sovereign nations.
A1. Do you favor or oppose a 1 percent tax by the United Nations on international air travel which by itself would produce half of all the revenue the organization needs and a savings for all members, which for the United States would be about a billion dollars a year. The tax itself would add about a billion dollars to America’s international travel and air freight bills. Do you favor or oppose that?
A2. Do you favor or oppose a 1 percent tax on international arms sales and transfers which by itself would produce 10% of all the revenue the United Nations needs and a savings for all members, which for the United States would amount to about $200 million dollars per year. Because the United States is the world’s largest international arms supplier, this tax would require U.S. arms dealers and their foreign customers to come up with a considerable sum, estimated at $400 million a year, to pay this tax. Do you favor or oppose that?[IF FAVOR OR OPPOSE] Is that strongly or somewhat (favor/oppose)?
A3. Do you favor or oppose a world-wide tax, proposed by Nobel economics prize winner James Tobin, on international financial speculation in currencies, of ONE-HALF OF ONE PERCENT of the value of trades, which would produce enough revenue to eliminate the dues and assessments of all member states and fund ALL of the United Nation’s needs? This tax would produce, in fact, a trillion dollars a year, which could pay for programs that in a few years would clean up all the world’s polluted drinking water supplies, reverse the destruction of all the world’s forests, and give a basic education to all the world’s children. Some of this tax would fall on international corporations, but, by far, the largest part would fall on financial companies and banks, which do over a trillion dollars worth of such trades EVERY DAY. Do you favor or oppose that? [IF FAVOR OR OPPOSE] Is that strongly or somewhat (favor/oppose)?
Pre-amble for B4-B6. Collecting dues from sovereign nations is difficult and may never work very well. There are ways that the United Nations could become less dependent on dues. Let’s look at these.
B4. Do you favor or oppose a tax on some forms of international pollution, such as ocean dumping of toxic wastes? [IF FAVOR OR OPPOSE] Is that strongly or somewhat (favor/oppose)?
B5. Do you favor or oppose a tax on some forms of international pollution, such as carbon emissions by each country’s power plants, cars, and industries?[IF FAVOR OR OPPOSE] Is that strongly or somewhat (favor/oppose)?
B6. Do you favor or oppose a world-wide tax on international financial trading in currencies?[IF FAVOR OR OPPOSE] Is that strongly or somewhat (favor/oppose)?
E7. Do you favor or oppose allowing member states to purchase a kind of national security insurance? In return for paying the United Nations an annual premium, UN peacekeeping forces would come to their aid if they were ever invaded by their neighbors. Premiums would be tailored to risk — very high for excessively aggressive or militaristic nations and low for low-risk, peaceful nations. The premiums could be a bargain for many countries that live in fear of some of their neighbors and otherwise would have to spend more on defense than they would like to. All the premiums could add up to enough to fund the necessary UN peacekeeping forces and maybe even have money left over to pay for other UN operations as well as peacekeeping. Do you favor or oppose this idea for national security insurance?[IF FAVOR OR OPPOSE] Is that strongly or somewhat (favor/oppose)?
Analysis of A1-A3, B4-B6, and E7.
E7 is a good example of a tightly worded question (138 words) describing a proposal that was earlier presented to the world in a 44 page, 16,526 word, document endorsed by many world figures, including Nobel Laureates. Of course, E7 cannot cover all the important points of a well-written document over 100 times longer, but it does get out the key ideas.
In each of the long form descriptions of the three proposals, A1-A3, the last sentence before, “Do you favor. . . .” gives an argument against the proposal, and from the point of view of Mitofsky’s editorial in the New York Times, which explains the position endorsed by mainstream commercial pollsters (see Chapter 8, discussion of Perot campaigns) that these are balanced, unbiased questions. The contra-argument in A3 is weak, only because nobody on the ATI survey design team at that time came up with a stronger argument.
Compare A3 with B6, two questions testing support for a currency tax. The 144 word version A3 achieves 69% support and has only 4% DKs, while the 14 word version B6 has only 45% support and a much larger 14% DK. The short version is clearly too short and inadequate, while the long one is about as short as a version can be and still adequate to capture the public’s preference.
Discontinuances. Experience in question design and survey completion shows that discontinuances (respondents hanging up on the interviewer) which can be reduced by special and expensive techniques, normally run about 40% to 50% of those respondents meeting qualifications such as adult over 18, and occur mainly at the beginning of the interview, near the point when the social contract is tacitly made. The rare discontinuances thereafter are not significantly associated with question length.
Internal Consistency. Further evidence that the support for detailed proposals may be reliably captured by carefully worded questions comes from the internal consistency of such support from sample to sample and from survey to survey, by testing the effects of wording variations and pro and con argument evaluations (the debate format, described on Chapter 4).
The responses to two questions, A8-A9 from ATI#25, illustrate how good polling can come at the above four proposals A1-A3 and E7 in different ways to show internal consistency. (Four different proposals B4-6 and E7 were similarly tested of the B half sample.) The two half samples were asked of the four proposals for funding the United Nations, which they liked best and which least. The results confirmed the previous findings. Best liked in the A sample were the arms sales tax at 36% and in the B sample, pollution taxes at 53%. The most impressive result of these questions was that after each of the four proposals were presented in random order, as follows:
A: Arms sales tax, Air travel tax, Currency trading tax, UN Security Insurance Plan
B: Pollution taxes: Coal, oil and gas; Pollution taxes: ocean dumping; Currency trading tax; UN Security Insurance Plan.
Respondents were asked “Or just the current dues system.” At a time when Sen. Jesse Helms, D-N.C., assured the world that the United States opposed UN funding by any means other than direct contributions from member nations, only 10% in the A and 14% in the B sample agreed that the last-asked “just the current dues system” was best. Helms was totally off the mark.