Keeping the Subject Happy
The importance of good survey construction
Surveys remain one of the most popular ways for market researchers to tap into the ideas and preferences of consumers. And with good reason – they're cheap to administer, provide solid data, and they're simple to create. Throw in the fact that surveys are perfectly suited to on-line presentation, and there's nothing that can impede the flow of information between marketer and consumer, right?
Well … not exactly.
The rise and rise of information technology, internet penetration, and usage proficiency has created an environment in which the consumer is virtually (pun intended) bombarded with requests for information. Surveys are everywhere on the net, and they always seem to find some way onto your screen (or into your inbox). And the more prosaic methods persist; hard copy surveys still float into your physical mailbox, phone surveys still interrupt your dinner. People will pick and choose which surveys they're going to do, meaning competition amongst researchers. How can we maximize the likelihood of a subject completing a survey? Or, more accurately, how can we minimize the likelihood of a subject not completing a survey?
While there are a huge number of surveying organisations and survey types, the entire process is psychologically quite simple, from the subjects' point of view. Two simple principles guide whether someone will complete your survey: interest in the survey material, and ease with which it can be completed. Interest is simple to manipulate via incentives, but ease of completion is more of a grey area. Interest may be the key to starting, but user-friendliness is the key to finishing.
Non-response in surveys is a recognized source of potential bias. If your survey receives a 10% response rate, is it likely that the 10% who responded differ in any major way from the 90% who failed to respond? Possibly, but this often glossed over. However, if you discovered that the commencement rate (i.e. subjects who started answering the survey) was 15%, then some important issues arise. Firstly, the drop from 15% to 10% is annoying – it represents a loss of generalizing power. Secondly, and potentially more damning, is the likelihood that the remaining 10% do meaningfully differ from the whole because the dropouts probably ceased involvement for a reason. If, for example, they dropped out due to frustration at the survey's length or structure, then the remaining completing respondents probably lack that trait underlying the dropouts' decision. Such an event seriously questions the validity of the survey.
Understanding the psychological processes survey respondents undergo is a vital ingredient in constructing a good survey. Survey writers can use this knowledge to their advantage, and construct questionnaires that minimize respondents' frustration, confusion, and resultant non-completion.
Most researchers appreciate that surveys that take longer than fifteen minutes to complete result in significant reduced response rates. Many researchers are also aware that different question types can result in radically different response times – choosing a preferred option from a list, for example, is easier and quicker than preference ordering the same list. The number of list items a person can keep in short-term memory (STM), however, varies wildly across individuals. As a rule of thumb, a person can keep seven items in STM, give or take two. If a person with a STM capacity of five tackles a list with eight items, it will take them at least twice as long to complete the task as it would a person with STM storage of eight or nine. The result? Frustration, and a greatly increased likelihood of non-completion. The solution? Limit lists to five or six items as often as possible, and only include longer lists if there are no alternatives. Two lists of five are greatly preferable to one list of ten.
Question structure is a critical issue in survey design, and it is this area where most survey writers are found wanting. I am continually astonished at the number of organisations that conduct surveys that aren't worth the paper (or code) they are written on. Consider this scenario: a market researcher wishes to determine if Coke brand cola beverages are generally preferred to Pepsi. The researcher poses the following yes/no question:
Do you prefer Coke or Diet Coke to Pepsi?
Subjects who prefer Coke to Pepsi, but Pepsi to Diet Coke (i.e. dislike the category of “diet drinks”) might have great difficulty answering. At best, they may leave the answer blank. Some might quit the survey then and there, frustrated at their inability to comprehend a poorly-worded question. Others may answer “yes” or “no”, leading the researcher to false conclusions about drinker preferences. Decisions and actions based on incorrect information can cost staggering amounts to rectify.
Nuance and command of language are also vital aspects to quality survey construction. Even punctuation can play a crucial role. Compare these two sentences:
- A woman without her man is nothing.
- A woman: without her, man is nothing.
Again, the problem is twofold: some respondents will quit the survey through irritation or frustration, lowering response rate and subsequent generalisability. Others may respond to a question different to the one asked (or intended). Either way, the survey is compromised.
Respondent mood states can greatly affect surveys as well. It is already established that a poorly designed survey can result in low commencement and completion rates. However, an annoyed or frustrated subject will still sometimes complete the survey – but in a more aggressive or angry frame of mind, leading to data of questionable merit. In some cases a disgruntled subject will deliberately provide false data to “get back” at the survey administrator; such cases are far from rare.
There are a host of other issues that can impact upon the survey taker's probability of providing useful and unbiased data, but for brevity's sake they will not be discussed here. Ultimately, the utility of any given survey is dependent upon its creators – not the subjects. While anyone can write a survey, it is those who understand that a survey represents a different, distinct process of communication that will create a worthwhile questionnaire that people will happily complete.