Wednesday, November 16, 2011

The Art of Survey Design

While reading this week’s article, “Capitalizing on Capabilities”, one thing that I realized was the immense need of a good survey design. The goal of any survey is to identify exactly what people feel about a particular topic i.e. in our case the various capabilities that the firm seeks to get more information about. However, an improperly designed survey may lead to widely varying and confusing outcomes. In the worst-case scene, this could also lead the organization to arrive at incorrect conclusions leading to focus on the wrong capabilities.

As part of my Human Computer Interaction course, Prof. Brad Myers had mentioned some key points to consider in designing good surveys. I write this blog to share those ideas. The following are the three major points that form the crux of a good survey design.


Relevance and Accuracy

The survey design starts from identifying which questions are the most relevant for the results that it seeks to achieve. Application of this in real-life is seldom as easy as expecting it. External consultants, industry experts and key players from the company’s various business units are important here. Consultants who specialize in this framework would be better able to guide the management through the process, while the management would be the best to know the internal operations. Once the questions have been shortlisted, it is important to make sure they are of the highest levels of accuracy without any ambiguity. Creating focus groups and testing out the surveys by means of pilot programs are essential here.

Question Intent and Approach

Often times, the view of the survey designers and the view of the general population completing the survey are tangential. Questions that lend themselves to differential interpretation benefit nobody. Some points to remember while making the questions:

  1. Are the questions legible/understandable to the various business units?
  2. Are all of them relevant to a particular business unit?
  3. Would people get frustrated at the length of the questionnaire and hence fill incorrect results?
  4. Are my questions designed in a way that might lead people onto choosing a particular answer?
  5. Is there any single question that asks viewpoints on two-different topics?
  6. Are my questions specific or open-ended? Is it aligned to the req.?
  7. Are there any questions that assume existence of certain knowledge?


Survey Question Types and Structure

Though there are various ways in which a survey can be designed, it is important that it has the right types of questions. There are various ways in which the survey questions can be designed like Likert Scales, Semantic Differential Scale, etc. However, which ones to use it total dependent on the survey designer based on his/her understanding of the question and what exactly it tries to convey.

  1. Likert Scale – Propose something and let people agree or disagree
  2. Semantic Differential Scale – Two opposite feelings
  3. Choice questions – Choose one or more options based on user preference
  4. Open ended – To be used very cautiously as this is the most time consuming (usually a text field) and is usually overlooked or not filled accurately. Also going through the responses also becomes troublesome.
  5. Ranking questions – Rank the options based on user preferences

Like any designing endeavor, survey design is part science and part art. People get increasingly better at it as time progresses. Also, the criticality of the survey should never be underestimated as that forms the basis for data collection from which useful information can be gleaned.


References:

  1. Smart Survey
  2. User Study, Brad Myers Lecture Notes



No comments:

Post a Comment

Note: Only a member of this blog may post a comment.