The Worst Survey Ever?

I recently popped into a branch of one of the main high street banks and couldn’t resist picking up the in-branch service questionnaire. These things are relatively common these days, with every organisation from high street shops to the police asking you to rate the quality of the service you received. This one, though, has issues than most.

The first issue is the poor design and quality of the survey. It has been roughly torn along one edge, uses too many colours (including red and green together, which is a disaster for many red-green colour-blind readers), and has a large organisational chart taken straight from Microsoft Word in the middle of the page. Any literature you produce and provide to customers reflects your brand and image, so the poor presentation and finish of this survey is bound to reflect poorly.

The first sentence is one of the most obvious examples of a loaded question I’ve seen, one of the cardinal sins of survey and questionnaire design.

The next four questions are similarly problematic. I don’t think it’s clear that they are questions, as the alignment (centre-aligned) is ambiguous, and there are no places to mark an answer, like a simple empty box. The provided answers are also entirely arbitrary: why can you answer ‘excellent/very good’ for queue experience and not for ‘making you feel valued?’ Is the queue experience actually how long you had to queue, or is it the entertainment provided while you queue that respondents are asked to comment on? And, finally for this section, ‘making you feel valued’ is so vague it’s practically meaningless.

Advertisement

Survey Question Design

I received a link to this survey recently, asking me to rate my experience with a popular brand of sofas and furnishings. While it got a lot of things right, it got a few of the fundamentals wrong.

In case you can’t read the screenshot, the two questions are:

1. Thinking about your overall customer experience, how likely are you to recommend [the company] to friends, family and colleagues? Answers given on a scale of 0 to 10.
2. Please tell us a little more about why you have given this score.

Using a scale of 0 to 10 is good, and the fact that it’s not part of a matrix is even better. The problem with its implementation here is that the scale reads backwards, i.e. from 10 on the left to 0 on the right. I assume this is either an oversight or, if I was feeling cynical, a suggestive way to improve the score.

The scale should always read naturally, i.e. from 0 on the left to 10 on the right, as someone would read a series of numbers in ascending order.

A progress indicator was good to see, as well as the general lack of matrices, so it wasn’t all bad. In fact this is one of the better laid-out surveys I’ve seen recently.

The Importance of Encrypting Online Survey Data

I co-convene the Sociologists Outside Academia group of the British Sociological Association, and as part of that role I volunteered to help the BSA with their annual membership survey.

One of the recommendations I made to the BSA is that because they intended to collect personal and sensitive information from respondents, it is a good idea to encrypt the online survey to make sure you or your organisation complies with the Data Protection Act 1998 (other countries will invariably have similar legislation).

The encryption is carried out usually by a technology called transport layer security (TLS), and is similar to that used to encrypt your credit card data when you pay for goods online, complete forms requesting personal information, or log in to your online banking service. You know your connection in encrypted because your browser usually displays a padlock logo similar to that in the image below:

 

If you build your survey online, encrypting your survey is usually just a matter of ticking the relevant box in your survey software. In SurveyGizmo, for example, when you have launched your survey, edit your survey link under ‘My Links and Campaigns’ and under ‘Link Protocol’ change this to ‘Secure (https://)’. Simple!

 

 

Other software, such as SurveyMonkey, will have a similar option. If you are using another tool, check the manual or online help.

Why You Should Encrypt

If you are not collecting personal or sensitive data, then you don’t need to encrypt the connection. However, most online survey packages offer encrypted connections with their most basic packages (except their free packages) which cost a minimal amount of money. I’m not a lawyer or solicitor so can’t advise you on whether you should encrypt your connection or not, but if you have the option to enable an encrypted connection anyway, I recommend you do it. It doesn’t cost any more, it doesn’t perceptibly slow down the user’s connection, and it demonstrates to the respondent that you value their data enough to secure it, earning you trust in the process. And it helps you comply with the Data Protection Act if you do decide to collect personal information later.

World Giving Index – Methodology

Image via Wikipedia

Part two of (probably) three discussing the World Giving Index, published by the Charities Aid Foundation (CAF). This post covers the report’s methodology.

An Unanswerable Question

Yesterday I stumbled across another poorly designed question when answering one of the surveys I received.

Only Be As Specific As You Need

I think this question is definitely too specific for anybody’s needs:

In case you can’t see the screencapture, the question asks, “When did you start using the internet?” Possible responses are:

• 2010
• 2009
• 2008
• 2007
• 2006
• 2005
• 2004
• 2003
• 2002
• 2001
• 2000
• 1999
• 1998
• 1997
• 1996
• 1995 or before

I’m afraid I just don’t remember. I’ll have to check my diary for ‘started using internet today.’

This question is asking for a very, very specific answer. One that the respondent is going to struggle to remember, and isn’t really going to give much useful insight anyway. Does it really matter if there’s a difference between users who have been using the internet since 1997 and 1998?

To me, this question is like asking how far it is between London and Tokyo, to the nearest centimetre.

Use a level of specificity that is appropriate for its use and that your respondent might be able to remember.

In case you’re wondering, this is a bone fide survey question sent to me to answer from one of the many survey sites out there, but I won’t tell you which one.