December 15th, 2019
Alexandria Grimaldi and I both work in an innovation lab — so, we often find ourselves in the company of amazing people from very diverse backgrounds. But this also means that many of them have never ran a survey. While guiding our lab members to put together the survey they needed, we realised that (1) it can be very stressful and frustrating for them, (2) it’s really hard to find in-depth yet accessible help online and (3) our tips could be very helpful for a larger community. So in this series, we wanted to talk about how to design your very first survey from scratch.
Surveys are an important research tool to leverage. They can help to quickly derive quantitative insights to inform major decisions. Some successful examples are
But, if you don’t have an experienced researcher, the budget to outsource or hire, you end up having to run surveys all by your lonesome. Huh? 🤯
🧐 How do I know what type of survey to run?
🤨 What questions should I be asking?
😣 Where do I even start?
😫 What choices do I have?
There are survey platforms we love, like Typeform, Survey Monkey, and Qualtrics, that are awesome at answering these questions. However, there is also an underlying assumption that if you are using their services, you likely have fundamental knowledge of survey design. This is great for people like Alexandria Grimaldi and I, who have years of research experience and may need quick refreshers on learned methodologies. However, if you’re completely new to this field, getting started probably feels like a shot in the dark. 😶 🔫
We’ve left the table stakes deliberately brief to concentrate more on some of the ickier points and also to talk about points that readers’ have brought up around why no one trusts surveys. If you are looking to dig into some of the above points, here is a great article.
The very first step when writing a survey is to ask yourself the following questions:
To keep things simple, the answers to these questions will inform your goals. (These “goals” are also sometimes known as assumptions, hypotheses, objectives, etc.) Throughout the survey creation process, your goals are your lighthouse; they will keep you focused and guide you back on course when you go astray. 🐈
The easiest way to write questions is to work backwards from your set of goals. It’s important to remember that you have to make a very conscious effort not to bias the survey while doing this. It’s a tough balancing act, which is why writing surveys is much more of an art than science.
For example, if you want to find out how many people love the colour red, the appropriate question to ask should be “What is your favourite colour?” Then, list a few colours for the participant to select from. It becomes a much more compelling story when red is selected against all other options. Directly asking people whether or not they like the colour red creates a false dilemma. These are the sorts of logical fallacies that hurt the credibility of survey results. (Read more about creating unbiased options later)
Your survey will undergo a lot of revisions, either from yourself, other team members or stakeholders. Sometimes in the torrent of edits, you’ll lose sight of the original goals. Loose questions will stick out like sore thumbs. Be firm and remove or re-write these.
This is especially true if you are working with external stakeholders or teams that have other priorities besides your own. If there isn’t a clear alignment, you’ll have multiple stakeholders using the survey to achieve different outcomes, which makes things extremely messy and counter-productive. The revision process becomes exponentially easier when everyone is aligned from the start on the goals of the survey.
If your survey has more than one goal, these goals need to work together in a way that makes sense. If they don’t, you either need to re-define your goals or run separate surveys instead. Cramming everything into one survey may seem as if you save time and money, but it will inevitably create a lot of unnecessary noise. You’ll find yourself endlessly splicing and digging into data that may not be relevant. But because you have all of this data, it momentarily seems important.
So, remember, just because you have 95% stat on something, doesn’t mean it’s necessarily useful or meaningful.
The reality is: no one wants to take a survey.
Do you remember the last time you took a survey? I’m sure you were extremely engaged and considerate of your responses in the beginning, but the more time you spent on it, the less you read the questions or thought through your answer. The longer it is, the higher your drop-out rate will be, and the lower quality of responses you’ll receive. (More about survey bloat here)
To keep your survey short but comprehensive: only include responses that help with your goal (#1). You can, and should, include questions with data exploration in mind. For example, when validating a persona, do include demographic and behavioural questions. And, if you are managing disparate stakeholder goals, remember, running a separate survey is always an option.
If the question doesn’t make or break things right now: take them out. Less is more.
This is one of the trickier bits in research — balancing of qualitative vs quantitative. Often times, to have a rich understanding of our users, you need to collect varying proportions of both.
Most of the time when we use open-ended responses it’s because we want to hear the answer described in the participants’ own words and / or don’t want to eliminate any possible options for them. We can get a lot of inspiration from people that think outside the box!
But, let’s not make the participants hate you.
Open-ended questions are very taxing on the participants. The more someone has to think and explain themselves, the longer the survey will feel. You’ll begin to see a lot of nonsensical responses like “n/a,” “ok,” “idk,” “asdfg” and “i’m just going to copy the question straight into the response section.”
Not very helpful.
Prior proper research is very important, and it shows.
You’ve probably already done a ton of research around your survey topic, so you should have a good idea of what the options should be. Use them! If you have no idea what some of the potential responses could be, think about doing more research or delay asking that question.
Save yourself a lot of time and effort
When you turn an open-ended response into a close-ended one, you’ll also save yourself time when you’re analysing the results. Including qualitative responses means that you have to read through and quantify qualitative responses, which can be a time-consuming endeavour. Why not make it much easier for yourself to splice and analyse the results?
We want to continue discussing how to design a survey, but Tzeying is going to kill Alexandria Grimaldi for how long this is already. The next article will focus on the topics of:
If there’s anything else you’d like us to explore or elaborate more on, let us know!
Liked this? Drop your email here and we'll send you something interesting whenever we can.
Your missing survey design toolkit.
You’re not alone anymore.
© 2020 Bellini Slushie
This is a database of basic survey questions. Free for anyone to access and use. New questions added everyday!
Tzeying has been doing UX Strategy & Research across 7 countries in the Asia-Pacific for 10+ years.
Alexandria has been doing concept validation and development in the US and Asia for 3+ years.