July 1st, 2020
Quarantine has created unusual circumstances for everyone, and borne of it: new phenomena. The Tiger King. Virtual happy hours. Online schooling and graduation. And, of course: Animal Crossing: New Horizons.
I’ve been obsessed with playing Animal Crossing since I bought my Switch early March. It’s a great way to forget about the craziness outside and connect with friends.
As a researcher, I’ve also been obsessed with Animal Crossing.
How does Nintendo sift through legitimate gameplay concerns vs concerns that may only be relevant in today’s environment? More pertinently, how EXACTLY has COVID-19 affected gaming behaviors and how permanent is this change?
While Animal Crossing is thriving during these unusual times, the same circumstances make it hard to decipher and protect its success in a post-COVID world. Being launched during a pandemic, it lacks a probable baseline comparison to benchmark its performance against a pre-COVID world.
It’s easy to find endless complaints / requests if you frequent Animal Crossing Reddit, Twitter, Discord, etc. To be fair, there’s mainly love and support for the game. Yet, it’s hard to ignore the constant posts about improvements that need to be made to the game.
The lists go on and on, ranging from practical to completely ridiculous.
Therefore, I thought it’d be interesting to run through a thought experiment to answer my earlier questions. To start, I'll outline what it takes to answer them:
In this article, I'm going to talk through each of these and put it all together in practice.
You can’t change the product simply because people yell repeatedly and loudly. Instead, Nintendo will probably work to understand how big of an issue X is for gamers and which changes have longevity.
The severity that X is affecting gameplay will determine feature prioritization. This can range from “X would be nice to have” to “I can’t play the game anymore unless X is fixed.” In other words, is the issue a slight inconvenience that can mostly be ignored OR is it negatively affecting gameplay so much that people have stopped playing?
But, the most important thing to talk about here is: how does COVID-19 affect gameplay?
This is a factor that’s unique to this pandemic: Nintendo needs to determine which changes will forever improve gameplay and which changes will only improve gameplay during covid-19. They need to outline the use cases of X and closely examine whether each use case will continue to occur even after quarantine ends.
Just because you and 10K internet strangers agree about something, doesn't mean your opinion is representative of the entire community. Thus, when prioritizing changes, Nintendo will test it with their Animal Crossing personas.
The personas are usually created with real user data and represent common types of Animal Crossing players. Possible data in personas could be demographic such as geography (Asia, North America) or age (Less than 10, 20-25), behaviours (plays mostly online, never plays online) or playtime (plays 5+ hours a day, plays once a week), gaming goals (plays to relax, plays because bored), AC experience (first time playing, played all AC games), etc. We created Bellini Slushie survey question bank to make building these personas easier.
The personas created will be used to test changes and understand how the changes would be perceived across most of the community. Once we understand who the gamers are, we can understand why they’re experiencing these issues and why they feel it is / isn’t an issue.
Once Nintendo prioritizes game changes, they can test them with users that match their example personas. The feedback they get from these users can illustrate how changes could be made across most of their players.
We can walk through an imaginary example using changes I've crowdsourced from friends and simplified personas I've created.
Potential changes to Animal Crossing:
The actual testing would be more complex, but for this exercise I’ll make it up. Players are asked to rate each of the problems on a 3 point scale with the follow options:
When we analyze the results, two issues jump out:
The majority of the players say these are affecting their ability to play the game, therefore these issues would move to the top of our prioritization list. Although some other issues also pop, they either aren’t significant problems or representative enough of the population for us to categorize them as major issues right now.
This particular exercise was definitely not crafted so I could prioritize my two biggest issues in the game. Not at all!
User testing helps a company validate that it's making products that users want and love! This often serves as the foundation and guardrails for product development. However, users' wants don't always equate to practical decisions for the company.
Thus, once user features are prioritized, they are often aligned across the rest of the business. This topic bleeds into a whole other field of work, namely product management, but evaluation can include the technical requirements and impact of the change, potential monetization strategies and revenue affects, etc.
Our takeaway as researchers is to understand the importance of identifying and advocating for the right user requests / feedback. This way, we can ensure the users’ voices aren't lost.
Liked this? Drop your email here and we'll send you something interesting whenever we can.
Your missing survey design toolkit.
You’re not alone anymore.
© 2020 Bellini Slushie
This is a database of basic survey questions. Free for anyone to access and use. New questions added everyday!
Tzeying has been doing UX Strategy & Research across 7 countries in the Asia-Pacific for 10+ years.
Alexandria has been doing concept validation and development in the US and Asia for 3+ years.