Quarantine has created unusual circumstances for everyone, and borne of it: new phenomena. The Tiger King. Virtual happy hours. Online schooling and graduation. And, of course: Animal Crossing: New Horizons.

I’ve been obsessed with playing Animal Crossing since I bought my Switch early March. It’s a great way to forget about the craziness outside and connect with friends.

As a researcher, I’ve also been obsessed with Animal Crossing.

I’m so fascinated thinking about how Nintendo is dealing with user feedback right now. Especially since, Nintendo could have never anticipated how popular the game would be.

How does Nintendo sift through legitimate gameplay concerns vs concerns that may only be relevant in today’s environment? More pertinently, how EXACTLY has COVID-19 affected gaming behaviors and how permanent is this change?

While Animal Crossing is thriving during these unusual times, the same circumstances make it hard to decipher and protect its success in a post-COVID world. Being launched during a pandemic, it lacks a probable baseline comparison to benchmark its performance against a pre-COVID world.

It’s easy to find endless complaints / requests if you frequent Animal Crossing Reddit, Twitter, Discord, etc. To be fair, there’s mainly love and support for the game. Yet, it’s hard to ignore the constant posts about improvements that need to be made to the game.

The lists go on and on, ranging from practical to completely ridiculous.

Therefore, I thought it’d be interesting to run through a thought experiment to answer my earlier questions. To start, I'll outline what it takes to answer them:

1. Forming hypotheses and assumptions about COVID impact on gaming
2. Sifting through user feedback
3. Figuring out who the REAL audiences are
4. Weighing out the impact on these audience segments

In this article, I'm going to talk through each of these and put it all together in practice.

How Do We Prioritize Change?

You can’t change the product simply because people yell repeatedly and loudly. Instead, Nintendo will probably work to understand how big of an issue X is for gamers and which changes have longevity.

The severity that X is affecting gameplay will determine feature prioritization. This can range from “X would be nice to have” to “I can’t play the game anymore unless X is fixed.” In other words, is the issue a slight inconvenience that can mostly be ignored OR is it negatively affecting gameplay so much that people have stopped playing?

But, the most important thing to talk about here is: how does COVID-19 affect gameplay?

This is a factor that’s unique to this pandemic: Nintendo needs to determine which changes will forever improve gameplay and which changes will only improve gameplay during covid-19. They need to outline the use cases of X and closely examine whether each use case will continue to occur even after quarantine ends.

Who Are We Improving For?

Just because you and 10K internet strangers agree about something, doesn't mean your opinion is representative of the entire community. Thus, when prioritizing changes, Nintendo will test it with their Animal Crossing personas.

The personas are usually created with real user data and represent common types of Animal Crossing players. Possible data in personas could be demographic such as geography (Asia, North America) or age (Less than 10, 20-25), behaviours (plays mostly online, never plays online) or playtime (plays 5+ hours a day, plays once a week), gaming goals (plays to relax, plays because bored), AC experience (first time playing, played all AC games), etc. We created Bellini Slushie survey question bank to make building these personas easier.

The personas created will be used to test changes and understand how the changes would be perceived across most of the community. Once we understand who the gamers are, we can understand why they’re experiencing these issues and why they feel it is / isn’t an issue.

Putting It All Together

Once Nintendo prioritizes game changes, they can test them with users that match their example personas. The feedback they get from these users can illustrate how changes could be made across most of their players.

We can walk through an imaginary example using changes I've crowdsourced from friends and simplified personas I've created.

Potential changes to Animal Crossing:

• Fix the slow loading time travelling on/off different players islands
• Make it easier to skip through repetitive dialogue
• Ability to buy multiple clothing items during one shopping trip at Ables
• Improve gameplay for 2nd player
• Ability to mass terraform sections at once
• Ability to craft multiple DIYs at once

Simplified Personas

The actual testing would be more complex, but for this exercise I’ll make it up. Players are asked to rate each of the problems on a 3 point scale with the follow options:

• ⭐️ = This is nice to have
• ⭐️⭐️ = This is a problem but I can mostly ignore it
• ⭐️⭐️⭐️= I can’t play the game unless this is fixed

Testing Results

Prioritization

When we analyze the results, two issues jump out:

• Ability to mass terraform sections at once
• Fix the slow loading time travelling on/off different players islands

The majority of the players say these are affecting their ability to play the game, therefore these issues would move to the top of our prioritization list. Although some other issues also pop, they either aren’t significant problems or representative enough of the population for us to categorize them as major issues right now.

This particular exercise was definitely not crafted so I could prioritize my two biggest issues in the game. Not at all!

That Was Fun...But Now What?

User testing helps a company validate that it's making products that users want and love! This often serves as the foundation and guardrails for product development. However, users' wants don't always equate to practical decisions for the company.

Thus, once user features are prioritized, they are often aligned across the rest of the business. This topic bleeds into a whole other field of work, namely product management, but evaluation can include the technical requirements and impact of the change, potential monetization strategies and revenue affects, etc.

Our takeaway as researchers is to understand the importance of identifying and advocating for the right user requests / feedback. This way, we can ensure the users’ voices aren't lost.