It helps validate many of your hypothesis about your idea or product
One of the key responsibilities of a product team is user research. The idea is to get actionable data that can be used to do changes to the current release or for a future iteration.
It helps the Product team understand who the product is for and why it is valuable to them. It helps validate many of your hypothesis about your idea or product
The most commonly used research methods currently are:
A/B Testing
User Surveys
User interviews
Moderated User Research
The most critical output of user research for me has been to use the signals for making the product simpler and to drop / tweak some features.
I had primarily used A/B tests to take product decisions in my product roles at Yahoo, AOL, Dell. Undeniably the data from these tests are some of the strongest signals you can use.
I am a huge huge fan of running these tests and ideally some bandwidth in every sprint should be dedicated to this. Of course this makes sense only if your product has enough traffic for the test to be statistically significant.
Mildly Amusing Anecdote: While working for Yahoo Search Product team I always had A/B test results of multiple variants of the background colour for the Ads on Yahoo Search ready. The tests were to measure which background colours gave us the maximum uplift in ad clicks without degrading the user experience too much. Legend has it that when she was at Google, Marissa Mayer ran tests on 41 variants of the colour blue to find the best shade for the Google Search Ads background. I can honestly say that I tested more variants than that during my stint at Yahoo :). In my defence that was not the only work I did, but once at a family dinner a teenaged cousin was incredulous that my "job" was choosing colours. She thought it was the best job ever. I recall one day or rather night there were multiple calls at 3am from my project manager before I finally woke up to answer the phone. She had just got out from a meeting where the decision was taken to launch a new ad background colour variant which could give us a significant he biggest short term uplift as there was some significant revenue pressure ( She called at 3am because she was based in the bay area I was based in Bangalore and the decision was to launch the change immediately. I was the only one with the data - there was no Slack in 2011 !)
I'm sure almost all product teams know enough about A/B testing. And most of them would have also used surveys and interviews to their advantage.
But in this post I primarily want to touch upon two things
1> How to use Moderated User Research to improve your product iteration
2> How to talk to customers to figure out if your new product idea is worth it? (Primarily inspired by the best book on this subject The Mom Test )
Moderated User Research
As I mentioned earlier I had primarily depended on A/B testing in my career and the power of quantitative data. But in 2013 during my stint at Target.com I had the chance to experience how powerful moderated user research can be. Our UX team organised a 2 day session where users tried out new features / redesign of our website with a trained moderator asking them questions. We just sat and observed the users from behind a one way mirror wall (yes a little bit like police interrogation where we could see the "suspects" but they couldn't see or hear us). Some of the qualitative inputs I heard over those 2 days were eye opening to say the least. There are some nuances which will always be missed even if you use the best online data analysis framework.
The tools tell you what the users did. But many times you cannot figure out why they did it. That's where some of this qualitative information can fill the gaps
If you decide to go down this path then here are a few important points:
Communicate the need for this exercise to stakeholders. If this is something your company has not being doing before
Go with the mindset that user research will always provide value.
Your "test" conditions don't accurately represent real life but this direct interaction with users will help you understand their challenges, expectations and moments of delight in a much better way.
Pair a UX researcher with the product manager and let the UX researcher "lead" things for the purpose of the study. I understand small startups may not have the budget for a UX researcher and PM so the founder could work with UX/Product for this. I really liked this post by Alëna Louguina that gives a framework on how the UX researcher and PM can partner.
The truth sometimes hurts. Users may not really understand or care about a feature that you thought was super cool and useful or they totally hate it. The A/B test may still show clicks because they have to finish the task and that button is the only alternative to move forward but they absolutely hate it. Without taking names some of the biggest private sector bank websites in India are guilty of this. Try to login and you are always taken to an intermediate page with "ads" before you can login to your account. Of course the click data shows that almost all users click the button but is it a good experience?
How to talk to your customers a.k.a. The Mom Test
After I read The Mom Test book last year, I don't think anyone can explain this process better that Rob Fitzpatrick. So I highly recommend that you buy his book if you want to figure out whether your startup idea / product idea has merit or not
At a very high level instead of directly asking them about your idea you should focus on the user's life and not directly mention your product.
Some rules of thumb from the book:
If you don’t know what you’re trying to learn, you shouldn’t bother having
Customer conversations are bad by default. It’s your job to fix them.
Give as little information as possible about your idea while still nudging the discussion in a useful direction. Keep an eye out for the people who get emotional about what you’re doing.
Ideas and feature requests should be understood, but not obeyed.
So net net figure out what kind of user research fits the requirements of your team / product and then try to do it in an organised fashion.
This post has been published on www.productschool.com communities