Learnings from 40+ User Interviews
Conducting User Interview is one of the primary skill for Leaders/Founders/Product Managers to develop.It is one of those skills which theory/articles/tutorials can not prepare you for in entirety. I recently had to conduct over 40 user interviews & with time was able to synthesise few major takeaways on “What not to do for sure”.
The “What to do” question has constantly evolving answer with myriad possibilities & vectors. We will attempt to cover one such approach derived from experience that makes sense.
Before moving forward, a little bit on category of user interviews. B2B & B2C category have very different requirements & processes required. A B2B setting will require taking into account very different set of variables such as designation of individual being interviewed , final authority on “Yes/No” & B2B customer relationships are often very long term in nature. B2C tends to be more unpredictable & competitive.
B2B2C is a burgeoning new category wherein Individuals are Businesses in themselves & it appears to have mix of both B2B & B2C properties.In this article, i am trying to capture this category i.e. individuals , as sole authority, deciding on Problems/Solutions for themselves.
In my personal experience, B2C imprint in more prominent in B2B2C category.
We are also covering “Idea Validation” phase of User Interview with no built out product yet as the primary theme of the article.
A built out product wherein tinkering/growth is required will require different approach.
Primary Challenges
The abstractness & complexity of all variables in User Interviews presents unique challenges , specially if you starting out for first time. Here are few sample questions to consider :-
Where do you find the users for interview?
What is your aim for the user interview ? “Idea validation” or “Insight Generation” or “Measuring willingness to pay” or something else.
What does the user profile look like? Depending on the source of User, you may not have all the necessary info.
How long the the interview might run for? Accordingly you may not to make major trade-offs.
But the most important one i will try to cover in detail is :-
Broadly, How to conduct the interview itself or what questions to ask & subsequently, how to separate noise from the signal?
Each stage of User Interview are interlinked & has its pitfalls. Before , we try to answer some of the questions , it is vital to understand the severe Impact of Bad user interview.
Risks/Impact of Bad User Interviews
While you might imagine a bad user interview is the one wherein customer outright refuses to constructively engage with you or the one wherein you generate “no new insights”, the biggest & the most harmful impact is actually of collecting “False Positives” i.e. you incorrectly conclude you idea is useful.
The reasons again could be varied - from polite users unwilling to disappoint you to your own inability to decipher signals correctly.
The core issue is also very difficult to diagnose & solve.
We as Humans are “confirmation bias” machines. Once you have an idea that you personally find valuable or has made some progress on , you will effortlessly & subconsciously look for signals that confirm that idea further. Warning signals are often overlooked or downplayed.
Not to mention, if you are part of an organisation, there is ever more pressure to show “Progress”. It is inherently difficult & effortful at times to say “No“ to an idea you might have invested lot of time into & start from scratch looking for new things.
If you have ever conducted “Interviews” & reviewed CVs for evaluation , none or very few talk( have personally never found anyone) about how over the last year they worked on 3 major ideas & decided/concluded to not move further on it. The profile highlight is often “worked on & launched X product/feature” because it is concrete & easier for others to measure.
It is vital to mention that most of these actions/results are not “actively planned” by anyone involved in ensuring Products success. It often default path, unless active effort is put in place to avoid it.
Product Manager role already has very few “visible” outputs on which to measure performance. If you have not written PRD over the last year , because no ideas were useful, it appears as if you have not done anything major. Writing “worked on, Launched & Learned from” appears more concrete on annual evaluation sheet.
But the “App Store Graveyard” should serve as a reminder to avoid such risks. So many apps launched don’t reach even double digit downloads before vanishing.
If you avoid short term pains, the “pain” of launching something that nobody wanted is more expensive in terms of resources & time. Not to mention, learnings from such launches itself might not be useful. A PM is unlikely to conclude that “We built something none wanted” , instead the most likely learnings will focus on Execution risks, Marketing issues , Operational or Competitive issues or that “next killer feature” that will make the product prominent.
The “opportunity costs” are generally not actively considered. The fact that you focused on prematurely on an Idea & selectively confirmed proofs to convince yourself of the idea, you may have missed out something that had more potential.
While there are no easy answers & a Product failure may have variety of sources of failure, but if the genesis or first major step of idea validation i.e. “User Interview” is flawed, the chances of success will dim subsequently.
The article is on how we can minimise risks inherent with Idea Validation.
Let’s start with some of the questions :-
Where do you find the users for interview?
Probably another book is required to cover this in detail. But let’s cover it briefly since it does impact the most important question we want to cover later i.e. “How to conduct the Interview & What questions to ask”?
The 3 broad sources can be :-
Source 1 :- Friends & Family
Pros :- The easiest one to start with so that you can get initial validation.
Cons :- Sample is likely to be very low for you to conclude anything decisively.
They are unlikely to be completely honest with you. Incentives, be default , are to support you on your idea. The risks of “False Positive” is extremely high.
They might not be part of Early Adopter segment to start with & hence the conversation might not amount to much.
Let’s say , you are thinking of creating a tool for “older generation, say , 60+, to become digital savvy”.
You reach out to your grandparents & friend’s & colleague’s grandparents to validate the idea. The issue with this is that age is only one variable of the potential user profile - Past education, work experience ( Bank worker vs Auto Driver) , current location , living alone or with family ( Grandparents living alone might be more motivated to become digital savvy to ensure more independence)- matters a whole lot more.
If you are not actively accounting for such inputs, you might incorrectly conclude that the product has no market as such since most people i talked to are not motivated enough to learn new skills at this stage.
Alternately, if i try to actively define who my “Early Adopters” might be & refine the user profile to “60+ Parents whose Children live abroad & currently live alone”. Surely, you have shrunk the target market. But, your chances of making real progress is higher here.
Time is also a factor to consider. While today the defined market is small, the trend of people having single kids is increasing & a larger proportion of such single kids might go abroad for studies/work in future.
This does impact the target market size in coming years. Personally, i find today’s “Smaller markets” with “Limited active competition” but potentially large growth in coming years-as best bet for making big outsized gains. Early mistakes in such markets have limited downside due to limited alternatives available for users & through “Trial & error” you are likely to accumulate externally unknown “Competitive Know-How Advantage” of cracking the market.
Source 2 :- Organic Channels - Landing Pages, Social Media Reach-out
Pros :- Depending on the idea, you may be able to reach out larger target segment for interviews & Validation.
Cons :- Very few users might be willing to engage & give you time for user interviews. In absence of limited credibility, such users have no motivation to engage.
Source 3 :- Inorganic Channels ( Paid Ads)
Pros :- You can potentially reach the largest segment through paid ads & unlike other sources, you can make quick progress.
Cons :- Pitfalls include too much noise vs useful signal, high probability of user misinterpreting the product theme, Limited understanding of user profile & inability to conduct in-depth user interviews.
As an organisation or team, focusing on 1-2 channels can lead you astray. Hence, continuous diverse focus on all channels & synthesis of all the learnings is best suited.
How to go about the user interview?
Before jumping into what we should do? A little bit on things to avoid for sure.
As an observation it is easy to notice that users being interviewed at Idea Validation stage have nothing at stake - if the product is not built out yet & you are proposing on ideas that might someday in future they might download & try .
Hence, they would hardly ever outright refuse to try out a potential new idea you are proposing. The onus lies with the Interviewer to bring out the true intention.
A question such as - “ Would you try out a product which will do X?”- The answer will most probably be yes.
Key Takeaway :- Users are lot more familiar with & aware of the problems. However, they have very little clue on solution.
Below is an anecdote to illustrate this further.
Anecdote
With the initial 10+ user interviews that we had collected from friends/family contacts , it was obvious in few cases that users were merely attending the call for obligation. They were not interested in the offering at all , however found it difficult to articulate the same openly.
At this time, there was an opportunity to for me to attend few user interviews as Interviewee - wherein people had reached out to me with their product offerings.
Attending user interview as “Interviewee” was an eye opener of sorts.
It provided a much needed perspective on where “Interviewers”, despite their best intentions, go awry in the process.
Let’s take one the Products.
The Product, as per the pitch, was for Early Professionals ( PMs, Marketers, Writers etc.) wherein they could “Supercharge their capabilities & create CVs for the Internet Age”.
The interviewer started out be pitching few features such as - “A repository which has all articles, major tweets & linkedin posts by you” at one place that could be shared with “Potential employers”.
Possible Hypothesis :- The core idea was that CVs were a legacy offline world approach & that Internet world CVs should look different.
The hypothesis was solid . But i was probably not the Early Adopter.
Below is the summary of my “Interviewee” experience :-
I found it easy to say yes to the product without much surety if i need it at all. Saying “No” felt rude though i was not convinced fully of the offering.
The interviewer did not even question me on my “writing habits, activity on twitter, linkedin etc”. The “Early Adopter” would have been someone who is already very active on Twitter, Linkedin Or writes & Share more frequently.
Plus, there were lot of jargons that he used i did not get at all.
I found the interview process very effortful , since the interviewer spoke most of the time on the Product/Feature & i had to process all the new information.
The interview was certainly very excited about the idea & talked about other few features i do not recall.
He did ask me few questions on the most challenging problems i faced. To be honest though, i was blank for 30 seconds. I felt as if i am put in a spot & gave some answer that popped up first in my head quickly without much thought.
Later on i realised, i did not share the most relevant answers to the above question of “Most difficult problem i face” . If the questions had been asked , say a day later in the same manner, my answer would have been probably very different.
Such questions also increase risks of your Product becoming “Feature Factory”. You will gradually accumulate laundry list of diverse “Features” with little signal on which features might work across the widest segment userbase.
Users also cannot visualise the right solution. As Henry Ford quoted over a century back - “ If i had asked people what they wanted, They would have said faster horses”. It is the team’s job to tinker with the solution piece & focus on problem only during interviews.
The Possible Approach
The learnings on “What to avoid” from experience provided the below core approach to take :-
“Simply discuss the Users profile, their day to day activity wherein your product is relevant, unearth problems deeply & do not pitch your Product at all, unless absolutely important”.
The idea is not perfect. You cannot implement it with leads collected form FB ads properly or your landing page has to mention Product one way or the other.
However, There are few advantages to such approach that outweigh the cons :-
The User being interviewed can effortlessly talk about his daily routine. This way you are likely to be able to get more time & hopefully go back to them again later on .
You have high chances of unearthing new problems & insights that the user might not be actively aware of. For example, a user might be performing menial manual jobs repeatedly & might have become used to it. But , there can be an automated feature for the same that our solution can cater to, saving valuable time for the user.
You also have possibility of understanding GTM related aspects such as “how does the user currently discover new Products”. In the example mentioned earlier of Digital Savvy Tools for Grand-parents , You may figure out that such users often use their “Grandkids help” with installing new Products or they view “useful apps” section of an elderly magazine. In absence of such insights, you might only consider usual digital channels. Whereas , there might be cheaper & easier to exploit mediums to consider in plain sight.
Let us attempt to narrate how the User interview for “Digital Age CV” might have looked like under the new approach recommended :-
Scenario-1
Interviewer :- How did you get your last Job?
Interviewee :- I applied through a friend’s reference to the organisation.
Interviewer :- Who was first contact point? What all did you share with the Organisation for applying ?
Interviewee :- A individual from HR was the first contact point. I shared my CV & a cover letter.
[Insights to consider :- While the digital CV product will have applicants using it, the first receiver/evaluator of such new tool( HR department here) also needs familiarity with this new tool. Else, the traction might be limited & Product might not take-off.]
Interviewer :- What was major aim to highlight as major achievement in CV & cover letter? I mean, what did you want your Interviewer to notice the most?
[ People generally like talking about their achievements without much effort]
Interviewee :- Hmm, My last major project wherein i achieved over 5k+ active downlaods of a new product within 6 months. I even received appreciation from my organisation/Team Lead for the same.
Interviewer :- That’s cool. How did you get your appreciation - was it verbal , through certificate or on linkedin?
Interviewee :- It was an internal mail.
[At this stage you may consider probing him further]
Interviewer :- Did you feel the “appreciation part” registered actively with the interviewers ? Or Do you think a linkedin appreciation might have been more valuable?
Interviewee :- Not sure on both the questions.
Interviewer :- Are you active on Linkedin/Twitter & do you write blogs often?
Interviewee :- No , not so much. I mean i am there on those platforms but i simply passively consume other’s content. I am not very active with my blog as well.
Interviewer - How frequently are you writing blogs - a broad range will help?
Interviewee - Very rare actually. I have not written in last 6 months.
[ Tentative Conclusion :- The current user being interviewed is not an “Early Adopter”. We can consider ending the interview at this stage only. ]
An early adopter already needs something to be able to capitalise on in his “Digital Internet age CV”. Our MVP is unlikely to change deep ingrained behaviours, say becoming active on Social Media or becoming active blogger.
The user above most probably belongs to “Early Majority”. Such users will generally try out a new solution once there is some social proof. Say, a colleague, an early adopter tries out your new product & recommends it to him, then only you can expect such users to download it.
There can be hundreds of variants of the above interview. But the main takeaway is that the focus exclusively is user himself/herself. Notice how above we did not even pitch the Product at all. In practice, though, it is almost impossible to achieve in initial interviews. However, the Interviewer should be aware whenever this happens & try to bring back the interview discourse on its original intended path.
Scenario-2
Interviewer :- How did you get your last Job?
Interviewee :- I applied on Linkedin.
Interviewer :- Who was first contact point? What all did you share with the Organisation for applying ?
Interviewee :- I am not sure. I had received call from HR for scheduling Interview.
[ Insights to consider :- Irrespective of the User being interviewed & end conclusion, you now have more proof of the need to focus on HR side of your Product. This is very important since sole focus on Candidate might lead to half-baked product with puzzle arising later on as to why we do not have active users.]
Interviewer :- What was your aim to highlight as major achievement in CV & cover letter? I mean, what did you want your Interviewer to notice the most?
Interviewee :- My ideas led to over 30% growth in transactions in the last product i owned. In addition, i actively assisted with user research on a new product.
Interviewer :- That’s fantastic. Did you receive any appreciation for the same? If yes, how did you receive it?
Interviewee :- Yes, My team leader appreciated me verbally & i received Kudos from my organisation on Linkedin/Twitter.
Interviewer :- Did you feel the “appreciation part” registered actively with the interviewers?
Interviewee :- Not sure.
Interviewer :- Are you active on Linkedin/Twitter & do you write blogs often?
Interviewee :- Yes , i am quite active on Linkedin mainly.
Interviewer :- What is the nature of your posts?
Interviewee :- Quite a few things. I share advice on tips to becoming effective PM, insights from successful people, takeaways from books i read recently. Quite a few PM aspirants reached out to me for suggestions. One of my posts was even liked by our Founder/CEO/”Someone Important”.
Interviewer :- Wow, that’s quite impressive. Would you say your CV currently captures such dynamic qualities & achievements.
Interviewee :- Don’t think so. CV is recommended to be 1-pager & hence there is always a challenge on what to include. I tend to focus on work experience.
Interviewer :- How has your linkedin status/account evolved since joining the new organisation ? I mean, the new team members must be your connections now?
Interviewee :- Yes, i have added them. Infact, my Team lead, who was my hiring manager, & fellow members regularly like my new posts.
Interviewer :- Do you think prior to applying/joining the current organisation, if your linkedin profile activity was captured in the CV somehow & your interviewers were familiar with it , your chances of selection would have improved?
Interviewee :- Yes, definitely. I am aware of quite a few instances wherein individuals have received new offers out of the blue because of their insightful linkedin/Twitter posts.
Tentative Conclusion :- The user is certainly an “Early Adopter”. He understands the value of “Digital Internet age CV” & will likely try out our mvp. I would probably stop the interview at this stage & mention that i will get back to him.
Let’s assume we had no idea on what the product looks like prior to the interview , but we have received some insights from the interview above.
To start with, i would think of the MVP broadly with below components :-
Onboarding module for applicant.
Ability to connect Linkedin/Twitter account.
Based on Linkedin/Twitter account connected, we would auto-fetch most liked/Commented posts.
On Home Profile page, a “Major Highlights” section would display top 3 fetched posts from Linkedin/Twitter in real time.
There will be a “Share Link” that the Applicant can copy & share it with others.
In parallel though, i will start visualising HR side of things & the actions they perform while discovering/evaluating & scheduling interviews with panels. You may have unexpected findings that may be consequential to your product.
For simplicity, we have kept out other important questions such as “alternate solutions that the user might have already tried”, “Competition in the industry” , willingness to pay etc.
Remember that the current widespread CV model is free of cost in monetary terms & has only costs involved w.r.t time/effort saved . Competing against free solutions with inbuilt behaviour of decades is difficult.
The new Product value proposition needs to build around - “Limited effort in upkeep since APIs will keep all content auto-updated & new opportunities that Digital Age CV opens up for the applicants”.
Unless the new product is 10x, the inertia involved with existing behaviour of working with CVs might be too powerful for a mildly useful “Digital CV” to make any dent into. This insight should help you align the Team on roadmap items that help you achieve the 10x experience & the expectation that mvp is just the first step towards realising the 10x UX.
Let me your thoughts in comments below.