I have answered this question in a Slack group for Product Managers. The question was:
What you would respond in an interview if the question is How to Improve Product X?
Since my reply got a bunch of likes there, I have decided to share it here so it may help to Product Owner fellows with their interviews.
In my company, we are interviewing many people recently for Product Owner role. How you would improve product X is a common question. I always start with a product which is known to the wider audience, or it is mainly known to a candidate, so one shall not be stressed talking about product/industry which he heard about for the first time.
What I am expecting to hear ( an example is for Udemy service, the question we have asked in the last interview):
1. There is no golden rule, right or wrong answer, just process of thoughts and ability to respond to “hard” situations in the process.
2. Assuming metrics and market state is more than ok because you do not have access to analytics nor any real insights from a company, but explain how you would find out data in real life situation.
3. Start with a problem of a user:
People tend to abandon course on Udemy once they have paid it. They start, and easily get distracted by other things, and never reach the end.
4. A problem for the business:
a. Users do not get the full value of the course, so they tend to abandon the whole service.
b. Business cannot upsell next level of the same course because they have not finished the previous one.
c. Business will lose word of mouth from a user who did not feel that he gets value from the service.
5. Metrics which will be improvement indicators:
a. Percentage of users who finished at least one course and bought another one VS percentage of users who abandoned course (they do not view any lecture in past month) and bought another one
b. Total percentage of abandoned courses
c. Critical period of inactivity which leads to churn, etc
6. An idea for solving a problem (I would like a candidate to have, down to the ground ideas, something that they saw from competition and some experimental ideas which can lead to 100x metric movement or fail):
a. Gamification / Mini Certificates for each part of the course
b. Email campaign with reminders
c. Unlocking free materials with progressing
d. A campaign to make one day in a week learning day
e. If you finish whole course service will give you back 50% of the money you have paid
7. How you would test an idea before grooming with a team: user surveys, interviews, etc.
8. How you would communicate an idea with engineering team and UX -> Rough mockups, problem we are solving, and what we are expecting to accomplish
9. What if engineering says that something is impossible -> break into phases, and make different MVP
10. What will MVP include -> As small as it can be, to accomplish just core of the problem. Even include things that not scale, and you will need to rebuild them if feature succeed.
11. How you would put the feature on the market -> A/B test with small sample size.
12. How do you know the feature has a better result than a current solution -> Metrics from 5. Explain statistical side of making a decision. It is not so important to know exact formulas, but it is a big plus to know in theory for how long you need to run an experiment, what is type 1 and type 2 errors (false positive and false negative).
Some of the questions that can come in the process:
1. What if your selling cycle is 60 days i.e. a user who bought the first course no matter of finishing he is buying next one no before 60 days from first. How would you test your idea earlier?
Invent new metrics, which is correlating to Buying another course metrics e.g. Activity Frequency – User which frequency of activity is less than seven days has more chances to buy from one which activity frequency is bigger than seven days.
2. What if engineering said to you that shortening 7 days of MVP is doable but will introduce 14 days of Technical debt.
Go for it, shorten MVP as much as you can, and later on tackle technical debt, because no matter of time for development there will always be technical debt.
3. What if right after release you figure out that doing one feature will almost certainly increase metrics 10% but will pile even more technical debt
Don’t do that, postpone it, and research more in deep, while engineering solves the debt. Make exception is this is something that is crucial to users, and they are churning about it.
4. What if one of the course lecturers contact you to and want to pay for certain features for their courses e.g. ability to run specific programming language code in mini tests.
Don’t do that; custom features are almost always significant dead weight in future. Focus on things which can help to more people than one.
and so on, and so on 🙂
Hope I’ve helped you a little a bit, google all things from the answer that are not familiar to you, and there is a lot of material to learn, and remember, someone somewhere made a mistake, so you do not need to.