Product Problem

A problem can be broken down into smaller parts using an iterative method.  This allows each sub section to be analysed to a greater amount of detail.  The process helps to remove ambiguity in determining why something is happening, speeds up problem solving and removes wasted resources needed for solving similar problems in the future.

The procedure below begins by assessing the impact of a problem in order to mitigate against its immediate effect.  It then investigates internal and external causes.  One question to consider is whether to investigate internal or external causes first.  There is no right answer and the best strategy is likely linked to the change that has been observed from the data.  I believe that by starting with an internal assessment, some obvious causes can be ruled out e.g. new feature X was released yesterday which changed the data in metric Y to change.

Procedure

1. Impact:
   1. Understand the problem: What metric has been affected and what could this mean E.g. Has the user sign up or activation rate changed.  Have any other metrics been affected? For example, a reduction in time spent using an app could be due to a recent update that increased the efficiency of performing a task?
   2. Assess Scale: Is the problem affecting a single location or product line?
   3. Inform Stakeholders: Do other stakeholders need to be informed about the problem? 
   4. Mitigation: What procedures need to be activated to stop or minimise the impact of the problem?
2. Internal Causes:
   1. Cross Analyse: Check if the problem has occurred cross-product/platform or with a different team/location?
   2. Historical Data: Check past data of the product for similar irregularities.  When did the anomaly occur (time of day and day of week)?  Is there an annual trend to the data?
   3. Assess growth models: Does the growth model of the product predict the trends in the data until now?  How did the anomaly affect the data (gradual/sharp change)?  Do extrapolations from the old data set predict this new change?
   4. Internal Developments: Has there been a new feature release, a defect that has been discovered, expiration of feature, security change?  Check for user feedback to see what the customers are saying.  Are there any common issues that users are complaining about?
   5. Demographic affected: Is it only a certain demographic of users that have been impacted e.g. new users from the past three months, users in a certain age category?
   6. Business model: Has there been any significant changes to the business model of the product e.g. the licensing model, seasonal changes to a products services, a free service has changed to a paid subscription.
3. External Causes:
   1. Competition: Has a competitor changed their product or business model?  Has a new competitor emerged in the market? 
   2. Geopolitical: Have there been any recent changes to government policy that could have restricted/altered access to a platform?  Has there been a power outage or natural/human disaster?
   3. Network Issue: Has there been an issue with a telecom provider that could have resulted in accessibility issues?   
   4. PR: Has there been a bad PR case recently that affected the company?
   5. Regulations: Has there been any legal changes that have come into effect?
4. Internal Mitigations:
   1. Internal changes are likely to cause a sharp change in the data. A new feature, a defect or a sudden change to the business model are likely to causes a sudden reaction from the users of the product.
   2. For a problem that has been identified as internal, a good strategy would be to work closely with the engineering team to isolate the problem and implement a fix.
   3. The severity of the problem determines the response needed. A critical problem might require the current work to be interrupted whereas a minor problem could be implemented in future work. 
   4. Problems with new features can be mitigated by reversing the change to restore a previous version of the product. The new feature can then be reviewed to determine the root cause.
5. External Mitigations:
   1. External problems are more likely to cause a gradual change in the products data. For example, a competitor’s gradual rise of market share or the slow implementation of a new law.  This isn’t a strict rule as many external factors can have sudden effects e.g. a natural disaster.
   2. For a problem that has been identified as external (e.g. a new competitor), a good strategy would be to continue the development procedure of the product whilst continuously assessing what changes could be introduced as mitigation.
   3. For external problems that have a temporary effect (e.g. seasonal weather), there may not me much action required and a gradual return to business is likely to follow.

 Example:

The performance of Uber went down in Norway?  Identify the problem and form a plan to fix it?

1. Assessing the Impact.

A: What do you mean by the 'performance' of Uber in Norway? Does this refer to the user sign up rate, activation rate, time spent using the app, logins per day?  Is this affecting both riders and drivers or just one party?

Q: The issue concerns a decrease in the number of drivers using the app. 

A: What % decrease in drivers has been recorded? Is this a trend that concerns Month on Month (MoM) or Year on Year (YoY) data?

Q: There has been a 8% decrease in drivers MoM for the past three months. 

A: Is the issue only affecting Norway?  Are there any other countries within Europe or elsewhere that have been affected?  I’m not familiar with the number of cities that Uber operates in Norway? Is Oslo the only city operating the service?

Q: Only Norway has been affected and you can assume Uber only operates in Oslo.

A: Is it just jus the ride sharing service that has been impacted or have other services been affected e.g. UberEATS, Bike/Scooter sharing?

Q: Just the ride sharing service has been affected.

A: Okay, the issue appears to be a regional problem but I will need to assess it further to make sure.  For now, I will inform other product managers in Europe about the problem that we are experiencing to make them aware.

2. Assess Internal Causes.

A: Have there been any recent feature updates to the app in the last week? Are there any recent comments in the product store that provide useful information?  Is the performance decrease seen on both Android and iOS platforms?  Has there been an update to the terms and conditions for using the app?

Q: There have been no major feature updates recently or changes to the terms and conditions.  The trend is affecting the app on both platforms and we do not have any data on user comments.

A: Prior to the three months in which the number of drivers have been decreasing, what has the month on month trend been for drivers registering/leaving the app? What has the trend in drivers been for the last year? 

Q: Prior data shows that the number of drivers using the app was increasing at 5% per month (on average) for the past 18 months.

A: Okay, it sounds like the issue might be caused by an external factor.  I am going to explore some external issues briefly before analysing internal issues further.

3. Assess External Causes.

A: Are we aware of any recent developments in a competitor’s business model? Are competitors incentivising drivers to switch to their product?

Q: You don’t need to worry about competitors for this question.

A: I think the change is unlikely to be caused by a network issue as the trend has been ongoing for three months.  Similarly, I don’t believe the problem to be caused be a geographical issue for similar reasoning.  Has there been any regulatory change in Norway that could have affected the way Uber and other ride sharing apps operate in the country?

Q: This is not a factor to be concerned about.

A: Have we received any bad PR in Norway that could be affecting the region?

Q: There has been no significant change in the PR of Uber in Norway recently.

A: Has there been an update to the terms and conditions for using the app that has come into effect?

There doesn’t seem to be an obvious external factor causing the decline in drives.  Due to the sharp decrease in users and since it is affecting drives specifically, I believe that the problem might be a bug affecting the user experience of drives.  It might be an issue affecting a certain version of the app only and therefore only a certain percentage of drivers.

Q:  You are in the right area.  What would be your plan to mitigate against this?

4. Mitigation Strategy.

A: Here is the strategy that I want to employ:

• This might be an issue which is unique to a certain version of the app that drivers are using. I want to start contacting drivers using various versions of the app and ask about their experience using the app over the past couple months.  This might indicate a common frustration that drivers are experiencing.
• I could prompt drivers who are using older versions of the app to upgrade to the most recent update. However, I can’t be sure that the problem is with an older version of the app so I think it is best to wait for the user feedback first. Once received and if the feedback is correct to the cause of the problem, I would like to slowly rollout an upgrade (or downgrade) of the app to a percentage of drivers and receive feedback on this change. If the subsequent feedback alleviates the problem, then I will enforce the change across all users.