Google displays personalized search suggestions, Netflix populates recommendations for shows, and Amazon provides products you may like widget are notable examples of machine learning or big data products. These product features will not show accurate results until algorithm understand end-user pattern. Therefore, the fundamental question for machine learning/AI/Big Data product managers is whether to ship or to learn?
Every product manager’s biggest question is whether to ship or to learn? This debate becomes further challenging when it comes to products with machine learning, big data, or artificial intelligence as a base technology. Because machine learning, big data, or AI will not perform until these technologies understand end user behavior. As a product manager you will have million dollar question shall I release new feature with default behavior or shall I learn more to understand end user patterns and tune my algorithm to display user-specific results? However, further you learn; more resources you consume. It increases cost. Thus it will impact a company from time and budget perspective. On the other side, until you understand if a new feature is doing well and providing an appropriate result; you will not be ready to take a risk and release new functionality considering end user experience.
Let’s consider you are a product manager at Amazon for “You May Like” widget. John signed up as a new user on Amazon.com and did not browse or searched for any new product. Now which products algorithm will populate under “You May Like” widget? You might display generalized product set based on default attributes, and you might use default input to an algorithm to generate default result. However, default result will not create significant impact as you have expected. It means product needs an additional time to learn John’s search pattern or buying behavior. Now to make it further complicated; let’s say John is buying Minnie Mouse birthday mug for his daughter. He searched for Disney products. Does that mean he will buy Disney products next time he visits Amazon.com? Maybe not. Then, as a PM you might make a decision and say, we will not display “You May Like” widgets for first two weeks of an user life cycle to understand user’s behavior. Now there could be multiple variables involved in this situation.
- How many times John visited site in last two weeks
- Did he purchase anything or just did window shopping
- Every search he did with an intention to buy a product
- Does he interested in buying products for himself or he is buying as gifts to friends and family?
List continues and there could be few more questions. It means as a PM you need to define a matrix to make a decision – whether to ship your product now because algorithm knows enough to deliver product set based on user behavior or still give more time for an algorithm to learn. Above example is about widget so it might not create any negative impact on business. However, if we consider an example of self driving car do you think PM will be willing to launch a product without multiple learning cycles?
We all know, Google is working on a self driving car and they are testing it for a while now. Once in awhile, you will find it in a testing mode in Mountain View, California. In this case product is ready and Google is in learning face to understand various situations so that algorithm will learn each case and handle it without causing any human or property damage. Self driving car should be risky affair as it might impact someone’s life if things go wrong. Thus it makes complete sense in this case to perform extensive learning before you ship a product.
In summary, every PM stuck in a dilemma of whether to Ship or to Learn during their PM tenure. I would say this is very subjective question and one must evaluate following factors before making a decision;
- Cost associated with a product development/integration – Resource, time
- Impact of launching a new feature
- Risk associated with a product/ feature
- Probability of each risk occurrence and impact of risk on consumers
- Evaluate how much risk company is ready to support if company decide to move forward