In 2006, when I was attending Stanford University School of Education for my master’s degree, we were working on our master’s project. One of our professors, who is an expert on evaluation, came to our class to discuss evaluation of our projects. He started by saying:
“Don’t fool yourself into believing that you are developing a great product.”
It was a shocker for us as we all completely believed that we were indeed on our way to developing great products. My project was intended to make it easy for school children to view 3D animations on 3D monitors on concepts that were difficult to visualize in 2D, for example, the rotations and revolutions of planets and the atomic structure of molecules. It was the greatest idea of all time. Why did I need to conduct an evaluation for such a simple, yet powerful concept?
We learned a lot that day, especially not overestimating the usefulness and effectiveness of our ideas. Unfortunately, we didn’t have much time to evaluate our projects so we completed our program and set out, starry-eyed. to change the world.
However, people in real world who develop products and services don’t always care about the users of those products and services. They come up with innovative ideas (or so they think), and start working on the ideas with the premise that they are developing great products. Case in point: Windows 8, iTunes Ping, Google Plus. They ask the users what they think about those ideas, and, thanks to confirmation bias, come back with user feedback that confirm the greatness of their ideas.
They fool themselves into believing that they are developing a great product. They don’t want to look at the analytics because that would disprove their hypothesis. And I have been as guilty of this crime as anyone else.
The right way: Startup companies
Startup companies too start with the premise that they are working on the world’s greatest idea. And they are right. If they don’t think that way, they wouldn’t risk everything to work on that idea. They spend years developing, launching, evaluating, and iterating on their idea. Their best friend? Usage analytics, which gives them insights into how many people are using their products, where, and when. How many people are visiting what parts of the product more? What macro-level trends are visible? Does making a minor change in a color shade increase or decrease the usage of a page or a field?
More than 90% of startup companies die within two years of launching because of several reasons. But they all know how useful and effective their products are (or not) thanks to analytics data. Analytics do not always save an idea but they help fail early, fail fast, fail often, and help people learn and move on. They can fool themselves but not for long. And that’s why evaluation is critical to developing great products.
In the Arena 