Too busy to read?

When testing leap-of-faith assumptions, it's tempting to ask customers directly what they want, either through individual customer interviews, a focus group, or a survey. Many of us were taught to do this kind of market research. But there's a problem with this approach: people often think they know what they want, but it turns out that they're wrong.1 That's why the College Scorecard team went out to the Washington Mall not with a survey but with a prototype product. They could observe what customers actually did with it. The reason to run experiments is to discover customers' revealed preferences through their behavior. In other words, don't ask customers what they want. Design experiments that allow you to observe it.

My proposal was simple: create a one-page guidance document that laid out, in plain English, a series of parameters within which innovation teams would be pre-cleared to work. 1. If you're doing a tiny MVP experiment with fewer than X customers possibly affected, total liability of Y, and a cost of Z, you're pre-approved. 2. If the experiment is a success and you want do a "scaling MVP" of a little bit greater complexity and bigger numbers for X, Y, and Z, it's deemed approved in advance as long as: (a) it's built on an initial MVP and (b) you have managerial sign-off. 3. If you want to exceed these guidelines for something larger or more complex, you need to discuss it with legal. Here's the hotline to call ... You probably won't be surprised to learn that the first version of this one-page document that the legal team created was ... ten pages long.

Every modern startup possesses a metrics dashboard that the team and board revisit on a regular cadence (schedule). The even more recent trend is to post real-time versions of this dashboard up around the office, on large flat-screen monitors that are visible to everyone. This is part of the transparency that startups tend to favor and that many large enterprises find frightening. But as a coordinating device, it's extremely helpful. There can be no question about how well the company is doing when everyone shares the same set of facts.

Per-customer learning metrics include: Conversion rates (such as the percentage of customers who try a free trial of a product who subsequently become paying customers). Revenue per customer (the amount of money customers pay for a product on average). Lifetime value per customer (the amount of money the company accrues from an average customer over the entire "life" of his or her relationship with the company). Retention rate (what percentage of customers are still using the product after a certain amount of time). Cost per customer (how much it costs to serve a customer on average). Referral rate (what percentage of existing customers refer new customers to the product, and on average how many referrals they make per unit of time). Channel adoption (what percentage of the relevant distribution channels carry the product).