Surveys, Data, and Vibes
It's okay to ask people what they want and believe them
I’ve become less skeptical of surveys. Economics, as a field, has a bias for revealed preferences—what people do—over what they say, and I mostly inherited the biases of the field because, by default, you believe whatever you read when you were twenty-three.
But I’m becoming a fan of surveys.
This post is about both sides of the revealed vs. stated-preference divide. I talk about the problems with using surveys to package and price products and ways to replace some common survey use cases with revealed preference methods. But also about why sometimes it pays to just ask people what they’d do, and believe them.
The difference between saying “I’d pay $100” and writing a check for $100 is the difference between the plan to run a marathon and lacing up. This is the fundamental problem with surveys.
Conjoints don’t solve this problem. Artificially creating a realistic choice set doesn’t bridge the gap between stating and paying.
[My little take here is that for the narrow task of pricing (as opposed to understanding the determinants of demand), conjoints are worse than just asking “would you pay $5?” My evidence is classified, but I’ve seen many price elasticities from conjoints, and they are always wrong to the point of being useless, while I’ve seen decent-enough results from the “would you pay $X for ___?” question.]
The right argument for running a survey is that we don’t have a good answer to the question: “what else could we do?” When there’s nothing else to do, you run a survey. But there’s a temptation to throw in the towel early because (we imagine) a survey is a time-bound, well-defined task that we can probably ship in a week or two.
But you can learn a lot from revealed preferences before you have to resort to stated preferences.
It’s rare to launch a genuinely new product, a product that’s completely unrelated to whatever else your company does, so you don’t have any relevant data. Most new products are re-packaging your existing product or a small new add-on. You want to offer an all-you-can-eat subscription, alongside your current à la carte, transactional business. You’re going to start charging for a type of product usage that was previously free. Etcetera.
When revealed preference data exists that answers the relevant question, it’s better because it represents real checks. For example, it might look like you don’t have any data on how people will respond to a price for a previously free product, but you can triangulate.
When you introduced this product, how many more people signed up for the subscription?
When you raised the price by $5 last year, how many fewer people did?
Combining those two answers tells you something about how valuable customers find the feature. You can say things like “people value the feature like a $3 reduction in the product’s price, so…” [You do need both parts here: variation in the presence of the feature and variation in the price.]
The point is that, if you’re a wee bit creative, you can use data to answer a broad class of questions before you need to make your own data via surveys.
Surveys are worse than behavioral data, but they are 1000x better than vibes (the 1000x, however, was measured by vibes).
Analysis is a competitive market. If there’s no data analysis or survey, decisions will be made via vibes. There’s always an outside option. People don’t have to use any evidence at all to make decisions. You can just decide things. Vibes are always a backup plan.
[This is why I lean towards “give stakeholders something” instead of “we don’t really have any ready data on that” or “oh, that’ll be involved, we might be able to work that into the next sprint” (at which point the data is, to a rough approximation, always useless: the decision’s been made). The question is: “Do you have something that’s probably better than vibes?” If so, ship it.]
The point of a survey is to create data where we don’t have it. The typical types of missing data are:
We don’t have price variation. The price has never changed before.
We don’t have product variation. Either everyone has always had the product, or no one has ever had it. So we don’t know how much anyone will like it.
We have no way to connect the new product to an existing product (it’s not just a re-packaging of goods we’re already selling).
It’s surprising how many problems in statistics amount to missing data problems. Causal inference is all about the fact that we don’t observe the potential outcome for treatments that people never experienced. Pricing new products is the same kind of problem. We don’t know what people will pay for products they’ve never been offered.
This is the key frame to use when writing surveys: what data is missing? We don’t just want to run survey to run a survey. We have a goal. We’re trying to fill in the gaps in our data. Answering this question makes writing a focused, short survey easy and natural. We need to ask questions that extract the missing data.
We don’t have price variation? Then we need to ask people, “Would you buy it at price X?” and “Would you buy it at price Y?” Bam. Now the price varies between X and Y.
We don’t have product variation? Suppose we’re looking at adding a new feature to a bundled subscription. Then, we need to ask the questions “Would you buy it at price X if it included feature Z?” and “Would you buy it at price X if it didn’t?” i.e., we need to keep the pricing constant and vary the feature set.
Another kind of missing data we need to extract from a pricing survey: the respondent's size. If you’re selling, say, a B2B product, the size of the company purchasing the product will be relevant to how much this customer’s price sensitivity matters. If you’re doing seat-based pricing, then you need to know the number of seats the respondent represents. If you’re selling a transaction-based or usage-based consumer product, you have to get a sense of how many potential transactions or how much potential usage the respondent represents. The particulars of how to do this depend on how demand scales for the product, but it’ll usually be something like “how many times do you run workflow X” or “how many employees work at your company”.
From there, you can maximize revenue/profit (depending on your cost structure) weighted by the respondent’s demand size (a lot of survey tools don’t do this sadly: they’ll just maximize revenue for you without taking into account demand size—alas). The demand curve is just given to you by the highest prices that people say they’re willing to accept.
I like to solve a problem that looks like this:
Where P is the price, W(i) is the weight associated with the size of the respondent’s demand (maybe based on their answer to company size questions, etc), WTP(i) is the highest price folks say they’re willing to pay (the survey usually gives an upper and lower bound on WTP, so it makes sense to solve this at each of those extremes), a is a discount or premium on people’s stated willingness to pay. I like to check for robustness to deviations between stated willingness to pay and true willingness to pay, or even just to downweight WTP if you’re trying to price for growth. So you can plot the optimal price as a function of a and find a price that doesn’t vary a lot for small deviations in a.
There’s a bit of an art to this.
For a well-targeted survey, a very high percentage of responses to the “Any other thoughts?” prompt are really thoughtful. This isn’t a real quote, but it’s illustrative of responses you’ll get: “Seat-based pricing wouldn’t really track the value here. We’d really only need one seat because only one engineer needs to manage this. The usage option you presented makes more sense, but only if there’s some cap or it gets cheaper as we scale…”
Humans like being helpful, and even though it takes them time, and it doesn’t really affect whether they get the $25 gift card, respondents take surveys seriously. Because humans are generally good folks who like to help their fellow corporate drones out. Go figure. The kids call this a “white pill” (I think?).
It’s not like you have to blindly roll out the survey’s optimal price. A couple of in-depth interviews with potential customers can at least make sure you’re not going off the deep end. If someone says, “That sounds reasonable,” it probably is. If they say, “lol what?” something’s probably gone wrong.
You don’t have to completely discard vibes. Vibes contain useful data. They’re a statistic of past experience. So, you can use them to discipline the survey, while still basing pricing (or other decisions) on data. If some respondents’ stated willingness to pay makes no economic sense, just drop them. If you’re worried people are overstating their willingness to pay—before they have to actually convince Finance to sign off on it—then just underprice it (set a < 1 in the above problem).
The point is that you can build trust in survey results. You don’t have to treat them as a machine that takes responses and converts them to decisions. This makes them infinitely more usable, despite their limitations.
I’ve been thinking a lot about surveys recently. There might be a few more survey-focused techniques in upcoming blogs (thinking through some issues).
Zach
Connect at: https://linkedin.com/in/zlflynn