Price Testing
More information is better than less. Capital Allocation Matters. Etc.
Instacart recently ended customer-level price tests.
This blog is a reaction to that, but it’s also an attempt to change attitudes towards these tests more generally. Thesis: price tests are good and make the world a better place. Bad philosophies of “fairness” make us poorer than we have to be.
I also have a take on press releases.
What we’re talking about
A customer-level price test works like this: we randomly assign some customers a price vector (a mapping from each product to its price), and assign the other customers alternative prices. We then run the experiment for a few weeks and see which prices make more moo-lah.
These tests are extremely powerful because not only do we learn which prices perform better, we learn how elastic demand is near current prices and, if designed correctly, how various goods substitute for each other. This helps us move prices in the right direction, getting closer to maximizing revenue given demand.
What’s the objection
The objection — as I understand it — is that folks don’t think they should pay different prices for the same good at the same time. This happens every day in an unending list of industries, but it’s an “ick” in other industries, evidently including consumer packaged goods and groceries.
Why do I care what other people pay? I care what I pay. You tell me the price, and I decide whether to buy the thing at that price. Why would it matter to me that someone else got a “better deal?” Good for them?
Another theory is that businesses do this to charge customers higher prices. If they wanted to charge higher prices, they could just go ahead and do that! They don’t need to run an experiment! In fact, if they are considering increasing prices, running an experiment decreases prices in expectation. Without the experiment, they’d just increase prices. That’s their prior. With the experiment, they might learn, “Woah, we can’t increase prices.”
Whenever I think an argument is silly, I come up with a model of the world where I’m wrong, and the argument works. There’s always a model. I think this helps to avoid dismissing things reflexively. Sometimes, an argument only sounds silly.
My best attempt to steelman this: yes, consumers can be better off if firms are ignorant.
Steelman: Is ignorance bliss (for consumers)?
Suppose we’ve got a monopolist — the worst case because they can fully exploit the demand curve, if they know it — do they price higher or lower if they have full information?
Chalkboard model. Our monopolist produces a good with zero marginal cost. They face a linear demand curve:
Q = C— DP
Now, suppose that our monopolist doesn’t know this demand curve. They have some beliefs about it, of course, but they don’t know it. They’re risk-neutral. They maximize expected profits.
If they knew C and D, they’d just replace the expectations above with the truth.
So the monopolist price with perfect knowledge is P* = C/2D, and we have that:
On average, is the perfect knowledge price greater than the imperfect knowledge price?
Let’s assume the monopolist has rational expectations. Then, this is just a question of whether the expectation of the ratio is greater than the ratio of the expectation.
If our beliefs about (C,D) were independent, for example (unlikely, but just to make the point), then, on average, the monopolist price with perfect knowledge would price above the monopolist under uncertainty because…
D>0, so the Harmonic Mean is less than the Arithmetic Mean, and we have:
So: E[C/D] = E[C]E[1/D] (by independence) ≥ E[C]/E[D]
Ergo, the monopolist prices higher with knowledge (on average).
I’ll emphasize on average one more time because, even in this case, it is still possible for C/D < E[C]/E[D] (perfect knowledge pricing is less than pricing under uncertainty) to happen frequently.
This is only a special case and depends on many detailed assumptions about the nature of uncertainty and how it affects demand, but even so…
But what if we have competition?
We get a very different result when we combine risk aversion and perfectly competitive firms.
This part comes from Sandmo (1971). I’m stealing it whole cloth. It’s a good paper. You should read it.
Suppose we have a competitive industry, i.e., firms take prices as given, but they are uncertain about what those prices will be. Firms are risk-averse, and they have a baseline income M. Firms choose output to solve:
Where P is random with E[P] = r. U is increasing and concave (firms are risk-averse). C is increasing and convex. M is the baseline income, and F is the fixed cost.
The first order condition of output choice is:
Subtract E[U’ x r] from both sides to get:
Write profits as V = M + PQ — C(Q) — F = E[V] + (P — r) Q.
So, suppose P ≥ r. Then, V ≥ E[V] and because U is concave:
Because P ≥ r, we can write:
In fact, this inequality holds for all P because if P < r, then U’(V) ≥ U’(E[V]) but (P-r) < 0, so the above still holds.
Okay, so we can take expectation across P on both sides of the inequality:
Going back to the first order condition, this means:
E[U’(V) (P — r)] = E[U’(V) (C’(Q) — r)] ≤ 0, which since U’ > 0 must mean that at the optimal Q,
C’(Q) ≤ r
i.e., at the optimal output, the marginal cost is less than the expected price. If prices were known with certainty, the firm would choose output to set marginal cost equal to the known price. Because marginal cost is an increasing function of Q (more or less necessary for price-taking models), this means that:
Uncertainty about price reduces output.
Equilibrium prices will be higher because firms restrict output relative to perfect knowledge.
i.e., consumers actually want firms to know more! It leads to lower prices!
Note. It’s a little funky to think about price uncertainty. It’s more natural to think of firms picking a price and seeing how much demand they get. But this model works at the limit of competition, so output is the firm’s only choice.
Policy
So, depending on our model of the world, we get very different implications for consumers when firms learn more about the demand curve.
The point is that it’s far from clear how firms having better information plays out, and, in any case, we shouldn’t base policy (governmental, social norms, or biz decisions) on the idea that firms will remain ignorant of their demand curve. They will learn it from experience! They don’t need experimental data to learn demand. It’s just a better and more accurate way to do so. Enforcing ignorance is a bad solution to a market power problem…
Profit-maximizing prices matter. They tell folks where to put capital. They tell us how valuable a product really is. They allocate resources. The right prices matter for more than the individual firm trying to make a buck. When resources are misallocated across an economy, it lowers aggregate output, lowers productivity (because capital matches with the wrong labor/other factors), and prevents economic growth. Sad!
Experiments give us something clear. A solid ground and a judge of what works and what doesn’t — instead of having to base all pricing analysis on tortured regressions or highly parameterized discrete-choice models.
Consider what would happen in the absence of experimentation. If firms don’t experiment, they’ll use non-experimental analysis methods to set prices (this is what most pricing analyses are exactly because of the Instacart experience — there’s this vague sense that randomized pricing isn’t “fair”). If this works well, then it’ll lead to the same outcome as the experimentation approach. So, there’s no harm in the firm experimenting.
If the assumptions underlying the non-experimental analysis are wrong, who knows what will happen? Maybe we get lucky, and firms don’t recognize their full market power because of their ignorance. Maybe resources are misallocated. Maybe the firm underprices, and a valuable good or service doesn’t get the capital it needs? Maybe the firm overprices? Who knows?
Does more information make pricing more pro-consumer or less? It depends. But I find it difficult to imagine that an economy operating under greater uncertainty is better than one operating with more information, all else equal.
Press Releases
I’m going to close by discussing an exciting new topic, not yet explored on this blog. Press releases.
“At a time when families are working exceptionally hard to stretch every grocery dollar, those tests raised concerns, leaving some people questioning the prices they see on Instacart. That’s not okay — especially for a company built on trust, transparency, and affordability.”
?
Why can’t we write this?
“We enabled our retailers to run price experiments where one customer might see a different price than another customer for the same good at the same time. Our customers said they didn’t like that. The upside of running those experiments isn’t worth the blowback. So, we won’t enable our retailers to run price experiments like that anymore. Thanks.”
Everyone reading knows that’s the truth — and it’s a fine truth! There’s nothing wrong with it! Not saying it clearly and introducing vague “concerns” just creates this sense of impropriety where none exists. The reader wonders how Instacart’s pricing experiments hurt families struggling to pay for groceries. Why are people questioning the prices? Were the prices “fake” in some way? Etc.
Not to bring up politics, but this is one thing Trump did effectively in the 2016 GOP Primary. His opponents would catch him contradicting a previous position, and then he would say, “Yeah, well, I changed positions because this one polls better.” And his opponents were caught flat-footed. “Wait, you can just admit that? You don’t have to come up with a fake justification for changing your position?” Of course! It’s obviously true!
You wanted us to stop it, so we’re stopping it. End of story.
(I realize I just lost everyone with the absolute unit of a hot take that “we should handle PR like Trump,” but I only mean in this specific way! We should feel less shame when we have nothing to be ashamed of. Democracy requires aligning yourself with voters. Business requires trying things to make more moo-lah.)
Thanks for reading!
Zach
Connect at: https://linkedin.com/in/zlflynn
Visit me on the World Wide Web: https://zflynn.com