I’ve become less skeptical of surveys. Economics, as a field, has a bias for revealed preferences—what people do—over what they say, and I mostly inherited the biases of the field because, by default, you believe whatever you read when you were twenty-three.

But I’m becoming a fan of surveys.

This post is about both sides of the revealed vs. stated-preference divide. I talk about the problems with using surveys to package and price products and ways to replace some common survey use cases with revealed preference methods. But also about why sometimes it pays to just ask people what they’d do, and believe them.

Subscribe now

The difference between saying “I’d pay $100” and writing a check for $100 is the difference between the plan to run a marathon and lacing up. This is the fundamental problem with surveys.

Conjoints don’t solve this problem. Artificially creating a realistic choice set doesn’t bridge the gap between stating and paying.

[My little take here is that for the narrow task of pricing (as opposed to understanding the determinants of demand), conjoints are worse than just asking “would you pay $5?” My evidence is classified, but I’ve seen many price elasticities from conjoints, and they are always wrong to the point of being useless, while I’ve seen decent-enough results from the “would you pay $X for ___?” question.]

The right argument for running a survey is that we don’t have a good answer to the question: “what else could we do?” When there’s nothing else to do, you run a survey. But there’s a temptation to throw in the towel early because (we imagine) a survey is a time-bound, well-defined task that we can probably ship in a week or two.

But you can learn a lot from revealed preferences before you have to resort to stated preferences.

It’s rare to launch a genuinely new product, a product that’s completely unrelated to whatever else your company does, so you don’t have any relevant data. Most new products are re-packaging your existing product or a small new add-on. You want to offer an all-you-can-eat subscription, alongside your current à la carte, transactional business. You’re going to start charging for a type of product usage that was previously free. Etcetera.

When revealed preference data exists that answers the relevant question, it’s better because it represents real checks. For example, it might look like you don’t have any data on how people will respond to a price for a previously free product, but you can triangulate.

• When you introduced this product, how many more people signed up for the subscription?

• When you raised the price by $5 last year, how many fewer people did?

Combining those two answers tells you something about how valuable customers find the feature. You can say things like “people value the feature like a $3 reduction in the product’s price, so…” [You do need both parts here: variation in the presence of the feature and variation in the price.]

The point is that, if you’re a wee bit creative, you can use data to answer a broad class of questions before you need to make your own data via surveys.

Surveys are worse than behavioral data, but they are 1000x better than vibes (the 1000x, however, was measured by vibes).

Analysis is a competitive market. If there’s no data analysis or survey, decisions will be made via vibes. There’s always an outside option. People don’t have to use any evidence at all to make decisions. You can just decide things. Vibes are always a backup plan.

[This is why I lean towards “give stakeholders something” instead of “we don’t really have any ready data on that” or “oh, that’ll be involved, we might be able to work that into the next sprint” (at which point the data is, to a rough approximation, always useless: the decision’s been made). The question is: “Do you have something that’s probably better than vibes?” If so, ship it.]

The point of a survey is to create data where we don’t have it. The typical types of missing data are:

1. We don’t have price variation. The price has never changed before.

2. We don’t have product variation. Either everyone has always had the product, or no one has ever had it. So we don’t know how much anyone will like it.

3. We have no way to connect the new product to an existing product (it’s not just a re-packaging of goods we’re already selling).

It’s surprising how many problems in statistics amount to missing data problems. Causal inference is all about the fact that we don’t observe the potential outcome for treatments that people never experienced. Pricing new products is the same kind of problem. We don’t know what people will pay for products they’ve never been offered.

This is the key frame to use when writing surveys: what data is missing? We don’t just want to run survey to run a survey. We have a goal. We’re trying to fill in the gaps in our data. Answering this question makes writing a focused, short survey easy and natural. We need to ask questions that extract the missing data.

We don’t have price variation? Then we need to ask people, “Would you buy it at price X?” and “Would you buy it at price Y?” Bam. Now the price varies between X and Y.

We don’t have product variation? Suppose we’re looking at adding a new feature to a bundled subscription. Then, we need to ask the questions “Would you buy it at price X if it included feature Z?” and “Would you buy it at price X if it didn’t?” i.e., we need to keep the pricing constant and vary the feature set.

Another kind of missing data we need to extract from a pricing survey: the respondent's size. If you’re selling, say, a B2B product, the size of the company purchasing the product will be relevant to how much this customer’s price sensitivity matters. If you’re doing seat-based pricing, then you need to know the number of seats the respondent represents. If you’re selling a transaction-based or usage-based consumer product, you have to get a sense of how many potential transactions or how much potential usage the respondent represents. The particulars of how to do this depend on how demand scales for the product, but it’ll usually be something like “how many times do you run workflow X” or “how many employees work at your company”.

From there, you can maximize revenue/profit (depending on your cost structure) weighted by the respondent’s demand size (a lot of survey tools don’t do this sadly: they’ll just maximize revenue for you without taking into account demand size—alas). The demand curve is just given to you by the highest prices that people say they’re willing to accept.

I like to solve a problem that looks like this:

Where P is the price, W(i) is the weight associated with the size of the respondent’s demand (maybe based on their answer to company size questions, etc), WTP(i) is the highest price folks say they’re willing to pay (the survey usually gives an upper and lower bound on WTP, so it makes sense to solve this at each of those extremes), a is a discount or premium on people’s stated willingness to pay. I like to check for robustness to deviations between stated willingness to pay and true willingness to pay, or even just to downweight WTP if you’re trying to price for growth. So you can plot the optimal price as a function of a and find a price that doesn’t vary a lot for small deviations in a.

There’s a bit of an art to this.

For a well-targeted survey, a very high percentage of responses to the “Any other thoughts?” prompt are really thoughtful. This isn’t a real quote, but it’s illustrative of responses you’ll get: “Seat-based pricing wouldn’t really track the value here. We’d really only need one seat because only one engineer needs to manage this. The usage option you presented makes more sense, but only if there’s some cap or it gets cheaper as we scale…”

Humans like being helpful, and even though it takes them time, and it doesn’t really affect whether they get the $25 gift card, respondents take surveys seriously. Because humans are generally good folks who like to help their fellow corporate drones out. Go figure. The kids call this a “white pill” (I think?).

It’s not like you have to blindly roll out the survey’s optimal price. A couple of in-depth interviews with potential customers can at least make sure you’re not going off the deep end. If someone says, “That sounds reasonable,” it probably is. If they say, “lol what?” something’s probably gone wrong.

You don’t have to completely discard vibes. Vibes contain useful data. They’re a statistic of past experience. So, you can use them to discipline the survey, while still basing pricing (or other decisions) on data. If some respondents’ stated willingness to pay makes no economic sense, just drop them. If you’re worried people are overstating their willingness to pay—before they have to actually convince Finance to sign off on it—then just underprice it (set a < 1 in the above problem).

The point is that you can build trust in survey results. You don’t have to treat them as a machine that takes responses and converts them to decisions. This makes them infinitely more usable, despite their limitations.

I’ve been thinking a lot about surveys recently. There might be a few more survey-focused techniques in upcoming blogs (thinking through some issues).

Zach

Connect at: https://linkedin.com/in/zlflynn

I’ve decided to write well.

Now, I know the first step traditionally is to move to Paris or a small American town—you can get away with NYC if you’ve got old money—and preferably not be employed in B2B SaaS, but I don’t have that kind of dedication. I like money, I have suburban tastes, and I like my neighbors like I like my comments—anonymous.

So, I’m going to try another tactic: practice.

Subscribe now

I’ve been blogging once or twice a month on statistics (which you call “data science” when you’re trying to get a job) for a few years now, but I’m going to write more frequently now. My goal is to put out a blog three out of every seven days.

Of course, that doesn’t mean I’ll write up three experimentation techniques every week, like in the old blogs. I have a limited supply of these! But I realized I had a lot of thoughts about industry tools, practices, news, etc., and just listing the ideas in my little Kanban board made it clear there was enough content to do it.

So, we’re broadening our horizons, limited by the following principles:

1. I have to care about the topic.

2. I have to know enough about it to not sound stupid (hard!).

Especially looking forward to doing more tools-of-the-trade reviews! I did one recently on ggsql. There’s a ton of products, tools, techniques, etc that I’ve never really used because they weren’t used at prior jobs—or, they’re new—and it’d be fun to take a look at them.

I’ve got some initial ideas, but feel free to pitch me on tools to review! Including ones you wrote! (With the warning that I will say what I think of it. My rate for lying about how great your tool is $1 million. So readers should know that either I think the tool is really good or you paid me $1 million.)

Looking forward to the challenge! My current plan is to write the posts mainly over the weekend and schedule them throughout the week, but we’ll see how it evolves…

Zach

Connect at: https://linkedin.com/in/zlflynn

Posit released a new library for graphics programming (plotting) in a SQL-ish style language called ggsql. It’s aiming to be like its ggplot project for R/Python, but with a SQL-ish syntax, which, I guess, is becoming more and more the language of everything (remember the NoSQL moment in the early 2010’s… yeah, that’s dead).

Subscribe now

I gave it a spin over the weekend on some “ancient” data I re-discovered in an old Dropbox folder. In grad school, I scraped the internet for this data on radio station coverage and formed markets from sequences of intersecting stations for this paper I was working on.

The site was trying to stop scrapers, so I had a Perl script randomly cycle through a set of User Agents and switch Tor proxies frequently while downloading thousands of pages (it blocked every 1-5 and the algorithm tried to find the right combo of waiting + user agent switching + tor proxy restart that would unblock me). The site’s solution for stopping scraping was pretty good. It took my little eeePC netbook (remember those!) weeks to get all the data with all the switching, random pausing, and delays I threw in there to get past the detection automatically. I just kept the laptop open night and day. The big hole in their scraping-prevention technique was that they assumed the scraper valued their time and sanity more than paying them $100 (I think it was about that much to get a computer-readable form of the data). Their model of the world did not include a graduate student making ~$10k a year, who was perfectly content to parse HTML and wait weeks to save $100.

In any case, I haven’t looked at this data in forever, so it felt like the perfect laboratory to poke around with ggsql.

Funny coincidence: this paper was the last project I worked on, where I used lattice for the graphs, not ggplot.

What ggsql does (with a few examples)

ggsql uses a “reader” that processes the pure SQL parts of the statement and the ggsql extensions. The easiest way for me to test things was with the DuckDBReader, which lets me visualize the output of DuckDB queries. I imagine there are other readers for different databases (or, there soon will be).

ggsql outputs Vega (a JSON format for graphics). Vega makes a lot of sense as an output target because it’s flexible, low-level, and plays nicely on the web side + it’s got some institutional support, and tools on top of this are sort of the key to making ggsql valuable. We can transform it directly on the web via the JavaScript library, so you can send the Vega output to the front end and render it on the fly.

To get a sense for what it looks like, here’s the syntax for plotting a histogram of the number of radio stations within each market.

SELECT * FROM radio_markets
VISUALIZE Firms AS x
DRAW histogram
LABEL title => 'Radio Station Distribution Across Markets',
x => 'Radio Stations',
y => 'Markets'

This is what it looks like after rendering:

The part preceding VISUALIZE is just standard SQL. You can add WHERE, JOIN stuff, etc.

After VISUALIZE, we enter ggsql mode and start providing (approximately) the ggplot arguments in this SQL-ish syntax.

If you’ve used ggplot before, there’s this concept of “aesthetics” where we define the x-axis, the y-axis, colors, etc. VISUALIZE Firms AS x puts the SQL variable Firm into the x part of the aesthetic. You can also do VISUALIZE Firms As X, blah AS Y, blah2 AS color, etc, etc.

The DRAW clause tells it what kind of visualization to make (you can have multiple DRAW layers, of course), and LABEL… labels things (it can also change the tick labels, not just the axes, and alter any legends, etc).

By the way, to actually run this, I wrote a Python script. That’s clearly not the vision for how to use this. The idea is more of a notebook or other UI that can parse Vega output automatically to show the visualizations as output from an SQL cell or as results in an SQL-running UI, like Databricks/BigQuery/etc. But for messing around with it, you can do something like this:

import ggsql
import pandas as pd
import duckdb

df = pd.read_csv("radio_market_selected_30.csv", sep=":")

with duckdb.connect("radio.db") as con:
    con.sql("CREATE OR REPLACE TABLE radio_markets AS SELECT * FROM df")
    
reader = ggsql.DuckDBReader("duckdb://radio.db")

viz_query = """
SELECT * FROM radio_markets
VISUALIZE Firms AS x
DRAW histogram
SCALE y
LABEL title => 'Radio Station Distribution Across Markets',
x => 'Radio Stations',
y => 'Markets'
"""

viz = reader.execute(viz_query)

writer = ggsql.VegaLiteWriter()
vegalite_json = writer.render(viz)
print(vegalite_json)

And then, I just did:

python3 histogram.py | xclip -selection clipboard

And rendered it in Vega’s online demo thing: https://vega.github.io/editor/

[PS. Did you notice the source file is colon-separated lol? As best I can figure, given how long ago this was, I did it that way because the city names had commas in them (Peoria, IL), and I didn’t want to quote the strings :) ]

To show the SQL elements of this, here’s the same chart, deleting the long tail that starts at around 200 stations.

SELECT * FROM radio_markets
WHERE Firms <= 200
VISUALIZE Firms AS x
DRAW histogram
LABEL title => 'Radio Station Distribution Across Markets',
x => 'Radio Stations',
y => 'Markets'

The best way to explore a new tool and figure out if it’s really a sensible system worth adopting is to just try to make something more involved from the basic getting-started examples, without referencing the docs until you absolutely have to. What you’re looking for is whether things work logically together. You start to intuit the logic of a system (or lack thereof) when you force yourself to try and figure it out instead of just Googling/LLM’ing for “how do I…”

ggsql definitely passes that test. It’s natural and intuitive. Here’s a slightly more involved example to see what I mean:

SELECT * FROM radio_markets
WHERE Firms <= 200
VISUALIZE Area AS x, Firms AS y
DRAW point
   MAPPING Population AS size
   SETTING fill => null
LABEL title => 'Radio Stations and Physical Area',
x => 'Radio Stations',
y => 'Land Area (sq. mi.)',
size => 'Population (100k)'

My Takes

First off, this is super smooth, and I’ve always liked the “grammar of graphics” idea. It’s conceptually clear, easy to use, and easy to extend. It’s a great way to add plots to SQL. The part I’m stuck thinking about after playing with it is whether that’s what I want to do…

I can’t see this being very useful as a standalone thing. Even in a notebook, the utility seems limited. Being in SQL in-and-of-itself seems like it would only matter to the set of people who can be bothered to learn this new syntax but can’t be bothered to learn/don’t already know basic Python/R. I haven’t run a survey, but I’m pretty sure that’s an empty set. If I’m in a notebook, I already have access to ggplot/matplotlib/whatever, and I probably know how to use it.

One standalone use case that is valuable: it should make it easier to run visualizations on your main database compute engine, avoiding first loading the query results into your instance and then running Python/R/etc. You can return the Vega from the compute, plot it, and never touch the full dataset locally. That is super valuable if you’re plotting a larger dataset.

But the better use case really needs a full system where folks—or their Robots—write visualizations inline in a BI tool. ggsql (in its current state) isn’t a real replacement for the kind of viz you’d want in a BI tool, though. It’s not a language that can express (as far as I can tell) table viz, pivot tables, single value elements, layouts, etc. So, it’s got some of the elements, but not enough to replace a dashboard-like system.

On the other hand, it seems “right” to extend SQL with a declarative visualization layer on top of it. A good direction to go—and I have a feeling this is the plan—would be to add their grammar of tables as well to get a fuller system for expressing visual output in SQL.

Another thing I noticed was that I don’t think the VISUALIZE syntax is exactly a SQL extension in the sense that I don’t think it really has the same semantics or processing. Here’s an example I stumbled upon because this old dataset still has the base-R-like periods substituting for spaces in variable names:

import duckdb
import ggsql
import pandas as pd

df = pd.DataFrame({"numbers": [1,2], "variable": [3,4], "variable.dotted": [3, 4]})
with duckdb.connect("reproduce.db") as con:
    con.sql("CREATE OR REPLACE TABLE test AS SELECT * FROM df")

reader = ggsql.DuckDBReader("duckdb://reproduce.db")

## Quoting the dotted variable name works in DuckDB.

with duckdb.connect("reproduce.db") as con:
    print(con.sql("SELECT \"variable.dotted\" FROM test"))

### But not in ggsql
viz_query = """
SELECT * FROM test
VISUALIZE numbers AS x, "variable.dotted" AS y
DRAW line
LABEL title => 'variable ~ numbers'
"""
viz = reader.execute(viz_query)

writer = ggsql.VegaLiteWriter()
vegalite_json = writer.render(viz) # errors

I haven’t done a deep dive into the code base, but this very minor bug (?) suggests to me that ggsql sits outside the SQL engine and isn’t transparently working on the SQL elements. At least, that’s my first guess at why quoting wouldn’t work in its context. I could be wrong here, and maybe it’s just a bug on the Writer side (I filed an issue with the GitHub project, so maybe that’ll be all it is). It’s a small thing anyway.

Conclusion

In general, this is a nice solution to the problem of wanting ggplot in SQL. It solves that problem fantastically. Very clean. Very intuitive if you already know ggplot. I could guess how to do things after an hour or two playing with it, and I was usually right. So, that’s an incredibly strong win.

I’m less convinced it’ll become anyone’s daily driver unless it delivers a full BI experience. In the short run, I see the main use case being to create visualizations for large datasets that you don’t want to pull into memory. That’s a real use case, but a bit narrower than I was hoping for when I started playing with it.

As of today, I think it’d be faster for me, in most cases, to just copy-paste the query output into Google Sheets and plot it there if I want a quick visualization, but we’ll see where things go!

Things are happening—even in SQL.

Thanks for reading!

Zach

Connect at: https://www.linkedin.com/in/zlflynn/

Reading Club! April Edition!

I’m starting a new thing. I’m going to write about a recent paper that strikes my fancy once a month. More as a commitment device for myself to keep up with the lit than anything. I won’t just dive into the paper. I’ll talk about the relevant literature beforehand so the post is more or less self-contained.

This month’s paper is from a little-known, small journal from the fringes of Economics academia: “Econometrica.” It’s about figuring out how best to use the data to inform model estimates when we’ve got enough evidence to know the model’s “wrong.”

Subscribe now

It’s one of those excuses you’ll hear. “I know the model’s wrong, but all models are wrong,” and you, the listener, who is both smarter and better-looking, think, “Well, yes, but, like, clearly there’s a limit to that. If it’s really wrong about important things, that matters!” Wouldn’t you know it? There is a whole literature about estimation when we know the model is sort of wrong. Here’s part of it and the links!

The Paper: March 2026, Econometrica: https://www.econometricsociety.org/publications/econometrica/2026/03/01/Optimally-Transported-Generalized-Method-of-Moments

Free ArXiv Link: https://arxiv.org/pdf/2511.05712

Generalized Method of Moments

To grok the paper, we have to start with the kind of estimation problem we’re looking at. It’s a Generalized Method of Moments problem, a very common method in economics, but the thing about statistics is that many fields teach it and they all do it a little differently—mostly by historical accident. I think I’d like this framework, even if I wasn’t brainwashed into liking it by my education, but I guess that counterfactual is unobserved… It’s a nice, flexible way to look at the empirical implications of a model.

Suppose we have a vector of parameters b. The Generalized Method of Moments (GMM) thinks about problems where the parameters are identified by “moment restrictions” like this:

You can write a slew of common estimators like this.

• OLS: E[X(Y - X’b)] = 0

• IV: E[Z(Y - X’b)] = 0

• Maximum Likelihood: E[ (dlog f/db)(X, b)] = 0

• Quantile regression: E[X (p - 1(Y <= X’b))] = 0

• Etc.

The idea is that Models produce moments, so by finding the model parameters that satisfy the moment conditions, we identify the version of our model that matches the data.

For another example, in an older post, I used GMM to build this estimator that reduces the variance of quantile treatment effects in experiments.

Usually, we estimate the model by solving some version of this problem:

For some weighting matrix W. The estimator is consistent for any positive definite W, so the choice comes down to efficiency.

Overidentification

But sometimes models produce more moments than we need to estimate the parameters. This “overidentification” is useful because it allows us to use the extra moments to test whether the model is true. Intuition: a subset of the moment conditions pins down the model parameters, but we still have other moment conditions left over after fitting that we can plug the model parameters into to see whether all the restrictions can hold simultaneously.

But what if the model fails the test? What if it’s rejected by the data? It’s not terribly surprising that a model doesn’t hold exactly. So, how can we decide whether the model is wrong in the sense that it’s a simplification of a messy reality or if it’s wrong in the sense that we’re missing something critical?

One idea that emerged in the literature was: what if we could minimize the distance between our modeling assumptions and the data? We could capture some “pseudo-true” parameter, i.e., the most realistic parameter for the model given the data.

The parameter estimated by the GMM procedure above doesn’t really reflect that goal. It finds the parameter that minimizes the weighted norm of the moment conditions, which can have strong efficiency properties if the model is correctly specified. But if the model is wrong? Well, then the GMM estimator doesn’t really connect well with intuition about a parameter that minimizes the distance between the data and what the model requires. It’s just minimizing the norm of the moment conditions.

Empirical Likelihood

One idea in this vein is known as Empirical Likelihood, or EL. The empirical distribution assigns probability 1/n to each X(i), and we know this distribution doesn’t satisfy the moment conditions for any parameter b, but what if we take the mass points of the empirical distribution and tilt them until the moment conditions are satisfied for some b? What if we estimated the distribution of the data, subject to the constraint that it satisfies our model?

The empirical likelihood estimator does exactly like that. It maximizes the likelihood of the data subject to the moment condition:

Note that without the moment conditions, the solution to this problem is p(i) = 1/n, i.e., the empirical distribution. The non-negativity constraints can’t bind because the derivative of log approaches infinity as its argument approaches 0, so the first order condition is:

1/p(i) = lambda => all the p(i)’s are equal so p(i) = 1/n.

That means that if there were a b parameter that satisfied the moment conditions, we would just set p(i) = 1/n and not make any adjustments to the data. We’re effectively maximizing the log likelihood of the empirical distribution subject to the constraints of our model.

Even though there are n + dimension(b) number of parameters, there are nice tricks for computing the estimator from the first order conditions, i.e.

So,

Summing across i gives (from sum p(i) = 1 and sum p(i) g_k(X(i),b) = 0:

For any given “b”, we can then solve for mu via the equations:

Define the solution as \mu(b).

So we have this lower-dimensional estimation problem:

Optimally-Transported GMM

EL is what the maximum likelihood estimator of the empirical distribution would be if we added our model’s constraints, but this isn’t the only way to get at this problem. What if, instead of looking for the closest distribution to the empirical distribution by adjusting probabilities, we changed the mass points?

So, now we’re keeping the empirical distribution's even weighting but perturbing the data points to make the moment conditions hold, minimizing the distance we travel to “transport” X to Z.

Like EL, this problem also has a convenient solution despite being a high-dimensional optimization problem, but this time based on successive linear approximation, see Algorithm 2.1 in the paper.

The estimator is intuitive. It finds a distribution of the data that minimizes change, as measured by a clear metric that satisfies the moment conditions. You get numbers for Z(i), so you can start asking questions like: if X(i) had to change that much, do I think I’m really on the right path in modeling this? Or: yeah, that seems like it’s just a little off—fine. It’s a nice way to quantify how bad a problem the misspecification is.

Empirical Likelihood can do something similar, but to me, anyway, I have a lot more intuition about variable-space than about probability weights—your mileage may vary. “You’re telling me I have to make revenue numbers twice as large… weird. I don’t know if I believe this. No, I don’t think I do.”

Downsides here are that I’m not sure E[||X-Z||^{2}] is really the loss function we have in mind when we’re thinking about minimizing how much we change covariates—I don’t think it’s crazy or anything, but I think we’re usually thinking about something a little less… “cardinal”, is maybe the word I’m looking for. There are some variables that I can’t imagine being mismeasured, so I’m suspicious of moving them around. The method doesn’t really depend on the loss function here, but once you’re out of the least squares-ish world, you lose the nice computational trick, and you’ve got an n x dimension(X) parameter to optimize over…

One idea I had here was to make the problem a little more specific and, based on that, identify more intuitive restrictions for the particular problem.

For example, the overidentified linear IV model:

And the misspecification we’re worried about is that some of the IV’s are invalid. So, write:

And so, we’ve got:

Maybe the misspecification problem is that U = E + U* where E is correlated with W but U* isn’t, so if instead the world had produced U* as the structural residual we wouldn’t have any misspecification. We could solve for the minimum “E” that solves the problem, like so:

So now we have a structural interpretation of the error. If I had to give any modeling advice… knowing what the error is is the first step toward wisdom…

Anyway, I think there are many other interesting applications of this idea and Empirical Likelihood, but I don’t think I’ve ever seen it used in industry, despite seeing many examples of overidentified IV models rejected by the data. If you’ve got a GMM problem and the time, it’s worth taking a look at these methods to get a sense of the size of the misspecification problem, and, if it’s small, using pseudo-true estimates that balance the model and the data.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Once upon a time, there was a “Two-Language Problem” (which led to the creation of Julia; see the old texts). The problem was that while most analysis took place in high-level languages like Python or R, both were too slow, so most of the core logic of the various mathematical and statistical packages is not actually written in either language. It’s written in C (or in Fortran for many R packages).

Share

One of the main selling points of Julia was that it avoided needing two languages by using a compiler with multiple dispatch to mimic a lot of the niceties of a dynamically-typed, flexible language while still being performant because it could act more like a compiled, statically-type language (each function is compiled once types are known, so the code generated knows the data type).

I like Julia because multiple dispatch and the language's more functional nature fit mathematics and statistics well. I was never bothered by using two languages in a project. It’s actually kind of a nice experience to write the parts of an analysis that need to be performant in Fortran, and handle the data cleaning, etc., in R.

But I admit to liking C and Fortran because they were the first two languages I learned to program in. So, I’m biased.

And yet… like when Nixon said “We’re all Keynesians now”, we’re all two-language people now. The dream of one language has been deferred. The new dual-language world is Natural Language and X, where X is one of those old programming languages that requires you to think logically and carefully about what you’re doing (lame).

Before, the distinction was between a language where you needed to think in bytes and data types and a language where you could be a little more loosey-goosey about such things. Now, the distinction is between a language where you need to think in syntax and semantics and structure, and a language where you don’t.

We won’t be able to ditch structured language entirely. Natural language is a lossy representation of computing, just as the flexible scripting languages before it were. But it loses precision that can’t be recovered by spending performance.

The lossyness of Python and Ruby hurt performance. The lossyness of Natural Language hits business logic.

I predict a language will emerge that solves the modern two-language “problem”. It’ll be a more structured version of Natural Language that adds back some information, while still partially keeping the loosey-goosey feel of “build Spotify, make no mistakes.”

What I have in mind. Imagine you’re building a web app purely with AI today. You’re going to type lots of prompts to fix various things as you debug the app, but that doesn’t really scale for an app of any real complexity. Natural Language isn’t precise enough. In most cases, it’s quicker and better to just fix the code directly once the project is closer to being done.

But let’s say you defined things a little more precisely, then maybe you could keep it in Natural Language longer, i.e., in some kind of MVC-ish language:

• My app has a Model for users. They have passwords and emails and have to log in. Users can’t access other users’ stuff unless it’s been shared with them.

  • We need endpoints for users to sign-up, log-in, log-out,…

• My app has a Model for books. Users can write books or upload them as .epubs and share them with each other. There is an in-app reader for books.

  • We need endpoints for …

• Etc

So, I’m specifying the Model and a bit about the Controller components of MVC, and letting the LLM fill in what the views and controllers look like.

This scaffolding holds a lot more information than “clone Kindle.” I can add a model for pages read and mention that I want to bill by that metric rather than fixed prices, and I think the LLMs would do a decent job.

But this doesn’t really approach a maintainable language. I think we’ll get there. Someone will come up with a language that uses this newfangled LLM thing to both keep the expressiveness of Natural Language and recover some of the precision of structured language.

Or at least that’s where I hope it goes, because I doubt the old structured languages will ever gain market share.

(Alternatively, we’ll drown in endless AI slop, where nothing is built to last, and the past severs from the future, so that we exist in a never-ending present, neither returning nor progressing, until our souls, which yearn to plant some small monument, vanish into the void…)

I bet the language will look kind of like YAML, if I had to guess.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I love McDonald’s. I eat there 3-4 times a week—down from 5 pre-Covid, which is how we’re all marking time (from “since the birth of God” to “since the Plague”…). I couldn’t do that back in grad school (no money for such extravagances), but now that I have a real job...

Share

I’ve eaten just about everything on the menu locally, but they add things… If you’ve seen the awkward vid of the McDonald’s CEO eating a hamburger, you might’ve caught that what he’s promoting is the Big Arch, a new burger. The idea, if you listen to this great segment from the WSJ (free on YouTube) with the CEO, is to try to capture a slice of the premium burger market.

It’s going to work. Now, I know it’s in keeping with this blog’s usual thing to do an actual analysis with some math and a little theory, but this time I’ve got a better, more convincing reason. I tried it tonight, and I have news to report:

It tastes like the fruit must have tasted in the garden.

Dear God.

Homer sang of the Lotus Eaters, a people who were lulled into a state of perfect apathy where they forgot their homes, their goals, and their loves upon tasting the fruit of the Lotus and desired only to eat it and bask in the bliss it offered for the rest of their days. I now understand why such a life might be desirable.

In a world where everyone’s out there making the robots write their CRUD apps, it’s a breath of fresh air to see true innovation in the wild.

Let’s get into specifics:

• The patty is truly big. Much larger and thicker than you’d expect from McDonald’s. A bit off-brand in this respect, but it’s so well done. Delectable.

• They have these little fried onion things for some crunch. It crunches.

• The sauce is very Big Mac-ish, so if you’re a fan of that (like me), you’ll dig it.

• I was almost full before the fries (I eat in order: first burger, then fries. I’m not an animal).

This is my true and honest review. Sadly, McDonald’s did not pay me for this. If they want to send money, I won’t decline. I am 100% for sale. I want to be clear on that.

The Big Arch will be my go-to for as long as they have it. Fabulous work, gang.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

We all know that Data Scientist is a silly name. What kind of science doesn’t use data? Vagueness is the first sin of good science, and we’re putting it right in the name!

Share

I’m proposing a new job title: Table Reader. Now, I know what you’re thinking. “Table Reader is reductive. I don’t just read Tables! I do other things!” Okay. I’m sure you do. Really. I believe that is your truth.

By “Reader”, I do not mean a “Book Reader,” of course. I mean it in the Mystical sense. You aren’t “Analyzing” (vague and Latinate). You’re taking a Reading, tracing the palm lines of the Table and considering what the flight paths of birds might have to say (using weather as an instrument). Google Sheets has been missing a spiritual layer.

Data Scientist doesn’t say anything about what we actually do. Table Reader evokes.

A question arises. You consult the Tables for guidance, perform the Rites, Read the signs, and tell folks what the Tables say.

Each word is a strict improvement. Table is better than Data because it specifies the kind of data we look at. Data covers a lot more than tables, but the job doesn’t.

Reader is better than Scientist because it evokes Magic and Mysticism, which is both more inspiring and accurate. I’m not saying the Art has a lot in common with Divination, but it has more in common with that Practice than Chemistry.

For example, a computer is a Technology. If we both give it the same instructions, it’ll do the same thing. My nature and experience don’t affect the way it operates.

Magic is different. If the extensive literature on the working of Magic has established anything, it is that the Practice is particular to the individual. For example, Magic can be acquired via being born a wizard (Rowling, J.K. 1997), or you can be gifted powers by a strange lady who spends a lot of time in the woods (Map, Walter c. 1220), or be possessed by the Gods (Pythia, c. 1400BC), or learn to speak the language and meanings of the Nonmen, all but destroyed in the First Apocalypse, when the No-God, Mog-Pharau, rose millennia ago (Bakker, Scott R. 2004)…. we’re getting off track.

The point is that if we give the same Tables to two Table Readers, we’ll get different answers. It requires experience and intuition to know which signs are real and which are misleading, which models “make sense” and which don’t. So, Table Reading is an individual Art.

But, you say, “There is Math involved!” Sure, the Practice may use aspects of Science or Technology, like Mathematics, but consider:

If an Augur foretells the future by learning the biology necessary to identify the different parts of an animal’s entrails and from these gains insights into the length of Caesar’s reign, do we say he is doing Science? In part. But the Science is only necessary to construct the components of the Magic. The Science serves the Magic. So, clearly, Augury is a Magic.

In a similar way, the Science serves the Table Reader’s judgement as he or she divines whether the treatment is good or bad, whether to run this in logs or levels, whether we’re going to make next quarter’s targets or not, what kinds of fixed effects to use, and whether there are any plausible excuses to explain this product’s failure that should not obviously have been foreseen by you or your stakeholders (I kid, I kid).

So, we need a name that evokes Judgement and Practice, not Science: Table Reader.

I also considered “Table Teller”, but it sounded too authoritative, too crafted. In the good version of our profession, we merely read what the Tables tell us. We don’t instruct the tables, like the Medium doesn’t instruct the Spirits. We listen to Them.

Thanks for reading!

Zach

Connect on LinkedIn: https://linkedin.com/in/zlflynn.

We’re going to talk mainly about the general problem of understanding censored relationships, but specifics make generality easier to grok…

Share

Suppose we want to understand how the time until a Free/Basic Plan customer signs up for the Premium/Ultra/Mega Package varies with different actions or incentives the customer takes or receives. For customers that do eventually sign up for Premium, we observe the time it took them to do so from when they first signed up for the Free Plan. But for customers that haven’t signed up for Premium, the only thing we know about their Time to Premium is that it is greater than however long it’s been so far, i.e. we only have a lower bound on their Time to Premium. We’re missing data.

Most of causal inference is about making assumptions to fill in missing data. We don’t know what so-and-so would have done if they hadn’t been treated, so we find people who weren’t treated and say they’re kind of like so-and-so and fill in the treated subject’s missing outcome with theirs. In censored problems, like this one, we come up with an assumption about what the censored data would look like.

The classical methods (recall that “classical” is math-speak for “lame”) use parametric assumptions—impose a functional form on the censored variable’s distribution—to make the censored variable informative about the full distribution.

The classical example looks like this:

Where we only observe y* = max{y, 0} and y is unobserved. b and s are the parameters to estimate.

The parametric assumption, that y is normally distributed, allows the uncensored data to be informative about the distribution of the censored data because we know:

And we can identify the parameters of this distribution just from data on (y*, x). The normality of e allows us to connect the error from the uncensored data to what the error would be on the censored data. But this is a very strong assumption and we’re essentially letting functional form tell us about the extent of the censoring problem. Not very satisfying!

What is the most we can identify without this kind of functional form assumption?

What can we learn about how the distribution of a latent, censored variable changes with a covariate without making distributional assumptions about what we can’t see?

Fundamentally, the question of how the distribution of y changes with x is a question about how quantiles change. So, write:

f is the conditional quantile of y given x.

Let v be a uniform random variable drawn on (0,1). Then, here’s a random variable that has the same distribution as y given x = x:

So, if we can identify this quantile function, we’ll know a lot about how x affects y.

The problem is that we don’t observe y, so we can’t just estimate the model using our favorite quantile regression method.

Instead, we observe y* = max{y, 0}.

What we’re going to do is show that for a certain set of (x,t) we can identify f(t,x). Maybe that set is good enough that we don’t need to make any other assumptions. We can answer whatever question we’re asking the data just from identified points of the quantile function. If we need more, then we can extrapolate, i.e. make the dirty parametric assumptions, but we can do so, starting from points that are identified without having to make such compromises.

Whenever I’m thinking about how to identify something, I start with the distribution function. The reason to start there is that it fully characterizes the information available in the data. So it gives you the most identifying power to play with.

The data we have available to us is (y*, x). So let’s do a little algebra and see what y* given x tells us.

Let p(x) = pr[f(x,t) + e(t) ≥ 0 | x] be the probability of not being censored, conditional on x. The nice thing here is that we can estimate this function from the data because we know whether each observation is censored.

Let C be the event of an observation being censored. Then we can write the above as.

At all points on y*’s support, u is nonnegative because y* is nonnegative, so:

Let’s substitute in the true quantile function of y for u here:

Because f is the conditional quantile function, we know:

So substitute for that term in the conditional CDF of y* expression:

Now, we’re cooking. We’ve got the true quantile that we want (t) plus a bias term introduced by the censoring. It turns out that we can actually identify the set of (x,t) where the bias term is 0.

Suppose that for a certain (x,t): p(x) = 1 - t + c where c > 0.

Because:

We must have that f(t,x) > 0 because c > 0.

If the data is censored, then y < 0. Because f(t,x) > 0, the probability of y being less than f(t,x) given that y is censored is 100%. So:

So, for any (x,t) with sufficient probability of not being censored, we can identify the conditional quantile of y given x just from the conditional quantile of y* given x.

Cool. What does that buy us? This tells us the set of quantiles and covariate variables for which we do not face a real censoring problem and can identify the quantile function directly from data on (y*, x).

Unfortunately, this will usually not be enough. We’ll need to know how the quantile function responds at other (x,t) besides the ones that aren’t identified.

My preferred method for doing this is to do one of the following, depending on what you need: simply parameterize f(x,t; r) with a vector of parameters r. This form of parametric assumption is more natural and easier to motivate than assumptions about the functional forms of residual densities because you’re choosing how a response function varies as the quantile increases and as covariates change, and the shape is strongly informed by the actually-identified points instead of being a functional form assumption without strong justification, i.e. in a simplified example: suppose you had the following point-identified points and one non-identified point:

It seems sort of reasonable to think it’s roughly linear with x in this example so that x=4 should have a median around 4. The point is that you can use the nonparametrically-identified shapes to inform what parametric restrictions you put on the problem to identify the rest.

Anyway, this is just a tool that I think is less well-known than it deserves to be. So, I’m sharing it. Enjoy!

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Look. The first thing I’m going to do is hook you. I’m going to tell you this is about AI and LLMs. You have to believe me that we’ll get there. If, along the way, this starts to look like it’s about time series regressions (“boring shit”), just trust me. Believe me. Keep the faith.

Share

Computation and Statistics

Suppose we have a time-series regression (remember: trust me) where we don’t know the lag structure.

We find the “best” model for the time series in terms of out-of-sample prediction error by using the following computational procedure:

1. Fix an L, require that p cannot be greater than L, i.e. for every p greater than L, r_p = 0.

2. For each of the 2^L candidate models, compute the cross-validation score by leaving out future data and evaluating how well the model predicts it.

The reason we fix an L is that 2^L gets awfully large awfully fast. Computationally, we just can’t evaluate that many models. L models our computational constraint. We have two possible scenarios:

1. The optimal model selected via this criterion has max{p} = L.

2. The optimal model selected via this criterion has max{p} < L.

In scenario 1, we are computationally constrained. If we could increase L and make our model more flexible, our predictions would improve. We just need to call up NVIDIA and get more, better chips.

In scenario 2, we are statistically constrained. The problem isn’t that we don’t have enough computational power. The problem is that, given the data available to us, the best models we can construct are less flexible. We need more data before we can improve the model. NVIDIA can’t help us. If we spend billions of dollars on compute, the return on it will be zip, zilch, nada.

LLMs

The key question of how much core AI capability will improve over the next few years is whether we are computationally or statistically constrained.

Until now, LLMs have been primarily computationally constrained. When we got better chips, when folks spent more on compute, the result was better models. But I think anyone who’s used the tech over the past few years has noticed the core technology leveling off.

The difference between GPT 3.5 and 4.0 was enormous. I recall asking 3.5 to explain statistical significance in a stakeholder-friendly way, and it responded with gibberish that was so bad it didn’t even rise to the level of being wrong—and to a simple, textbook question! 4.0 was so much better that it boggled the mind. But 5.x… It’s fine. I’m sure it’s better, but I hardly notice.

I have no inside info, but it does make me wonder whether we aren’t moving into the statistically constrained regime—or, at least, will be in the next couple of cycles. And if we are… it’ll be very difficult to improve the core model. Throwing more parameters and more compute into the mix won’t change the mathematics.

Unlike other technologies, the returns to more data are not increasing. They decrease rapidly. You need exponential increases in data to generate sublinear improvements in performance.

I think we’ll run into the statistical constraint sooner than the market’s priced in. Word prediction is a hard game. The difference between the right word and the wrong word is a big thing.

The difference between the almost right word and the right word is really a large matter—'tis the difference between the lightning-bug and the lightning.

Mark Twain

In math, the loss function is steep. Small errors are big errors. A steep loss function, combined with a very high dimensional, nonsmooth problem, implies we’ll need a lot of data. We’ll have to get it from somewhere.

I think I can see how it happens.

Skills

We can’t generate vast sums of additional, real data. The AI companies are already using as much as is plausibly available. So, we need to make better data, not just more. Structured data. We have to add single observations that are worth billions of unstructured rows. We need to generate data for LLMs rather than rely on data originally generated for other purposes.

This is where folks are going with “skills” and related ideas. Provide tight context in a structured format to give LLMs data that is worth a vast amount of unintentional data.

Going back to our time series example. Suppose we knew that there was a seasonal component that hit every four time units. So all the models that can’t capture that structure are right out. This tells us information that might take a long time series history to discover from the data alone.

In other words, skills are parameter restrictions. If the restrictions are true, they dramatically improve model performance.

In five years, I think the vast improvement in perceived LLM performance will come from these kinds of purpose-built skills, not from core model performance. The core model will soon run into statistical constraints, if it isn’t there already, but there will be skills that help it write better code, better emails, etc, by applying rules and logic that humans already know in a structured way.

Like in classical statistics, the way to learn more from a fixed data set is to make more credible assumptions about the data-generating process. Give the problem structure by applying what you know about how the world works, and stop requiring the data to learn everything.

The trick is to tell the data what to think when you know what it ought to think and to let it tell you what to think when you don’t.

Anyway, I think that’s why it’s going where it’s going.

Fewer chips, more context.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Look at my website: https://zflynn.com

I live in Lake County, Illinois, just north of Chicago. That fact hasn’t mattered for the past six years — except at tax time and whenever I look at that obscene property tax bill (obscene in the old sense: indecent, an offense to good morals, perverse).

I’ve worked either remotely or from home since moving here a month before COVID. I’m not from here, and my daily wanderings to Walgreens, Target, and the occasional restaurant (McDonald’s) do not exactly immerse me in the local culture.

None of the work I do “contributes” to the local economy. I’m pretty sure I’ve single-handedly put the Dominos guy’s kid through college, but that’s via my consumption, not my labor.

I’ve never worked for a company with a local focus. Why would you hire a data scientist if 1-5% revenue increases wouldn’t justify the salary? You can’t make a product good enough via data science. You can only make it better once it’s good enough. You’ve got to see some action…

All of which is to say, I have very little connection to where I live—and this is the place I’ve lived the longest since leaving home! Six years! I don’t keep up with local politics or have opinions on the Parks District race. But I want to! I like living here. It’s lovely for one thing, and there are rabbits that follow me around whenever I go outside (because I give them food).

A subspecies of science fiction talks about this. Disconnection from the physical. Folks augment themselves with machines, becoming human-shaped Ships of Theseus, severing the self from the body and the physical environment. The protagonist does this to himself in search of security, riches, respect, or meaning, but it causes problems. The writers solve these problems in different ways, but part of the solution typically involves an attractive woman with a katana, some unresolved issue with her dead father, and an alternative aesthetic. She reminds him of what it means to be human. That there’s some flesh you don’t want to part with even for cybernetic glory…

Okay, I’m stretching the analogy. What I’m trying to say is that I think it’d be good if where I lived affected what I do in some small way. So, I think I’ve got a solution...

Forecasting models.

Hear me out:

Respectable people forecast the national accounts, unemployment rates, inflation rates, etc. They put vast sums of money on the line, betting on how those series will move. We pay less attention to local forecasts, which, as far as I can tell, are primarily generated by a city or state paying some consulting company to do it.

The kind of person who would execute a weekend project to forecast local economic conditions, without hope of financial reward, clearly could not be called rootless. No, he would have to be very attached. A pillar of the community, even. You see my plan.

So, I started this project that I’m calling ZILCAST, Zach’s Illinois Forecasts, to forecast various things of local concern to Illinois and the Chicago area. I started with the Illinois Unemployment Rate because it’s an easy series to understand and build my model on, but the goal is to expand this to local series that are a bit off the beaten path.

The mathematical idea is to create solid local forecasts using a kernel-weighting strategy that borrows data from similar locales, reducing forecasting error from otherwise noisy series. It also uses (frequentist) model averaging to average over different model types, lag structures, and bandwidth selection strategies.

I’ll write up more about the statistical method at some point, but I put it all together technically over the last few weekends. You can watch my forecasts evolve, fail, and succeed here: https://zflynn.com/zilcast. The forecasts and actuals for the Illinois Unemployment Rate will update as the BLS posts more data.

The tech stack is simple but fun. I wrote the whole workflow from fetching the data to running the fancy regressions in R, which I think has been devalued far too quickly in tech-land, but that’s another post. I run the actual forecasting on AWS because it’s fairly computationally-intensive, but the whole thing is “orchestrated” via cron jobs running on my laptop. I archive the results on this tiny Postgres box I use for all my side projects.

Here’s the current median forecast for 2026 (the latest available data is December 2025; it should update soon with January 2026 actuals).

It would have absolutely crushed in 2025, except at the end. You should always be skeptical of backtests, but the forecast below is from data up to December 2024. The gap in the series is because the BLS didn’t produce the data for October 2025 because of the budget “issue”.

Anyway, I hope my little attempt to connect to where I live via fancy regressions is useful to at least a few people! Check it out at https://zflynn.com/zilcast.

Thanks for reading!

Zach

I’m on LinkedIn at: https://linkedin.com/in/zlflynn

And I have a normal, personal website at: https://zflynn.com

PS. If you’re a local government that’d like a forecast like this for planning, etc, feel free to reach out. I’m looking at what would be most useful to add to the site.

Instacart recently ended customer-level price tests.

This blog is a reaction to that, but it’s also an attempt to change attitudes towards these tests more generally. Thesis: price tests are good and make the world a better place. Bad philosophies of “fairness” make us poorer than we have to be.

I also have a take on press releases.

What we’re talking about

A customer-level price test works like this: we randomly assign some customers a price vector (a mapping from each product to its price), and assign the other customers alternative prices. We then run the experiment for a few weeks and see which prices make more moo-lah.

These tests are extremely powerful because not only do we learn which prices perform better, we learn how elastic demand is near current prices and, if designed correctly, how various goods substitute for each other. This helps us move prices in the right direction, getting closer to maximizing revenue given demand.

What’s the objection

The objection — as I understand it — is that folks don’t think they should pay different prices for the same good at the same time. This happens every day in an unending list of industries, but it’s an “ick” in other industries, evidently including consumer packaged goods and groceries.

Why do I care what other people pay? I care what I pay. You tell me the price, and I decide whether to buy the thing at that price. Why would it matter to me that someone else got a “better deal?” Good for them?

Another theory is that businesses do this to charge customers higher prices. If they wanted to charge higher prices, they could just go ahead and do that! They don’t need to run an experiment! In fact, if they are considering increasing prices, running an experiment decreases prices in expectation. Without the experiment, they’d just increase prices. That’s their prior. With the experiment, they might learn, “Woah, we can’t increase prices.”

Whenever I think an argument is silly, I come up with a model of the world where I’m wrong, and the argument works. There’s always a model. I think this helps to avoid dismissing things reflexively. Sometimes, an argument only sounds silly.

My best attempt to steelman this: yes, consumers can be better off if firms are ignorant.

Steelman: Is ignorance bliss (for consumers)?

Suppose we’ve got a monopolist — the worst case because they can fully exploit the demand curve, if they know it — do they price higher or lower if they have full information?

Chalkboard model. Our monopolist produces a good with zero marginal cost. They face a linear demand curve:

Q = C— DP

Now, suppose that our monopolist doesn’t know this demand curve. They have some beliefs about it, of course, but they don’t know it. They’re risk-neutral. They maximize expected profits.

If they knew C and D, they’d just replace the expectations above with the truth.

So the monopolist price with perfect knowledge is P* = C/2D, and we have that:

On average, is the perfect knowledge price greater than the imperfect knowledge price?

Let’s assume the monopolist has rational expectations. Then, this is just a question of whether the expectation of the ratio is greater than the ratio of the expectation.

If our beliefs about (C,D) were independent, for example (unlikely, but just to make the point), then, on average, the monopolist price with perfect knowledge would price above the monopolist under uncertainty because…

D>0, so the Harmonic Mean is less than the Arithmetic Mean, and we have:

So: E[C/D] = E[C]E[1/D] (by independence) ≥ E[C]/E[D]

Ergo, the monopolist prices higher with knowledge (on average).

I’ll emphasize on average one more time because, even in this case, it is still possible for C/D < E[C]/E[D] (perfect knowledge pricing is less than pricing under uncertainty) to happen frequently.

This is only a special case and depends on many detailed assumptions about the nature of uncertainty and how it affects demand, but even so…

But what if we have competition?

We get a very different result when we combine risk aversion and perfectly competitive firms.

This part comes from Sandmo (1971). I’m stealing it whole cloth. It’s a good paper. You should read it.

Suppose we have a competitive industry, i.e., firms take prices as given, but they are uncertain about what those prices will be. Firms are risk-averse, and they have a baseline income M. Firms choose output to solve:

Where P is random with E[P] = r. U is increasing and concave (firms are risk-averse). C is increasing and convex. M is the baseline income, and F is the fixed cost.

The first order condition of output choice is:

Subtract E[U’ x r] from both sides to get:

Write profits as V = M + PQ — C(Q) — F = E[V] + (P — r) Q.

So, suppose P ≥ r. Then, V ≥ E[V] and because U is concave:

Because P ≥ r, we can write:

In fact, this inequality holds for all P because if P < r, then U’(V) ≥ U’(E[V]) but (P-r) < 0, so the above still holds.

Okay, so we can take expectation across P on both sides of the inequality:

Going back to the first order condition, this means:

E[U’(V) (P — r)] = E[U’(V) (C’(Q) — r)] ≤ 0, which since U’ > 0 must mean that at the optimal Q,

C’(Q) ≤ r

i.e., at the optimal output, the marginal cost is less than the expected price. If prices were known with certainty, the firm would choose output to set marginal cost equal to the known price. Because marginal cost is an increasing function of Q (more or less necessary for price-taking models), this means that:

1. Uncertainty about price reduces output.

2. Equilibrium prices will be higher because firms restrict output relative to perfect knowledge.

i.e., consumers actually want firms to know more! It leads to lower prices!

Note. It’s a little funky to think about price uncertainty. It’s more natural to think of firms picking a price and seeing how much demand they get. But this model works at the limit of competition, so output is the firm’s only choice.

Policy

So, depending on our model of the world, we get very different implications for consumers when firms learn more about the demand curve.

The point is that it’s far from clear how firms having better information plays out, and, in any case, we shouldn’t base policy (governmental, social norms, or biz decisions) on the idea that firms will remain ignorant of their demand curve. They will learn it from experience! They don’t need experimental data to learn demand. It’s just a better and more accurate way to do so. Enforcing ignorance is a bad solution to a market power problem…

Profit-maximizing prices matter. They tell folks where to put capital. They tell us how valuable a product really is. They allocate resources. The right prices matter for more than the individual firm trying to make a buck. When resources are misallocated across an economy, it lowers aggregate output, lowers productivity (because capital matches with the wrong labor/other factors), and prevents economic growth. Sad!

Experiments give us something clear. A solid ground and a judge of what works and what doesn’t — instead of having to base all pricing analysis on tortured regressions or highly parameterized discrete-choice models.

Consider what would happen in the absence of experimentation. If firms don’t experiment, they’ll use non-experimental analysis methods to set prices (this is what most pricing analyses are exactly because of the Instacart experience — there’s this vague sense that randomized pricing isn’t “fair”). If this works well, then it’ll lead to the same outcome as the experimentation approach. So, there’s no harm in the firm experimenting.

If the assumptions underlying the non-experimental analysis are wrong, who knows what will happen? Maybe we get lucky, and firms don’t recognize their full market power because of their ignorance. Maybe resources are misallocated. Maybe the firm underprices, and a valuable good or service doesn’t get the capital it needs? Maybe the firm overprices? Who knows?

Does more information make pricing more pro-consumer or less? It depends. But I find it difficult to imagine that an economy operating under greater uncertainty is better than one operating with more information, all else equal.

Press Releases

I’m going to close by discussing an exciting new topic, not yet explored on this blog. Press releases.

“At a time when families are working exceptionally hard to stretch every grocery dollar, those tests raised concerns, leaving some people questioning the prices they see on Instacart. That’s not okay — especially for a company built on trust, transparency, and affordability.”

?

Why can’t we write this?

“We enabled our retailers to run price experiments where one customer might see a different price than another customer for the same good at the same time. Our customers said they didn’t like that. The upside of running those experiments isn’t worth the blowback. So, we won’t enable our retailers to run price experiments like that anymore. Thanks.”

Everyone reading knows that’s the truth — and it’s a fine truth! There’s nothing wrong with it! Not saying it clearly and introducing vague “concerns” just creates this sense of impropriety where none exists. The reader wonders how Instacart’s pricing experiments hurt families struggling to pay for groceries. Why are people questioning the prices? Were the prices “fake” in some way? Etc.

Not to bring up politics, but this is one thing Trump did effectively in the 2016 GOP Primary. His opponents would catch him contradicting a previous position, and then he would say, “Yeah, well, I changed positions because this one polls better.” And his opponents were caught flat-footed. “Wait, you can just admit that? You don’t have to come up with a fake justification for changing your position?” Of course! It’s obviously true!

You wanted us to stop it, so we’re stopping it. End of story.

(I realize I just lost everyone with the absolute unit of a hot take that “we should handle PR like Trump,” but I only mean in this specific way! We should feel less shame when we have nothing to be ashamed of. Democracy requires aligning yourself with voters. Business requires trying things to make more moo-lah.)

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Visit me on the World Wide Web: https://zflynn.com

My most Luddite opinion (tough competition): AI art won’t work. Not even if the models get better. Beauty isn’t enough.

No one talks about art without talking about who made it. It’s an essential part of the thing. We connect with it because we know someone made it. It matters who Van Gogh is. We put ourselves in the artist’s shoes, or they repulse us. We react. You can feel what they do and identify with it, or be stupefied by an experience so alien to your own… If you know it’s a machine, the whole thing falls apart.

There’s a genre of pro-AI Art take. The writer points out that people can’t tell the difference between AI-generated images and paintings by human artists, but the test, like the Turing Test, misses the point. It’s not whether people can be fooled that matters, but what happens when they know the truth. We’ll know the robots are true intelligences when you can tell a human being that they’ve been talking to a robot, and they respond, “So? He seems like a nice fellow.”

If you show folks two pictures and tell them AI made one and a human artist made the other, they’ll choose the human one — and mean it. It’s not just loyalty to carbon over silicon. It’s a sense of connection. They know that any artistic insight generated via AI comes from endless training data and some opaque process of aggregating that information, not from a particularly foggy morning or falling in love.

It’s an uncanny valley: the models produce something that looks like art, but it has no biography, no connection, and look, frankly, the robots have never smoked clove cigarettes or said anything unbearably pretentious but also sort of true, so I don’t think they’ll make it in the art world.

Zach

LinkedIn: https://linkedin.com/in/zlflynn

Website: https://zflynn.com

Udemy course: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/

You might have heard we don’t “accept” the null hypothesis. We merely “find no evidence” to reject it.

There’s a version of the world where this framing of statistical inference makes sense: where the chief task of the researcher is to gather information about the world and conclude whether it can tell us anything, and if not? Oh well. We’ll get ’em next time.

This world exists, and we’ll vacation there in this blog, but I’ve never lived there, and you probably don’t either.

When you are the policymaker — even the policymaker’s assistant’s understudy — you can’t avoid… making policy. You either ship A or you ship B.

If you “found no evidence to reject the null,” and shipped A, then you looked at the data and decided that A was better than B by some metric. Why else wouldn’t you go with B? So, there’s a very real sense that you “accepted” the null.

The distinction is between decision-making and fact-finding.

Suppose we’ve got T = E[Y(B)] — E[Y(A)], a standard average treatment effect, and, if we knew T, we would want to do action B if T > 0, and action A if T < 0.

T > 0 is either true or false. It’s a fact. So, one framing of statistical inference: do we have enough evidence to conclude T > 0 is true? If we take this approach, we set up a hypothesis test with the null hypothesis that T ≤ 0 and evaluate whether to reject it. From this perspective, it really does make sense to say, “Oh well.” In fact, we must be willing to because we can’t assume there’s enough evidence to prove T > 0.

But this is an unusual way to state the problem if we’re deciding between A and B. We can’t avoid making a choice, so it doesn’t make sense to talk about the data “not having enough evidence” for us to decide. Some datasets support choice A, and others support choice B.

That isn’t to say the hypothesis testing framework and statistical significance aren’t useful. They are. They’re a nice way of quantifying how much status quo bias we want to incorporate into our decision (and we should have some: we have more experience with the past and more knowledge of its risks — better the devil you know…).

Statistical inference, like art and morality, requires us to draw the line somewhere. The hypothesis testing framework helps us find where to draw it.

But the key is to recognize that we aren’t actually testing a hypothesis. Not really. We’re not finding facts. We’re making decisions. So, we do accept nulls and alternatives as we please.

A side-benefit of treating statistical inference as a decision problem is that it’s much easier to explain. We’re looking at this dataset and deciding whether it favors A or B. You don’t need to try to explain the subtle asymmetries of nulls and alternatives.

Nice!

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Udemy course: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/?couponCode=928E40B177B5DB6F5266

If you would like my help with something (involving statistics, I should clarify): https://zflynn.com/consulting.html

I went through a few phases in grad school where I thought certain papers had the answer to everything — in particular, to the most critical question: “What should I write my thesis on?”

One of those phases sent me down this rabbit hole on Aghion and Howitt (1992) and the inverted U-shaped relationship between innovation and competition, creative destruction, Schumpeter, and all that.

Recently, Aghion and Howitt won the Nobel — largely because of that paper and research program.

So, we have two factors coming together:

1. I know the paper well because of the hours I spent thinking and writing about it — generating zero usable output — at many Madison-area coffee shops circa 2014.

2. I have a blog.

So, I have to write about it to cash in on those sweet Nobel clicks — maybe? We’ll see. In any case, it’s a great paper, and you should know about it.

The paper wrestles with the question of how innovation drives economic growth.

Economic growth is the most important thing in the entire world. And that’s understating it.

Throughout most of human history, economic growth measured by something like GDP per capita was anemic by modern standards. 2% in a year would have been considered insane progress. And it kind of is: 2% growth means the economy doubles every thirty-five years. Not doubles because of inflation — doubles in the sense that we really produce twice as much or stuff that’s twice as valuable. Nowadays, that’s the growth rate of even already very advanced economies like the United States.

Why? What changed? And what causes economic growth in any case? We only have one Earth and whatever resources God and the stars deigned to give us. And ourselves. So how do we produce more per person from the same materials and the same people?

At first, the answer seems obvious. Technological progress. We get better at making things. And, in fact, that answer is correct, more or less. There’s an old model that already won a Nobel Prize — before Aghion and Howitt even wrote their paper (well, maybe it was already a working paper, publication takes time) — by Mr. Solow.

Solow showed, among other things, that, in the long run, all economic growth comes via improved productivity.

But what is productivity?

Well, stuff we can’t measure and don’t know about. Factors of production besides capital and labor.

So, productivity is a great answer but a partial one. Economic growth can’t be driven by technological progress because progress is itself an outcome. It begs the question: why is there technological progress in the first place?

And so, economists started a literature on “endogenous growth theory,” which tried to — you guessed it — endogenize growth, i.e. make growth an outcome of an economy, not an exogenous technological improvement.

Aside: Economists as a species are horribly uninspired when it comes to naming things. Machine learning types would have written about Real Economic Learning or the Progress Function or something that would sell. There’s room for improvement in the Econ marketing department.

Aghion and Howitt (1992) is part of that literature. They propose a new avenue for economic growth: Creative Destruction.

Reading Schumpeter (1942) is the first step toward radicalization:

“This process of Creative Destruction is the essential fact about capitalism. It is what capitalism consists in and what every capitalist concern has got to live in… The problem that is usually being visualized is how capitalism administers existing structures, whereas the relevant problem is how it creates and destroys them. As long as this is not recognized, the investigator does a meaningless job. As soon as it is recognized, his outlook on capitalist practice and its social results changes considerably.” — Joseph Schumpeter (1942)

“The fundamental impulse that sets and keeps the capitalist engine in motion comes from the new consumers’ goods, the new methods of production or transportation, the new markets…. [This process] incessantly revolutionizes the economic structure from within, incessantly destroying the old one, incessantly creating a new one. This process of Creative Destruction is the essential fact about capitalism.” — Joseph Schumpeter (1942)

Model Setting

Aghion and Howitt’s model has three kinds of goods in it:

1. Labor

2. A Good People Want To Consume

3. An intermediate good used to make the consumption good

The world consists of a continuum of people who live forever (nice!), and don’t mind working (no disutility from labor).

There are three kinds of labor folks can do:

1. Unskilled labor, which can only make the Consumption Good.

2. Skilled labor, which can do “research” or produce the Intermediate Good.

3. Specialized labor, which does “research”.

A key simplifying assumption here is that the mass of each of these three labor types is fixed at (M, N, R). Note that this still allows for different research intensities because the Skilled Labor can choose to produce research or intermediate goods.

Because unskilled labor is fixed and supplied inelastically, we can write the production function for the consumption good without including it directly, i.e. y = A F(x), where x is the flow of the intermediate input and A is current productivity.

The flow of intermediate input is determined linearly by the amount of Skilled Labor working on the intermediate good (L): x = L.

Research produces innovations arriving randomly via a Poisson parameter that depends on the amount of labor allocated to work on research, i.e. q x g(n, R) where n is the skilled labor working on research.

This is the key part. When a new innovation arrives, it increases productivity by a constant factor A’ = v A, but the new innovation requires a new type of intermediate input. To be more productive, we need to do different things than we did before.

Property rights are perfect so that the innovative firm can capture the full rewards from the more productive technology, and the lower-cost product immediately monopolizes the industry.

There is only one firm active at any given point in time, and it’s a monopolist — but only until a new firm comes along with a better production process to replace it.

The monopolist, however, is not the only firm. There are other firms, waiting in the wings, deciding how much to invest in research to have a chance of being the firm to produce the next innovation.

The beauty of this model is that it’s trying to capture a very complex problem: how economies evolve with an endogenous innovation and, yet, at any point in time, there’s really only one decision for the economy to figure out: how much skilled labor to allocate to research and how much to allocate to production.

The model had such a great impact exactly because of this setup, I think. It’s an elegant way to model out and think seriously about Creative Destruction (discussed in economics since at least Marx, who viewed the process as eventually leading to the end of capitalism, etc).

Okay, so that’s the setup. What do Aghion and Howitt do with it?

Implications

So, first, we have some direct comparative statics about how much research skilled labor does in equilibrium.

Research increases when interest rates fall (because the marginal benefit of research rises as current consumption can be delayed), as the size of the innovation (v above) increases (again, because the benefit of research increases), and if innovations arrive more rapidly. That more rapid innovation leads to increased research is not obvious because more rapid innovation cycles decrease future profits from monopolizing the industry, so the marginal benefit of innovation falls. However, it turns out, the marginal cost of innovation falls more.

But the most interesting result is how the equilibrium research intensity differs from the socially optimal one.

When the size of innovations (v) are large, firms will underinvest relative to the social optimum because the discount rate of private firms is larger than the discount rate of the social planner, who understands that the innovation will have permanent effects on economic growth. The private firms understand they have only a finite lifespan until they are destroyed via competition.

When the size of innovations is relatively small compared to the size of monopoly power, then firms will overinvest because the rewards to winning the research race are greater and longer-lasting. In other words, less competition drives increased innovation!

The paper has a great slew of other interesting extensions to this core idea — you should check it out!

Conclusion

If you haven’t read it, or if you haven’t read it carefully, the paper and the entire growth research program really, is well worth it. Most of Modern Economics is a lot of statistics, demand models, details. This literature steps back and asks the big questions — which can be a nice break from figuring out how to calculate standard errors 

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Take my Udemy course at: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/?couponCode=1DD1B90E5C29DD496967

This blog is written in the context of a centralized Experimentation Team, but it can apply more generally if you are working to expand Experimentation within your particular team, as well.

The first temptation when trying to create an Experimentation culture is not unlike the temptation faced by the would-be utopian reformer: we should write some laws. There should be rules for how to experiment, we think.

Unlike the utopian reformer, we (thankfully) don’t have the option to convert into a dictator when our Enlightened Laws prove difficult to enforce.

Rules for experimentation are not enforceable and cannot be enforced because an Experimentation Team lacks the two necessary requirements for any law enforcement agency:

1. Punishment.

2. Borders.

We do not have a mechanism to penalize people for not following our rules and we have no means of forcing them to run an experiment at all. Teams can simply secede from Experimentation Nation.

Think about it: How could an Experimentation Team possibly “block” direct code changes? How would that even work? It would clearly be a bad thing, in any case, because an Experimentation Team doesn’t have the information and skillset necessary to do this in a smart way.

Be humble.

The essential thing about a good experimentation culture is that people run experiments. Bad experiments are better than no experiments. Get people to run experiments and look at the data. If you show up to the gym everyday, eventually you’ll lift and get stronger — even if you don’t use the “best” workout routine.

If we try to use rules to create an experimentation culture, we will end up doing one of two things:

1. We will make the rules so weak that the rules become meaningless, or

2. We will be forced to engage with a significant number of teams about why their individual experiments should not launch despite their desire to do so. We will need to be comfortable being and empowered to be a blocker for them despite us having no direct stake in their product or roadmap and a limited, narrow understanding of the constraints they face. Experimentation will become an obstacle to overcome. A road block, not a road sign to know whether we’re going the right way.

So, we should not make rules or write laws. We should provide guidance that we make no effort to enforce but significant effort to disseminate.

Instead of laws, we should write guidelines.

We can make guidelines strong and prescriptive (and, therefore, meaningful and useful) because teams have the option of simply not following them, when they run into an exception — or they just don’t want to.

The key thing is that they are experimenting. They are looking at the data, making tradeoffs. There is a record of the tradeoffs they made. And they have some advice about what to do.

The Main Problem is not that teams don’t use the right standard errors or peek too much. The Main Problem is when things launch without looking at any metrics about what the impact of the product change is, or we only look at pre-post effects where most product impacts are masked by daily demand fluctuations. The Main Problem is when we have no record and no idea of how what we launched this quarter affected our revenue.

Solve the Main Problem with Culture and make the UI helpful enough that the Secondary Problems aren’t a big deal. Don’t let the Secondary Problems get you off-message.

In real life — which, arguably, work is a part of — Culture changes Law, not the other way around (see: Uber, LLMs). Carry this over into your thinking about how to make experimentation a larger part of your org’s decision-making.

A strong Culture does not need strong Laws.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Check out my Udemy course on Causal Inference!: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/?couponCode=A105FEABA0A750B7BB41

If you want my help with any Experimentation, Analytics, etc. problem, click here.

The equations we estimate in data science, statistics, economics, etc., tend to look something like this:

Outcome = F(Observables) + Unobservable

We have a set of observed covariates or features that relate in some way to the outcome, but there’s another part of the data-generating process that we do not observe.

We call the unobserved part of the equation a “residual.” The name suggests it measures the limits of how well the covariates can predict outcomes. This intuition doesn’t apply when the equation we’re estimating is supposed to represent something — when the equation is not a merely statistical relationship.

We can think of outcomes as a function of the choices people make and the states they face, i.e., Y(S, C), where S is the complete vector of states and C is the complete vector of choices. If we observed S and C, we could perfectly explain how Y varies both in a predictive sense and in a causal sense because observing (Y, S, C) and (Y’, S’, C’) would be enough to tell you how moving from (S,C) to (S’,C’) changed Y.

[Causal inference is fundamentally a missing data problem: we don’t observe all potential outcomes.]

The problem we face is that our observed covariates are a subset of (S,C), and our residual is an index of the other state and choice variables that we do not observe.

If we think about residuals this way — from first principles — it is much easier to think about what the residual is and what identification assumptions make sense. For example: Does it make sense that F and the unobservable states and choices are additively separable?

When we think about residuals as states and choices, a lot of the “standard” assumptions start to sound a little… less satisfying?

Why are the observed states and choices treated so asymmetrically from the unobserved states and choices? Why does the fact that we can measure these variables and not those variables change how they enter the data-generating process? Etc.

Examples Of The Different Kinds Of Residuals

Prediction Error

The name “residual” makes the most sense in the regression problem, where we want to estimate E[Y|X=x] for some covariates X, i.e.

Y = E[Y|X=x] + V, where E[V|X=x] = 0.

V is the residual in the sense that it represents the part of Y ignored by X. Var[V] = 0 if Y=Y(X), so the residual exists and matters because Y is not a function of X.

So, our first definition of a residual is the part of the dependent variable we can’t predict.

In this model, the relationship of the residual to real-world states and choices is less important because our primary goal is to estimate the statistical object E[Y|X=x].

Effect Heterogeneity

The residual can also be unobserved heterogeneity in treatment effects. If we run an experiment, we might identify the average treatment effect like so:

Y(z) = E[Y(z)] + U(z)
Y = ZY(1) + (1-Z)Y(0)
Y = E[Y(0)]+ (E[Y(1)] — E[Y(0)]) Z + U(0) + Z(U(1) — U(0))

Because U(z) are independent of treatment assignment in an experiment: E[U(0) + Z(U(1)-U(0))|Z] = 0,

So, when we write our regression specification as Y = a + bZ + V, the residual (V) means U(0) + Z[U(1)-U(0)].

In this particular case, the residual is both the prediction error from regressing Y on Z, and it has a structural interpretation.

Structural Residuals

Now, suppose we’re estimating an equation from some theoretical model. Say a production function from economics:

log Q = log F(Z) + log A,

Where Q is output, Z is a vector of inputs, and A is total factor productivity.

We have data on inputs (Z) and output (Q), and we want to estimate this equation to recover the production function and productivity.

Productivity is a residual. It is the part of the production process that is unobserved, but its meaning comes from the behavioral model, not a statistical criterion.

The definition of productivity and how to measure it is part of a grand old literature (GOL) spanning more than 80 years. From Marshak and Andrews (1944) to Griliches (1994) to some great, recent papers and one okay essay. The one thing the literature agrees about is that productivity is not the error or a residual from regressing outputs on inputs.

To see why, notice that when Q = F(Z)A, greater productivity increases the marginal product of inputs. Therefore, the choice of inputs will be related to productivity and E[log A|Z] != E[log A].

Why A Careful Model Of The Residual Matters

How we identify structural functions or make causal inference depends crucially on how we define the residual: what states and choices are unobserved? For example:

1. In demand estimation, it’s common to treat each product as a bundle of product characteristics (to reduce dimensionality) and write something like this: Q = D(P, X, U), where X is a vector of observed product characteristics, and U is unobserved. We can treat U as an index of unobserved product characteristics. A common identification assumption is that U is independent of X. If we don’t think about the residual as unobserved choices and states, that assumption sounds pretty reasonable. “Shocks” sound like the sort of things that are at least roughly uncorrelated with product design decisions. But that’s not what U is in our model. U is an unobserved product characteristic. Are the product characteristics in X independent of each other? No. So why do we expect them to be independent of unobserved product characteristics?

2. When we write Q = F(Z)A for the production function, how should we think about “A”? One model treats A as a parameter, heterogeneity in how the production function translates Z into outputs. But: what is that heterogeneity? It doesn’t really make sense that some firms or countries can, by their very nature, produce more output. There must be a reason. Whatever that reason is is an unobserved factor of production. So, instead, let’s think about the residual as an index of unobserved inputs. Z is the vector of observed inputs, and Q is the observed output. What’s left over are the inputs we don’t observe. These different models of the residual produce different identification strategies.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Check out my Udemy course on Causal Inference: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/?couponCode=A105FEABA0A750B7BB41

If you want my help with any Experimentation, Analytics, etc. problem, click here.

If you’ve ever worked in an area of data science bordering on experimentation or if you’ve worked in Product, Marketing, etc., in an environment where experiments are possible, you’ve probably heard (or even said — it’s okay, this is a safe space) that ideally we’d run an experiment here, but we need to “move fast.”

I’m going to make the case that there isn’t a tradeoff. Experimentation maximizes velocity.

Admittedly, there is a tradeoff between Experimentation and Speed, but Speed is not Velocity!

Speed is how fast you’re moving, but Velocity has a direction. A sign. A destination you’re drifting towards.

If you don’t care whether you’re moving in the right direction, then sure, you don’t need to run experiments — or do any other analysis for that matter. Just move.

That’s not what we want to do.

“Never confuse movement for action.”

From a letter Hemingway allegedly wrote to Marlene Dietrich, probably not about A/B testing. Likely a quote of a quote. Unoriginal. An old proverb. A truism. The point is it’s a saying that I found on the Internet that made the point I wanted to make.

So, what you’ll hear people propose is: “let’s just launch it and see how it looks [as if it were easy!] after a few weeks.”

Now, if things go exceptionally well or exceptionally poorly, you probably will be able to just “see how it looks”. Sometimes, “sciency” folks get a little picky here, but if metrics suddenly jump up or down an enormous amount and stay consistently at those new levels for weeks, and the timing happens to line up with exactly when you launched the thing…

But 99.9% of product changes (conservatively) are not like this. They slightly improve conversion rates. They don’t cause massive jumps in time series. Product improvements shift company metrics by less than their normal day-to-day variation. For example, here is a made-up time series where there is a 2% average improvement in the metric at some point. That’s a big win! Can you spot the improvement in the series?

Another issue with analyzing just the time series: companies make many product and marketing changes at the same time, so how are we going to distinguish which change is responsible for improving metrics, and which is holding us back?

So, what’s the alternative to running an experiment if:

1. We’re maximizing velocity, not speed. We want to know that we’re moving in the right direction.

2. We’re not massively altering our product. So, we’re not expecting metric movements of +/- 20%.

3. We’re changing many product features at the same time.

The obvious candidate is a causal inference method based on observational data, but let’s think about how that would work in practice.

While there’s an enormous variety of causal inference methods (learn some of them in my Udemy course — look, I’m learning how to plug things!), they all share a similar structure:

1. There is a time before customers were treated with the new experience, and we’ll assume we can use that to establish a baseline for the treated population.

2. There is a set of customers who do not receive the experience at any time, so we can use changes in their experience over time to differentiate between changes from all the other things going on at the company — or in the broader economy/industry— from the actual product change.

So, it’s just another method of holding some people out of treatment — except that the resulting analysis is much less credible and depends on a bevy of assumptions. We have to wait to see results anyway. If the results look bad, there’s always a discussion of whether it’s really bad or if the method is bad…

Why bother, if you have the choice? Go with the finest and loveliest method, the original. The experiment doesn’t have to be 50–50, A vs B. Even a small “A” group protects against the worst outcomes, which are always more likely than you think.

Another way to think about it: why would the best allocation for the control group be 0%? Why not something else, just a little more?

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Check out my Udemy course on causal inference: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/?couponCode=A105FEABA0A750B7BB41

If you want my help with any Experimentation, Analytics, etc. problem, click here.

Medium recently published an argument for why writers without a large external audience should pick Medium over Substack. I fall into the “limited external audience” category, and I’m but a wee little blog, so I thought I’d write up my take on it.

It’s comfortably outside the blog’s normal focus on economics and statistics, but I’m a card-carrying member of the economics cult. So, I have to shoehorn it into everything.

Clearly, by revealed preference (see? I did it already), I agree with the conclusion of the post. I’m writing this on Medium.

But there’s one part of the post that people who start writing here should at least know is unlikely to hold true for them.

Re: “Write on Medium if you don’t have an existing audience”

I disagree with the post’s suggestion that you can just start writing on Medium and people will read it, i.e., that discovery on Medium will get you reads, even without an audience or trying to promote the blog.

Discovery on Medium is a multiplier. If you get a sufficient number of reads from some other source, then you’ll start getting discovery on top of what you can bring in — which is great! — but it’s not like you can hit publish and get any real exposure.

I’d prefer data from more diverse sources, but I only have one dataset to go off of: my own. And, as always, you go to the Data War with the Data Army you have.

I’ve written a couple blogs that I didn’t share on my LinkedIn for whatever reason. The total reads from those blogs are less than 20 — cumulative. One of the posts was in January 2025 where I had about 2300 reads in total. Maybe that gives a sense of how little action you’ll get on a post unless you have some alternative way of telling people about it.

Most of the great blogs on Medium I’ve found via LinkedIn posts, not on-site discovery.

[I think part of this is caused by Medium having these omnibus publications with limited editorial input. It’s the natural outcome of the market, of course. Getting more blogs improves the publication’s chances, and there’s basically no downside to including a weak blog in the pub. These posts flood the discovery feeds, and you can’t follow a pub for its good posts without getting spammed by its bad posts. Publications with editorial input could solve this problem by providing more of a filter — and a meaningful stamp of approval — but that’s not what the product encourages.]

You aren’t going to get readers from Medium’s discovery without first getting a seed amount of readers from elsewhere [probably? There’s one of everything]. So, I don’t think this is a great reason to choose Medium over Substack. You have to acquire the audience either way, and Substack also has discovery mechanisms.

So, why do you use Medium?

It has a nice, easy “follow” button for people to get notifications about my new posts. People don’t have to click a lot of buttons or give extra info. They can just follow. Substack feels more like I’m being moved through a conversion funnel. There are buttons saying “Subscribe now!”, e-mail addresses to enter, etc.

I don’t want to “monetize.” I want an experience that minimizes the friction to reading. So, I don’t want landing pages covering the post asking for e-mails, etc.

(My take: Direct monetization is a silly strategy for blogs about career stuff. Any direct money you might earn will be minuscule compared to the Right People seeing the post. You want to maximize the probability they see it, not try to make a hundred dollars by paywalling it.)

The SEO is strong if a post does well, too. I only have one post that’s done well enough to get picked up by search engines — In Defense of Statistical Significance — but that’s kind of neat to see.

It’s an easy writing experience. A couple clicks to get a landing photo from Unsplash. Simple to use. I don’t have to set anything up to start writing. People can comment easily. What’s not to like?

Ads

Of course, I know nothing about Medium’s traffic patterns to have a sense of whether it has the scale to work business-wise, but will that stop me from commenting on its business strategy? This is the Internet. Of course not. I don’t need “data” to drop a hot take.

On Medium I’d prefer the ads model, I think, without any direct monetization for bloggers. I think the vision the CEO put forward in this post is a good one: Making a site for people who aren’t “content creators.”

People who aren’t [aspiring] “content creators” aren’t looking to make money from writing blogs.

(I’m never going to stop scare-quoting that phrase — “content creator” — because it sounds alien — bizarre. A person makes a series of images shown in rapid succession to imitate movement, but we’re too embarrassed to call it a movie, a film, or a flick. So, we searched for a word to describe the Thing, and found one neutral, bland, and beige enough that it can refer to what’s inside a cardboard box, my stomach, and For Whom the Bell Tolls.).

So long as the revenue is via Premium subscriptions (I’m a subscriber!), there has to be a reason to make Premium posts, so you have to pay bloggers, and you’ll get the “hustle” types that try to hack Medium’s algo in some way to make a little dough. So, I think it’d be neat to move away from that.

In Conclusion…

Anyway, that’s my take. It’s definitely outside the normal topics I write about. There wasn’t a single equation!

But, as the post said, Medium is a good place to write outside your niche. I’m proving out the theory.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

My little Udemy course on Causal Inference: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/?couponCode=CHEAPCAUSALINF3

If you want my help with any Experimentation, Analytics, etc. problem, click here.

Bullshit producers produce their output at near zero marginal cost, but the social costs of bullshit are far greater. Some folks will hear the bullshit and believe it is true — perhaps supposing that people must be saying it for a reason besides vibes and hunches (the primary inputs to the bullshit production process). It’s impossible to look into everything you hear, so an oversupply of bullshit increases the probability that we make the wrong decision by inadvertendly “buying” the bullshit.

The key market failure that leads to an oversupply of bullshit is that the costs of dealing with bullshit fall almost entirely on economic agents that did not produce the bullshit. So, what we need to do to make the market efficient is to lower the costs of countering/dealing with bullshit and increase the costs of producing it.

Experimentation does a little of both. It provides data to refute bullshit, subsidizing the countering process, and it increases the costs to saying bullshit because someone might just run an experiment and show that the claim is bullshit.

VERY IMPORTANT: THE EFFICIENT LEVEL OF BULLSHIT IS NOT ZERO.

Some bullshit is socially valuable. For example, you have a hunch that something might work. It’s not based on anything but vibes, but you just want to test it out. It’s bullshit, right? Well, sometimes these ideas end up working out because vibes are sort of data… The point is that there’s a good level of bullshit. We need to take risks and not just wait until we can “prove” everything. That means allowing for a little bullshit.

I’ve yet to find a more efficient tax/subsidy scheme than Experimentation for approximating the socially efficient equilibrium supply of bullshit. It doesn’t overtax it to the point that no one’s willing to just “try things” or speculate, but it has enough of a tax to control the flow…

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Check out my Udemy course on causal inference: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/?couponCode=CHEAPCAUSALINF3

If you want my help with any Experimentation, Analytics, etc. problem, click here.

Causal inference methods focus on the “effect” on metrics and other high-level outcomes, but we need a model to translate treatment effects into a decision.

Structural econometric methods help us make decisions because they fully specify the economic nature of the decisions we’re making. This blog post shows how to combine these two approaches to make decisions grounded in both economic theory and data.

For example, if we know the causal effect of a price increase on demand for a product, that’s useful information, but it doesn’t — by itself — tell us what price we should charge. We need a model to turn that treatment effect into a decision.

If our cost of providing our products is roughly zero — we’re making websites, selling SaaS — then maybe we want to maximize revenue:

max P’Q(P)

Now we’re working in the space of structural parameters, the economic primitives. The next step is to link these to treatment effects.

The revenue-maximizing price solves:

DQ(P) P + Q(P) = 0

DQ(P) is the derivative of demand with respect to price, which maps onto the more causal inference problem of getting the treatment effect of shifting price on quantity, allowing us to translate treatment effects into pricing decisions.

We always put treatment effects through a model to make decisions. Whether we use an explicit economic model or not, we have some model in mind when we say that it’s a “good thing” to increase this metric and hurt that other metric. What structural methods do is make those assumptions explicit and ground those assumptions in economic theory.

But there is a downside to structural methods as they’re frequently used in the academic literature.

A common approach you’ll see:

1. Estimate a demand function Q(P) that we can evaluate at any price, given some functional form assumption.

2. Then, solve the first-order conditions DQ(P)P + Q(P) = 0 explicitly, which involves evaluating the demand function globally, at prices far from those in the data.

Hot take: this analysis is not credible enough for making business decisions at the accuracy level we need. The further we move from prices we have seen in the data, the less we’re using data to inform pricing (or other) decisions, and the more we’re relying on arbitrary functional form assumptions.

Anyone who’s tried the above approach in the wild has seen the model suggest wild prices that end up (rightly) ignored by stakeholders as unreasonable.

The trick to marrying more credible causal inference methods and structural methods is to use structural methods that can make use of local results — use methods that don’t rely on identifying the structural function globally.

Instead of trying to solve DQ(P)P + Q(P) = 0, instead recognize that DQ(P)P + Q(P) gives the marginal change in revenue for a small price change for each product. So we can “localize” the revenue maximization problem by doing something like the following:

1. Estimate the demand function at current prices Q(P) by using various causal inference methods — or experimentation, if feasible. This is more credible because we don’t need to try to map out the full demand function. We just need to measure the impact of increasing prices a little relative to current prices, i.e., estimate treatment effects.

2. Then, compute F = DQ(P) P + Q(P) at current prices.

3. If F(j) > 0, increase the price P(j) a little bit. If F(j) < 0, decrease the price P(j) a little bit. In practice, because F is estimated, it makes sense to add other restrictions, like: “we’ll only change prices if |F(j)| > e”.

4. Periodically re-run the analysis to update the demand function now evaluated at the new prices, and then we march towards the optimal prices.

So, we can combine more credible causal inference methods with economic models to help us make decisions that are both informed by economic theory and don’t overly rely on arbitrary functional form or other modeling decisions.

The key is to localize the structural model, using either the first-order method above or constraining decisions to lie within a tight neighborhood around the status quo.

Over time, we’ll move further and further from the initial status quo as we gather more evidence about other parts of the decision space.

And that’s it. A pragmatic way to use structural economic models and causal inference methods together.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Take my little Udemy course (on causal inference methods) at: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/?couponCode=CHEAPCAUSALINF3

If you want my help with any Experimentation, Analytics, etc. problem, click here.