Theory, Data, and Experiments

By James Bland

“It’s one of the few forms of interaction with people that I don’t find repellent.” – Sheldon Cooper (Jim Parsons) on human experimentation, The Big Bang Theory, series 4 episode 20.

Economic theory and empirical observation are very much complements. Economic theory not based on, or not supported by, some kind of evidence is at best an untested hypothesis, and at worst recreational mathematics with no value other than entertainment. Empirical observation without theory tells us nothing about why things are happening, or when the empirical regularities we observe at the moment might fail us. The interesting stuff happens when the two combine.

For those of you who have studied sciences such as physics, chemistry, or psychology, you may have noticed that the usual method of testing theory with data in these fields is different: experiments are designed to test specific hypotheses, and are conducted in a controlled laboratory environment which is easily re-constructed if one wishes to repeat the experiment. In contrast, economists rarely deal with data that are generated (1) specifically to test their hypotheses, (2) in a controlled environment, and (3) in a manner that is easily repeatable. It is therefore remarkable (and a tribute to the work being done in econometrics) that as economists, we are able to test so many ceteribus paribus predictions of our theory using data that clearly do not satisfy the ceteribus paribus condition.

As a science that studies the interaction of people in often very complex environments, it is at the same time both shocking and unsurprising to me that economic experiments have only recently taken off in a big way. On the one hand, holding anything, let alone all else, constant seems (at first) very difficult when people and economies are involved. On the other, when one dives into the experimental economics literature, one quickly realizes the gap experiments can bridge between theory and data.

What do economic experiments look like?

Laboratory experiments in economics usually investigate how people behave when performing tasks that could earn them real money. Economics is, after all, the study of how agents (usually humans, although monkeys have also been used as test subjects) respond to incentives. Money is used because it is reasonable to assume that all people will always want more of it (unlike for example, chocolate, as some people may not like it, or can only stand to eat so much). Participants are given tasks to perform, which usually involve making decisions at a computer terminal. These decisions, and the decisions of other participants, go on to determine the payments everyone receives when they leave the lab. Various devices, such as surveys and announcements, are used to establish that the participants understand what they and others are doing, and how their actions lead to their and others’ payments (otherwise our results could be attributed to confusion).

Why experiment?

Consider the following problem:

In a market, there is a group of buyers, and a group of sellers. Each seller owns an identical item, which he is willing to sell at a price known only to himself. Each buyer is looking to buy exactly one item, as long as she can obtain it for a price below her valuation, which is known only to herself.

Very standard economic theory says that there will be an equilibrium price at which the market will clear: at this price the number of buyers and sellers willing to trade is equal. We can work out this price, and the quantity traded, by constructing demand and supply curves and finding their intersection. The theory also tells us that at this price, total surplus is maximized (assuming no externalities, etc.). Furthermore, all we should observe from this scenario is the final price and quantity traded, and if we’re really lucky, the number of buyers and sellers who did not trade.

At this stage, note that (1) the theory makes a testable hypothesis: that the final price and quantity is determined by the private valuations; and (2) we cannot test this hypothesis without knowing the valuations. While there are some pretty cool techniques in the empirical industrial organization literature about how to back out supply and demand curves (and hence valuations) from observable information, these methods start by assuming the theory is correct, and then infer valuations based on this assumption.

This is where controlled experiments come in. Instead of observing a small fraction of what goes on in a market, why not construct our own market where we know, and in fact choose ourselves, all of the valuations? To do this, we could get a group of people in a room, and give them the following instructions:

You are a buyer [seller] in a market looking to purchase [sell] one item. This item is worth $V to you, so if you find a seller [buyer] and agree to trade at price $P, we (the experimenters) will pay you $V-P [$P-V].

Yes, real money is involved! This technique is called inducing preferences: loosely speaking, by paying buyers the surplus implied by their trade of a fictitious good (i.e. $V-P), they now have the same incentives as they would have had if they were in a non-laboratory market and came across a seller offering price $P for a good they valued at $V. As the experimenter, all we have to do now is observe which people trade, and at what price, then compare this to the theoretical predictions.

The first market experiment of this kind was conducted by Chamberlin (1948), and later followed up by Smith (1962, 1964), who was one of the participants in Chamberlin’s initial experiment. These experiments showed something that is unsurprising (to me at least), but lacking in the theory discussed above: institutions matter. Specifically, the way in which agents were allowed to trade affected the accuracy of the theory’s predictions: trading systems where prices were more publicly visible resulted in prices and quantities closer to the theoretical predictions. This information can be (and has been) used to modify theory to better capture the important determinants of market outcomes.

Hopefully this story highlights two things. Firstly, we can use controlled economic experiments to answer questions that may be impossible, or at least difficult, to answer with other methods. Secondly, the flow between theory and hypothesis testing (i.e. data) is by no means one-way: theory suggests empirical regularities that we can test, and data can help us how we could shape our theories in order to better capture the important aspects of economic activity.

Want to know more?

If you are interested in reading more about experimental economics, a great place to start is this article, written by my honors and masters thesis adviser Nikos Nikiforakis, published in the Australian Economic Review.

The University of Melbourne Experimental Economics Laboratory website. You can sign up to be a participant here.

The University of Melbourne runs a third-year subject on experimental economics in the second semester.

About the author

James Bland is a 2nd-year PhD student at the University of Melbourne. His research interests include experimental economics and game theory. When he is not studying he plays basketball, makes his own beer, and plays the drums.

References

Chamberlin, E.H. (1948) “An experimental imperfect market,” Journal of political economy, 56, 95-100

Smith, V.L. (1962) “An experimental study of competitive market behavior,” Journal of political economy, 70, 111-137.

Smith, V.L. (1964) “The effect of market organization on competitive equilibrium,” Quarterly journal of economics, 78, 181-201.