Many results in microeconomics are shaky

MICROECONOMISTS are wrong about specific things, Yoram Bauman, an economist and comedian, likes to say, whereas macroeconomists are wrong in general. Macroeconomists have borne the brunt of public criticism over the past decade, a period marked by financial crisis, soaring unemployment and bitter arguments between the profession’s brightest stars. Yet the vast majority of practising dismal scientists are microeconomists, studying the behaviour of people and firms in individual markets. Their work is influential and touches on all aspects of social policy. But it is no less fraught than the study of the world economy, and should be treated with corresponding caution.

For decades non-economists have attacked the assumptions underlying economic theory: that people are perfectly informed maximisers of their own self-interest, for instance. Although economists are aware that markets fail and humans are not always rational, many of their investigations still rely on neoclassical assumptions as “good enough” descriptions of the world. But this “101ism”, as Noah Smith, an economist and journalist, calls it, is less prevalent than it was in the 1950s and 1960s, when researchers like Gary Becker reckoned everything from crime to marriage could be described in terms of rational self-interest. Since the 1970s, as Roger Backhouse and Béatrice Cherrier describe in “The Age of the Applied Economist”*, a new collection of essays, the field has taken a decidedly empirical turn.

Get our daily newsletter

Upgrade your inbox and get our Daily Dispatch and Editor’s Picks.

Most influential economic work today involves at least some data from the real world. Many economists made their names by finding unique datasets containing “natural experiments”, in which a change in policy or conditions affects only parts of a population. This allows researchers to tease out the effect of the change. In a famous example, published in 2001, John Donohue and Steven Levitt used variations in abortion laws across states to conclude that legalising abortion had been responsible for as much as half of the decline in crime in America in the 1990s. Other economists used randomised controlled trials (RCTs) to generate experimental data on the effects of social and development policies. In RCTs randomly chosen subjects are given a “treatment”, such as a microloan or a school voucher, while those in a control group are not. The behaviour of the two groups is then compared.

These developments have led to better, more substantial research. Yet they have also exposed economics to the problems bedevilling most social sciences, and some hard sciences, too. Researchers can tweak their statistical tests or mine available data until they stumble on an interesting result. Or they read significance into a random alignment. Economics, like other social sciences, is suffering a replication crisis. A recent examination in the Economic Journal, of almost 7,000 empirical economics studies, found that in half of the areas of research, nearly 90% of those studies were underpowered, ie, that they used samples too small to judge whether a particular effect was really there. Of the studies that avoided this pitfall, 80% were found to have exaggerated the reported results. Another study, published in Science, which attempted to replicate 18 economics experiments, failed for seven of them.

Even when a study is perfectly designed and executed, the result is open to interpretation. Environmental factors such as changing institutions or social norms inevitably play some role, but researchers cannot fully account for them. The results of an experiment conducted in one country might not be relevant in another, or in the same country at a later date. Research may suffer from more than one of these problems. Critics of the paper by Messrs Donohue and Levitt reckon, for instance, that the authors’ computer code contained an error, that they used a measure of crime that flattered their results, and that they neglected the possibility that differences in the change in crime across states were caused by differences in factors other than abortion laws. (The pair conceded an error, but responded that taking better account of confounding factors did not weaken their conclusion.)

Small wonder that economists struggle to answer seemingly straightforward questions, such as how minimum-wage laws affect employment. In 2017 two teams of researchers released assessments of a change in Seattle’s minimum-wage laws within days of each other. Each came to wildly different conclusions (continuing an established pattern of such research).

New techniques could help. Machine learning, in which computer programs comb through vast datasets in search of patterns, is becoming more popular in all areas of economics. A future beckons in which retailers know virtually everything about every transaction, from the competing products buyers considered before their purchases to their heart rates at the moment of payment. That could mean better predictions and policy recommendations without a smidgen of economic analysis. But pitfalls are already apparent. The algorithms used are opaque. And getting access to the richest data will require researchers to work with, or for, giant tech firms which have their own interests.

Read the small print

Economics enjoys greater influence over policy than other social sciences. Striking new findings are publicised by researchers and their institutions, promoted by like-minded interest groups and politicians, and amplified by social media. Conflicting results and corrections are often ignored. Being alert to the shortcomings of published research need not lead to nihilism. But it is wise to be sceptical about any single result, a principle this columnist resolves to follow more closely from now on.

Sources:

“The age of the applied economist: the transformation of economics since the 1970s”, Roger Backhouse and Beatrice Cherrier, November 2016.
“The impact of legalised abortion on crime”, John Donohue and Steven Levitt, Quarterly Journal of Economics, May 2001.
“The impact of legalised abortion on crime: comment”, Christopher Foote and Christopher Goetz, Quarterly Journal of Economics, February 2008.
“Measurement error, legalized abortion and the decline in crime: a response to Foote and Goetz”, John Donohue and Steven Levitt, Quarterly Journal of Economics, February 2008.
“The power of bias in economics research”, John Ioannidis, T.D. Stanley and Hristos Doucouliagos, Economic Journal, October 2017.
“Evaluating replicability of laboratory experiments in economics”, Colin Camerer et al, Science, March 2016.
“Seattle’s minimum wage experience 2015-2016”, Michael Reich, Sylvia Allegretto and Anna Godoey, June 2017.
“Minimum wage increases, wages, and low-wage employment: evidence from Seattle”, Ekaterina Jardim et al, NBER working paper 23532, June 2017.

This article appeared in the Finance and economics section of the print edition under the headline “A little knowledge”