Nothing New And A Lot Of Flaws: Abhijit Banerjee’s RCT ‘Breakthrough’ Which Won The Nobel Prize

The 2019 Nobel Prize in Economics Sciences was won by Indian-American MIT professor Abhijit Banerjee, Esther Duflo and Michael Kremer for their research into the use of experimental approaches in designing welfare schemes.

Based on common sense, every intervention that a government plans in people’s welfare should be based on the evidence about effectiveness of that intervention, just as before administering a drug to a patient, we should be sure of its effectiveness. This is what drives clinical trials in pharmaceutical industry.

Banerjee’s approach is application of the Randomised Controlled Trial (RCT) - something that has been used for long in pharma industry - to the field of social sciences.

RCTs are an attempt to bring a certain rigour to social sciences research which otherwise are quite subjective and can depend heavily on the author’s ideological and other biases.

How RCTs work

In the social sphere, several factors are at work together. Therefore, it is hard to isolate the impact of a certain intervention carried out by the government. RCTs aim is to isolate the effect of a certain variable on overall outcome of the social scheme.

For example, take the PM Kisan scheme, under which a fixed sum will be transferred to the farmers.

To gauge the outcome of this intervention, a RCT can be set up which will remove impact of other factors, say, a good monsoon, new technology in farming, etc and focus specifically on the impact of the money transferred by the government.

The way RCTs claim to isolate the impact of a specific variable and remove other factors is through ‘randomisation’ - random allocation of subjects to different groups.

One group—the experimental group—has the intervention being assessed, while the other—usually called the control group—has a placebo or no intervention.

The argument is that since all random samples are subject to the same array of "confounding" factors, they are essentially identical to one another.

The randomness in the assignment of subjects to groups reduces selection bias and allocation bias, balancing both known and unknown prognostic factors.

In RCTs, the experiment may also be ‘blinded’, that is, information which may influence the participants is withheld until after the experiment is complete. Therefore, the participants/experimenter may have no knowledge of whether they are being administered actual drugs or placebos.

Blinded experiments reduce other forms of experimenter and subject biases.

RCTs in social sciences

RCTs in social sciences are harder to conduct due to sociological complexities. Blinded experiments are close to impossible and the interventions themselves can inadvertently induce socioeconomic and behavioural changes that can “confound” the relationships.

For example, let’s say, an RCT wants to test the effectiveness of providing assistant teachers for weaker and poorer students in a government school. The RCT would like to isolate and measure the impact of this on the outcome - the performance of these children.

Introducing assistant teachers for weaker students in the school may increase the sense of competition among school teachers leading to their better attendance, teaching quality, resulting in increased outcome.

The extra classes of assistant teacher may also force parents to keep the child studying for more time, while otherwise, they would have made the child help in household work etc. These are the inadvertent changes brought about by the experiment conditions.

The sources of funding may also create conflict of interest dangers. For example, an RCT sponsored by a political party or government - led by a political party would be expected to produce results amenable to its own interests.

The topics and interventions selected for studies will be informed by the sensibility of the funding agency.

In fact, the ideological and other biases of the experimenters can also easily affect the outcomes.

In his paper, ‘Why all randomised controlled trials produce biased results’, Alexander Krauss of London School of Economics, after assessing the 10 most cited RCT studies worldwide, concluded that the trials inevitably produce bias.

The paper stated that the “trials involve complex processes – from randomising, blinding and controlling, to implementing treatments, monitoring participants etc. – that require many decisions and steps at different levels that bring their own assumptions and degree of bias to results”.

Among other things, Krauss pointed out following limitations: “participants’ background characteristics are often poorly allocated across trial groups, participants at times switch between trial groups, trials often neglect alternative factors contributing to their main reported outcome, among others”.

2015 Economics Nobel prize winner Angus Deaton has also criticised Randomised Controlled Trials. In his paper 'Understanding and misunderstanding randomised control trials’ , Deaton argued that mere random selection of samples for an RCT doesn’t result in identical groups.

He argued that while two randomly chosen samples might turn out to be similar in some cases, there are greater chances that most samples are not really similar to each other.

Welfare economics

A great utility of using scientific methods to asses outcomes of welfare schemes is their capability to generate broad-based consensus. Strong evidence, not tainted by ideological biases, would result in less polarisation in political debates.

However, given the susceptibility of RCTs to above mentioned biases - including the ideological bias of the experimenter- their credibility will remain debatable.

A flagship proposal of Abhijit Banerjee -Congress’s NYAY scheme- was to be financed by raising taxes or increasing deficit. R Jagannathan noted that high government borrowings would have ultimately resulted in an 'inflation tax', and called the scheme ‘poor economics’.

NYAY scheme doesn't only flounder on some basic tests like efficient beneficiary targeting, as Jagannathan noted, but also bears the hallmark of a ‘compensatory’, rather than ‘developmental’ state - which pays people a meagre compensation for its failure - rather than rich dividends of policy-successes.