What evidence do MPs turn to when they make policy?

Anyone going to mention RCTs today? PA/PA Wire

MPs value the views of constituents and expert opinion more highly than evidence from randomised controlled trials, a new survey has found. However, the majority of the 104 Labour and Conservative MPs from the previous parliament who were questioned support the idea of using randomised controlled trials to evaluate policies and don’t believe they are too expensive.

The survey, conducted by Ipsos MORI and the charitable trust Sense about Science, also found that MPs were more inclined to act on their own principles than on the results of randomised controlled trials, which involve testing new interventions on a randomly selected group of people and comparing that with a control group of people who get the usual intervention.

Interestingly, the survey found that the views of experts and the opinions of constituents trumped those of practitioners, such as teachers or doctors. It also found that MPs give very little weight to the views of journalists.

There are signs that a lot of MPs harbour some erroneous misgivings about randomised controlled trials, suggesting a widespread lack of understanding of how the trials work and how they might be used to test the effectiveness of policies.

Many thought it unfair that some people would be randomly assigned to not receive the policy being investigated – 35% believed this, including 26% of the 74 MPs who supported the use of the trials. But this is a fundamental principle, and a strength, of this kind of trial.

The MPs quizzed were roughly as supportive of pilot schemes without a comparison group as they were for randomised controlled trials by 67% to 64%, misunderstanding the value that the addition of a randomly selected control group can bring.

Pilot studies also contain an aspect of “unfairness”, for people in areas where the scheme is not piloted. But pilot studies do not have the added benefit of collecting data in those areas to provide a comparison, generating stronger evidence as to whether a policy is effective.

Each word in the name RCT is important. Randomisation of the trial’s subjects mean that, in a health trial for example, there’s less chance of the sickest people being all in one group and warping the findings. Controlled studies compare the new treatment to a baseline group where the usual treatment is given to check whether there’s a real difference between the two. And they are a trial, rather than just implementing a new treatment without testing it first.

What’s important for MPs

The survey also highlighted the difference between where politicians think they should look for evidence when making policy decisions, and what evidence they have actually used to justify the decisions they’ve made in the past. Randomised controlled trials were rated as less important than uncontrolled pilot trials in both instances.

MPs put more trust in their own principles than RCTs. Ipsos MORI

Evidence from experts was voted top of what politicians felt they should consider, but this was beaten by “the views of constituents” when they were asked what they have used to justify a policy in the past. In both instances, the MPs' own principles were rated much higher than evidence from RCTs.

Political decisions will rarely be based on evidence alone. And it should absolutely be the case that politicians want to listen to their constituents and act accordingly. But where the impact of a potential policy change is not known, testing this out before it is widely implemented can save money long-term, and make sure only policies likely to be effective are implemented.

Although the background of the MPs surveyed is not reported, an interest in and understanding of science is somewhat lacking in the corridors of power. In 2010, the Campaign for Science and Engineering compiled a list of MPs with an interest or background in science, and it equated to roughly 10% of MPs in parliament at that time. The implication here is that the other 90% have little or no interest in science whatsoever, let alone a scientific background or understanding of its methods.

Randomised trials gaining ground

In medicine it seems obvious to test treatments before rolling them out, and there is a move to apply such techniques to policy too, aided by RCTs. In 2013, the then secretary of state for education, Michael Gove, enlisted the help of epidemiologist Ben Goldacre to help bring a strong evidence base to educational policy.

The Behavioural Insights Team work with the government to design and test policies or interventions, and along with Goldacre and David Torgerson have authored a policy paper called Test, Learn, Adapt, instructing how to run trials for policy.

The Behavioural Insights Team have already used randomised controlled trials to investigate the effectiveness of a number of policy changes or interventions. One such trial showed that a small change to a government website led to increases in organ donation sign-up.

A government-backed charity called the Educational Endowment Foundation have used randomised trials to show that teaching assistants can improve numeracy and literacy when used effectively, which had been doubted after evidence from earlier largely non-randomised research. Goldacre has himself said that if anyone wants to help bring RCTs to policymaking, he will “stand on the barricades” with them.

The results of this new survey suggest MPs would be receptive to this. Rather than smirking at politicians' failure to grasp the complexities of scientific trials, researchers need to explain their importance, design and limitations. Randomised controlled trials have changed medicine for the better, and if done properly can do the same for the way policies are developed.

MPs don’t need a scientific background to value evidence-based policy, but if they need help understanding how to get the strongest evidence, we should provide it.