Report says dozens of medical trials may contain inaccurate data

Dozens of professional papers published in highly respected academic medical journals may contain incorrect data according to Dr Andrew Klein, the Editor-in-Chief of Anaesthesia, the official peer reviewed journal of the Association of Anaesthetists of Great Britain and Ireland (AAGBI).

Dr Klein’s comments are based on a new study published today (Monday 5 June 2017) entitled Data fabrication and other reasons for non-random sampling in 5,087 randomised, controlled trials in anaesthetic and general medical journals. For the first time the study applies a new, mathematical screening tool to a large sample of scientific medical papers and concludes that over 80 of these papers need to be urgently reviewed for accuracy. It suggests that in over half these papers the chances of the key data actually being correct is trillions to one against.

The new screening tool is a sophisticated computer programme designed to spot statistical inaccuracies. It was developed by Dr John Carlisle, a consultant anaesthetist at Torbay Hospital. Previously the Carlisle screening tool has only been used to assess the work of specific authors but in his latest study – for the first time – Dr Carlisle applies the methodology to a large sample of papers both in anaesthetic journals and in general medical journals such as the New England Journal of Medicine.

Dr Carlisle said: “This screening tool raises questions about data in some studies, which on full investigation may turn out to involve misinterpretation, statistical error or plain simple mistakes. However, on the basis of previous studies it is likely that some of the data highlighted in this latest research may have been deliberately falsified. As a minimum it is clear that the reporting of some randomised, controlled trials may be seriously flawed.”

Dr Andrew Klein, the Editor-in-Chief of the journal Anaesthesia said: “This study is about working to correct the scientific record. The new Carlisle screening tool has been developed here in the UK and it is now clear that it should be used by medical publications around the world. The integrity of medical science demands that we do everything we can to ensure complete accuracy in the publication of research.”

“The Carlisle method is freely available online – along with the necessary computer code – so any medical journal reporting randomised controlled trials can use it. The editors, owners and publishers of these journals have an ethical duty to do everything in their power to ensure accuracy in scientific publishing. Patients expect accuracy and safety in the research on which their care is based.”

Dr Geraldine Pearson, Co-Chair of the Committee on Publication Ethics (COPE) said: “As an organisation which works to promote integrity in all aspects of publication, we are prepared to collaborate closely with the AAGBI on the findings of this study and to provide guidance and education if requested by editors of journals. COPE provides advice to editors and publishers on all aspects of publication ethics; therefore we welcome the ongoing work of Anaesthesia towards ensuring the scientific record is set straight.”

Chris Graf is Director of Research Integrity and Publishing Ethics at Wiley, the publisher of the journal Anaesthesia. He said: "Wiley supports developments which promote integrity in science, and applauds this research. Dr Carlisle’s methodology has potential to improve the quality of scientific publications.”


Notes to Editors
All editors-of-chief of the eight journals in question have been written to regarding this paper and have since responded.

'Data fabrication and other reasons for non-random sampling in 5,087 randomised, controlled trials in anaesthetic and general medical journals', JB Carlisle, Consultant, Department of Anaesthesia, Peri-operative Medicine and Intensive Care

For more information, please contact Saffron Pineger or Louise Harris at Freshwater:


Please note that it may take up to 5 minutes for your comments to appear.