Over the past decade or so, it’s come to light that biomedical research may not be as accurate as once presumed. “There is newfound recognition that many procedures, methods, and approaches to doing research produce findings or conclusions that aren’t as reliable as originally thought,” said Steven Goodman, MD, MHS, PhD, associate dean of clinical and translational research and professor of medicine and epidemiology at Stanford University School of Medicine in California.
In fact, a 2016 article reported that more than 70% of 1,576 surveyed researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments (Nature. 2016;533:452–454). The data reveal sometimes contradictory attitudes toward reproducibility. Of those surveyed, 52% agreed that there was a significant crisis of reproducibility.
Short-Term Implications
The short-term consequences of this breakdown in biomedical research are that promising clinical trials and research already in the pipeline will go unfunded or will be underfunded by the National Institutes of Health (NIH) and other large federal agencies. “This will delay the development of new therapies, which could lead to continued patient suffering and mortality,” said Lamont R. Jones, MD, MBA, vice chair of the department of otolaryngology at the Henry Ford Health System in Detroit. “Consequently, bright and innovative men and women interested in biomedical research who are faced with a mountain of school debt and the grim prospect of the future of research funding may not choose a research career,” he said.
Lisa Ishii, MD, MHS, associate professor of otolaryngology-head and neck surgery and chief quality officer for clinical best practices at Johns Hopkins School of Medicine in Baltimore and research and quality coordinator for the Academy of Otolaryngology–Head and Neck Surgery, believes that faulty studies and tight funding will result in the dissemination of flawed scientific data. “If those data are then applied in clinical care, for example, that could lead to waste or, in the worst case, actual harm to patients,” she said.
Long-Term Outlook
© totojang1977 / shutterStock.com
Kurt Amsler, PhD, associate dean of research and professor in the department of biomedical sciences at NYIT College of Osteopathic Medicine in Old Westbury, New York, pointed out that the documented and publicized issues of a lack of quality control and poor reproducibility in biomedical research have also led to a decline in public trust in the biomedical research enterprise, which could lead to the government questioning the value of continuing its strong support for the funding of biomedical research (F1000Res. 2014 May 28;3:119; EMBO Rep. 2017 Sep;18:1493-1496).
The uncertainty of financial support will likely drive funding to more seasoned researchers and conservative and safer projects, a move that would be detrimental to new investigators, Dr. Jones said. In addition, decreased research and development investments will slow the pace and make it harder to maintain the country’s current lead in biomedical research.
“The field has responded by instituting changes that have significantly expanded the administrative and compliance requirements associated with biomedical research studies,” Dr. Amsler said. “Altogether, the decline in public trust of biomedical research and other issues has dramatically decreased the pipeline of promising new researchers in the biomedical field, which will impact the field for many years to come. Timelines to new breakthroughs will either be lengthened substantially or missed. Public health will not continue to improve at its current pace.”
Dr. Ishii said the United States has distinguished itself on the world stage for its preeminence in biomedical research. “A systemic breakdown in scientific research threatens our international contributions and reputation,” she said.
Dr. Goodman, who co-founded and co-directs Meta-Research Innovation Center at Stanford (METRICS), a research-to-action center focused on transforming research practices to improve the quality of scientific studies in biomedicine and beyond, envisions a move toward more reliable research. He foresees fewer weakly informative studies and, instead, more contributors to larger, stronger studies. He also expects more openness in the research system, so others can view data and see if they can derive similar conclusions from it. “Ultimately, the future depends on the policies of funders, journals, and universities,” he said. “We are already in the midst of a sea change regarding what’s necessary to provide the most reliable research.”
Bright and innovative men and women interested in biomedical research who are faced with a mountain of school debt and the grim prospect of the future of research funding may not choose a research career. —Lamont R. Jones, MD, MBA
Improving Deficiencies in Biomedical Research
In the past five to 10 years, Dr. Goodman said some top-level solutions have emerged to deter faulty studies. Some of these include data sharing, better reporting methods, changes in the criteria by which results are declared statistically significant, and changes in the promotion’s criteria requirements for academic medical centers so that they more closely reflect whether the findings of academic research are reliable or are published in a transparent and reproducible way. Currently, no such tracking mechanisms exist. “We’re working to figure out if there are better systems that don’t incentivize the production of so many papers that may not be informative. It’s a complex problem,” he said.
One clear solution is to place a greater emphasis on data sharing using technologies, so other researchers can look at the primary data upon which others base their research. “Journal editors are increasingly asking authors for a declaration of their willingness to share data, protocols, and statistical codes, and are ensuring that trials are registered and reported in clinicaltrials.gov—which is legally required in the United States.
“A researcher must register any trial they plan to perform,” Dr. Goodman said. “They must report results, regardless of whether the trial was completed or not, within a year of it being concluded.” Institutions and companies that perform trials and fail to do this can be subject to fines of up to $10,000 a day for lack of reporting.
He also expects funders such as the National Institutes of Health (NIH) and National Science Foundation to possibly put forth more rigid requirements about data sharing and documenting publication of previously supported research. He says a movement toward required training in research design and methods could emerge—investigators might need to be credentialed before being allowed to submit a proposal to an institutional review board, for example.
Effects on the Field of Otolaryngology
The breakdown will have a halo effect, Dr. Jones said. As funding decreases, conditions with lower incidence rates, such as head and neck cancer, will receive less funding. “Because physician-scientists have dual pressures to be productive in both the laboratory and clinic, a lack of research will limit time spent doing the former,” he added. Also, he said that as research funding decreases, otolaryngology departments will be forced to make tough decisions regarding where to direct resources. “Because there is no immediate return on investments in research, it is easier to justify budget decreases or focus more on clinical rather than translational research.”
Dr. Ishii said otolaryngology will suffer as a specialty if the field cannot attract the brightest minds into scientific research. “Our next generation of clinician-scientists will not choose this path if they worry about resources and feasibility,” she concluded.
Karen Appold is a freelance medical writer based in Pennsylvania.
Key Points
- Many procedures, methods, and approaches to performing research produce findings or conclusions that are not reliable.
- One consequence is that promising clinical trials and research already in the pipeline will go unfunded or will be underfunded.
- As funding decreases, conditions with lower incidence rates such as head and neck cancer will receive less funding.
Rethinking “Statistical Significance”
A different approach to how p-values and the phrase “statistical significance” are viewed could be on the horizon.
In March 2016, The American Statistical Association (ASA) issued a statement on how to improve the conduct and interpretation of quantitative science and inform the growing emphasis on reproducibility of science research.
“The p-value was never intended to be a substitute for scientific reasoning,” said Ron Wasserstein, the ASA’s executive director, in the statement. “Well-reasoned statistical arguments contain much more than the value of a single number and whether that number exceeds an arbitrary threshold. The ASA statement is intended to steer research into a post p<0.05 era.”
The NIH has a web page dedicated to providing information about its efforts to enhance rigor and reproducibility in scientific research. “When a result can be reproduced by multiple scientists, it validates the original results and readiness to progress to the next phase of research,” the web page states. “This is especially important for clinical trials in humans, which are built on studies that have demonstrated a particular effect or outcome.”