P-Value Statistic Comes under Criticism

Despite this belief, he emphasized that many, if not most, clinicians are trained with only a superficial knowledge of statistical interpretation. “We’re armed with enough knowledge that we know we should be wary of it, but we’re not necessarily sure what other questions we should be asking or how we should be interpreting the data.”

According to Dr. Smith, other questions that clinicians need to start asking are those directed at understanding the clinical relevancy of the data instead of only looking at statistical significance. This includes asking questions about the effect size around the differences found between, for example, two treatment groups, as well as the width of the confidence interval around the observed difference. Once physicians become more comfortable asking these questions, he said, they can begin to draw their own conclusions about how to interpret the data.

“I think clinicians generally are a bit wary of p-values and significant findings. We’re armed with enough knowledge that we know we should be wary of it, but we’re not necessarily sure what other questions we should be asking or how we should be interpreting the data.” —Timothy Smith, MD, MPH

Zachary M. Soler, MD, associate professor in the department of otolaryngology–head and neck surgery at the Medical University of South Carolina (MUSC) in Charleston, reiterated the importance of asking about the effect size and confidence interval when interpreting a statistical finding in a study based on the p-value.

He acknowledged, however, that sometimes clinicians may not be up to speed when it comes to knowing how best to interpret complex statistical methods. In these cases, he encouraged clinicians to look to sources of post-publication peer review or published commentaries by leading experts for insight and interpretation of a study’s findings. In addition, he emphasized that the most important evidence supporting a specific finding is whether it can be replicated in an entirely new study.

Wasserstein also emphasized questions clinicians should begin thinking about to help determine whether a statistically significant finding is clinically relevant or meaningful (see “How to Determine Clinically Relevant Findings,”).

Research and Publication

While clinicians will be increasingly required to evolve in their thinking of how to interpret medical data, the real revolution in shifting from an overreliance on the p-value to other ways to demonstrate clinical relevance is and will be at the research level, said Dr. Smith. In part, this is being driven by editorial decisions at medical journals that are just beginning to establish policies for publishing research findings that don’t prioritize publishing “positive” data based on the p-value. “More and more journal editors are becoming more sophisticated about statistics and demanding different results reporting from their authors,” he said.