Reading and Assessing the Clinical Research Literature

The strategies that most people use to decide which paper to read are usually based on the title of the article; the author(s) and their institutions; or a quick review of the abstract. Unfortunately, none of these strategies is completely dependable in guiding the reader to a high-quality study. Titles may be uninformative or misleading (e.g., One gene: four syndromes, Nature 1994;367:319; Zinc: the neglected nutrient, Am J Otol 1989;10:56-60). Selecting by authors usually entails calling on the reputation of the authors and their institutions or placing greater emphasis on multi-institutional studies, neither of which guarantees a well-designed, well-conducted study. Although the structured abstract is now in common use, it does not allow for a critical examination of the analysis and is insufficient for drawing clinical conclusions apart from those offered by the author(s). It does, however, contain more information than one can derive from selecting on the basis of title or author alone.

Assessing the Major Parts of a Paper

Papers appearing in most peer-reviewed journals will have four major sections: the background or introduction; the methods; the results; and the discussion or conclusions. These sections should work together to allow you, in the end, to assess the three major attributes of a scholarly paper: its validity, its importance, and its reliability. A schematic depicting the sections of a paper, with their respective attributes, is presented in Figure 1.

click for large version

Figure 1. Evaluating the Medical Literature Critical Appraisal

Introduction and Background

First, are the clinical question of interest and the gap in knowledge the study is intended to address clearly identified in the beginning of the paper? Is the objective of the study or the hypothesis to be tested clearly stated? The hypothesis can be structured as a null hypothesis (which is the formal basis for testing statistical significance) or as an alternative hypothesis. Is the relevant literature fairly reviewed, up-to-date, with opposing viewpoints identified in an objective manner? If there is a prevailing conflict about the subject in the field, is there a statement about how this study may help to resolve it? This section’s purpose is to establish the relevance of the study for the reader and to review the past and current beliefs about the subject.

Methods and Procedure

Keeping in mind the study objectives and clinical question, the first question to ask is whether the study design is appropriate to achieve those objectives. A summary of the main categories of studies and the most appropriate study designs appears in Table 1. Next, you would want to know the sample size and how it was calculated. An adequate sample size, based on a power analysis, will be capable of detecting a true effect if one exists. Related to this issue is the sampling method: Was it a random sample, consecutive patients, patients with particular characteristics (in which case inclusion and exclusion criteria should be specified), and so on? A good paper will always indicate not only the sample size, but also the length of the study (including recruitment period, study period, and follow-up period), and completion rate-that is, the percentage of patients in the original group that completed the follow-up. A completion rate of 80% or more indicates a study with reasonable validity.2 The variables of interest should be specified, as well as the tools or instruments used to study them.

click for large version

Table 1. Categories of Studies and Appropriate Study Designs

The procedure section should contain enough detail so that another person could replicate the study in its entirety. This doesn’t mean that this section should constitute a procedure manual; rather, details that could influence the outcome should be given, including type/approach of surgery, stimulation and recording parameters of electrophysiological tests, title and validation status of outcome or quality-of-life instruments, key laboratory tests or procedures, and any nonstandard diagnostic or intervention procedures. In this section, the astute reader will also be alert to potential sources of bias in the design and execution of the study, such as selection bias, subject recall bias, measurement bias, investigator bias, and the like. The purpose of the methods and procedure section is to allow the reader to assess the quality and validity of the study. If there are major flaws in the procedure, the results and conclusions are inevitably called into question.

Results

Arguably the most important question that will guide your assessment of the results section is whether the results reported are valid. In order to reach this conclusion, look for several features that will allow assessment of the power of the study and the precision with which it was conducted. The results section should begin with a description of the demographic features of the subjects: their numbers, age, gender, and ethnic distribution. Accompanying this information, measures of central tendency (mean or median), the variance (standard deviation), and the range of any baseline numerical data (e.g., tumor size, pure tone average, apnea/hypopnea score, etc.) should appear. If a disease entity is being studied, look for an operational definition of that disease, such as For purposes of this study, chronic rhinosinusitis was defined as…. The same is true if levels of severity are an operational factor. If there is a comparison group-normal, different condition, different demographic, or different intervention-there should be some statement of how closely they were matched and whether their prognosis was the same at the beginning of the study. Pre- and post-intervention results are reported, along with complication rates and adverse events.