What's the Value of the P Value?

Many studies published in current scientific journals include a P value as part of the data analysis, often as an indication of statistical significance of the results. What does the P value mean? A P value is related to the null hypothesis (i.e., there is no difference between the no-treatment and treatment groups) and represents the probability of obtaining the observed results if the null hypothesis is true (1). For example, a P value of 0.05 represents a 5% probability of obtaining the observed results if the null hypothesis (no difference) is true. In a recent Viewpoint article published in JAMA (2), John P.A. Ioannidis discusses a proposal to lower the P value threshold for statistical significance from 0.05 to 0.005, while also highlighting many limitations with the use of P values. The first limitation is that P values are often misinterpreted as providing evidence that a given hypothesis is either true or false. The second limitation is that P values are overtrusted, when the P value can be highly influenced by factors such as sample size or selective reporting of data. The third limitation discussed by Ioannidis is that P values are often misused to draw conclusions about the research. Additionally, Ioannidis highlights that many results with P values <0.05, yet still close to that threshold, may not represent true effects. Lowering the P value threshold to 0.005 will likely increase the number of true effects that are reported in the literature.