People believe data from a sample of 30 nearly as much as data from a sample of 3000

The present research investigates whether people are sensitive to very large differences in sample sizes when making decisions. Using Bayesian analysis, Experiment 1a and 1b found that two orders of magnitude differences in sample sizes did not have any significant impact on the judgments of both experienced business executives and students with statistical training. Experiments 2 and 3 found that when deciding how much to believe or whether to act upon the findings from a sample, people were strongly influenced by differences in the sample mean but hardly influenced by differences in the same sample size, although statistically, differences in sample sizes were much more consequential than differences in sample means. Finally Experiment 4 identified an intervention that reduced people’s insensitivity to sample sizes–providing people with a p-value, along with a lay person interpretation of the p-value. Addressing a long standing debate, multiple experiments using a between-participant and thus demand-free design find that although people might be more confident in data from larger sample sizes when multiple sample sizes are simultaneously juxtaposed, people are nearly completely insensitive to very large differences in sample size when considering just a simple sample on its own.