Effects of measurement errors on linear calibration equations

This report summarizes the results of the developmental work in the area of statistical treatment of calibration data performed in the preparation of American National Standard N15.19. The effects of measurement errors in both the dependent and independent variables in linear calibrations are studied by examining (1) a simulation model in which the lack of fit was zero with assigned known values of the measurement errors, (2) data from a laboratory calibration of a process vessel where the measurement errors are small compared to the lack of fit, (3) data from a process vessel calibration where the lack of fit can be removed through the use of a cubic equation, and (4) data from a process vessel calibration where the major error appears to be in the dependent variable. The effect of measurement errors on the estimation of the standard deviation of the slope is found to be highly dependent on the experimental design used and on the subsequent treatment of the data.