OMB Meeting Book - January 8, 2015

40

GUIDELINES FOR REPEATABILITY ESTIMATION 1. No collaborators should be removed, except for known cause. Such causes may include: 1) does not meet inclusion criteria for protocol, if protocols used are known; or 2) provable contamination. Statistical identification of outliers or influential data is not grounds for removal, only for investigation. 2. Replication may range from 2 to 4 replicates per collaborator. Repeatability should be estimated in the usual way as the pooled standard deviation of the combined set of data. 3. Alternatively, replication may exceed 4 for some collaborators, but each estimate of repeatability standard deviation should be assigned the minimum degrees of freedom across all collaborators, and this number should be used in pooling and reporting. 4. The final number of degrees of freedom assigned to the pooled estimate must be 8 or more. 5. There must be at least 3 collaborators with replication. 6. Robust statistics associated with repeatability may be estimated and reported (such as interquartile range), but not estimates which attempt to convert such statistics to standard deviations by, e. g., constant factors under a normality assumption. Reporting such robust estimates for designed collaborative studies should be encouraged so that comparative results may be accumulated over time. 7. Boxplots and half-normal plots are encouraged. GUIDELINES FOR REPRODUCIBILITY ESTIMATION 1. No collaborators should be removed, except for known cause. Such causes may include: 1) does not meet inclusion criteria for protocol, if protocols used are known; or 2) provable contamination. Statistical identification of outliers or influential data is not grounds for removal, only for investigation. 2. The number of collaborators providing included data must be 8 or more. 3. If replication is present for most or all collaborators, repeatability, among-collaborator variability and reproducibility should be estimated as standard deviations estimated in the usual way from 1-way analysis of variance (additive model). No more than 4 replicates should be used for any collaborator. 4. If replication is not present, reproducibility only may be estimated (as the standard deviation of collaborator results). 5. Robust statistics associated with reproducibility may be estimated and reported (such as interquartile range), but not estimates which attempt to convert such statistics to standard deviations by, e. g., constant factors under a normality assumption. Reporting such robust estimates for designed collaborative studies should be encouraged so that comparative results may be accumulated over time. 6. Boxplots and half-normal plots are encouraged.

3

Recommended to OMB by Committee on Statistics: 07-17-2013 Reviewed and approved by OMB: 07-18-2013 by Committee on S atistics: 07-17-2 13

9