OMB Winter Mtg.-February 5-6, 2015

95

Estimating Reproducibility from Proficiency Test Data

Are there ways to implement proficiency models into an experiment to estimate reproducibility of a method? In order to experimentally estimate reproducibility, there must be an experiment which is operating under reproducibility conditions. Reproducibility conditions are defined by ISO as “conditions where test results are obtained with the same method on identical test items in different laboratories with different operators using different equipment.”

The AOAC Committee on Statistics has agreed to the following strategy for accepting proficiency test data as part of method validation.

1. Preferred Approach: A collaborative study (CS) is designed to follow AOAC guidelines with a study protocol developed in conjunction with AOAC Statistics Committee advisors.

2. In the event a collaborative study cannot be organized, Proficiency Test (PT) data may be used in partial substitution if the following important considerations are met. a. Acceptance criteria should be agreed on by the Expert Review Panel before the study is performed, or before the data are analyzed and reported. b. Laboratories in a PT study or check sample program can be used only if they follow the AOAC Method strictly, without modification. c. The minimum number of laboratories should be 8. d. Given the estimate for s(r), a 95% CI, and the degrees of freedom from replication, report s(R), a 95% CI, and the degrees of freedom from reproducibility. e. Primary decision criterion for Final Action should be based on the upper limit of the 95% CI of s(R) and s(r). 3. The statistical estimation techniques described in ISO 13528 are optimized for proficiency statistics. For validation statistics, AOAC INTERNATIONAL requires the use of the statistical procedures of ISO 5725, with the provision that no outliers be removed without an assigned cause. Recommendations on estimation techniques are as follows. a. Collect PT data from check sample programs or other PT data available. b. Use only results from labs that admit to running the AOAC Method. Results from other methods shall be removed from the data set. c. Do not trim, drop or exclude outliers except for justifiable cause such as an admission from the laboratory that the method was modified. d. Use standard AOAC/ISO5725 ANOVA model to estimate reproducibility standard deviation.