AOAC Statistics (Appendix S) Motions-September 2021

AOAC INTERNATIONAL

General Guidelines on Statistical Analysis and Reporting as Recommended by the Committee on Statistics (APPENDIX S)

MOTIONS: CS2021.01/CS2021.04/CS2021.05

REVISION DATE: September 22, 2021 APPROVAL DATE: August 11, 2022

Recommendation by the AOAC Committee on Statistics

Motion CS2021.01: Recommended Binary MLV Study Performance requirements for a microbiological analyte (passed April 21, 2021; OMB approval August 11, 2022) 1. The LPOD(R) for the reference method (“R”) must fall between 0.200 and 0.800 at the concentration used to compare methods (“fractional range“). If no reference method is used, the rule applies to the candidate method (“C”) LPOD(C). 2. Homogeneity (Reproducibility) is to be demonstrated by having the entire 90% confidence interval for intraclass correlation (“ICC”) for each of the compared methods falling below 0.333. If this does not occur, the method has not been shown to be homogeneous among collaborators. 3. For comparison of a reference method to the candidate method, the 90% confidence interval on dLPOD(C,R) must fall entirely within the interval -0.200 to 1.000 in order for the candidate method to be said to be ‘equal or superior’ to the reference method. (This interval applies solely to LPOD(R) = 0.500 and should be adjusted for other LPOD(R). See TR364 for tables and formulae.) If this does not occur, the candidate method has not been shown to be equal or superior to the reference method. 4. For comparison of the candidate presumptive method (“CP”) to the candidate confirmation method (“CC”), the 90% confidence interval on dLPOD(CP,CC) must fall entirely within the interval -0.200 to +0.200 in order for the candidate presumption method to be said to be “equivalent” to the candidate confirmation method. (This interval applies solely to the case LPOD(CC) = 0.500 and should be adjusted for other LPOD(CC). See TR364 for tables and formulae.) 5. It is recommended that each collaborator perform 12 or more replicates. Use of fewer replicates may materially increase the chances of the study failing by either by failing to declare methods equivalent when they are, or by rejecting methods that are homogeneous in performance among collaborators. RATIONALE: The principal purposes of a collaborative study on a binary candidate method for a microbiological analyte are to 1) demonstrate homogeneity in performance among collaborators (i.e., “reproducibility”); and 2) demonstrate that the candidate method is equal or superior to a reference method. Under current rules, the methods used to achieve these goals are ineffectual due to lack of study power, and lead to complete misinterpretation of the results. The proposed modifications reflect a change in posture in which positive declarations of equivalence replace lack of definitive conclusions. The changes recommended in the motion have been reviewed and accepted by the ERP for Microbiology.

AOAC Statistics Motions are still in progress or development

REVISION DATE: September 22, 2021 APPROVAL DATE: August 11, 2022

Recommendation by the AOAC Committee on Statistics

RELATED DOCUMENTS:

1. Task Group for Fractional Results (TGFR): Report to the Microbiology Expert Review Panel v.3. 2. LaBudde, RA. (2019). Statistical analysis of interlaboratory studies. LVII. Suggested performance requirements for proving equivalence of two analytical methods in chemical and microbiological collaborative and single-laboratory studies. Least Cost Formulations, Ltd., technical report TR353. 3. LaBudde, RA. (2020). Statistical analysis of interlaboratory studies. LXV. Setting a balanced equivalence interval on dPOD or dLPOD in a binary qualitative study. Least Cost Formulations, Ltd., technical report TR364.

AOAC Statistics Motions are still in progress or development

REVISION DATE: September 22, 2021 APPROVAL DATE: August 11, 2022

Recommendation by the AOAC Committee on Statistics

Motion CS2021.04: Specific language to be used in reporting statistical results comparing analytical methods in validation studies subject to margin for equivalence interval criteria (passed September 15, 2021; OMB approval pending). 1. EQUIVALENCE: If two methods ‘A’ and ‘B’ are to be compared in a validation study, and ‘equivalence’ is defined as the true difference between methods falling within the equivalence interval Δ 1 to Δ 2 (or typically – Δ to +Δ), where Δ 1 (or – Δ) and Δ 2 (or Δ) are lower and upper margins for equivalence (i.e., minimum size differences that are material) determined by subject-matter experts: a. If the entire 90% confidence interval for the true difference falls entirely within – Δ to +Δ, then the conclusion should be stated as “The study data indicate with 95% confidence that method A is equivalent to method B”. b. If the entire 90% confidence interval for the true difference falls entirely below – Δ, then the conclusion should be stated as “The study data indicate with 95% confidence that method A results are less than method B”. c. If the entire 90% confidence interval for the true difference falls entirely above +Δ, then the conclusion should be stated as “The study data indicate with 95% confidence that method A results are greater than method B”. d. If any part of the 90% confidence interval for the true difference falls both inside and outside the interval – Δ to +Δ, then the conclusion should be stated as “The study data fail to indicate with 95% confidence whether method A results are equivalent or not equivalent to method B, due to insufficient sample size”. 2. EQUAL OR GREATER: If two methods ‘A’ and ‘B’ are to be compared in a validation study, and ‘equal or greater’ is defined as the true difference between methods falling within the equivalence interval – Δ to ∞, where - Δ is the lower margin for equivalence (i.e., minimum size difference that is material) determined by subject-matter experts: a. If the entire 90% confidence interval for the true difference falls entirely above – Δ, then the conclusion should be stated as “The study data indicate with 95% confidence that method A results are equal or greater compared to method B”. b. If the entire 90% confidence interval for the true difference falls entirely below – Δ, then the conclusion should be stated as “The study data indicate with 95% confidence that method A results are less than method B”. c. If any part of the 90% confidence interval for the true difference falls both inside and outside the interval – Δ to +∞, then the conclusion should be stated as “The study data fail to indicate with 95% confidence whether method A results are equal or greater or not equal or greater compared to method B, due to insufficient sample size”. 3. EQUAL OR LESS: If two methods ‘A’ and ‘B’ are to be compared in a validation study, and ‘equal or less’ is defined as the true difference between methods falling within the equivalence interval – ∞ to +Δ, where +Δ is the upper margin for equivalence (i.e., minimum size difference that is material) determined by subject-matter experts: a. If the entire 90% confidence interval for the true difference falls entirely below +Δ, then the conclusion should be stated as “The study data indicate with 95% confidence that method A results are equal or less than method B”. b. If the entire 90% confidence interval for the true difference falls entirely above +Δ, then the conclusion should be stated as “The study data indicate with 95% confidence that method A results are greater than method B”. c. If any part of the 90% confidence interval for the true difference falls both inside and outside the interval – ∞ to +Δ, then the conclusion should be stated as “The study data fail to indicate

AOAC Statistics Motions are still in progress or development

REVISION DATE: September 22, 2021 APPROVAL DATE: August 11, 2022

Recommendation by the AOAC Committee on Statistics

with 95% confidence whether method A results are equal or less or not equal or less compared to method B, due to insufficient sample size”.

RATIONALE: History indicates study authors are creative in describing the conclusions based on statistical results in studies. This motion intends to regularize the language used when an equivalence interval is available so that 1) the language is consistent with good statistical principles, and 2) the language is uniform across studies.

AOAC Statistics Motions are still in progress or development

REVISION DATE: September 22, 2021 APPROVAL DATE: August 11, 2022

Recommendation by the AOAC Committee on Statistics

Motion CS2021.05: Specific language to be used in reporting confidence intervals comparing analytical methods in validation studies (passed September 15, 2021; OMB approval pending). If two methods ‘A’ and ‘B’ are to be compared in a validation study, and if a confidence interval (95% or some other level of confidence) on the difference in performance of the methods is given as “L to U” with estimate y (where L and U are the endpoints of the interval and y is the point estimate): 1. CONFIDENCE INTERVAL: Report the confidence interval by “The study data indicate with 95% confidence that the true difference between methods A and B could be as small as L or as large as U, with a best estimate of y”. 2. SIGNIFICANCE: If the value 0.0 is within the 95% confidence interval L to U, then it should also be reported that “The study data were unable to find a statistically detectable difference from zero between methods A and B, due to insufficient sample size”. RATIONALE: History indicates study authors are creative in describing the conclusions based on statistical results in studies. Typically, such language as “no statistically significant differences were found” is used, which misleads the reader to presume that it was shown no material differences exist. In fact, the language, while technically correct, actually means the study was poorly designed and cannot draw a clear conclusion about the size of any difference, other than any confidence interval reported. “Statistically significant” means “statistically detectable”. (Studies should be designed to detect a minimum material difference of a size determined by subject-matter expertise and fitness for purpose intended.) This motion intends to regularize the language used when an equivalence interval is available so that 1) the language is clear in meaning and consistent with good statistical principles, and 2) the language is uniform across studies.

AOAC Statistics Motions are still in progress or development

REVISION DATE: September 22, 2021 APPROVAL DATE: August 11, 2022