ERP Micro December 2019
EXPERT REVIEW PANEL MICROBIOLOGY FOR FOODS AND ENVIRONMENTAL SURFACES
MONDAY, SEPTEMBER 9, 2019
AOAC INTERNATIONAL OFFICIAL METHODS OF ANALYSIS SM (OMA) PROGRAM
The Official Methods of Analysis SM (OMA) program is AOAC INTERNATIONAL's premier methods program. The program evaluates chemistry, microbiology, and molecular biology methods. It also evaluates traditional benchtop methods, instrumental methods, and proprietary, commercial, and/or alternative methods. In 2011, AOAC augmented the Official Methods SM program by including an approach to First Action Official Methods SM status that relies on gathering the experts to develop voluntary consensus standards, followed by collective expert judgment of methods using the adopted standards. All methods in the AOAC Official Methods SM program are now reviewed by Expert Review Panels for First Action AOAC Official Methods of Analysis SM status, continuance, repeal, and/or to recommend for AOAC Final Action Official Methods status. The OMA program has undergone a series of transitions in support of AOAC's collaborations, evolving technology, and evolving technical requirements. Methods approved in this program have undergone rigorous scientific and systematic scrutiny such that analytical results by methods in the Official Methods of Analysis of AOAC INTERNATIONAL are deemed to be highly credible and defensible. The methods are published in the Official Methods of Analysis of AOAC INTERNATIONAL and supporting manuscripts are published in the Journal of AOAC INTERNATIONAL . AOAC Official Methods SM program allows for submissions for all proprietary, single and sole source methods. Methods submitted through the PTM-OMA harmonized process also will be reviewed through the O fficial Methods of Analysis SM (OMA) program. Other complementary products and services include expanded consulting services for validation protocol development and AOAC INTERNATIONAL Organizational Affiliate Membership.
AOAC INTERNATIONAL 2275 Research Blvd, Suite 300 Rockville, Maryland 20850 Phone: (301) 924-7077
AOAC EXPERT REVIEW PANEL (ERP) MICROBIOLOGY FOR FOOD AND ENVIRONMENTAL SURFACES
MONDAY, DECEMBER 2, 2019 1:00PM EST – 5:00PM EST AOAC INTERNATIONAL VIRTUAL MEETING GoToMeeting EXPERT REVIEW PANEL CO-CHAIRS: WENDY MCMAHON & MICHAEL BRODSKY AGENDA
I.
WELCOME AND INTRODUCTIONS Brodsky and McMahon will open the meeting by welcoming all attendees and initiating introduction of the AOAC Expert Review Panel members and meeting attendees.
II.
REVIEW OF METHODS FOR MODIFICATION For each method, the ERP members will present a review of the proposed modified collaborative study manuscript, after which the ERP will discuss the method and render a decision on the status for each method.
1) AOAC OMA 2005.04: Escherichia coli O157:H7 in Selected Foods, Assurance® GDS E. coli O157:H7 Tq Study Directors: Andrew Lienau, MilliporeSigma, BioControl Systems, Inc.
III.
REVIEW OF METHODS FOR AOAC FIRST ACTION OFFICAL METHODS STATUS For each method, the ERP members will present a review of the proposed collaborative study manuscript, after which the ERP will discuss the method and render a decision on the status for each method. 1) OMAMAN-50: EVALUATION OF THE GENE-UP ® LISTERIA SPP. METHOD FOR THE DETECTION OF LISTERIA SPECIES IN A VARIETY OF FOODS AND SELECT ENVIRONMENTAL SURFACES: COLLABORATIVE STUDY Study Directors: Ron Johnson, BioMerieux, Inc., 595 Anglum Road, Hazelwood, MO 63042 2) OMAMAN-51: EVALUATION OF THE GENE-UP ® LISTERIA MONOCYTOGENES METHOD FOR THE DETECTION OF LISTERIA MONOCYTOGENES IN A VARIETY OF FOODS AND SELECT ENVIRONMENTAL SURFACES: COLLABORATIVE STUDY Study Directors: Ron Johnson, BioMerieux, Inc., 595 Anglum Road, Hazelwood, MO 63042 ERP will discuss, review and track First Action methods for 2 years after adoption, review any additional information (i.e., additional collaborative study data, proficiency testing, and other feedback) and make recommendations to the Official Methods Board regarding Final Action status. 1) AOAC OMA 2017.09: Identification and Confirmation of Salmonella spp., C ronobacter spp. and Other Gram- Negative Organisms by the Bruker MALDI Biotyper Method Study Directors: Miles Murphy, Bruker Daltonics
IV. DISCUSS FINAL ACTION REQUIREMENTS FOR FIRST ACTION OFFICIAL METHODS (IF APPLICABLE)
*Agenda is subject to change. V1
AOAC INTERNATIONAL ● 2275 RESEARCH BLVD, SUITE 300 ● ROCKVILLE, MARYLAND 20850 USA
2) AOAC OMA 2017.10: Identification and Confirmation of Listeriamonocytogenes, Listeria spp. and Other Gram- Positive Organisms by the Bruker MALDI Biotyper Method Study Directors: Miles Murphy, Bruker Daltonics ERP TECHNICAL REVIEWS AND PROCESSES ERP members to discuss volunteers and reviews, including equity among members of doing reviews, timelines for receiving methods, new processes for increasing volunteer participation and for method review assignments; and ERP meeting participation. NEW ITEMS, ANNOUNCEMENTS AND ACTION ITEMS Relevant announcements and review the action item list, including any reviews in progress. Discussion of any potential new items to come forward for the ERP’s future consideration.
V.
VI.
VII.
ADJOURNMENT
*Agenda is subject to change. V1
AOAC INTERNATIONAL ● 2275 RESEARCH BLVD, SUITE 300 ● ROCKVILLE, MARYLAND 20850 USA
Official Methods of Analysis SM (OMA) Expert Review Panel MEETING AND METHOD REVIEW GUIDANCE
The AOAC Research Institute administers AOAC INTERNATIONAL's premier methods program, the AOAC Official Methods of Analysis SM (OMA). The program evaluates chemistry, microbiology, and molecular biology methods. It also evaluates traditional benchtop methods, instrumental methods, and proprietary, commercial, and/or alternative methods and relies on gathering the experts to develop voluntary consensus standards, followed by collective expert judgment of methods using the adopted standards. The Official Methods of Analysis of AOAC INTERNATIONAL is deemed to be highly credible and defensible. All Expert Review Panel (ERP) members are vetted by the AOAC Official Methods Board (OMB) and serve at the pleasure of the President of AOAC INTERNATIONAL. In accordance to the AOAC Expert Review Panel Member and Chair Volunteer Role Description all Expert Review Panel members are expected to 1) serve with the highest integrity, 2) perform duties and method reviews, and 3) adhere to review timelines and deadlines.
To assist the ERP Chair and its members, please note the following in preparation for Expert Review Panel meetings and method reviews.
Pre-Meeting Requirements 1. Confirm availability and plan to be present to ensure a quorum of the ERP.
(Please refer to page 25, Quorum Guidelines, Expert Review Panel Information Packet ) 2. Ensure that your laptop, CPU or mobile device can access online web documentation. 3. Be prepared for the meeting by reviewing all relevant meeting materials and method documentation.
In-Person Meeting and Teleconference Conduct 1. Arrive on time.
2. Advise the Chair and ERP members of any potential Conflicts of Interest at the beginning of the meeting. 3. Participation is required from all members of the ERP. All members have been deemed experts in the specific subject matter areas. 4. The ERP Chair will moderate the meeting to ensure that decisions can be made in a timely manner. 5. Follow Robert’s Rules of Order for Motions. 6. Speak loud, clear, and concise so that all members may hear and understand your point of view. 7. Due to the openness of our meetings, it is imperative that all members communicate in a respectful manner and tone. 8. Refrain from disruptive behavior. Always allow one member to speak at a time. Please do not interrupt. 9. Please note that all methods reviewed and decisions made during the Expert Review Panel process are considered confidential and should not be discussed unless during an Expert Review Panel meeting to ensure transparency. Reviewing Methods Prior to the Expert Review Panel meeting, ERP members are required to conduct method reviews. All methods are reviewed under the following criteria, technical evaluation, general comments, editorial criteria, and recommendation status. These methods are being reviewed against their collaborative study protocols as provided in the supplemental documentation. Note: The method author(s) will be present during the Expert Review Panel session to answer any questions.
Page 1 of 2
Version 1 – OMA ERP Meeting Conduct
Official Methods of Analysis SM (OMA) Expert Review Panel MEETING AND METHOD REVIEW GUIDANCE
Reviewing Methods (Cont’d)
Reviewers shall conduct in-depth review of method and any supporting information. In-depth reviews are completed electronically via the method review form. The method review form must be completed and submitted by the deadline date as provided. All reviews will be discussed during the Expert Review Panel meeting. Any ERP member can make the motion to adopt or not to adopt the method. If the method is adopted for AOAC First Action status, Expert Review Panel members must track and present feedback on assigned First Action Official Methods . Recommend additional feedback or information for Final Action consideratio n. Here are some questions to consider during your review based on your scientific judgment: 1. Does the method sufficiently follow the collaborative study protocol? 2. Is the method scientifically sound and can be followed? 3. What are the strengths and weaknesses of the method? 4. How do the weaknesses weigh in your recommendation for the method? 5. Will the method serve the community that will use the method? 6. What additional information may be needed to further support the method? 7. Can this method be considered for AOAC First Action OMA status? Reaching Consensus during Expert Review Panel Meeting 1. Make your Motion. 2. Allow another member to Second the Motion. 3. The Chair will state the motion and offer the ERP an option to discuss the motion. 4. The Chair will call a vote once deliberations are complete. 5. Methods must be adopted by unanimous decision of ERP on first ballot, if not unanimous, negative votes must delineate scientific reasons. Negative voter(s) can be overridden by 2/3 of voting ERP members after due consideration. 6. All other motions will require 2/3 majority for vote to carry.
Page 2 of 2
Version 1 – OMA ERP Meeting Conduct
9/13/2018
AOAC Expert Review Panels An Orientation
Deborah McKenzie רב Sr. Dir., Standards Development AOAC INTERNATIONAL Staff Liaison ‐ Official Methods Board
1
9/13/2018
As a
,
AOAC INTERNATIONAL advances and ,
members, organizations, and experts dedicated to developing and validating and of
by
Analytical Excellence
AOAC Strategic Goals
Core Programs
2
9/13/2018
AOAC STRATEGIC PLAN
Accessible at AOAC homepage www.aoac.org
Analytical Excellence addresses emerging issues and influence standards development as a global leader in analytical excellence
Standards Development
Official Methods of Analysis SM (OMA) & Performance Tested Methods SM (PTM)
Laboratory Proficiency Testing & Quality Management
Analytical Excellence
Journal of AOAC INTERNATIONAL, OMA, ALACC Guide
3
9/13/2018
AOAC Method Approval Programs
Official Methods of Analysis SM (OMA) • AOAC’s premiere methods program • Approved methods – published in the Official Methods of Analysis of AOAC INTERNATIONAL (print and online) – Manuscripts published in the Journal of AOAC INTERNATIONAL – First Action and Final Action status
Performance Tested Methods SM (PTM) • AOAC’s method certification program • Certified methods – Commercial/proprietary rapid methods (test kits) – Certifications published on AOAC website – Manuscripts published in the Journal of AOAC INTERNATIONAL – Method developers licensed to use certification mark – Annual review & recertification
AOAC Official Methods SM Program
Submit Methods Responding to issued Call for Methods • Adoption of methods as Official Methods is contingent upon standards development activities • No application fee required to submit methods in response to Call for Methods Submit Individual & Sole Source Methods • Adoption of methods as Official Methods is contingent upon data supporting applicability and community based validation guidance information • Including proprietary/commercial methods and harmonized PTM – OMA methods • Application fee required
4
9/13/2018
Status of Official Methods of Analysis First Action, Final Action, Repeal
AOAC Policies & Procedures
Policy on Use of Association Name, Identifying Insignia, Letterhead, Business Cards
Policy on Volunteer Conflict of Interest
Policy on Antitrust
OMA Appendix G ‐ Use of AOAC Voluntary Consensus Standards to Evaluate Characteristics of a Method of Analysis
Expert Review Panel Policies and Procedures
5
9/13/2018
Road to First Action OMA Status
1. PTM – OMA Methods 2. Other Sole Source Methods 3. Response to Call for Methods
Method submitted
Expert Review Panels review all methods submitted methods
Notify Method author
Reject
ERP
Adopt
Published First Action OMA
Road to Final Action OMA Status
Method reproducibility must be demonstrated before Final Action consideration.
ERP determines if sufficient evidence merits a recommendation for Final Action status or repeal. • Only the OMB promotes a method to “Final Action” status or repeal the method. • Methods that did not meet the bar would be repealed. • Same for all method submissions
6
9/13/2018
PTM Overview for PTM‐OMA Harmonized Process • Administered by the Research Institute in 2003. • Well established and streamlined • Original approved by consensus with the OAs, OMB, RI Board of Directors and AOAC INTERNATIONAL Board of Directors. • ERP may be formed during Consulting Service. • Criterion for OMA: manufacturer’s method claims.
Recruiting Experts and Methods • AOAC issues – Call for Methods (Stakeholder affiliated methods) – Call for Experts • Sole Source/Individual Method Submissions – Individually completed Application not associated with an open Call for Methods
7
9/13/2018
Qualifications for ERP Membership Candidate must meet one of the following: • Demonstrated knowledge in the appropriate scientific disciplines. • Demonstrated knowledge regarding data relevant to adequate method performance.
• Demonstrated knowledge of practical application of analytical methods to bona fide diagnostic requirements.
Candidate application package includes: • Statement of Expertise • Current Abridged CV or Resume
ERP Member Vetting Process
Approved roster sent to AOAC President for volunteer appointment
Candidate submits application package
Reviewed by AOAC staff with recommendation to OMB
Reviewed by OMB and roster approved
• All members serve at the pleasure of the AOAC President • OMB assigns a representative to serve as a resource for every ERP
8
9/13/2018
Candidate Method Assignments A minimum of primary and secondary reviewers may be assigned to every method. In depth review via review form Prepare to attend and speak on the method and make a recommendation for ERP discussion and consideration. Review forms are completed and returned to AOAC staff in advance of the meeting. An email is sent with information on how to access the candidate methods and how to submit reviews
Members of both Committee on Safety and Committee on Statistics serve as advisory resources for all ERPs
Candidate Method Reviews
In your judgment, does the method sufficiently meet the Standard Method Performance Requirements (SMPR) or community‐based guidance?
In your judgment, is the method scientifically sound and can be followed? In your judgment, what are the strengths and weaknesses of the method? In your judgment, how do the weaknesses weigh in your recommendation for the method? In your judgment, will the method serve well the stakeholder community that will use the method? In your judgment, what additional information may be needed to further support the method meeting the SMPR or community‐based guidance? Members of both Committee on Safety and Committee on Statistics serve as advisory resources for all ERPs
9
9/13/2018
ERP Meetings ERPs can meet in person at a minimum of twice a year and up to four times per year*: AOAC Mid‐Year meeting (DC metro area) AOAC Annual Meeting. *2 additional designated times for proprietary method Organziational Affiliates At the ERP meeting: Reviews will be presented and the reviewers can make a motion to the ERP whether to adopt the method as First Action OMA. ERP discusses the method. ERP renders a decision on First Action status. ERP renders decisions on modifications to Official Methods . If the method is adopted ERP decides on what additional information is needed to recommend the method for Final Action status
ERP Teleconferences • Only after the initial in‐person ERP meeting for First Action consideration of methods • Possible for some method modifications • Possible for First Action to Final Action ERP recommendations
10
9/13/2018
ERP Meetings
Quorum
Presence of 7 vetted ERP members
Presence of 2/3 vetted ERP members
OR
WHICHEVER IS GREATER IF NO QUORUM, NO OFFICIAL MEETING
Method Review Overview
Method authors may be invited to make a presentation on their method REVIEWERS PRESENT THEIR REVIEWS AND MAY INITIATE A MOTION TO ADOPT THE METHOD IF THEY CHOOSE Chair recognizes each reviewer Reviews are presented.
If in favor, reviewers may make and second a motion to adopt adopt the method Chair can then entertain discussion on themethod Chair can call for a vote once deliberation is complete
11
9/13/2018
Consensus – First Action Adoption
First Action Official Methods status is granted:
Method must be adopted by unanimous decision of ERP on first ballot, if not unanimous, negative votes must be based on scientific reasons.
Negative voter(s) can be overridden by 2/3 of voting ERP members after due consideration.
Method becomes First Action on the date when ERP decision is made.
Consensus – First Action to Final Action
The ERP may then reach consensus on any additional information that it needs to review to be able to make a recommendation for Final Action Official Methods status.
This is a separate motion.
12
9/13/2018
ERP Meetings – Review for First Action METHOD AUTHOR: present any method and any resulting changes to the method since submission for review, summary of SLV and/or reproducibility evaluation, any recognitions (from AOAC or external) and, final draft of method proposed for decision
ERP CHAIR & MEMBERS: present reviews and discuss any resulting issues or questions on the method, review and agree upon final draft of method proposed for decision, and chair calls for ERP decision in accordance to procedures.
CONSENSUS: Method must be adopted by unanimous decision of ERP on first ballot. If not unanimous, negative votes must delineate scientific reasons. Negative voter(s) can be overridden by 2/3 of non‐ negative voting ERP members after due consideration. Abstentions do not count towards vote; in case of multiple abstentions the results will need to be evaluated. Staff will monitor and record consensus voting.
STAFF: Will organize and coordinate meeting, record ERP actions and decisions, draft ERP report and distribute after chair approval, work with chair and OMB liaison to complete checklist and assemble recommendation package for OMB.
ERP Methods Review & Approval
Methods should be scientifically sound with demonstrating that it will meet the needs of those using the method (evidenced by meeting the standard, or other acceptance criteria)
ERPs have approved methods with evidence of high potential to First Action and request additional work or support be submitted for review prior to ERP convening to recommend an action to OMB
OMB requires a justification or rationale for methods that are deemed acceptable and adopted but may not fully meet the standard set or acceptance criteria.
13
9/13/2018
OMB Expectations for First Action
Safety review needed prior to First Action status
SLV type of supporting information available per the SMPR
• Applicability, Method Performance Requirements Table, System Suitability, Reference Materials, and Validation Guidance
• Documented method performance versus a SMPR • Document reasons for acceptability if method does not meet the SMPR
Comparison to SMPR
Any approved method(s) along with supporting manuscript(s) and documentation sent to AOAC Publications after the meeting.
Method incorporating ERP revisions (preferably in AOAC Format) Method Manuscript incorporating specified ERP revisions (in AOAC Format) Signed AOAC Copyright Authorization form
NO OMA NUMBER ASSIGNED UNTIL ALL DOCUMENTATION SUBMITTED
Publication of First Action Methods
Method and method manuscript prepared for publication in the Official Methods of Analysis of AOAC INTERNATIONAL and in Journal of AOAC INTERNATIONAL Updates on methods approved or status changes are published in the Inside Laboratory Management magazine and on the AOAC website
14
9/13/2018
ERP Meetings – Method Tracking METHOD AUTHOR: present any method feedback obtained and any resulting changes to the method, any reproducibility information, any implemented ERP recommendations, final draft of method proposed for decision ERP MEMBERS: present any method feedback obtained and
discuss any resulting changes to the method, any reproducibility information, any implemented ERP recommendations, review and agree upon final draft of method proposed for decision, and make a recommendation to OMB. CONSENSUS: 2/3 vote in favor of a motion. Abstentions do not count towards vote; in case of multiple abstentions. Staff will monitor and record consensus voting.
STAFF: Will organize and coordinate meeting, record ERP actions and decisions, draft ERP report and distribute after chair approval, work with chair and OMB liaison to complete checklist and assemble recommendation package for OMB.
Documentation Needed
Method Safety Evaluation
Reference Materials
Evidence of Single Laboratory Validation or equivalent
Evidence of Reproducibility Assessment
Published First Action OMA
Method Performance versus SMPR or acceptance criteria
Final draft of First Action OMA to be considered for status update
Rationale or Justification for Repeal or Continuance of First Action OMA
15
9/13/2018
OMB Meeting for Review of ERP Recommendations
OMB Review (renders decision on recommendation)
ERP Chair/or designee (addresses questions/comment)
OMB Liaison (presents recommendation)
Modifications to Official Methods • Types of Modifications – Editorial
– Major – Minor
• Applicable to First Action and Final Action OMA
• Relevant to all ERPs
16
9/13/2018
Editorial Modifications • The applicant must submit a written explanation of the change(s) including a statement that the modification does not alter the validated performance of the method.
• Examples include: Typos or editorial corrections or clarifications that strengthen instruction.
• Methods that have undergone an editorial modification will retain the same number.
Editorial Changes
• Editorial changes to methods only require AOAC staff review and the change is made to the OMA with changes noted in next printed edition of OMA. • A list of the methods with editorial modifications will be published in Inside Laboratory Management and on the Website.
17
9/13/2018
Minor Modifications • Results in no changes to the current validated performance. There is no significant effect to the results. The method will retain the original number. • Supporting data to justify the proposed modification must be submitted. Equivalency data is required unless adequate Justification to exclude this data is provided. • Examples include: Reagent change, a change in a column or consumables that do not impact the validated method performance.
Major Modifications • Results in a change to the current validated performance of the method. • This level of modification will result in a new method as part of AOAC standards development and will receive a new method number. • Examples include: significant change to the technology, sample preparation, or chemistry.
18
9/13/2018
Minor & Major Modifications
Based on AOAC staff review, a public comment period for the proposed modification is required.
Applicant Options
• Following the comment period, any comments are reconciled and recommends a response to the applicant. • The applicant can decide to proceed based on the reconciled comments
19
9/13/2018
Pathways for Minor & Major Modification • If applicant decides to
proceed, an ERP is formed – Level of modification determined by ERP
– Applies to
modifications of First Action and Final Action methods
Documentation and Communication • AOAC carefully documents the actions of Stakeholder Panel, Working Groups, and Expert Review Panels • AOAC will prepare summaries of the meetings – Communicate summaries to the stakeholders – Publish summaries in the Referee section of AOAC’s Inside Laboratory Management • AOAC publishes its voluntary consensus standards and Official Methods – Official Methods of Analysis of AOAC INTERNATIONAL – Journal of AOAC INTERNATIONAL • AOAC publishes the status of standards and methods in the Referee section of AOAC’s Inside Laboratory Management
20
9/13/2018
Requirements for ERP Service
Must have demonstrated expertise in the method, technology, analyte/matrix, etc… Be a subject matter expert. Must be able to attend ERP meetings Must be able to complete assigned reviews on time Must be prepared to speak on the method and share reviews during the meeting Must be proactive in tracking assigned First Action Official Methods Must be able to assist in peer reviewing paper for publication Must sign and submit AOAC Volunteer Acceptance Form
General Expectations for ERPs • You can expect to have a minimum of three weeks to review methods prior to ERP meeting. – You are requested to submit written reviews by specified deadline. Please alert staff if you are not able to complete on time. – You may have individually assigned methods to review or all of the methods to review. Please be prepared to discuss these methods during meeting. – You may use the OMA appendices as guidance for types of validation work that can be expected. If additional information is needed, please ask staff. • ERP Meeting Quorum – If there is no quorum, there is no official meeting. Please alert staff as early as possible if you are not able to attend a meeting. • ERP Consensus – ERP consensus may not reflect your own personal view – There may be times when a method may not meet all of the criteria exactly; however, the ERP can adopt the method.
21
9/13/2018
Ethical Expectations of AOAC Expert Review Panel Members • Respect for your peer ERP members and chair – Each member has been vetted for expertise relevant to the review of the method(s) in the ERP • Be considerate of each others perspectives and points of view • Be considerate of the ERP’s consensus even if you disagree – Inform staff as early as possible if you cannot attend the scheduled ERP meeting • Be considerate in that your absence can impact the quorum of the ERP and its ability to have an official meeting to make decisions – Notify staff and/or disclose in the ERP meeting if you have a direct or perceived conflict of interest for a specific method • Please review AOAC’s policy on Volunteer Conflict of Interest Ethical Expectations of Expert Review Panel Members (con’t) • Respect for Method Authors and Intellectual Property – Each Method Author is encouraged to attend the ERP meeting – Each candidate methods (not yet adopted or published as Official Methods of Analysis of AOAC INTERNATIONAL ) are still the intellectual property of the method author. Therefore, the information is shared only with the vetted ERP members and is available during the meetings. Please do not distribute the information without expressed written permission from an appropriate AOAC staff liaison. – Be clear about and justify how additional recommended work is a requirement for First Action, a requirement for Final Action consideration, or something recommended, but not necessary. – Keep your focus on the science
22
9/13/2018
ERP Chair Responsibilities
Before Meeting
During Meeting
Moderate discussions based on agenda
Work with staff on meeting coordination
Engage staff to encourage members to reach decision points
Review submitted and/or assigned methods
Engage staff on procedural questions
Review method reviews if applicable
Engage discussion on feedback mechanism
Review SMPR(s) and/or relevant guidance and criteria
ERP Chair Responsibilities
Other Efforts and Recognitions Can nominate methods for OMB Award
After Meeting Review Meeting Report and Approve Final Version
Can nominate ERP members for OMB Award
Assist with any follow up on methods
Can assist in identifying methods for review
Assist in Publication Reviews
Can serve as a guest editor for the Journal
23
9/13/2018
Roles and Responsibilities
AOAC Official Methods Board Vet and approve stakeholder panel chair & voting members Vet and approve ERP membership and AOAC Experts Render decisions on status of First Action methods (Final Action, repeal, etc…) Assign a liaison to each stakeholder panel and ERP Coordinate OMB Awards AOAC Expert Review Panels Review methods and meet in person to render decisions on methods for First Action Official Methods SM status. Track First Action Official Methods SM and modify, if necessary Recommend First Action methods after 2 years or less to OMB for Final Action, continuance, or Repeal Participate in Consulting Service and PTM reviews for OMA and harmonized PTM and harmonized OMA method studies AOAC Experts Review and approve PTM validationtesting protocol documentation Peer review of PTM validation manuscript and supporting documentation AOAC Research Institute ‐ PTM Expert Reviewers Peer Review of PTM validationmanuscripts and supporting documentation
AOAC Research Institute Independent Laboratories Conduct independent evaluation of candidate method using AOAC approved testing protocols AOAC Stakeholder Panels Develop voluntary consensus standards Assign working groups to draft standards method performance requirements Voting members demonstrate consensus on behalf of stakeholders AOAC Staff Coordinate method reviews and method approval activities Coordinate OMB meetings Provide trainings and orientations Maintain website and communication Document and publish actions and decisions Coordinate standards development activities Publish standards and methods AOAC Research Institute Technical Consultants Draft validation protocols in Consulting Service for assigned methods
Facilitate PTM evaluation of assigned candidate methods Facilitate comments/responses for assigned OMA reviews
Questions?
Thank you
24
Validation Study to Demonstrate the Equivalence of a Matrix Extension and Minor Modifications of Assurance® GDS for E . coli O157:H7 Tq Method to the Reference Culture Method Summary of Proposed Changes The objective of this study is to demonstrate equivalance of a matrix extension and two minor modifications for the GDS EHEC method to the reference methods. First is the extension of matrices to include finely textured beef (FTB, 375 g), carcass cloths and green onions (25 g). The FTB samples will be frozen when added to the media for enrichment. Secondly, a modification is the use of an automated DNA extraction system, Assurance® GDS PickPen TM PIPETMAX® (PPMX), for sample preparation. The PPMX is a modular automated system to complete the routine pipetting and the PickPen TM IMS concentration of target organism. The machine automates all reagent dispensing, performs the PickPen TM IMS protocols and dispenses concentrated DNA into the Amplification tubes. The PPMX will be validated as an alternative sample preparation method alongside the standard method for GDS EHEC for beef trim, FTB and carcass cloths. Finally, the third modification is an alternative amplification tube size for the assay. A smaller alternative tube size for the amplification tube (HT, High Throughput) has previously been developed for GDS EHEC. The internal amplification contents remain unchanged; only the exterior tube size is modified. The HT amplification tube will also be validated alongside the standard (size) amplification tube for GDS EHEC for use by the manual and automated DNA extraction methods by demonstrating consistency of the two tube formats. The HT amplification tubes will be analyzed on beef trim, FTB and carcass cloth.
commercially. (a) Assurance® GDS Rotor-Gene® Q thermocycler – (Cat. No. 73070BC) (b) Laptop computer with Assurance GDS Rotor-Gene software – (Cat. No. 73071BC), version 2.3.103 (c) Assurance GDS Rotor-Gene Q Rotor-Disc – (Cat. No. 73039BC), 36-well (d) Assurance GDS Rotor-Gene Q locking ring (Cat. No. 73040BC), 36-well, to lock the Rotor-Disc (e) Adhesive film strips – (Cat. No. 73026BC), 200/case (f) Sample wells and sample wells base – samples wells (Cat. No. 73044BC), 200/case and sample wells base (Cat. No. 73043BC), 10/case (g) Resuspension plate – (Cat. No. 73006BC), 120/case (h) Gel cooling block – (Cat. No. 73083BC), box of 2 units (i) PickPen TM device – (Cat. No. 73091BC), 8 magnets (j) PickPen TM tips – (Cat. No. 73087BC), racked, 96/box (k) Adjustable micropipette – capable of accurately dispensing 1000 µL (l) 8-channel micropipette – capable of dispensing 30 µL (m) Filter barrier micropipette tips – (Cat. No. 73034BC), 10 – 50 µL, 960/case, or equivalent and (Cat. No. 73030BC), 200–1000 µL, 800/case, or equivalent (n) Adjustable repeat pipette – capable of dispensing 20 µL, 45 µL, 500 µL, and 1000 µL (o) Repeat pipette tips – (Cat. No. 73001BC), 0.5 mL, 100/case, or equivalent and (Cat. No. 73036BC), 10 mL, 100/case, or equivalent (p) Incubator – capable of maintaining and 42 ± 1oC (q) Freezer – capable of maintaining -20 ± 2oC (r) Refrigerator – capable of maintaining 5 ± 3oC, for storing GDS kits and enriched samples (s) Microtiter plate vortex mixer – (Cat. No. 73063BC), IKA® MS 3 or equivalent, capable of maintaining 900 RPM (t) Laboratory paddle blender – Stomacher® 400 and 3500 or equivalent, for sample homogenization (u) Sterile laboratory filtered Stomacher bags – Nasco or equivalent (v) Top-loading balance – minimum 4,000 g capacity, sensitivity of 0.1 g (w) Digital countdown UP timer (a) Amplification and detection instrument. —Assurance GDS Rotor-Gene ® , or equivalent. (b) Computer and software. (c) Sample concentration device. —PickPen ® . (d) Vortex mixer. (e) Sample wells and sample wells base.
AOAC Official Method 2005.04 Escherichia coli O157:H7 in Selected Foods Assurance ® GDS E. coli O157:H7 Tq
First Action 2005 Final Action 2008
Revised First Action2011 Revised First Action2012 Revised First Action 2019
(Applicable to detection of E. coli O157:H7 in raw ground beef, beef trim, finely textured beef, carcass cloths, orange juice, apple juice, fresh vegetables, green onions and sprout process water . ) Caution : E. coli O157:H7 are pathogenic bacteria. Symptoms of infection include bloody diarrhea and cramping, little to no fever, and hemolytic uremic syndrome. Decontaminate all spent media and equipment used in test prior to disposal of media or re-use of equipment. Note : Use proper environmental and procedural precautions for the handling of reagents and equipment when performing genetic- based assays. See Table 2005.04 (original study, 2005) for the results of the interlaboratory study supporting acceptance of the method. A. Principle The Assurance ® GDS E. coli O157:H7 Tq method is a gene- based assay that uses specific primers and proprietary probes directed against a highly conserved DNA sequence in the target organism. The method uses enrichment in modified EHEC ( mEHEC ® ) broth. After enrichment, populations of target microorganisms are concentrated by using a proprietary concentration (PickPen TM ) device and reagents or processed with an automated concentration instrument. The concentrate is transferred to a conical reaction vessel containing amplification reagents (Amplification tube) . The vessel is sealed and placed in a rotary thermocycler PCR instrument (Rotor-Gene®) which uses proprietary software placed in an instrument which allows for simultaneous amplification and detection. All tests, positive and negative, are indicated at the end of analysis, as well as the results of a procedural control which is contained in every Amplification tubereaction vessel . Assurance ® GDS E. coli O157:H7 Tq detects strains of E. coli O157:H7 and toxigenic E. coli O157:NM. This method is designed to be highly selective, and does not detect microorganisms that are potential cross-reactors in antibody-based assays, including E. coli O157 that are not H7 or NM, and other microorganisms that express O157 antigen but are not E. coli O157:H7. B. Apparatus Items are available from MilliporeSigma (www.sigmaaldrich.com). Ordering information—SigmaAldrich, 12822 SE 32nd St, Bellevue, WA 98005, Tel: 425-586-3300, email: bcs_cust_svc_bellevue@milliporesigma.comItems ( a )–( l ) are available from BioControl Systems (Bellevue, WA, USA; www.biocontrolsys.com). Items ( m )–( n ) are available
Formatted: Superscript Formatted: Superscript
Formatted: Superscript
(f) Resuspension plate. (g) Gel cooling block. (h) PickPen tips in box. (i) Sealing film with adhesive backing.
(j) Dedicated pipets.— Eight-channel pipettor, capable of dispensing 30 µL; repeater pipettor; and micropipettors capable of accurately dispensing 10 and 35 µL. (k) Repeat pipettor tips.— 0.5 and 10 mL.
© 2012 AOAC INTERNATIONAL
1)
Amplification (Amp) tubes Tq, containing lyophilized
(l) Micropipettor tips.— Filter-barrier, volumes 20–200 and 200–1000 µL. (m) Incubator.— Maintaining 42 ± 1°C. (x) Masticator. —Seward 400 (Seward Ltd, Worthington, UK; www.seward.co.uk), or equivalent. Optional : The following apparatus are required for use with Assurance® GDS for E . coli O157:H7 Tq assay (576 tests, Cat. No. 71007-576) due to the reduced size of the Amplification tubes. (a) Amplification tube holder – (Cat. No. 79134BC), variable spacing, 72-well (b) Amplification tube holder lid – (Cat. No. 79184BC), variable spacing, 72-well (c) Amplification tube capping tool – (Cat. No. 79132BC) (d) Amplification tube cap rack – (Cat. No. 79197BC), 72-well (e) Aluminum cooling block – (Cat. No. 73093BC), 72-well (f) Assurance GDS Rotor-Gene® Q Rotor-Disc – (Cat. No. 73076BC), 72-well (g) Assurance GDS Rotor-Gene® Q locking ring (Cat. No. 73084BC), 72-well, to lock the Rotor-Disc For information on additional materials needed for sample analysis by the Assurance® GDS PickPen® PIPETMAX® (PPMX) please see the PPMX User Manual (No. 55240BC). (n) C. Media and Reagents Item ( a ) is available from BioControl Systems. Items ( b )–( e ) are available in the Assurance GDS E. coli O157:H7 Tq test kit from BioControl Systems. Item ( f ) is available commercially. Items are available from MilliporeSigma (www.sigmaaldrich.com). Ordering information—SigmaAldrich, 12822 SE 32nd St, Bellevue, WA 98005, Tel: 425-586-3300, email: Prewarm 1 L sterile deionized water at 42 ± 1˚C overnight. On day of use aseptically transfer 31.6 g mEHEC® media into the prewarmed sterile water. Gently mix to fully rehydrate the media. Use prepared broth within 6 h. Alternatively, mEHEC® broth can be prepared and autoclaved at 121˚C for 15 min. Autoclaved broth must be prewarmed to 42 ± 1˚C overnight prior to sample addition. (b) Assurance® GDS for E . coli O157:H7 Tq Assay – (100 tests; Cat. No. 71007-100), kit includes bcs_cust_svc_bellevue@milliporesigma.com. (a) modified EHEC broth (mEHEC®)— Optional : Assurance® GDS PickPenTM PipetMax® (Cat. No. 79100BC)
reagents for E . coli O157:H7, 1 pouch, Cat. No. 34739– 100 2)
O157 Concentration Reagent—One bottle, Cat. No.
Formatted: Justified, Indent: Left: 0.08", First line: 0.12", Right: 0.08", Space Before: 3.8 pt, Line spacing: Multiple 1.1 li, Tab stops: 0.38", Left + Not at 0.45"
34030 – 100 3)
Resuspension Buffer Tq—One bottle, Cat. No. 34724 –
100
Formatted: Font: Bold Formatted: Font: Italic Formatted: Font: Italic
4) Wash Solution—One bottle, Cat. No. 34043 – 100 (c) Assurance® GDS for E . coli O157:H7 Tq Assay – (576 tests; Cat. No. 71007-576), kit includes 1) Amplification (Amp) tubes Tq, containing lyophilized reagents for E . coli O157:H7, 1 pouch, Cat. No. 34739– 576 2) O157 Concentration Reagent—Two bottles, Cat. No. 34030 – 576 3) Resuspension buffer Tq—Two bottles, Cat. No. 34724 – 576 4) Wash solution—Three bottles, Cat. No. 34043 – 576 (d) Assurance® GDS for E . coli O157:H7 Tq Assay – (576 tests; Cat. No. 71007-576ATM), for use with automated (PipetMax) sample preparation, kit includes 1) Amplification (Amp) tubes Tq, containing lyophilized reagents E . coli O157:H7, 4 pouches, Cat. No. 34749-144 2) O157 Concentration Reagent—Two bottles, Cat. No. 34030 – 576 The following reagents are also necessary but sold separately: 3) Resuspension buffer Tq— Cat. No. 34724-100C 4) Wash solution— Cat. No. 61031-100 (a) BioControl mEHEC broth .—Prewarm 1 L sterile deionized water at 42 ± 1°C overnight. On day of use, aseptically transfer 31.6 g mEHEC into the prewarmed sterile water. Gently mix to dissolve thepowder.Usepreparedmediumwithin6 h.Alternatively, mEHEC can be prepared and autoclaved at 121°C for 15 min. Autoclaved media must be prewarmed to 42 ± 1°C overnight prior to sample addition. (b) Concentration reagent . (c) Resuspension buffer . (d) Amplification tubes containing lyophilized reagents for E. coli O157:H7 . (e) Wash solution. (f) Diagnostic media and reagents .—Necessary for cultural confirmation of positive Assurance GDS samples. Reagents ( b ) and ( d ) must be stored at 2–8°C when not in use. D. General Instructions Reagents (1) through (4) must be stored at 5 ± 3°C when not in use.
Formatted: Font: Bold
Formatted: Font: 9 pt Formatted: Indent: Left: 0.25", Space Before: 3.8 pt, No bullets or numbering, Tab stops: 0.38", Left + Not at 0.42"
Formatted: Font: Not Italic Formatted: Normal, No bullets or numbering
(a) Store Assurance® GDS for E . coli O157:H7 Tq kit components at 5 ± 3°C. Kit expiration is provided on the product box label. Do not use Assurance GDS for E E . coli O157:H7 Tq reagents that have expired. The quality of expired materials is not warrantied. (b) Read the kit instructions carefully before use. Failure to do
Formatted: Font: Italic Formatted: Font: Italic
© 2012 AOAC INTERNATIONAL
so may lead to inaccurate results. (c) When incubation indicates 42˚C, the allowed incubation range is 42 ± 1˚C.
potentially infectious materials. Decontaminate and dispose of materials in accordance with good laboratory practices and in accordance with local, state, and federal regulations. All enrichment broths should be sterilized following any culture- based confirmatory steps.
Safety Precautions
General Preparation
Assurance® GDS for E . coli O157:H7 Tq kit
(a) Use aseptic techniques. (b) Use filter laboratory bags during enrichment to minimize particulates. (c) Separate work areas for the following: media preparation, sample preparation, and pathogen detection. (d) Clean the work stations and lab equipment with a disinfectant of choice before and after use. (Sodium hypochlorite solution, phenol solution, Quaternary ammonium solution, etc.) (e) Do not reuse kit disposables.
This product is not intended for human or veterinary use. Assurance® GDS for E . coli O157:H7 Tq must be used as described in the package insert. Contents of the test may be harmful if swallowed or taken internally. If possible, maintain separate work zones and dedicated equipment and supplies for sample preparation and amplification and detection. It is recommended to utilize both positive and negative control samples. Do not use the test kit beyond expiration date on the product box label. Decontaminate and dispose of materials in accordance with good laboratory practices and in accordance with local, state, and federal regulations. Do not open or autoclave used Amplification tubes. After run is complete, place used Amplification Tubes into a sealed container with 10% bleach solution for a minimum of 15 min or else place the Amplification tubes into a double bag and dispose outside of the lab. Assurance® GDS Rotor-Gene® Improper use of the Assurance® GDS Rotor-Gene® thermocycler may cause personal injuries or damage to the instrument. Some components may pose a risk of personal injury due to excessive heat if improperly handled. For safe use, the instrument must only be operated by qualified laboratory personnel who have been appropriately trained. Servicing of instrument must only be performed by MilliporeSigma Service Engineers. Sample Enrichment The user should read, understand and follow all safety information in the instructions for the Assurance® GDS E . coli O157:H7 Tq Kit. Retain the safety instructions for future reference. To reduce the risks associated with exposure to chemicals and biohazards, perform pathogen testing in a properly equipped laboratory under the control of trained personnel. Always follow standard laboratory safety practices, including wearing appropriate protective apparel and eye protection while handling reagents and contaminated samples. Avoid contact with the contents of the enrichment media and reagent tubes after amplification. Dispose of enriched samples according to current industry standards. E . coli O157:H7 Precautions E . coli O157:H7 is a biosafety level 2 organism. Biological samples, such as enrichments, have the potential to transmit infectious diseases. Follow all applicable local, state/provincial, and/or national regulations on disposal of biological wastes. Wear appropriate protective equipment which includes but is not limited to: protective eyewear, face shield, clothing/lab coat, and gloves. All work should be conducted in properly equipped facilities utilizing the appropriate safety equipment (for example, physical containment devices). Individuals should be trained in accordance with applicable regulatory and company/institution requirements before working with
(f) Change pipette tips in between samples. (g) Wear personal protective equipment (PPE).
DNA Extraction and Purification
(a) Change pipette tips in between samples.
DNA Amplification
(a) Use aseptic technique. (b) Change pipette tips between samples. (c) Use gloves and protective laboratory wear. (d) Do not touch any PCR equipment and supplies without wearing gloves. (e) Avoid bubble formation. General Preparation (a) Ensure the gel cooling block has changed from pink (warm) to purple (frozen) in color and has been stored in the freezer (-20 ± 5 °C) for a minimum of 6 h. (b) Verify that the Assurance® GDS Rotor-Gene® program set up and data entry has been completed prior to transferring the samples. D. Sample Enrichment (a) (a) Prewarm mEHEC® broth to 42 C ± 1°C before use. (b) (b) 25 g sample: Aseptically weigh 25 g sample into 225 mL prewarmed mEHEC® broth. Stomach or homogenize for 2 min or mix well by hand. Incubate samples for 6.5–18 h at 42 ±1°C. For green onions or sprout irrigation water, incubate for 8 – 16 h. (c) (c) 375 g sample (raw beef, finely textured beef and leafy vegetables): Aseptically weigh 375 g sample into 1500 mL prewarmed mEHEC® broth. Stomach or homogenize for 2 min or mix well by hand. Incubate samples for 8–18 h at 42 ±1°C. For frozen finely textured beef, incubate for 10 – 18 h. (a) (d) carcass cloth: Use Fremonta MicroTallyTM (https://www.fremonta.com/microtally) cloth or equivalent for sampling. Collect carcass cloth per FSIS (FSIS Directive 10,010.1 Rev. 4). Add 200 mL pre-warmed mEHEC® broth to cloth contained in sample bag, assuring that cloth is completely
© 2012 AOAC INTERNATIONAL
immersed in media. Stomach or homogenize for 2 min or mix well by hand. Incubate for 8 – 16 h at 42 ±1°C.Aseptically weigh 25 g sample into 225 mL prewarmed (42 ± 1°C) mEHEC, C ( a ). Homogenize samples well with masticator, B ( n ). Incubate samples for 6.5–18 h at 42 ± 1°C. For sprout irrigation water, incubate for 8–18 h. (b) Aseptically weigh 375 g sample into 1500 mL prewarmed (42 ± 1°C) mEHEC. Homogenize samples well with masticator. Incubate samples for 8–18 h at 42 ± 1°C.
magnets and tap tips gently to release particles into the Resuspension Buffer Tq. Cover the resuspension plate with adhesive film. E. 10) Repeat steps (6) through (9) for all samples using new tips for each strip of samples. Note : Wear new gloves prior to handling reagents. (a) Mix concentration reagent, C ( b ), in a vortex mixer. Transfer 20 µL to each of the required number of Assurance GDS sample wells (one well/sample), B ( e ), using a repeater pipettor, B ( j ), and 0.5 mL pipet tip, B ( k ). Cover sample wells with adhesive film strips, B ( i ). (b) Transfer 1.0 mL of wash solution, C ( e ), to additional sample wells (one well/sample), using a repeat pipettor and 10 mL pipet tip. Cover sample wells with adhesive film strips. (c) Transfer 45 µL of resuspension buffer, C ( c ), to the sample wells in the resuspension plate, B ( f ), using a repeat pipettor and 0.5 mLpipet tip. Cover resuspension plate with adhesive film strips. (d) From one strip of sample wells, ( a ), carefully remove adhesive film. Using a micropipettor, add 1 mL of incubated sample to each sample well containing concentration reagent. Avoid transferring food particles. A new pipet tip must be used for each sample. Cover each strip of sample wells with a new adhesive film strip prior to adding samples to a new row. Immediately return samples to incubator. (e) Place sealed sample wells held in the sample wells base atop the vortex mixer, B ( d ), and vortex at approximately 900 rpm for 5–15 min. If necessary, adjust vortex speed to avoid liquid contact with adhesive film. After vortexing is completed, remove sample wells base from vortex mixer.
Formatted: Normal, Indent: Left: 0.08", No bullets or numbering Formatted: Font: Arial, 8 pt, Italic
Note:
Contact Technical Services for recommended
procedures for testing alternate sample sizes.
Formatted: Indent: Left: 0.2", No bullets or numbering Formatted: Font: Arial, 8 pt, Bold, Italic Formatted: Font: Not Bold Formatted: Font: Not Bold, Superscript Formatted: Font: Not Bold
F. Sample Preparation Protocol
Note: Sample prep can also be completed using the Assurance ® GDS PickPen TM PIPETMAX ® (PPMX). For automation setup procedures please see the Assurance ® GDS PPMX User Manual (Cat. No. 55240BC).
(a) Sample Preparation Procedure
Note : Change gloves prior to handling reagents.
Formatted: Font: Not Bold, Not Italic Formatted: Font: Not Italic Formatted: Font: Not Bold
1) Aliquot 20 µ L, using a 0.5 mL pipette tip and the repeater pipette, of homogenized O157 Concentration Reagent into the Assurance ® GDS sample wells (1 well/sample). Cover sample wells with adhesive film strips. 2) Add 1.0 mL, using a 10 mL pipette tip and the repeater pipette, of Wash Solution into another set of Assurance ® GDS sample wells (1 well/sample). Cover sample wells with adhesive film strips. 3) Add 45 µ L of Resuspension Buffer Tq, using a 0.5 mL pipette tip and repeater pipette, to the required number of wells in the Resuspension plate. Cover resuspension plate with adhesive film strips. 4) Carefully remove adhesive film strip from 1 strip of sample wells. Remove enriched sample after incubation. Add 1.0 mL of the enriched sample to each sample well that contains the Concentration Reagent. Avoid transferring food particles. A new pipette tip must be used for each sample. Seal each row of the sample wells with new adhesive film strip. Immediately return samples to 42 ° C incubator. 5) Place the sealed sample wells containing O157 Concentration Reagent on the plate vortex mixer at approximately 900 RPM for 5- 15 min. Note: If necessary, adjust the RPM to be certain that liquid does not contact adhesive film. 6) Carefully remove and discard adhesive film strip from 1 strip of samples. Remove corresponding film strip from sample wells containing the Wash Solution. 7) Load tips onto the PickPen TM device, ensuring that the tips are firmly in place on the PickPen TM tool. Extend the PickPen TM magnets and insert tips into the first strip of sample wells. Stir gently for at least 30 sec while continually moving up and down from the surface to the bottom of the wells. Gently tap the PickPen TM tips against the side of the sample wells to remove excess media droplets. 8) Transfer the PickPen TM tips to corresponding sample wells containing Wash Solution and gently swirl for 5 – 10 sec. Tap the PickPen TM tips against the side of the wells to remove excess Wash Solution droplets. Note: Do not release particles into Wash Solution. 9) Transfer PickPen TM tips to the corresponding row of the prepared resuspension plate. With tips submerged, retract the PickPen TM
Formatted: Font: Not Bold, Superscript Formatted: Font: Not Bold Formatted: Font: Not Bold, Superscript Formatted: Font: Not Bold Formatted: Font: Not Bold, Superscript Formatted: Font: Not Bold Formatted: Font: Not Bold, Superscript Formatted: Font: Not Bold Formatted: Font: Not Bold, Superscript Formatted: Font: Not Bold Formatted: Font: Not Bold, Superscript Formatted: Font: Not Bold Formatted: Font: Not Bold, Superscript Formatted: Font: Not Bold Formatted: Font: Not Bold, Superscript Formatted: Font: Not Bold
© 2012 AOAC INTERNATIONAL
Made with FlippingBook Online newsletter