title: Setting objective thresholds for rare event detection in flow cytometry.
abstract: The accurate identification of rare antigen-specific cytokine positive cells from peripheral blood mononuclear cells (PBMC) after antigenic stimulation in an intracellular staining (ICS) flow cytometry assay is challenging, as cytokine positive events may be fairly diffusely distributed and lack an obvious separation from the negative population. Traditionally, the approach by flow operators has been to manually set a positivity threshold to partition events into cytokine-positive and cytokine-negative. This approach suffers from subjectivity and inconsistency across different flow operators. The use of statistical clustering methods does not remove the need to find an objective threshold between between positive and negative events since consistent identification of rare event subsets is highly challenging for automated algorithms, especially when there is distributional overlap between the positive and negative events ("smear"). We present a new approach, based on the Fβ measure, that is similar to manual thresholding in providing a hard cutoff, but has the advantage of being determined objectively. The performance of this algorithm is compared with results obtained by expert visual gating. Several ICS data sets from the External Quality Assurance Program Oversight Laboratory (EQAPOL) proficiency program were used to make the comparisons. We first show that visually determined thresholds are difficult to reproduce and pose a problem when comparing results across operators or laboratories, as well as problems that occur with the use of commonly employed clustering algorithms. In contrast, a single parameterization for the Fβ method performs consistently across different centers, samples, and instruments because it optimizes the precision/recall tradeoff by using both negative and positive controls.
authors.affiliation: Department of Biostatistics & Bioinformatics, Duke University, Durham, NC, USA; Duke Center for AIDS Research, Duke University, Durham, NC, USA; Duke External Quality Assurance Program Oversight Laboratory, Duke University, Durham, NC, USA. Electronic address: adam.richards@ecoex-moulis.cnrs.fr.
authors.forename: Adam J
authors.initials: AJ
authors.lastname: Richards
authors.affiliation: Duke Center for AIDS Research, Duke University, Durham, NC, USA; Duke External Quality Assurance Program Oversight Laboratory, Duke University, Durham, NC, USA; Department of Surgery, Duke University Medical Center, Durham, NC, USA.
authors.forename: Janet
authors.initials: J
authors.lastname: Staats
authors.affiliation: Duke Center for AIDS Research, Duke University, Durham, NC, USA; Duke External Quality Assurance Program Oversight Laboratory, Duke University, Durham, NC, USA; Department of Surgery, Duke University Medical Center, Durham, NC, USA.
authors.forename: Jennifer
authors.initials: J
authors.lastname: Enzor
authors.affiliation: Vaccine Branch, Center for Cancer Research, NCI, Bethesda, MD, USA.
authors.forename: Katherine
authors.initials: K
authors.lastname: McKinnon
authors.affiliation: Institute for Genome Sciences and Policy, Duke University, NC, USA.
authors.forename: Jacob
authors.initials: J
authors.lastname: Frelinger
authors.affiliation: Duke External Quality Assurance Program Oversight Laboratory, Duke University, Durham, NC, USA; Duke Human Vaccine Institute, Duke University, Durham, NC, USA.
authors.forename: Thomas N
authors.initials: TN
authors.lastname: Denny
authors.affiliation: Duke Center for AIDS Research, Duke University, Durham, NC, USA; Duke External Quality Assurance Program Oversight Laboratory, Duke University, Durham, NC, USA; Department of Surgery, Duke University Medical Center, Durham, NC, USA.
authors.forename: Kent J
authors.initials: KJ
authors.lastname: Weinhold
authors.affiliation: Department of Biostatistics & Bioinformatics, Duke University, Durham, NC, USA; Duke Center for AIDS Research, Duke University, Durham, NC, USA; Duke External Quality Assurance Program Oversight Laboratory, Duke University, Durham, NC, USA.
authors.forename: Cliburn
authors.initials: C
authors.lastname: Chan
mesh_terms: D000465:Algorithms; D057205:Automation, Laboratory / Q000592:standards; D015415:Biomarkers / Q000097:blood; D016207:Cytokines / Q000097:blood*; D005434:Flow Cytometry / Q000592:standards*; D019983:Guideline Adherence / Q000592:standards; D006801:Humans; D007753:Laboratories / Q000592:standards*; D059021:Laboratory Proficiency Testing / Q000592:standards*; D007963:Leukocytes, Mononuclear / Q000276:immunology*; D015166:Monitoring, Immunologic / Q000592:standards*; D015588:Observer Variation; D017410:Practice Guidelines as Topic / Q000592:standards; D011237:Predictive Value of Tests; D016730:Program Development; D011786:Quality Control; D019984:Quality Indicators, Health Care / Q000592:standards; D015203:Reproducibility of Results; D013048:Specimen Handling / Q000592:standards
chemical_list: D015415:Biomarkers; D016207:Cytokines
keywords: Automated analysis; ICS; Positivity; Rare events; Reproducibility; Standardization
references.citation: Bioinformatics. 2012 Apr 1;28(7):1009-16
references.pmid: 22383736
references.citation: BMC Bioinformatics. 2009 Apr 09;10:106
references.pmid: 19358741
references.citation: Cytometry A. 2011 Feb;79(2):167-74
references.pmid: 21265010
references.citation: Bioinformatics. 2012 Aug 1;28(15):2052-8
references.pmid: 22595209
references.citation: PLoS Comput Biol. 2013;9(7):e1003130
references.pmid: 23874174
references.citation: Nat Rev Genet. 2006 Feb;7(2):119-29
references.pmid: 16418747
references.citation: Nat Methods. 2013 Mar;10(3):228-38
references.pmid: 23396282
references.citation: Nat Protoc. 2006;1(3):1522-30
references.pmid: 17406444
references.citation: Adv Bioinformatics. 2009;:247646
references.pmid: 20049161
references.citation: Nat Biotechnol. 2011 Oct 02;29(10):886-91
references.pmid: 21964415
references.citation: BMC Bioinformatics. 2010 Jul 28;11:403
references.pmid: 20667133
references.citation: Cytometry. 1993 Oct;14(7):702-15
references.pmid: 8243200
references.citation: Cytometry A. 2008 Aug;73(8):693-701
references.pmid: 18496851
references.citation: Nat Immunol. 2010 Nov;11(11):975-8
references.pmid: 20959798
references.citation: Cytometry A. 2011 Jan;79(1):6-13
references.pmid: 21182178
references.citation: BMC Immunol. 2005 Jun 24;6:13
references.pmid: 15978127
references.citation: Am J Transplant. 2011 Mar;11(3):553-60
references.pmid: 21219584
references.citation: Cytometry A. 2006 Sep 1;69(9):1037-42
references.pmid: 16888771
references.citation: BMC Bioinformatics. 2009 May 14;10:145
references.pmid: 19442304
references.citation: Proc Natl Acad Sci U S A. 2009 May 26;106(21):8519-24
references.pmid: 19443687
references.citation: J Immunol Methods. 2011 Jan 5;363(2):143-57
references.pmid: 20727897
references.citation: J Immunol Methods. 2014 Jul;409:54-61
references.pmid: 24727143
references.citation: Cytometry A. 2006 Jun;69(6):541-51
references.pmid: 16604519
references.citation: Cytometry. 1996 Sep 15;26(3):227-30
references.pmid: 8889396