Crowdsourcing as a Novel Method to Evaluate Aesthetic Outcomes of Treatment for Unilateral Cleft Lip.
BACKGROUND
Lack of convenient and reliable methods to grade aesthetic outcomes limits the ability to study results and optimize treatment of unilateral cleft lip. Crowdsourcing methods solicit contributions from a large group to achieve a greater task. The authors hypothesized that crowdsourcing could be used to reliably grade aesthetic outcomes of unilateral cleft lip.


METHODS
Fifty deidentified photographs of 8- to 10-year-old subjects (46 with unilateral cleft lip and four controls) were assembled. Outcomes were assessed using multiple pairwise comparisons that produced a rank order (Elo rank) of nasal appearance and, on a separate survey, by Asher-McDade ratings. Both surveys were repeated to assess reliability. A group of expert surgeons repeated the same tasks on a smaller subset of photographs.


RESULTS
The authors obtained 2500 and 1900 anonymous, layperson evaluations by means of crowdsourcing on each Elo rank and Asher-McDade survey, respectively. Elo rank and Asher-McDade scores were highly reproducible (correlation coefficients, 0.87 and 0.98), and crowd evaluations agreed with those by expert surgeons (0.980 and 0.96 for Elo rank and Asher-McDade score, respectively). Crowdsourcing surveys were completed within 9 hours, whereas the expert surgeons required 3 months. On further analysis of their cleft subject sample set, the authors found that greater initial cleft severity was associated with worse aesthetic outcome.


CONCLUSIONS
Outcomes assessed by crowds were reliable and correlated well with expert assessments. Crowdsourcing allows acquisition of massive numbers of layperson assessments on an unprecedented scale, and is a convenient, rapid, and reliable means of assessing aesthetic outcome of treatment for unilateral cleft lip.


METHODS
Diagnostic, IV.
Seattle, Wash.|From the Division of Plastic Surgery, Department of Surgery, University of Washington; and the Division of Craniofacial and Plastic Surgery, Seattle Children's Hospital.
Raymond W
RW
Tse
Seattle, Wash.|From the Division of Plastic Surgery, Department of Surgery, University of Washington; and the Division of Craniofacial and Plastic Surgery, Seattle Children's Hospital.
Eugene
E
Oh
Seattle, Wash.|From the Division of Plastic Surgery, Department of Surgery, University of Washington; and the Division of Craniofacial and Plastic Surgery, Seattle Children's Hospital.
Joseph S
JS
Gruss
Seattle, Wash.|From the Division of Plastic Surgery, Department of Surgery, University of Washington; and the Division of Craniofacial and Plastic Surgery, Seattle Children's Hospital.
Richard A
RA
Hopper
Seattle, Wash.|From the Division of Plastic Surgery, Department of Surgery, University of Washington; and the Division of Craniofacial and Plastic Surgery, Seattle Children's Hospital.
Craig B
CB
Birgfeld
D016022:Case-Control Studies; D002648:Child; D002971:Cleft Lip / Q000601:surgery*; D063045:Crowdsourcing; D004954:Esthetics; D005260:Female; D006801:Humans; D008297:Male; D017063:Outcome Assessment, Health Care / Q000379:methods*; D010781:Photography; D019651:Plastic Surgery Procedures; D015203:Reproducibility of Results
Asher-McDade C, Roberts C, Shaw WC, Gallager C. Development of a method for rating nasolabial appearance in patients with clefts of the lip and palate. Cleft Palate Craniofac J. 1991;28:385390; discussion 390.
Tobiasen JM, Hiebert JM. Reliability of esthetic ratings of cleft impairment. Cleft Palate J. 1988;25:313317.
Okkerse JM, Beemer FA, Cordia-de Haan M, Heineman-de Boer JA, Mellenbergh GJ, Wolters WH. Facial attractiveness and facial impairment ratings in children with craniofacial malformations. Cleft Palate Craniofac J. 2001;38:386392.
Prahl C, Prahl-Andersen B, van ’t Hof MA, Kuijpers-Jagtman AM. Infant orthopedics and facial appearance: A randomized clinical trial (Dutchcleft). Cleft Palate Craniofac J. 2006;43:659664.
Mosmuller DG, Bijnen CL, Kramer GJ, et al. The Asher-McDade aesthetic index in comparison with two scoring systems in nonsyndromic complete unilateral cleft lip and palate patients. J Craniofac Surg. 2015;26:12421245.
Paiva TS, Andre M, Paiva WS, et al. Developing a visual rating chart for the esthetic outcome of unilateral cleft lip and palate repair. Cleft Palate Craniofac J. 2015;47:5561.
Kuijpers-Jagtman AM, Nollet PJ, Semb G, Bronkhorst EM, Shaw WC, Katsaros C. Reference photographs for nasolabial appearance rating in unilateral cleft lip and palate. J Craniofac Surg. 2009;20(Suppl 2):16831686.
Mosmuller DG, Griot JP, Bijnen CL, Niessen FB. Scoring systems of cleft-related facial deformities: A review of literature. Cleft Palate Craniofac J. 2013;50:286296.
Sharma VP, Bella H, Cadier MM, Pigott RW, Goodacre TE, Richard BM. Outcomes in facial aesthetics in cleft lip and palate surgery: A systematic review. J Plast Reconstr Aesthet Surg. 2012;65:12331245.
Mercado A, Russell K, Hathaway R, et al. The Americleft study: An inter-center study of treatment outcomes for patients with unilateral cleft lip and palate part 4. Nasolabial aesthetics. Cleft Palate Craniofac J. 2011;48:259264.
Paiva TS, Andre M, Paiva WS, Mattos BS. Aesthetic evaluation of the nasolabial region in children with unilateral cleft lip and palate comparing expert versus nonexperience health professionals. Biomed Res Int. 2014;2014:460106.
Sitzman TJ, Allori AC, Thorburn G. Measuring outcomes in cleft lip and palate treatment. Clin Plast Surg. 2014;41:311319.
Galaxy Zoo. Available at: https://www.galaxyzoo.org/. Accessed February 1, 2016.
Khatib F, DiMaio F, Cooper S, et al. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nat Struct Mol Biol. 2012;19:364364.
Chen C, White L, Kowalewski T, et al. Crowd-sourced assessment of technical skills: A novel method to evaluate surgical performance. J Surg Res. 2014;187:6571.
Lendvay TS, White L, Kowalewski T. Crowdsourcing to assess surgical skill. JAMA Surg. 2015;150:10861087.
Cooper S, Khatib F, Treuille A, et al. Predicting protein structures with a multiplayer online game. Nature 2010;466:756760.
Fisher DM, Tse R, Marcus JR. Objective measurements for grading the primary unilateral cleft lip nasal deformity. Plast Reconstr Surg. 2008;122:874880.
Tse R, Booth L, Keys K, et al. Reliability of nasolabial anthropometric measures using three-dimensional stereophotogrammetry in infants with unrepaired unilateral cleft lip. Plast Reconstr Surg. 2014;133:530e542e.
Elo A. The Rating of Chess Players Past and Present. 1978.New York: Arco Publishing.
Asher-McDade C, Brattström V, Dahl E, et al. A six-center international study of treatment outcome in patients with clefts of the lip and palate: Part 4. Assessment of nasolabial appearance. Cleft Palate Craniofac J. 1992;29:409412.
Gkantidis N, Papamanou DA, Christou P, Topouzelis N. Aesthetic outcome of cleft lip and palate treatment: Perceptions of patients, families, and health professionals compared to the general public. J Craniomaxillofac Surg. 2013;41:e105e110.
Desmedt DJ, Maal TJ, Kuijpers MA, Bronkhorst EM, Kuijpers-Jagtman AM, Fudalej PS. Nasolabial symmetry and esthetics in cleft lip and palate: Analysis of 3D facial images. Clin Oral Investig. 2015;19:18331842.
Stebel A, Desmedt D, Bronkhorst E, Kuijpers MA, Fudalej PS. Rating nasolabial appearance on three-dimensional images in cleft lip and palate: A comparison with standard photographs. Eur J Orthod. 2016;38:197201.
Edler R, Rahim MA, Wertheim D, Greenhill D. The use of facial anthropometrics in aesthetic assessment. Cleft Palate Craniofac J. 2010;47:4857.