title: User evaluation of the effects of a text simplification algorithm using term familiarity on perception, understanding, learning, and information retention.
abstract: BACKGROUND
  Adequate health literacy is important for people to maintain good health and manage diseases and injuries. Educational text, either retrieved from the Internet or provided by a doctor's office, is a popular method to communicate health-related information. Unfortunately, it is difficult to write text that is easy to understand, and existing approaches, mostly the application of readability formulas, have not convincingly been shown to reduce the difficulty of text.
  
  
  OBJECTIVE
  To develop an evidence-based writer support tool to improve perceived and actual text difficulty. To this end, we are developing and testing algorithms that automatically identify difficult sections in text and provide appropriate, easier alternatives; algorithms that effectively reduce text difficulty will be included in the support tool. This work describes the user evaluation with an independent writer of an automated simplification algorithm using term familiarity.
  
  
  METHODS
  Term familiarity indicates how easy words are for readers and is estimated using term frequencies in the Google Web Corpus. Unfamiliar words are algorithmically identified and tagged for potential replacement. Easier alternatives consisting of synonyms, hypernyms, definitions, and semantic types are extracted from WordNet, the Unified Medical Language System (UMLS), and Wiktionary and ranked for a writer to choose from to simplify the text. We conducted a controlled user study with a representative writer who used our simplification algorithm to simplify texts. We tested the impact with representative consumers. The key independent variable of our study is lexical simplification, and we measured its effect on both perceived and actual text difficulty. Participants were recruited from Amazon's Mechanical Turk website. Perceived difficulty was measured with 1 metric, a 5-point Likert scale. Actual difficulty was measured with 3 metrics: 5 multiple-choice questions alongside each text to measure understanding, 7 multiple-choice questions without the text for learning, and 2 free recall questions for information retention.
  
  
  RESULTS
  Ninety-nine participants completed the study. We found strong beneficial effects on both perceived and actual difficulty. After simplification, the text was perceived as simpler (P<.001) with simplified text scoring 2.3 and original text 3.2 on the 5-point Likert scale (score 1: easiest). It also led to better understanding of the text (P<.001) with 11% more correct answers with simplified text (63% correct) compared to the original (52% correct). There was more learning with 18% more correct answers after reading simplified text compared to 9% more correct answers after reading the original text (P=.003). There was no significant effect on free recall.
  
  
  CONCLUSIONS
  Term familiarity is a valuable feature in simplifying text. Although the topic of the text influences the effect size, the results were convincing and consistent.
authors.affiliation: Information Systems and Technology, Claremont Graduate University, Claremont, CA 91711, United States. gondy.leroy@cgu.edu
authors.forename: Gondy
authors.initials: G
authors.lastname: Leroy
authors.forename: James E
authors.initials: JE
authors.lastname: Endicott
authors.forename: David
authors.initials: D
authors.lastname: Kauchak
authors.forename: Obay
authors.initials: O
authors.lastname: Mouradi
authors.forename: Melissa
authors.initials: M
authors.lastname: Just
mesh_terms: D000328:Adult; D000368:Aged; D000465:Algorithms; D005260:Female; D006801:Humans; D007255:Information Services; D008297:Male; D008875:Middle Aged; D017432:Unified Medical Language System; D014956:Writing; D055815:Young Adult
keywords: consumer health information; evaluation study; health literacy; natural language processing; text simplification
references.citation: Int J Med Inform. 2013 Aug;82(8):717-30
references.pmid: 23639262
references.citation: Br J Soc Psychol. 2002 Mar;41(Pt 1):101-21
references.pmid: 11970777
references.citation: Read Writ. 2010 Jul 1;23(6):701-717
references.pmid: 21691452
references.citation: J Am Med Inform Assoc. 2010 Mar-Apr;17(2):220-3
references.pmid: 20190068
references.citation: AIDS Patient Care STDS. 2003 Jun;17(6):283-98
references.pmid: 12880492
references.citation: J Health Soc Behav. 1983 Dec;24(4):385-96
references.pmid: 6668417
references.citation: Int J Med Inform. 2005 Aug;74(7-8):573-85
references.pmid: 15897005
references.citation: J Am Med Inform Assoc. 2008 Jul-Aug;15(4):496-505
references.pmid: 18436906
references.citation: Res Social Adm Pharm. 2013 Sep-Oct;9(5):503-16
references.pmid: 22835706
references.citation: Digit Libraries Cult Herit Knowl Dissem Future Creat (2011). 2011 Oct;7008:307-310
references.pmid: 26618205
references.citation: AMIA Annu Symp Proc. 2012;2012:522-31
references.pmid: 23304324
references.citation: J Arthroplasty. 2012 May;27(5):716-9
references.pmid: 22000573
references.citation: Stud Health Technol Inform. 2004;107(Pt 2):869-73
references.pmid: 15360936
references.citation: AMIA Annu Symp Proc. 2007 Oct 11;:418-22
references.pmid: 18693870
references.citation: Health Educ Q. 1984 Spring;11(1):1-47
references.pmid: 6392204
references.citation: J Gen Intern Med. 1995 Oct;10(10):537-41
references.pmid: 8576769
references.citation: J Am Med Inform Assoc. 2008 May-Jun;15(3):349-56
references.pmid: 18308983
references.citation: J Vasc Surg. 2012 Jul;56(1):21-6
references.pmid: 22521801
references.citation: Man Ther. 2012 Aug;17(4):318-24
references.pmid: 22464886
references.citation: J Am Med Inform Assoc. 2008 Jul-Aug;15(4):473-83
references.pmid: 18436895
references.citation: Br J Sports Med. 2012 Jul;46(9):675-83
references.pmid: 21504964
references.citation: Perspect Psychol Sci. 2011 Jan;6(1):3-5
references.pmid: 26162106
references.citation: Health Promot Int. 2011 Sep;26(3):338-50
references.pmid: 21149317