title: Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps.
abstract: BACKGROUND
  There are over 165,000 mHealth apps currently available to patients, but few have undergone an external quality review. Furthermore, no standardized review method exists, and little has been done to examine the consistency of the evaluation systems themselves.
  
  
  OBJECTIVE
  We sought to determine which measures for evaluating the quality of mHealth apps have the greatest interrater reliability.
  
  
  METHODS
  We identified 22 measures for evaluating the quality of apps from the literature. A panel of 6 reviewers reviewed the top 10 depression apps and 10 smoking cessation apps from the Apple iTunes App Store on these measures. Krippendorff's alpha was calculated for each of the measures and reported by app category and in aggregate.
  
  
  RESULTS
  The measure for interactiveness and feedback was found to have the greatest overall interrater reliability (alpha=.69). Presence of password protection (alpha=.65), whether the app was uploaded by a health care agency (alpha=.63), the number of consumer ratings (alpha=.59), and several other measures had moderate interrater reliability (alphas>.5). There was the least agreement over whether apps had errors or performance issues (alpha=.15), stated advertising policies (alpha=.16), and were easy to use (alpha=.18). There were substantial differences in the interrater reliabilities of a number of measures when they were applied to depression versus smoking apps.
  
  
  CONCLUSIONS
  We found wide variation in the interrater reliability of measures used to evaluate apps, and some measures are more robust across categories of apps than others. The measures with the highest degree of interrater reliability tended to be those that involved the least rater discretion. Clinical quality measures such as effectiveness, ease of use, and performance had relatively poor interrater reliability. Subsequent research is needed to determine consistent means for evaluating the performance of apps. Patients and clinicians should consider conducting their own assessments of apps, in conjunction with evaluating information from reviews.
authors: Payer+Provider Syndicate, Boston, MA, United States. powell@payerprovider.com.
authors: Adam C
authors: http://orcid.org/0000-0001-6519-3120
authors: AC
authors: Powell
authors: John
authors: http://orcid.org/0000-0002-5362-7937
authors: J
authors: Torous
authors: Steven
authors: http://orcid.org/0000-0002-1696-3122
authors: S
authors: Chan
authors: Geoffrey Stephen
authors: http://orcid.org/0000-0001-5407-8725
authors: GS
authors: Raynor
authors: Erik
authors: http://orcid.org/0000-0001-9026-1623
authors: E
authors: Shwarts
authors: Meghan
authors: http://orcid.org/0000-0003-2986-3937
authors: M
authors: Shanahan
authors: Adam B
authors: http://orcid.org/0000-0002-2166-0521
authors: AB
authors: Landman
keywords: evaluation studies; health apps; mental health; mobile applications; ratings