title: Outcome-driven Evaluation Metrics for Treatment Recommendation Systems.
abstract: Treatment recommendation systems aim to providing clinical decision supports, e.g. with integration of Computerized Physician Order Entry (CPOE). One of the most significant issue is the quality of recommendations which needs to be quantified, before getting the acceptance from physicians. In computer science, such evaluations are typically performed by applying appropriate metrics that provides a comparison of different systems. However, a big challenge for evaluating treatment recommendation systems is that ground truth is only partially observed. In this paper, we propose an outcome-driven evaluation methodology, and present five metrics (i.e. precision, recall, accuracy, relative risk and odds ratio) with highlight of their statistic meanings in clinical context. The experimental results are based on the comparison of two well-developed treatment recommendation systems (one is knowledge-driven and based on clinical practice guidelines, while the other is data-driven and based on patient similarity analysis), using our proposed evaluation metrics. As a conclusion, physicians are less prone to comply with clinical guidelines, but once following guideline recommendations, it is much more likely to get good outcomes than not following.
authors.affiliation: IBM Research - China.
authors.forename: Jing
authors.initials: J
authors.lastname: Mei
authors.affiliation: IBM Research - China.
authors.forename: Haifeng
authors.initials: H
authors.lastname: Liu
authors.affiliation: IBM Research - China.
authors.forename: Xiang
authors.initials: X
authors.lastname: Li
authors.affiliation: IBM Research - China.
authors.forename: Yiqin
authors.initials: Y
authors.lastname: Yu
authors.affiliation: IBM Research - China.
authors.forename: Guotong
authors.initials: G
authors.lastname: Xie
mesh_terms: D000465:Algorithms; D000066491:Clinical Decision-Making / Q000379:methods*; D003627:Data Interpretation, Statistical; D020000:Decision Support Systems, Clinical / Q000458:organization & administration*; D003661:Decision Support Techniques; D006801:Humans; D017063:Outcome Assessment, Health Care / Q000379:methods*; D013813:Therapy, Computer-Assisted / Q000379:methods*