title: Optimization of anemia treatment in hemodialysis patients via reinforcement learning.
abstract: OBJECTIVE
  Anemia is a frequent comorbidity in hemodialysis patients that can be successfully treated by administering erythropoiesis-stimulating agents (ESAs). ESAs dosing is currently based on clinical protocols that often do not account for the high inter- and intra-individual variability in the patient's response. As a result, the hemoglobin level of some patients oscillates around the target range, which is associated with multiple risks and side-effects. This work proposes a methodology based on reinforcement learning (RL) to optimize ESA therapy.
  
  
  METHODS
  RL is a data-driven approach for solving sequential decision-making problems that are formulated as Markov decision processes (MDPs). Computing optimal drug administration strategies for chronic diseases is a sequential decision-making problem in which the goal is to find the best sequence of drug doses. MDPs are particularly suitable for modeling these problems due to their ability to capture the uncertainty associated with the outcome of the treatment and the stochastic nature of the underlying process. The RL algorithm employed in the proposed methodology is fitted Q iteration, which stands out for its ability to make an efficient use of data.
  
  
  RESULTS
  The experiments reported here are based on a computational model that describes the effect of ESAs on the hemoglobin level. The performance of the proposed method is evaluated and compared with the well-known Q-learning algorithm and with a standard protocol. Simulation results show that the performance of Q-learning is substantially lower than FQI and the protocol. When comparing FQI and the protocol, FQI achieves an increment of 27.6% in the proportion of patients that are within the targeted range of hemoglobin during the period of treatment. In addition, the quantity of drug needed is reduced by 5.13%, which indicates a more efficient use of ESAs.
  
  
  CONCLUSIONS
  Although prospective validation is required, promising results demonstrate the potential of RL to become an alternative to current protocols.
authors.affiliation: Intelligent Data Analysis Laboratory, University of Valencia, Av. de la Universidad, s/n, 46100 Burjassot (Valencia), Spain. Electronic address: pablo.escandell@uv.es.
authors.forename: Pablo
authors.initials: P
authors.lastname: Escandell-Montero
authors.affiliation: Healthcare and Business Advanced Modeling, Fresenius Medical Care, Else-Kröner-Strasse 1, 61352 Bad Homburg, Germany.
authors.forename: Milena
authors.initials: M
authors.lastname: Chermisi
authors.affiliation: Intelligent Data Analysis Laboratory, University of Valencia, Av. de la Universidad, s/n, 46100 Burjassot (Valencia), Spain.
authors.forename: José M
authors.initials: JM
authors.lastname: Martínez-Martínez
authors.affiliation: Intelligent Data Analysis Laboratory, University of Valencia, Av. de la Universidad, s/n, 46100 Burjassot (Valencia), Spain.
authors.forename: Juan
authors.initials: J
authors.lastname: Gómez-Sanchis
authors.affiliation: Healthcare and Business Advanced Modeling, Fresenius Medical Care, Else-Kröner-Strasse 1, 61352 Bad Homburg, Germany.
authors.forename: Carlo
authors.initials: C
authors.lastname: Barbieri
authors.affiliation: Intelligent Data Analysis Laboratory, University of Valencia, Av. de la Universidad, s/n, 46100 Burjassot (Valencia), Spain.
authors.forename: Emilio
authors.initials: E
authors.lastname: Soria-Olivas
authors.affiliation: Healthcare and Business Advanced Modeling, Fresenius Medical Care, Else-Kröner-Strasse 1, 61352 Bad Homburg, Germany.
authors.forename: Flavio
authors.initials: F
authors.lastname: Mari
authors.affiliation: Intelligent Data Analysis Laboratory, University of Valencia, Av. de la Universidad, s/n, 46100 Burjassot (Valencia), Spain.
authors.forename: Joan
authors.initials: J
authors.lastname: Vila-Francés
authors.affiliation: Healthcare and Business Advanced Modeling, Fresenius Medical Care, Else-Kröner-Strasse 1, 61352 Bad Homburg, Germany.
authors.forename: Andrea
authors.initials: A
authors.lastname: Stopper
authors.affiliation: Healthcare and Business Advanced Modeling, Fresenius Medical Care, Else-Kröner-Strasse 1, 61352 Bad Homburg, Germany; Centre for Biomedical Technology at Danube, University of Krems, Dr.-Karl-Dorrek-Strasse 30, 3500 Krems, Austria.
authors.forename: Emanuele
authors.initials: E
authors.lastname: Gatti
authors.affiliation: Intelligent Data Analysis Laboratory, University of Valencia, Av. de la Universidad, s/n, 46100 Burjassot (Valencia), Spain.
authors.forename: José D
authors.initials: JD
authors.lastname: Martín-Guerrero
mesh_terms: D000368:Aged; D000465:Algorithms; D000740:Anemia / Q000097:blood / Q000188:drug therapy* / Q000209:etiology; D001185:Artificial Intelligence; D002908:Chronic Disease; D003661:Decision Support Techniques; D005260:Female; D006397:Hematinics / Q000627:therapeutic use*; D006441:Hemoglobin A / Q000378:metabolism; D006801:Humans; D007676:Kidney Failure, Chronic / Q000150:complications / Q000628:therapy; D008297:Male; D008390:Markov Chains; D008875:Middle Aged; D018579:Patient Selection; D012054:Reinforcement, Psychology; D006435:Renal Dialysis / Q000009:adverse effects*
chemical_list: D006397:Hematinics; D006441:Hemoglobin A
keywords: Chronic kidney disease; Darbepoietin alfa; Fitted Q iteration; Markov decision processes; Reinforcement learning; Renal anemia