title: Adaptive diagnosis in distributed systems.
abstract: Real-time problem diagnosis in large distributed computer systems and networks is a challenging task that requires fast and accurate inferences from potentially huge data volumes. In this paper, we propose a cost-efficient, adaptive diagnostic technique called active probing. Probes are end-to-end test transactions that collect information about the performance of a distributed system. Active probing uses probabilistic reasoning techniques combined with information-theoretic approach, and allows a fast online inference about the current system state via active selection of only a small number of most-informative tests. We demonstrate empirically that the active probing scheme greatly reduces both the number of probes (from 60% to 75% in most of our real-life applications), and the time needed for localizing the problem when compared with nonadaptive (preplanned) probing schemes. We also provide some theoretical results on the complexity of probe selection, and the effect of "noisy" probes on the accuracy of diagnosis. Finally, we discuss how to model the system's dynamics using dynamic Bayesian networks (DBNs), and an efficient approximate approach called sequential multifault; empirical results demonstrate clear advantage of such approaches over "static" techniques that do not handle system's changes.
authors.affiliation: IBM T.J. Watson Research Center, Hawthorne, NY 10532, USA. rish@us.ibm.com
authors.forename: Irina
authors.initials: I
authors.lastname: Rish
authors.forename: Mark
authors.initials: M
authors.lastname: Brodie
authors.forename: Sheng
authors.initials: S
authors.lastname: Ma
authors.forename: Natalia
authors.initials: N
authors.lastname: Odintsova
authors.forename: Alina
authors.initials: A
authors.lastname: Beygelzimer
authors.forename: Genady
authors.initials: G
authors.lastname: Grabarnik
authors.forename: Karina
authors.initials: K
authors.lastname: Hernandez
mesh_terms: D000465:Algorithms; D016477:Artifacts; D001185:Artificial Intelligence; D003195:Computer Communication Networks; D003198:Computer Simulation; D016247:Information Storage and Retrieval / Q000379:methods*; D015233:Models, Statistical; D010363:Pattern Recognition, Automated / Q000379:methods*; D012815:Signal Processing, Computer-Assisted; D013685:Telecommunications