title: Visual Turing test for computer vision systems.
abstract: Today, computer vision systems are tested by their accuracy in detecting and localizing instances of objects. As an alternative, and motivated by the ability of humans to provide far richer descriptions and even tell a story about an image, we construct a "visual Turing test": an operator-assisted device that produces a stochastic sequence of binary questions from a given test image. The query engine proposes a question; the operator either provides the correct answer or rejects the question as ambiguous; the engine proposes the next question ("just-in-time truthing"). The test is then administered to the computer-vision system, one question at a time. After the system's answer is recorded, the system is provided the correct answer and the next question. Parsing is trivial and deterministic; the system being tested requires no natural language processing. The query engine employs statistical constraints, learned from a training set, to produce questions with essentially unpredictable answers-the answer to a question, given the history of questions and their correct answers, is nearly equally likely to be positive or negative. In this sense, the test is only about vision. The system is designed to produce streams of questions that follow natural story lines, from the instantiation of a unique object, through an exploration of its properties, and on to its relationships with other uniquely instantiated objects.
authors.affiliation: Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD 21287; and.
authors.forename: Donald
authors.initials: D
authors.lastname: Geman
authors.affiliation: Division of Applied Mathematics, Brown University, Providence, RI 02912 stuart_geman@brown.edu.
authors.forename: Stuart
authors.initials: S
authors.lastname: Geman
authors.affiliation: Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD 21287; and.
authors.forename: Neil
authors.initials: N
authors.lastname: Hallonquist
authors.affiliation: Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD 21287; and.
authors.forename: Laurent
authors.initials: L
authors.lastname: Younes
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000379:methods*; D021621:Imaging, Three-Dimensional; D015233:Models, Statistical; D010363:Pattern Recognition, Automated; D015203:Reproducibility of Results; D012984:Software
keywords: Turing test; binary questions; computer vision; scene interpretation; unpredictable answers