title: Towards brain-activity-controlled information retrieval: Decoding image relevance from MEG signals.
abstract: We hypothesize that brain activity can be used to control future information retrieval systems. To this end, we conducted a feasibility study on predicting the relevance of visual objects from brain activity. We analyze both magnetoencephalographic (MEG) and gaze signals from nine subjects who were viewing image collages, a subset of which was relevant to a predetermined task. We report three findings: i) the relevance of an image a subject looks at can be decoded from MEG signals with performance significantly better than chance, ii) fusion of gaze-based and MEG-based classifiers significantly improves the prediction performance compared to using either signal alone, and iii) non-linear classification of the MEG signals using Gaussian process classifiers outperforms linear classification. These findings break new ground for building brain-activity-based interactive image retrieval systems, as well as for systems utilizing feedback both from brain activity and eye movements.
authors.affiliation: Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland; Helsinki Institute for Information Technology HIIT, Department of Computer Science, University of Helsinki, Helsinki, Finland. Electronic address: jukka-pekka.kauppi@helsinki.fi.
authors.forename: Jukka-Pekka
authors.initials: JP
authors.lastname: Kauppi
authors.affiliation: Heidelberg University HCI/IWR, Heidelberg, Germany; Helsinki Institute for Information Technology HIIT, Department of Computer Science, Aalto University, Espoo, Finland. Electronic address: melih.kandemir@iwr.uni-heidelberg.de.
authors.forename: Melih
authors.initials: M
authors.lastname: Kandemir
authors.affiliation: Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland. Electronic address: veli-matti.saarinen@aalto.fi.
authors.forename: Veli-Matti
authors.initials: VM
authors.lastname: Saarinen
authors.affiliation: Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland. Electronic address: lotta.hirvenkari@aalto.fi.
authors.forename: Lotta
authors.initials: L
authors.lastname: Hirvenkari
authors.affiliation: Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland; MEG Core, Aalto NeuroImaging, Aalto University, Espoo, Finland. Electronic address: lauri.parkkonen@aalto.fi.
authors.forename: Lauri
authors.initials: L
authors.lastname: Parkkonen
authors.affiliation: Helsinki Institute for Information Technology HIIT, Department of Computer Science, University of Helsinki, Helsinki, Finland. Electronic address: arto.klami@cs.helsinki.fi.
authors.forename: Arto
authors.initials: A
authors.lastname: Klami
authors.affiliation: Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland; MEG Core, Aalto NeuroImaging, Aalto University, Espoo, Finland. Electronic address: riitta.hari@aalto.fi.
authors.forename: Riitta
authors.initials: R
authors.lastname: Hari
authors.affiliation: Helsinki Institute for Information Technology HIIT, Department of Computer Science, University of Helsinki, Helsinki, Finland; Helsinki Institute for Information Technology HIIT, Department of Computer Science, Aalto University, Espoo, Finland. Electronic address: samuel.kaski@hiit.fi.
authors.forename: Samuel
authors.initials: S
authors.lastname: Kaski
mesh_terms: D000328:Adult; D000465:Algorithms; D001499:Bayes Theorem; D001921:Brain / Q000502:physiology*; D005133:Eye Movements; D005260:Female; D005403:Fixation, Ocular; D006801:Humans; D007091:Image Processing, Computer-Assisted; D015225:Magnetoencephalography / Q000379:methods*; D008297:Male; D008606:Mental Processes / Q000502:physiology*; D016011:Normal Distribution; D010775:Photic Stimulation; D055815:Young Adult
keywords: Bayesian classification; Gaussian processes; Gaze signal; Image relevance; Implicit relevance feedback; Information retrieval; Magnetoencephalography