title: On-the-fly learning for visual search of large-scale image and video datasets.
abstract: The objective of this work is to visually search large-scale video datasets for semantic entities specified by a text query. The paradigm we explore is constructing visual models for such semantic entitieson-the-fly, i.e. at run time, by using an image search engine to source visual training data for the text query. The approach combines fast and accurate learning and retrieval, and enables videos to be returned within seconds of specifying a query. We describe three classes of queries, each with its associated visual search method: object instances (using a bag of visual words approach for matching); object categories (using a discriminative classifier for ranking key frames); and faces (using a discriminative classifier for ranking face tracks). We discuss the features suitable for each class of query, for example Fisher vectors or features derived from convolutional neural networks (CNNs), and how these choices impact on the trade-off between three important performance measures for a real-time system of this kind, namely: (1) accuracy, (2) memory footprint, and (3) speed. We also discuss and compare a number of important implementation issues, such as how to remove 'outliers' in the downloaded images efficiently, and how to best obtain a single descriptor for a face track. We also sketch the architecture of the real-time on-the-fly system. Quantitative results are given on a number of large-scale image and video benchmarks (e.g.  TRECVID INS, MIRFLICKR-1M), and we further demonstrate the performance and real-world applicability of our methods over a dataset sourced from 10,000 h of unedited footage from BBC News, comprising 5M+ key frames.
authors.affiliation: Visual Geometry Group, Department of Engineering Science, University of Oxford, Oxford, UK.
authors.forename: Ken
authors.initials: K
authors.lastname: Chatfield
authors.affiliation: Visual Geometry Group, Department of Engineering Science, University of Oxford, Oxford, UK.
authors.forename: Relja
authors.initials: R
authors.lastname: Arandjelović
authors.affiliation: Visual Geometry Group, Department of Engineering Science, University of Oxford, Oxford, UK.
authors.forename: Omkar
authors.initials: O
authors.lastname: Parkhi
authors.affiliation: Visual Geometry Group, Department of Engineering Science, University of Oxford, Oxford, UK.
authors.forename: Andrew
authors.initials: A
authors.lastname: Zisserman
keywords: Convolutional neural networks; Face retrieval; Object category retrieval and recognition; Object instance retrieval; On-the-fly