title: Space-time behavior-based correlation-Or-how to tell if two underlying motion fields are similar without computing them?
abstract: We introduce a behavior-based similarity measure which tells us whether two different space-time intensity patterns of two different video segments could have resulted from a similar underlying motion field. This is done directly from the intensity information, without explicitly computing the underlying motions. Such a measure allows us to detect similarity between video segments of differently dressed people performing the same type of activity. It requires no foreground/background segmentation, no prior learning of activities, and no motion estimation or tracking. Using this behavior-based similarity measure, we extend the notion of 2-dimensional image correlation into the 3-dimensional space-time volume, thus allowing to correlate dynamic behaviors and actions. Small space-time video segments (small video clips) are "correlated" against entire video sequences in all three dimensions (x,y, and t). Peak correlation values correspond to video locations with similar dynamic behaviors. Our approach can detect very complex behaviors in video sequences (e.g., ballet movements, pool dives, running water), even when multiple complex activities occur simultaneously within the field-of-view of the camera. We further show its robustness to small changes in scale and orientation of the correlated behavior.
authors.affiliation: Department of Computer Science and Applied Mathematics, The Weizmann Institute of Science, 76100 Rehovot, Isreal. eli.shechtman@weizmann.ac.il
authors.forename: Eli
authors.initials: E
authors.lastname: Shechtman
authors.forename: Michal
authors.initials: M
authors.lastname: Irani
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D016002:Discriminant Analysis; D007089:Image Enhancement / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D021621:Imaging, Three-Dimensional / Q000379:methods*; D009038:Motion; D010363:Pattern Recognition, Automated / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D013223:Statistics as Topic; D014741:Video Recording / Q000379:methods*