Vision Group Journal Club

Probabilistic Context-Free Grammars (PCFGs) induce distributions over strings.
Strings can be viewed as observations that are maps from indices to terminals.
The domains of such maps are totally ordered and the terminals are discrete.
We extend PCFGs to induce densities over observations with unordered domains and continuous-valued terminals. We call our extension Spatial Random Tree Grammars (SRTGs). While SRTGs are context sensitive, the inside-outside algorithm can be extended to support exact likelihood calculation, MAP estimates, and ML estimation updates in polynomial time on SRTGs. We call this extension the center-surround algorithm. SRTGs extend mixture models by adding hierarchal structure that can vary across observations. The center-surround algorithm can recover the structure of observations, learn structure from observations, and classify observations based on their structure. We have used SRTGs and the center-surround algorithm to process both static images and dynamic video. In static images, SRTGs have been trained to distinguish houses from cars. In dynamic video, SRTGs have been trained to distinguish events such as entering, exiting, picking up, putting down, sitting down, and standing up. We demonstrate how the structural priors provided by SRTGs support these tasks.
Joint work with Charles Bouman, Shawn Brownfield, Bingrui Foo, Mary Harper, Ilya Pollak, and James Sherman.

In this talk, I will present an implemented system, called Leonard, that learns to classify simple spatial motion events, such as `pick up' and `put down', from video input. Unlike previous systems that classify events based on their motion profile, Leonard uses changes in the state of force-dynamic relations, such as support, contact, and attachment, to distinguish between event types. Since force-dynamic relations are not visible, Leonard must construct interpretations of its visual input that are consistent with a physical theory of the world. Leonard models the physics of the world via kinematic stability analysis and performs model reconstruction via prioritized circumscription over this analysis. Leonard represents the lexical semantics of verbs as temporal logic expressions over the sequence of force-dynamic interpretations of the video produced by model reconstruction. Leonard can both recognize occurrences of known event types in video input as well as learn new event types from video input. Representing event classes as temporal logic expressions over force dynamic relations has several advantages. Among them, it allows new event types to be learned from a small set of training examples and produces perspicuous multimodal representations, ones that can be understood by humans and used for inference as well as perception. In this talk, I will present an overview of the entire system, along with the details of both the model reconstruction process and the temporal-logic learning algorithm. I will illustrate the advantages of force-dynamics over motion profile for representing the lexical semantics of simple spatial motion verbs and present a live example illustrating the end-to-end performance of Leonard learning and classifying events from video input.
Part of this talk describes joint work with Alan Fern and Robert Givan.