Toward motion picture grammars

We are interested in processing video data for the purpose of solving a variety of problems in video search, analysis, indexing, browsing and compression. Instead of concentrating on a particular problem, in this paper we present a framework for developing video applications. Our basic thesis is that video data can be represented at a higher level of abstraction as a string generated by a grammar, termed motion picture grammar. The rules of that grammar relate different spatiotemporal representations of the video content and, in particular, representations of action.