SEMCOG: A hybrid object-based image and video database system and its modeling, language, and query processing

Multimedia data types, such as image and video, are structurally more complex than traditional data types. We view an image as a compound object containing many sub-objects. Each sub-object corresponds to image regions that are visually and semantically meaningful (e.g., car, man, etc.). In this paper, we first present a framework for modeling images and videos using object-based hierarchical structures. The proposed structure supports image retrieval at both whole-image and object levels. We show how to extend the object-based image modeling framework to video retrieval at object, frame, action, and composite action levels. The advantages of our approach are as follows: (1) hierarchical model allows translation of video retrieval queries to image retrieval queries with additional temporal relationships; (2) concept definitions release users from learning the system and implementation details and the language syntax; and (3) IFQ, a visual query interface, specifically designed for object-based image retrieval is extended for video retrieval query interfaces.