Domain knowledge extension with pictorially enriched ontologies

Classifying video elements according to some pre-defined ontology of the video content is the typical way to perform video annotation. Ontologies are built by defining relationship between linguistic terms that describe domain concepts at different abstraction levels. Linguistic terms are appropriate to distinguish specific events and object categories but they are inadequate when they must describe video entities or specific patterns of events. In these cases visual prototypes can better express pattern specifications and the diversity of visual events. To support video annotation up to the level of pattern specification enriched ontologies, that include visual concepts together with linguistic keywords, are needed. This paper presents Pictorially Enriched ontologies and provides a solution for their implementation in the soccer video domain. The pictorially enriched ontology created is used both to directly assign multimedia objects to concepts, providing a more meaningful definition than the linguistics terms, and to extend the initial knowledge of the domain, adding subclasses of highlights or new highlight classes that were not defined in the linguistic ontology. Automatic annotation of soccer clips up to the pattern specification level using a pictorially enriched ontology is discussed. © Springer-Verlag Berlin Heidelberg 2005.