WP4: Ontology infrastructure, content processing, knowledge-assisted content analysis and inferencing

At the heart of the R&D objectives of the aceMedia project, the goal of WP4 is to develop the necessary infrastructure, technology and tools for knowledge representation, extraction and usage throughout the aceMedia value chain. This technology will permit structured knowledge representation in aceMedia applications, provide tools to automatically analyse ACE content, extract knowledge and generate ACE metadata and annotation.

Objectives

Innovation

Although description of multimedia information has recently seen significant progress, the pace of automatic extraction of such a description, and especially of its semantic part, is rather slow and still remains an open issue. On the other hand, considerable achievements have been made in speech/text-based knowledge representation, extraction and management techniques. A main novelty and objective of the work in WP4 is to bring these two communities' work together and develop a common knowledge representation framework for automated and efficient extraction of semantic information (objects, events, properties, relations). This will result in automatic semantic metadata extraction and annotation, indexing and retrieval of multimedia content.

Workplan

In months 13-30 of the project, WP4 will continue refining and extending the knowledge infrastructure, but it also will shift to focus more on analysis of ACE content, with or without the use of knowledge, for the automatic generation of semantic metadata to be part of the ACE metadata layer. This change of focus is reflected by the content pre-processing task moving from WP3 to WP4, the search/retrieval task moving from WP4 to the new WP6 (which is now focusing on ACE usage), and the focus of the reasoning and context tasks to mainly assist multimedia content analysis. The output of WP4 developments will be used to deploy a set of Application Modules (AMs) that will provide an integrated set of semantic metadata. These AMs will be included and tested in the Personal Content Services (PCS) and Commercial Content Management (CCM) prototypes that will be developed in WP5. To conclude, WP4 in the second project period is more focused and concentrates on ACE content analysis, with or without the use of knowledge, for the generation of ACE semantic metadata.

Results

In WP4, the knowledge representation infrastructure for semantic multimedia content analysis and reasoning has been already developed. This is one of the major objectives of the aceMedia project where ontologies are extended and enriched to include low-level audiovisual features, descriptors and behavioural models in order to support automatic content annotation. More specifically, the developed infrastructure consists of the core ontology based on extensions (including extensions to accommodate directional and topological relationships in the spatial and temporal domains) of the DOLCE core ontology and the multimedia-specific infrastructure components. These are, the Visual Descriptors Ontology (VDO), which is based on an RDFS representation of the MPEG-7 Visual Descriptors and the Multimedia Structure Ontology (MSO) for description of the Semantic ACE metadata, similarly based on an RDFS representation of the MPEG-7 Multimedia Description Schemes.

Furthermore, a graphical user-friendly Visual Descriptor Extraction tool has been developed to support the initialization of domain ontologies with multimedia features. The integrated framework is called M-OntoMat-Annotizer (M stands for Multimedia) and is based on an extension of the CREAM (CREAting Metadata for the Semantic Web) framework supporting the extraction of the core MPEG-7 Descriptors included in the VDO. For this purpose, it uses aceToolbox for the low-level MPEG-7 features extraction to an XML file. The extracted descriptors are then transformed into RDFS instances of the visual descriptors defined in the VDO and automatically linked to the selected domain concept prototype instances. M-OntoMat-Annotizer is publicly available as free software. For more information visit the tool's homepage.

Based on the above ontology infrastructure, the first versions of multimedia-enriched domain ontologies have been constructed, to be used in tasks including content analysis and reasoning, as well as content search, retrieval and navigation. These include domain ontologies for Personal Content Services ("family" and "holidays"), and Commercial Content Management ("tennis" and "motor sports").

Finally, a number of WP4 AMs have been already integrated and tested in the first Personal Content Services (PCS) prototype currently being developed in WP5: Specialised image-understanding tasks, such as ontological text analysis, high level feature detection (face detection and person detection), ontology-based image cataloguing (e.g. indoor-outdoor decision), and knowledge-assisted analysis, capable of assigning semantic labels to segmented region images (e.g. sky, sand, beach, person for a holiday ontology).