WP 2: Evaluation, Integration and Standards

• WP Leader: Allan Hanbury (TU Vienna-PRIP)

The principal objective is to co-ordinate activities that facilitate within and between research workpackage collaboration. This principal objective can be divided into several specific objectives that are placed into 3 groups: evaluation, standards and computation.

Evaluation (or benchmarking)

• The collection of a large test database of copyright-free multimedia data and video surveillance data.
• The manual annotation of some of the data in the test database. This will allow objective evaluation of automatic algorithms
• The organization of evaluation campaigns.

• Development of an Infrastructure for MultiMedia Metadata Management based on Semantic Web technology

• To make available software for free use between network members.
• To publish resources on computational methods, such as software repositories and tutorial material.

WP 3: Visual Content Indexing

• WP Leader: Nicu Sebe (UVA)

The aim of this work-package is to prepare numerical visual data to knowledge extraction and multimodality semantics association. Tasks will focus on visual content indexing and mining. We aim to:

• Develop novel, innovative methods and algorithms for processing visual information from still images and video sequences, that can enhance symbolic and semantic object/event detection and description to allow efficient indexing of visual repositories;
• Participate with WP2 to evaluate the new algorithms on standardized (if possible) datasets;
• Foster integration through e-teams close cooperation;
• Disseminate our work through international community

WP 4: Content Description for Audio, Speech and Text

• WP Leader: Andreas Rauber (TU Vienna-IFS)

The aim is to prepare numerical multimedia data to knowledge extraction and multimodality semantics association. Tasks will focus on each single modality of text, audio as well as combinations thereof to extract semantic concepts.

Concerning audio processing, we aim to:

• develop reliable techniques for indexing audio;
• evaluate the performance of audio feature sets for different tasks such as genre analysis, sound classification, audio stream segmentation and audio retrieval;
• participate in respective evaluation campaigns such as MIREX

• address the problem of robustness of automatic speech recognition systems
• evaluate, in how far specific speech analysis can contribute to extracting and describing the semantics of audio data in general.

• study the specific problems of natural language processing for describing multimedia content (descriptive vocabulary, ontology)
• analyze textual aspects of multimedia such as scripts, lyrics, etc. accompanying multimedia data
• advance such techniques
• take part in world-wide competitions such as TREC or CLEF;

WP 5: Multimodal Processing and Interaction

• WP Leaders: Petros Maragos (ICCS-NTUA) and Alexandros Potamianos (TSI-TUC)

Multimedia information retrieval via an interactive human-computer interface is a complex task that requires feedback from the user and a complex negotiation between the user and the machine. We propose to research, design and build natural and efficient human-computer interfaces for performing multimedia information retrieval tasks that allow for negotiation (dialogue) between the user and the system. Human-computer interfaces research will focus in three broad research thematic areas: multimodality, adaptivity and mobility. The research performed in these tasks works towards the grand challenges of MUSCLE.
This work package addresses research on the theory and applications of multimedia analysis approaches that improve robustness and performance through cross-modal integration. It also focuses on interaction with multimedia content, with special emphasis on multimodal interfaces for accessing multimedia information. Its general research objectives include several scientific and technological goals and can be grouped into the following categories:

• Book on Multimodal Processing and Interaction
• Audio-Visual Speech Analysis and Recognition
• Multimodal Integration for Multimedia Analysis and Recognition
• Interfaces to Multimedia Content
• Coordination of research, E-teams, and Dissemination of results

WP 6: Machine Learning and Computation Applied to Multimedia Description 

• WP Leaders: Pádraig Cunningham and Simon Wilson (TCD)

The objectives of this workpackage include:

• development of new ML techniques specific for multimedia understanding;
• extension and integration of existing ML techniques; and
• coordination of the application of ML methods by the network to different tasks of multimedia understanding.

WP 7: Showcases and Dissemination towards Industry: Bringing the Grand Challenges to End Users

• WP Leader: Enis Cetin (Bilkent)

The objective of this workpackage is reflected in its subtitle "Bringing the Grand Challenges to the End Users": i.e. developing targeted research projects designed to deliver highly demonstrable results for showcasing, while facilitating the creation of multi-team showcases which show proof-of-concept with regard to major challenges in content indexation and audio-visual search. WP7 will be relying on research activities that will generate results and demonstrations to fuel the dissemination and showcasing activities. As such the principal objective of WP7 is twofold:

• Research Integration: by actively pooling software resources and by integrating research results, WP7 will establish (or reinforce) durable integration between different groups and research communities within the consortium, thus ensuring continuation of the collaboration beyond the Network's lifetime. To this end WP7 will provide a framework for the development of small internal research projects, the results of which will be destined essentially to showcasing activities.
• Dissemination: The above-mentioned showcases offer opportunities to intensify the Network's knowledge transfer and forge new research partnerships in preparation of the upcoming 7th Framework which need to be pursued. In particular, the showcases should be used to attract more substantial involvement from commercial and industrial partners;