Multimodal Surveillance

Lead Investigator: Mohsen Naqvi

Project Team: Zeyu Fu, Yang Sun, Federico Angelini, Yang Xian and Jiawei Yan

Academic Collaborators: University of Surrey, Sheffield University, University of Leicester, Universidad Carlos III de Madrid, Harbin Engineering University

Funding: The project is co-funded by Newcastle University and the industrial partner Thales, under the EPSRC - iCASE Award scheme. It is also aligned with activities within the University Defense Research Collaboration (UDRC).

Project Synopsis

Signal and Information Processing is particularly well suited to deal with multimodal data. The current and future sensor systems e.g. sensors in smart devices, will provide ever more data for subsequent analysis; hence, the research motivation of this project is multimodal (audio-video-infrared) data processing for next generation of artificially intelligent automated surveillance systems.


We aim to contribute algorithms and technologies that will enable significant advances towards the vision. The specific aims for multimodal surveillance are:

1. robust multimodal human action recognition and human behaviour analysis.
2. multiple human localization and tracking in cluttered and congested environments.
3. multimodal speech enhancement and separation in relatively large environments.