Multimodal Surveillance

Lead Investigator: Mohsen Naqvi

Project Team: Federico Angelini and Jonathon Chambers

Funding: The project is co-funded by Newcastle University and the industrial partner Thales, under the EPSRC - iCASE Award scheme and aligned with activities within the University Defense Research Collaboration (UDRC).

Project Synopsis

Multimodal Surveillance in the Wild is a research project focusing upon machine learning and fusion techniques for the automatic analysis of multimodal sensor measurements. The aim is to obtain a robust model for human activity based on multimodal data (RGB-depth-skeleton) for intelligent systems development, with special focus on abnormalities and potentially risky situations detection.

How effectively the machines can perceive us, detecting our presence in the space and responding coherently to our body movements are central questions for this project. Among these, this project is focusing on how effectively recognise human behaviours and perceive abnormalities in human activities for surveillance purposes.

New powerful technologies and disciplines such as intelligent sensors and machine learning are nowadays pushed forward by both the innate technological progress and the needs of modern society, among which security and surveillance have crucial importance. Private companies, governments and researchers from all around the world are nowadays committed in developing reliable and efficient systems for fast-responses to critical situations, for dangerous events prevention and for reaching highly effective investigation systems. Human behaviour modelling and recognition are both key tasks for these purposes.

The main approaches for human behaviour and action recognition depend on which sensors are involved in the data acquisition task. This project relies on facilities provided by Intelligent Sensing Lab for multimodal and synchronised human data acquisition, including

1. Basler cameras for RGB and background subtraction data (Fig. 1);
2. Basler Time-of-Flight cameras for IR data;
3. Multi-cameras recording workstation;
4. Kinect workstation for IR and skeleton data (Fig. 2);