Multi-Modal Patient Distress Detection System
Audio-triggered patient monitoring on Raspberry Pi 5 pairing FFT voice analysis with DeepFace emotion recognition
View repositoryPlatform
Raspberry Pi 5
Audio
FFT · 85–5000 Hz bands
Vision
DeepFace emotion
Result
90% acc · ~1 s alert
01Overview
An applied embedded-systems project: a Raspberry Pi 5 monitors a patient in real time, pairing audio signal processing with machine-learning computer vision to detect signs of distress and raise alerts continuously instead of on a fixed schedule. It was built as part of a 5-person team.
It is included here to show breadth in real-time signal processing, computer vision, and hardware/peripheral integration, alongside the firmware and PCB work that is the main focus of this portfolio.
02Problem Statement
Manual patient checks happen on fixed intervals, leaving gaps where distress can go unnoticed. Continuous, automated monitoring narrows that gap with low-latency alerts.
03Architecture
Audio path: 16 kHz mono audio is read in 1024-sample frames and passed through a SciPy FFT. Energy in the voice fundamental (85–500 Hz), harmonic (500–2000 Hz), and high (2000–5000 Hz) bands is scored as a ratio of total spectral energy, combined with a spectral-centroid check, into a voice-confidence score that screens for distress before any heavier inference runs.
Vision path: a LISTENING to ANALYZING state machine opens the camera only once audio crosses the threshold, then runs DeepFace over three frames sampled 200 ms apart. Frames classified angry, fear, sad, or disgust above 60% confidence count as distress, and the vote across frames lifts reliability to about 90% detection accuracy while suppressing false positives.
Hardware and integration: USB microphone, webcam, and speaker subsystems are driven on the Pi, delivering roughly 1 second trigger-to-alert latency end to end, validated across testing scenarios.
04Technical Challenges
Reliable hardware across configurations
Demo hardware varied between runs, so the system uses defensive USB device handling with a priority-based microphone fallback (Razer Seiren → USB webcam mic → system default) and camera retry logic with buffer-flush sequences, keeping it running across hardware configuration changes.
Suppressing false positives
A single noisy frame should not raise an alarm. A multi-frame voting state machine aggregates DeepFace classifications over time, trading a small amount of latency for materially fewer false positives.
Real-time on constrained hardware
Running FFT analysis and a facial-emotion model on a Raspberry Pi 5 while holding roughly 1 second latency required a staged pipeline where cheap audio screening gates the heavier vision step.
05Implementation Details
FFT voice screening
Implemented FFT spectral analysis in Python over the voice fundamental (85–500 Hz), harmonic (500–2000 Hz), and high (2000–5000 Hz) bands, weighting band-energy ratios and a spectral centroid into a voice-confidence score, with a silence gate so background noise never triggers the vision stage.
DeepFace with multi-frame voting
Integrated DeepFace facial-emotion classification behind a three-frame voting state machine keyed on angry, fear, sad, and disgust above 60% confidence, reaching about 90% detection accuracy with reduced false positives across testing scenarios. Alerts beep through ALSA and save a timestamped capture of the triggering frame.
Defensive hardware integration
Drove USB microphone, webcam, and speaker subsystems with priority-based mic fallback (Razer Seiren → USB webcam mic → system default) and camera retry with buffer flushing, then ran end-to-end validation at roughly 1 second trigger-to-alert latency.
06Key Engineering Decisions
Two-stage audio-then-vision pipeline
Gating expensive vision inference behind cheap audio screening keeps the system real-time on the Pi while controlling false positives.
Multi-frame voting over single-frame decisions
Voting across frames was chosen deliberately to make the alert robust to transient misclassifications, which matters more than raw per-frame speed for a monitoring use case.
Edge inference on the Pi
Running detection on-device keeps patient data local and avoids network round-trips, which helps both latency and privacy.
07Future Improvements
- Tune the audio and vision fusion thresholds across more scenarios.
- Add on-device model optimization for lower latency.
- Harden the alerting path for clinical reliability.
08Media & Documentation
Visual documentation for this project. Placeholders below are reserved for assets being produced during development.
Open to opportunities
Looking for a Summer 2027 internship.
If you're hiring for embedded, firmware, or hardware roles, I'd be glad to talk through my projects and how I work. The fastest way to reach me is email.
