Multi-Modal Patient Distress Detection System
Audio-triggered patient monitoring on Raspberry Pi 5 pairing FFT voice analysis with DeepFace emotion recognition
Platform
Raspberry Pi 5
Audio
FFT · 85–2000 Hz bands
Vision
DeepFace emotion
Result
90% acc · ~1 s alert
01Overview
An applied embedded-systems project: a Raspberry Pi 5 monitors a patient in real time, pairing audio signal processing with machine-learning computer vision to detect signs of distress and raise alerts continuously instead of on a fixed schedule. It was built as part of a 5-person team.
It is included here to show breadth in real-time signal processing, computer vision, and hardware/peripheral integration, alongside the firmware and PCB work that is the main focus of this portfolio.
02Problem Statement
Manual patient checks happen on fixed intervals, leaving gaps where distress can go unnoticed. Continuous, automated monitoring narrows that gap with low-latency alerts.
03Architecture
Audio path: FFT spectral analysis in Python (SciPy) extracts features from the voice fundamental (85–500 Hz), harmonic (500–2000 Hz), and high-frequency bands to screen for distress before any heavier inference runs.
Vision path: when the audio stage triggers, DeepFace facial-emotion classification runs on the webcam feed. A multi-frame voting state machine combines decisions over several frames, which lifts reliability to about 90% detection accuracy while suppressing false positives.
Hardware and integration: USB microphone, webcam, and speaker subsystems are driven on the Pi, delivering roughly 1 second trigger-to-alert latency end to end, validated across testing scenarios.
04Technical Challenges
Reliable hardware across configurations
Demo hardware varied between runs, so the system uses defensive USB device handling with a priority-based microphone fallback (Razer Seiren → USB webcam mic → system default) and camera retry logic with buffer-flush sequences, keeping it running across hardware configuration changes.
Suppressing false positives
A single noisy frame should not raise an alarm. A multi-frame voting state machine aggregates DeepFace classifications over time, trading a small amount of latency for materially fewer false positives.
Real-time on constrained hardware
Running FFT analysis and a facial-emotion model on a Raspberry Pi 5 while holding roughly 1 second latency required a staged pipeline where cheap audio screening gates the heavier vision step.
05Implementation Details
FFT voice screening
Implemented FFT spectral analysis in Python over the voice fundamental (85–500 Hz), harmonic (500–2000 Hz), and high-frequency bands to detect candidate distress and trigger the vision stage.
DeepFace with multi-frame voting
Integrated DeepFace facial-emotion classification behind a multi-frame voting state machine, reaching about 90% detection accuracy with reduced false positives across testing scenarios.
Defensive hardware integration
Drove USB microphone, webcam, and speaker subsystems with priority-based mic fallback (Razer Seiren → USB webcam mic → system default) and camera retry with buffer flushing, then ran end-to-end validation at roughly 1 second trigger-to-alert latency.
06Key Engineering Decisions
Two-stage audio-then-vision pipeline
Gating expensive vision inference behind cheap audio screening keeps the system real-time on the Pi while controlling false positives.
Multi-frame voting over single-frame decisions
Voting across frames was chosen deliberately to make the alert robust to transient misclassifications, which matters more than raw per-frame speed for a monitoring use case.
Edge inference on the Pi
Running detection on-device keeps patient data local and avoids network round-trips, which helps both latency and privacy.
07Future Improvements
- Tune the audio and vision fusion thresholds across more scenarios.
- Add on-device model optimization for lower latency.
- Harden the alerting path for clinical reliability.
08Media & Documentation
Visual documentation for this project. Placeholders below are reserved for assets being produced during development.
Open to opportunities
Seeking Fall 2026 embedded / firmware / hardware co-op.
If you're hiring for embedded, firmware, or hardware roles, I'd be glad to talk through my projects and how I work. The fastest way to reach me is email.
