
Hearing Heartbeat in Audio and Video: A Deep Learning Project - CaHoop
http://www.samcoope.com/posts/reading-faces
======
brandonb
What a cool project! I'm working with UCSF cardiology on various projects
involving Apple Watch's heart rate sensor, and here are a few applications you
might consider for your technology:

    
    
      1. Atrial fibrillation (you mentioned arrhythmias—this is probably the highest-impact heart arrhythmia since it causes about a quarter of strokes and is frequently undiagnosed)
    
      2. Sleep apnea (if you can also measure oxygen saturation)
    
      3. Heart rate variability for stress reduction or mindfulness.
    
      4. Tracking tachycardias from Adderall usage.
    
      5. Heart failure exacerbations (probably if paired with a scale)
    
      6. (Perhaps) other arrhythmias like supraventricular tachycardia or atrial flutter.
      
      7. Irritable bowel syndrome. Believe it or not, IBS is connected to heart rate variability via the action of the vagus nerve.
    
    

There are also all sorts of non-medical, but fun, applications you could
pursue. For example, rating which scenes in Game of Thrones get your heart
pumping the most: [http://blogs.wsj.com/digits/2015/08/13/what-game-of-
thrones-...](http://blogs.wsj.com/digits/2015/08/13/what-game-of-thrones-does-
to-your-heart-rate/)

------
kajecounterhack
The idea is from Eulerian video magnification

[http://people.csail.mit.edu/mrub/vidmag](http://people.csail.mit.edu/mrub/vidmag)

------
jimmyswimmy
I'm not very up on CNN/RNN models yet, beyond the basics, but I do a fair
amount of signals analysis. Is your truth data in git? I'd love to work with
it. Second, did you try whitening the [audio] data prior to training your
network? That might help with exposing some more salient features of your
signal. Pretty neat work.

~~~
CaHoop
Thanks for the ideas!

The data we used is not on our repo. It came from a database given to us by
our university, and there was confidentiality issues when it came to making it
publicly available.

------
Animats
That's nice. It doesn't work well enough to be useful yet, but has promise. If
this can be made to work, it will be a feature in video chat apps.

~~~
harigov
Or maybe security trying to detect if someone is in stressful situations. Or
maybe in some sort of augmented reality device. May valid use cases.

------
tuyguntn
What is the accuracy of detecting?

~~~
Kip9000
From the video they have on youtube, Arnold has over 200bpm and the kid at the
end has a 0bpm. That's a pretty useless territory.

~~~
CaHoop
That video was just a mock of our product. If you would like to see the
results of the performance of the models, have a look at the slides we made
for the project (at the bottom of the post). The best performance we got was
about 0.7 r.

~~~
kren1
0.7 r is a bit optimistic I would say. We only got 0.7 for people we have been
able to train on. For completely different people it learned nothing with
audio spectrograms. I speculate that convolutions on raw audio would be
better.

Estimation from video seemed much better. There we actually got 0.5 r on
unseen people, which I find very promising.

------
cyogee
Just curious on why your username is shown in green on HN homepage?

~~~
dang
New users don't make the front page very often, which is maybe not something
to be proud of. You'll see more at
[https://news.ycombinator.com/newest](https://news.ycombinator.com/newest),
and
[https://news.ycombinator.com/noobstories](https://news.ycombinator.com/noobstories)
if you want the motherlode. It looks like one has to be logged in to see green
usernames, which I'd forgotten.

In the current case we saw CaHoop's earlier submission of this project, which
got killed by an overzealous spam filter, and invited him to repost it.

