
Show HN: NeuralCam Live – Using ML to Turn iPhones into Smart Webcams - soonpls
https://neural.cam/news/
======
jimmies
Nice work!

I have been researching and working on this webcam quality problem for a
while. I have tried a couple of webcam-over-phone solutions and while they
solved the image quality issue, they tend to introduced quite some lag.
Especially when you use your dedicated microphone, then the lag becomes a huge
issue (because the microphone has minimal lag and the webcam has a significant
lag leading to the voice not matching the lips).

Have you thought of the issue and have you done an analysis on that?

Besides, with this setup, you have to rely on a driver that turns the video
from whatever phone to a webcam. That in my experience is quite flaky, but
perhaps not inherently problematic. So I wonder if you could comment on OS
support for people who want to use your software with say, Linux?

Disclaimer: I use and package a software suite to turn a raspberry pi high
quality camera to a webcam. The lag seems pretty good and it appears as just a
good old usb webcam on linux and windows (although I haven’t managed to make
it work on macOS) [1].

1:
[https://github.com/showmewebcam/showmewebcam](https://github.com/showmewebcam/showmewebcam)

~~~
microcolonel
I wonder how hard it would be to get a MIPI capture device that presents v4l2,
I guess it's almost certainly more expensive right now than a Pi Zero.

Anyone know of a SDI or MIPI capture card that works off the shelf with v4l2?

~~~
jimmies
I haven’t tried looking for it and it seems like minimizing latency on the pi
0 is more fruitful and flexible than trusting such an adapter to work. And as
you said, cheaper.

------
phlyingpenguin
Note for anybody thinking they want to use this with Zoom/Slack/Discord/etc:
It appears that this has the same restrictions that other DAL-based apps have
and you'll need to re-sign the app or disable SIP to use it that way. I've
been using an OBS virtual camera for this purpose and have had to do this a
few times now. It will depend on the target app if they have this in their
allowed plugins list.

[https://github.com/johnboiles/obs-mac-
virtualcam/wiki/Compat...](https://github.com/johnboiles/obs-mac-
virtualcam/wiki/Compatibility#apps-dont-allow-dal-plugins)

~~~
xoa
On the SIP workaround, it says disabling SIP for the filesystem is enough, but
even though that's less broad then disabling SIP entirely it's still a pretty
wide brush. Have you (or anyone else) tested whether it's enough to disable
SIP just for specific applications or folder locations? IIRC, all definitions
and exceptions for SIP fs restrictions are still in two locations:

    
    
      /System/Library/Sandbox/rootless.conf
    
      /System/Library/Sandbox/Compatibility.bundle/Contents/Resources/paths
    

Disabling SIP entirely allows editing these as root, and then reenabling
allows more fine grained control. Of course like everything else in an Apple-
owned location, changes have a high chance of being blown away any full system
update, but those are rare enough (and always require a reboot anyway) that it
doesn't seem like a major pain to just redo changes a few times a year (with a
script).

There are still of course some security implications for any additional
exceptions you add but a lot less then a full fs disable. Further it's cool to
have control over it anyway because you can even extend SIP to additional
locations of your own which could be handy for some scenarios.

~~~
phlyingpenguin
I can't comment to disabling SIP as I haven't taken that route. I've generally
taken the codesign route instead, which does require re-signing the app after
every update. In the case of Zoom, that's gone away in recent versions since
the app now has the virtualcams I'm using in its whitelist. For anything else,
it's a very manual process.

------
jjbolaygaj
Quality enhancement of the face region is to my knowledge the best use
regarding ML and real time communication.

The idea is to detect the region where the user's face is and then transmit
that region at a higher quality than the rest of the video image.

This vastly improves the video call experience as the face is almost always
the most important part of any video call.

~~~
wombatmobile
Yes, face is the most important part of most video.

80% of the intellectual content of most telepresence is transmitted via audio.
That's why when your local TV news goes to a live cross and there's no sound,
they go back to the studio until the audio is fixed.

------
tito
I've tried EpocCam Pro [1] which is good at the basics. The biggest challenge
with these "phone as webcam" setups is getting a good mounting system. A
tripod is hard to get in the right spot, and stacks of books are fail. Neither
EpocCam or this software solution mention mounting which makes me think
they're all not putting themselves in their user's shoes. My sense is the team
that wins here get the "basics" right of this -- flawless connection to
computer, good awareness of Do Not Disturb/it's a phone, and suggestions for
mounting. Very practical. Phone camera is way better than webcam so the rest
takes care of itself.

[1] [https://www.kinoni.com/](https://www.kinoni.com/)

~~~
michaelmior
They're not cheap, but I hear the Mountie[0] works well for physically
mounting your phone (or iPad) to your laptop screen.

[0]
[https://tenonedesign.com/mountie.php](https://tenonedesign.com/mountie.php)

~~~
michaelmior
Update: I have since decided to purchase a Mountie and I haven't used it much
yet, but I'm pretty happy.

------
ramzyo
This looks awesome - unfortunately, my 2017 MBP running the latest MacOS 10.15
and iPhone 11 don't seem to register with this software when plugged in after
installing the app and drivers. The Support Center doesn't have anything
useful for troubleshooting, so...YMMV?

~~~
aframe
Same. MacBook Pro 16 & iPhone 11 Pro.

Video source didn’t show up in Zoom but did in QuickTime, however there was no
video - only a static turquoise image.

Had to faff about parsing the Mac installer with pkgutil once I decided to
remove it as there were no uninstall instructions or mention of what was
installed where.

~~~
soonpls
Make sure the app is running on your iPhone and your iPhone is connected to
your Mac with a USB cable before you select the virtual webcam in video call
apps. Regarding Zoom support, the latest version Zoom version (5.2.0) should
work with no issues. If you're still encountering any sort of issues, send us
an email at support@neural.cam and we'll look into it asap.

------
paulcarroty
Well, sounds interesting. I've heard some people use old smartphones as cheap
webcams & LTE modems.

~~~
vijaybritto
Yes, I have been using DroidCam and its been great. I am using my 3 year old
phone for this and the camera quality is still better than the laptop webcam
bought last year

------
ramzyo
What's the pitch for the subscription model for NeuralCam Live +PLUS other
than "New Styles added regularly" (which I don't think I care about, although
I've used the product for less than a day so maybe with time)? The other +PLUS
features (additional video styles, special soften style, low light mode, no
ads promoting other NeuralCam Apps) seem like one-time purchases, not
something that as a consumer I could justify paying a recurring for.

~~~
soonpls
we want to make this app usable for most people in the free version too. we
see styles as the core component of the overall UX and being able to select
the right style for a given setting is quite useful. we think that many users
will appreciate a wider choice. we’ll be adding additional features to the
Plus version in the future while also continuously improving the tech.

~~~
ramzyo
Got it, thanks for responding. So my takeaway is that today the pitch for the
recurring is that you'll continue to add new styles, and users can pay a
monthly to access those as they come out. Just one potential customer's
opinion, but I'd be much more likely to purchase styles and +PLUS features if
they were one-time purchases.

------
juliann
Looks good OP, just a suggestion, I opened the app and continued through
(clicked continue to live feed) without airdropping the driver or clicking the
link. Now i can't find a way to get back to that screen on the app. There
should be a way to go back to that onboarding so that the user can both
preview the image quality without having completed the setup but also continue
it later. Just a suggestion! I'll try it later.

~~~
soonpls
thanks for the feedback, juliann. we'll be adding a "restart setup" button in
Settings. in the meantime, you can download the plugin directly from here:
[https://neural.cam/live/NeuralCamLive-
installer-1.0.0.pkg](https://neural.cam/live/NeuralCamLive-
installer-1.0.0.pkg)

~~~
robterrell
BTW your installer includes a gitkeep file.

------
nlh
Genuine question for people: When I read about things like gesture guard, head
bubble, virtual backgrounds, etc., I cant help but wonder why they’re so
important.

Is it really a problem if you sneeze or cough during an online meeting? Are
you colleagues really making this an issue?

Likewise, I’m fascinated by the focus (ha ha) on hiding backgrounds. Why is it
so bad to show people where you are?

I’m clearly in the minority in that neither of those scenarios are even
remotely (ha ha) bothersome to me or my colleagues. But from their overall
thematic popularity, obviously these are real issues for people.

I’d love to be enlightened...

~~~
guessbest
Many people don't have a room in their house or apartment that they can
dedicate to creating a home office, especially people with kids, and have to
use a shared room. Blocking the background is essential. Also, sneezing during
a pandemic sounds like something that could get an employee or contractor a
furlough.

~~~
thrav
Haven't gotten to try it yet, but is head bubble actually black around the
edges, or is that just to show off how it works?

First thought was it would be nice to combine head bubble with blur, so you
avoid the stark background -- that's probably already what you're doing
though. Will give it a shot later today.

~~~
soonpls
yes, the Head Bubble puts your head in a bubble, making your background black
and tracking your head. it's a great way to hide your background and it really
takes away any distractions from video calls.

------
xutopia
I tried the software but was so put off by the ads in the app and the
inability to really try the app out before paying. I they had a 10 day full
trial I would be in a better position to decide if I buy or not.

~~~
soonpls
hey xutopia, thanks for the feedback. the only ad in the app is towards our
other app, NeuralCam NightMode. we do have a 7-day trial option available, you
can try out all the features with it and cancel at any time if they're don't
bring you enough value.

------
abda0180
Wow. Looks interesting. I'm curious, why don't you use the built-in Macbook
webcam? - does Apple limit its use?

~~~
soonpls
The Macbooks webcam is a lot lower in quality when compared to even older
iPhones, so the image quality would be really low. Also, Macbooks do not have
a Neural Engine at the moment, so running advanced machine learning algorithms
in real-time is not really an option, when compared with the low quality
webcam. The iPhones & iPads, though, have both a Neural Engine and a very high
quality camera, and we're making full use of them with our solution.

------
aantix
You mention background segmentation - is there any way to replicate a shallow
depth of field/large aperture with the iPhone's dual lenses?

I'm sure this is quite challenging to get the segmentation accurate in real-
time.

~~~
soonpls
We'll be looking to build these types of features in the near future, as our
engine evolves. It's challenging yes, but not impossible.

------
simsla
An interesting idea (which would need to be integrated in the conferencing
apps) might be too do quality improvement on the receiver side. Turning low
bandwidth and compressed video into higher quality video.

------
adsjhdashkj
This sounds interesting - but how are people setting this up? I don't think
i'd be able to use this out of the box, since i don't have a stand for my
phone to position it like a webcam.

------
mosselman
> Gesture Guard

> Detects and blurs different types of gestures that don't have their place in
> a video call.

> Currently, it has two main components: detecting and blurring face touches
> (scratching face, picking nose, yawning, etc.) and detecting nudity. These
> help you to not get caught off guard.

Finally I can do my stand up in the shower.

This sort of feels like an April fools page.

~~~
soonpls
The 'nudity' scenario is somewhere along the lines of you being in a video
call, and your loved one walking naked in the background, or just forgetting
to close the webcam while you change, etc.

We realize it's not a common scenario, but even if it would work once or twice
per year, it would still be saving you from a very awkward situation, thus the
value we see in this.

~~~
bcraven
_Politician offers to resign after showering during live video meeting_

[https://www.independent.co.uk/news/world/europe/councillor-s...](https://www.independent.co.uk/news/world/europe/councillor-
shower-video-meeting-bernardo-bustillo-spain-torrelavega-a9601021.html)

~~~
kohtatsu
Pretty wholesome.

Unfortunately he believed minimizing the app would cut the video.

~~~
newsbinator
There's no substitute for a hardwired camera-on indicator light.

Otherwise politicians are going to keep showering on camera during meetings,
instead of showering off-camera during meetings, like they're supposed to.

~~~
tachyonbeam
I know that the webcam USB API requires the on-off light to be controlled in
software, but can't webcams just... Ignore that, and just control it in
hardware? It seems to me that would be better for everyone. Heck, we could
mandate webcams have hardware indicator lights by law. You can always tape
over them if they bother you.

~~~
microcolonel
It's kinda a mix of things. Webcams built with IR sensors and emitters (like
the Logitech BRIO 4K) don't use the indicator while using the IR, though you
can clearly see the pulsating near-infrared LED while it's on.

On Linux at least, if you open the v4l2 device for a device like this, it will
turn on the indicator directly, and I don't think you can attach to it with
libusb while it's otherwise in use by v4l2. In practice, I think the worst you
could do from userspace on Linux is show the indicator while not recording.

------
tomphoolery
NeuroCam Returns!
[https://www.youtube.com/watch?v=q9XmXad_wmo](https://www.youtube.com/watch?v=q9XmXad_wmo)

------
tomp
at first this looked like a spammy product advertisement page, but actually,
there's some interesting tech here! I'd suggest you make things like _face
bubble_ , _low light_ , _gesture guard_ and _denoising_ more prominent for the
HN audience, instead of "marketing material" like your future products that
don't yet exist.

~~~
soonpls
thanks for the feedback, tomp. we didn't tailor the article for a specific
audience, we tried to be as transparent as possible with our vision & mission.
we'll think of ways of improving communication both for casual users and users
like the HN audience.

------
rexreed
Dare I ask about Android support?

~~~
soonpls
yes you may :) however, Android is much more fragmented platform and it’s only
really possible to run the advanced ML algorithms we’re using on the newest
Android phones. so.. we’ll probably have some level of Android support in the
future, but it’s not clear which of the features on which devices.

