Hacker News new | past | comments | ask | show | jobs | submit login
How iOS 13 redraws your eyes so you're looking at the camera (twitter.com/schukin)
275 points by bobsil1 on July 3, 2019 | hide | past | favorite | 116 comments

I am reminded of a brief story about video calls in David Foster Wallace's Infinite Jest. Some excerpts:

Good old traditional audio-only phone conversations allowed you to presume that the person on the other end was paying complete attention to you while also permitting you not to have to pay anything even close to complete attention to her...Video telephony rendered the fantasy insupportable.

The proposed solution to what the telecommunications industry’s psychological consultants termed Video-Physiognomic Dysphoria (or VPD) was, of course, the advent of High-Definition Masking;

Mask-wise, the initial option of High-Definition Photographic Imaging — i.e. taking the most flattering elements of a variety of flattering multi-angle photos of a given phone-consumer and — thanks to existing image-configuration equipment already pioneered by the cosmetics and law-enforcement industries — combining them into a wildly attractive high-def broadcastable composite of a face wearing an earnest, slightly overintense expression of complete attention — was quickly supplanted by the more inexpensive and byte-economical option of (using the exact same cosmetic-and-FBI software) actually casting the enhanced facial image in a form-fitting polybutylene-resin mask,


Yeah this whole snapchat phenomenon with filters is literally out of Infinite Jest. It's almost hilarious how exact he got the fact that people will be talking with each other on the camera, and it will literally be a visually enhanced version of themselves communicating with each other. I mean people have no shame in photoshopping their entire form and passing it off as themselves. We are a sad bunch. Except for the ones who don't comply.

I was very surprised when I downloaded zoom yesterday and noticed that an “appearance enhancement filter” is enabled by default. This is something that I am not used to outside of social and camera apps from (mostly) Asian countries

Sad state of the world where the default in a business related App is essentially telling you that this is necessary. Can’t be healthy for society

Eh, it really doesn't do much but smooth skin.

I'd rather a client/vendor/coworker not be distracted if someone on the call has big nasty pimples, and I'd rather have a coworker with big nasty pimples feel more comfortable getting on a video call.

The existence of the feature is not a problem, that’s just where technology is heading. For individual use it could be enabled, if desired

But it being made default and thereby de facto saying that everyone needs it is absurd. There is no way that something like this is not damaging to society in the long run

I wonder, however, if smoothing skin makes for a lighter (compressed) image, resulting in bandwidth savings. We don't need to see skin details in a work conversation, just expressions.

Perhaps the feature should just be renamed/rebranded to something like "smartly reduce image details to save bandwidth"?

I think there's hard to draw boundaries. Small burnishes I think everyone would be okay with a touch-up. But on the other hand large changes people will feel deceived if they meet that person irl. The difficult is in the middle, which is going to have some distribution of perception of deception.

I think this is different than touching up models because we never end up seeing their true form. Though I remember when the fappening happened a lot of people mentioned how they felt better about themselves after seeing these people's true selves.

Shouldn’t we expect this to become normalized the moment AR becomes somewhat ubiquitous? People will happily pay for always-on augmentations to make themselves look more beautiful, younger, stronger, etc.

> Except for the ones who don't comply.

Don't worry, we're sad in our own way.

I also thought about this when I saw it. It's incredible how ahead of the times DFW was!

I can't help but think all this image recognition/manipulation tech being silently applied is a tad creepy. IMHO going beyond things like automatic focus/white balance or colour adjustments, and identifying more specific things to modify, crosses the line from useful to creepy.

I know what you mean, but this is basically just correcting for a deficiency in current technology—i.e. the ability to place the front-facing camera directly behind the image of the remote person's eyeballs.

The solution is a little creepy I'll admit. But the fact that we are becoming increasingly accustomed to talking to people who appear to be staring at our necks is even creepier, in my opinion.

> But the fact that we are becoming increasingly accustomed to talking to people who appear to be staring at our necks is even creepier, in my opinion

There have been a few comments now that mention people looking at necks instead of eyes as being creepy.

Now I wonder if anyone finds it creepy to talk to me in person, because I usually look at the speaker's mouth, not their eyes, at least when they are speaking. (I'm not actually sure where I look when I'm doing the speaking).

Looking at the mouth allows better error correction if the conversation is competing with noise.

What's weird is when someone only looks at one spot. If you stare at people's eyes some will become uncomfortable, because naturally we make slight side movements. I'm sure the same it's with mouth. But most people can't tell if you're looking at their mouth or at them.

People do notice direct eye contact though. Maybe someone could explain this. I'm kinda curious why directly in the eye we notice but there's more wiggle room around the face.

Phone manufacturers are trying to figure out how to make their phones as much screen as possible, while still having a front facing camera, so they have either a notch, a slide-out camera, or a hole for the camera in the screen.

I propose they put that hole not in a corner, but in the middle of the screen, and position the remote person's face so that the pupil of one of their eyes is positioned exactly over the camera. That should give you the most natural interaction without distorting anything.

They're already putting fingerprint scanners actually under the screen somehow (TIR?) - I doubt it'll be long before they figure out how to get a camera to look through a running screen while being invisible from the viewing side. Ultimate Big Brother tech right there.

Cameras are already being placed under the screen e.g. Xiaomi have implemented it and it probably won't be long before it's public.

I personally think it's too far, we should stick to bezels so we can have great hardware for cameras, speakers and screens.


I agree, I think phone industrial design peaked with the iPhone 5S/SE, Nexus 5, and Pixel 1. I'd like that design forever please.

I would love to see a successor to my old, rubberized steel Motorola Milestone with keyboard, but with a larger screen and unlocked bootloader, please. All these modern phones are way too flimsy.

I agree that dedicated hardware is better, but then again I used to think standalone cameras were always going to be better than a phone camera and (while price-for-price it's still true) phone cameras are "good enough" and the best camera is the one you have with you.

I don't think that's a good idea. What about the majority of the phone's use when it's not showing someone's face? What about when the person moves their head and the whole frame has to pan?

Not to mention having a hole in your screen when you just want to watch a video. It's a ridiculous idea, but funny to think about.

More seriously, I'm wondering if at some point the front-facing camera may be an intrinsic part of the screen. I've read that on-screen fingerprint sensors already use a kind of camera in the screen. If that were to become good enough to see images at a distance, that might actually work.

Here’s the current state of cameras in screens.


Both Xiaomi and Oppo shared videos of the in-display cameras working, but the videos are too low quality to make any kind of image quality determinations.

That coupled with a lack of even controlled lighting, best case sample images probably tells a lot about the image quality.

Everyone would complain about the "dead pixel" in the middle of their screen.

When I last had to replace my phone, because it fell out of my pocket exiting my car, I received a new phone, not with a dead pixel, but an entire column of dead pixels. No visible damage, straight up just powered it on after openning the box. Immediately called my phone company to request a replacement and they wanted me to buy a new phone. Said fuck that, I paid full retail for the new phone amd its defective out of the box. Send me a new phone and a prepaid shipping bill for the return.

hmm. i cant agree with this as a person who does phone calls and then sees people in real life. i guess there is a market for facetimers like yourself that i must respect. if you care that much about appearing properly on a video call with grandma, sure let apple redraw your eyeballs haha. The only people accustomed to talking to people who are looking at each other's necks are the people who live on the cell phone.

Like the girl who was doing a video facetime call on her cell phone all the way from waiting at the gate , boarding, sitting on plane. Call me an old grumper if you want, downvote if you want. but damn , da fugg is the deal get off my lawn with your video chats kids.

There's a difference between having the opinion that "I don't use it, so it shouldn't exist" & I don't have a problem as long as the people who use it aren't obsessed with it. Honestly I'm not sure which opinion brootstrap holds, but he's entitled to it either way. For those who can't stand selfies/facetiming there's a perfect song from Brad Paisley called "selfie":https://www.youtube.com/watch?v=6rGlcScJ0lE

I don't see your point. Is your argument "I don't use this feature, therefore it shouldn't exist"?

I think you missed his point (whether it's strong or not). He's saying there's an irony in the idea of focusing on proper appearance to the person on the phone when the boundaries of what's appropriate for loud speakerphone calls in a public environment have been lowered too far. Like worrying about whether you look nice while walking off a cliff. I think that's the point, whether one agrees with him or not.

It looks like there's a toggle https://twitter.com/flyosity/status/1146136279647772673

But yes, it crosses the line where it's no longer a camera (as a feature, it does not capture reality)

What is truth?

When the person on the other end of the call is looking at the image of me on their screen, the physical truth is that they're looking away from the camera, but the emotional truth is that their eyes are looking at me.

When I'm chatting with loved ones, which truth is more important? Physical truth, or emotional truth?

Consider for a moment that an offset front-facing camera might be just as serious a distortion of truth.

You've lost me at "emotional truth". The camera captures a specific scene. The result is processed to correct for lens distortions / colour / lighting issues. But none of those changes the content.

I don't think an offset camera is distorting anything. It captures the image it's expected to capture.

But it doesn't capture the image the user expects it to capture - when on a Facetime call, they expect (or at least want, if they think about it at all) to be looking directly at the other parties to the call. All this is doing is correcting for that, hence "emotional truth". Reality is always shaped by our emotions and our mental state, and thus there is merit to the idea that a representation of reality that isn't 100% physically accurate can still be closer to what we perceive as reality.

I see where you're going, but I disagree with the "emotional truth" idea. It crosses the line for me: fixing eyes is on the same level as a gender swap or cat face filter. It modifies content and meaning and I'm against this being normalised in normal communication.

I don't see how the eye correction is any less of a "modification of content and meaning" than the use of an offset camera angle which it's correcting for. Just because the offset angle is currently a technical necessity and an "analogue" distortion, doesn't make it any less of a manipulation.

> doesn't make it any less of a manipulation.

I think we've got a fundamental disagreement here. We'll just have to agree to disagree :-)

In an effort to be more agreeable, I really do understand where you're coming from. I'm just openly wondering if there is a technology bias at play making stuff we're already accustomed to and which feels normal to us (e.g. cameras with rectilinear lenses capturing images from arbitrary angles) feel less manipulative than stuff which is new and technically sophisticated.

what up SJ. I have to comment on you again my friend. Why are you so passionate about this eyeball re-drawing technology? Do you video call your family and friends all the time because you live far away?

Reading these conversations i'm going a bit crazy as you try to rationalize why this eyeball redrawing business is a good and necessary tech. I think it's complete trash. Is this the tech that makes apple worth billions of dollars?

Not trying to stir shit up with you. just generally curious as I have an extremely opposite viewpoint. I am 28, prime millenial love to use apps and phone age. I am a huge nerd and geek (clearly as we are commenting on HN lol). I have literally never face-timed anyone except when other people are doing it and I am nearby.

I first used a Macintosh 512k when I was around 5 years old. I had my own PC by the time I was 18 (a dual celeron 300A at one point) and I now spend my days doing programming and web development.

I’m now 38 and I’ve seen my two year old son’s face light up when he sees his mum through the phone screen.

My wife’s parents are on the other side of the planet and their intercontinental video chats connect them to our family.

I love science, math, philosophy and photography. I should care about pixel perfect truth. But to me the value of FaceTime is entirely emotional. I say that when my wife looks into my eyes, I want to see her eyes looking at mine. The fact that this isn’t how it works now is because of optical distortion. While this isn’t a true holistic fix for the underlying distortion—that would be amazing—it looks like a neat solution.

I’ve not seen it live and I’ll reserve final judgment till then. If it’s even slightly noticeable I’ll absolutely hate it. But if I can see no sign of trickery when staring into my wife’s eyes, it might just make the world a happier place, even if only a tiny bit.

Here's another anecdote to help satiate your curiosity: I'm a 27 year old software developer, lifelong nerd and all around hacker, and I love FaceTime to the point where it's inclusion in the iPhone is a defining feature of my experience with the product.

After relocating for college and again for work, I have family across four timezones and friends even farther abroad. In my experience, nothing short of a flight bridges that distance better than video calling. I can see my mother's eyes light up when we haven't spoken in a while, my younger brother can hold up his latest pair of sneakers to the camera while he geeks out about them, and watching my best friend's new puppy bounce around his apartment is much more interesting than hearing it bark in the background.

I'll probably make a FaceTime call 3-4 times a week, which from observation and casual conversation appears to be a similar frequency to the rest of my friend group (mostly coastal millennials, largely Bay Area + NYC, ages 24-31).

With regards to the eye repositioning I agree with the grandparent poster. All captured images are reproductions of reality, and if I were to place this one on a spectrum between corrective glass elements for lens distortion and cat-person filters I would say it's much closer to the former. In my opinion the correction captures the intent of the user, and I feel that's truthful enough.

Fair enough, I disagree with your premise but I respect your viewpoint nonetheless. I do feel that we should be vigilant at monitoring which such usages we are comfortable with - while I might be comfortable with this being used for something innocuous like eye offset correction, I would oppose its use for something more nefarious.

It's not modifying content and meaning, it's preserving content and meaning. When you're on FaceTime, the image you're seeing is not captured from where your eyes are presented on the screen, it's captured from somewhere above. This is a technical defect and it means that when the person you're talking with looks you in the eye, you don't see that. This is correcting that flaw and makes it so when the person you're talking with looks you in the eye on their end, you see that on your end too.

This is more akin to doing a white balance adjustment, such that you see what the scene is expected to look like rather than the raw data captured by the camera, it's just correcting for camera location instead of exposure.

Another way to think about it is this is a software fix for a hardware limitation. In an ideal world we'd fix the hardware limitation, remove the software fix, and your experience would be unchanged (or actually, even better, because of the lack of distortion on stuff crossing your eyes).

The camera doesn’t capture a “scene.” It’s all just electrical impulses on a CCD. You the perform extensive processing to turn those impulses into something approximating that someone might see. It doesn’t seem a big stretch to, while you’re at it, interpret the signals in a way that reflects where the person would actually be looking.

“Emotional truth” is the wrong phrase. It’s the actual truth—the person on the other side is looking at your eyes.

The real issue is that the processing isn’t sophisticated enough. Instead of just moving the eyes, it should calculate what the camera would show if it were located in the screen.

minor nit: Almost no camera use CCDs any more.

Maybe instead of emotional truth, how about the actual truth is that most people are actually looking the other person dead in the eye, as is appropriate for engaging with people. This camera magic just makes that more clear.

The offset camera position also "changes the content" just as surely as this new process which compensates for it.

Yes, but FaceTime is not attempting to be a camera; it is attempting to simulate a face-to-face conversation. That it uses a camera is an implementation detail. The fact that the distance between the camera and the face on the screen makes it look as though the person you are conversing with isn't quite looking at you is an implementation detail leaking out of the intended abstraction.

No camera "captures reality", ask any experienced photographer.

But yeah, I think it's creepy too.

It captures reality. It doesn't capture the image you see with your eyes, it distorts shapes, it messes with light, etc. But it captures the light coming in. So far we called any post-processing which changes that reality either a filter/effect or a bug. Eyes correction is as much reality as deepfakes.

> Eyes correction is as much reality as deepfakes.

I think you're making 2 mistakes:

- first, thinking that the line is between what we think of as normal camera usage, and eye correction & deep fakes. But the image that we see now is already a result of image correction—white balance, exposure, lens adjustments, and and more secret processing that we don't even get told about.

- second, that it's a hard line. The possibilities aren't "no adjustment whatsoever" and "deepfakes". It's a gradient; there are minor adjustments (like white balance), moderate adjustments (like eye correction), and even greater adjustments (like deep fakes). It's not a hard line line at all. Eye correction is much closer to reality than deep fakes, and a lack of eye correction is closer to reality still.

It doesn’t capture reality. Our own eyes don’t even do that. We don’t even know what “reality” is because we only experience it through our the filters of our mind.

CCDs are just stupid light buckets and the reality of them is, at best, a spreadsheet showing those counts. Everything beyond that is a choice someone made to present things a certain way.

And everything before the photograph is taken is that way too. Framing, exposure time, time of day, choice of subject... etc. all influence the “reality” captured in the photograph.

Half the reason you probably think they capture reality in the sense you mean is because photographers not shooting for Instagram generally try very hard to make it appear that way. But... they don’t.

On a technical level every photographic print or even SOOC jpeg has some level of image processing applied to it during development, whether analog or digital.

On an artistic level: https://toinformistoinfluence.files.wordpress.com/2012/06/19...

I see no philosophical difference between adjusting in-camera settings (aperture, focus, white balance) and this. In both cases, you're modifying the "feel" of the image (in fact, with preprocessing, you're generally affecting the image in a wholesale manner).

Digital cameras don't "capture reality", either. They scan the image over a small period of time, same as your average display in reverse. It's essentially a really fast time lapse, on the pixel level. Arguably, that's as much "not reality" as what Apple is doing here.

I mean, I guess I understand the idea of "capturing reality as close as possible". I just don't see much merit in caring too much outside of an academic context. This isn't deepfakery. This is shifting eyes to provide the much needed feedback of eye contact while on a video call.

There is certainly something about reality that cameras reflect, namely, that looking at the images they produce results in an approximation of seeing what one would see if one’s eye was where the camera was.

I suppose that goes without saying, seeing as that is sorta what cameras are (well.. many cameras. The central examples of cameras. IR cameras don’t quite have this property, but you know what I mean.), but sometimes it is “necessary to state tautologies” (though this isn’t a tautology exactly).

A linear representation of a group needn’t distinguish between every pair of group elements for me to say that it “captures” some aspect of the group.

"But it captures the light coming in. So far we called any post-processing which changes that reality either a filter/effect or a bug."

No, even analog photography, choices in developing certainly effect how the "light coming in" turns into a picture. As well as the particular camera used, and it's settings (aperture, ISO, shutter speed), and the film chosen.

The choices the digital photography system makes are no less choices with effects on the "captured light" than the ones made after you get bits out of the camera. They're just choices made by the engineers who designed the digital camera. There is no single 'true' way to "capture light".

> When a photographer chooses a particular brand or type of film, he’s already deciding what kind of reality he wants to show. Different types of film show reality and colors differently. Photographers pick film based on what they want the result to look like. Film photography never ever showed reality, not even close.


> But despite the presumption of veracity that gives all photographs authority, interest, seductiveness, the work that photographers do is no generic exception to the usually shady commerce between art and truth. Even when photographers are most concerned with mirroring reality, they are still haunted by tacit imperatives of taste and conscience… Although there is a sense in which the camera does indeed capture reality, not just interpret it, photographs are as much an interpretation of the world as paintings and drawings are. Those occasions when the taking of photographs is relatively undiscriminating, promiscuous, or self-effacing do not lessen the didacticism of the whole enterprise. This very passivity -- and ubiquity -- of the photographic record is photography's "message," its aggression.


But I don't actually disagree that there is something new going on lately with new kinds of digital processing. I don't think it's "all the same" exactly, we definitely have new techniques of altering and manipulating photos that give us SO MUCH more power than we had before, that I think it's reasonable to say it's a new thing.

But the original photographs never "captured reality" without intervention or manipulation. A photo is necessarily an intervention and a manipulation, in a variety of different ways. A photo never truly "captured reality" in an objective sense, it always depended on the photographer's choices, and the details of the technology used, in ways that mattered for what was perceived in the photo. But yeah, we have whole new orders of magnitude of it available, I'd agree.

Do you have the same complaint about automatic red-eye elimination?

Red-eye is induced by the flash and was never part of the reality being captured.

1. Engineering constraint: the camera is not as sensitive to light as humans

2. Workaround: it emits a flash which creates an undesirable side effect

3. Fix: we clean up in software

Is like?

1. Engineering constraint: The camera can't be positioned behind the LCD panel the user is actually looking at

2. Workaround: it's placed 1 inch above which creates an undesirable side effect

3. Fix: we clean it up in software


In both cases we get closer to where we'd be if the engineering constraint did not exist.

What is your opinion on the selfie wide angle correction Android does for faces? https://people.csail.mit.edu/yichangshih/wide_angle_portrait...

> But yes, it crosses the line where it's no longer a camera (as a feature, it does not capture reality)

This is incredibly funny.

Reality perception is subjective. Thus there is no true way to "capture reality".

My opinion is that it's not creepy. It's a simple, predictable thing.

At a minimum, the user should be able to turn it on and off.

Edit: Just saw where someone else posted there's a toggle.

Why is this creepy?

exactly. What other features of our faces will software "correct" next ?

Maybe they'll turn electrical signals into a picture using complex image processing that varies between sensor manufacturers.

To make sure I don't get banned for being ironic, that's intended light-heartedly to say lots of processing already goes on. The tool is attempting to represent what both sender and recipient usually intend.

Literally just “what a slippery slope!”

Well, doesn’t whether that is reasonable depend on how slippery the slope is?

That's not how the slippery slope fallacy works.

You can elaborate on what you think will happen and why, if this gains traction. Anything else is just... literally FUD.

Yes, the operation should be same on all pixels, regardless of position, and regardless of the values of other pixels.

What about algorithms that correct for optical lens aberrations like distortion and vignetting?

That is not how color/lighting correction works. Also video compression will affect different pixels differently.

I fail to see how this is creepy (outside of potential uncanny valley issues in edge cases). There is a toggle to disable it, and this is something that most average non-savvy users would either want by default, or wouldn't even notice happening (because the end result will look natural to most).

I don't buy the slippery slope argument here, yet. All they're doing is shifting eyes and nose to compensate for physical offset from the camera. Seems like Animoji would be more creepy, as it's capturing your entire face movement.

Invoke a slippery slope argument by name irritates me because the name comes from the slippery slope fallacy. It's poor faith debate because it's relies on the unfalsifiable notion that thing A will lead to slipping to thing B. Then you're just discussing feelings about how likely it is.

I agree with you. This seems more like a "quality of usage" type of thing. Now when you look at who your Facetiming, it's like your actually looking at them, instead of their chin. Something like skin smoothing in the regular camera would be worse, in my opinion.

Doesn't the camera already have skin smoothing? Or you mean for facetime?

I think people who find it creepy are the ones who rarely if ever use Video Conferencing or FaceTime.

And It would have been creepy if this was coming from anyone else. But knowing this would only be available on FaceTime I don't see what's wrong with it.

> I don't buy the slippery slope argument here, yet.

That's how the slippery slopes get you. They look all gentle and inviting at the beginning like you can totally stand on them no problem, then the slippery gets you and you're sliding right on down.

The real problem with talking about slippery slopes in tech is that we're already on an unrecoverable slide into the unknown. So the question is just "does this new thing make the slide even faster?" Not, "we're on stable ground and we need to be careful about beginning a slide."

There's no going back and we have no idea where we're headed.

Right, I understand that. I just don't feel this particular use of the technology pushes us significantly closer to that point.

> or wouldn't even notice happening

That's the problem.

I really want to see this, but I opt out of having a twitter account :(

Funny... It must need to synthesize the obscured portion of the eye highlight. I wonder if it grabs it from the back-facing camera, or if it's using stock images/texture synthesis?

Just waits for when you look at camera and grabs that texture.

Hunh? You can view tweets just fine without twitter accounts.

This also happens with posts from Facebook often. People just want to show the world how virtuous they are by not having accounts in popular networks, so they don’t even click the link but complain they can’t read the content. Reminds me of this:


Not on mobile perhaps?

When I click it I just get a sign in page.

Huh. I was on a team that experimented with this a few years ago, and the results were so uncanny-valley that we quickly stopped going down the path -- despite it looking "real", people wound up looking like slightly possessed demons because there was something "just off" about their eyes. I mean, it seriously freaked us out.

I'm really wondering if/how iOS avoids this, or if it took a ton of extra work.

The feature is only available on the iPhone Xs so I assume it takes advantage of the iPhone's 3D depth sensing features. When it comes to eye and face tracking Apple has a lot of hardware at their disposal. The photos I've seen so far look perfectly normal, not uncanny at all.

I'm just hoping that it will correct my eyes so I can watch TV in the background while my girlfriend thinks I'm looking at her during the videochat.

I don't think it works like that. iOS is adjusting your eyes by a specific angle to account for the distance from the screen to the camera. If you look away it will appear to the person on the other end of the call that you are looking away.

I understand why this feels creepy in our tech bubble, but I think it's worth noting how popular apps like FaceTune have become the past few years [0] [1].

I'm certainly not defending this trend and I think it's incredibly unhealthy — especially for the average teenager who's already naturally self-conscious about their appearances. But a minor eye correction will be peanuts in the eyes of this crowd (no pun intended) compared to the amount of processing that most of their Instagram and Snapchat photos go through before being uploaded.

[0] https://www.theguardian.com/media/2018/mar/09/facetune-photo...

[1] https://abcnews.go.com/GMA/Living/photo-retouching-apps-affe...

Weird, what happens when you are actually trying to look at something else?

Insert image of Clockwork Orange film scene where a machine holds your eyes open and forces you to look.

Maybe it still re-paints your eyes, relative to the person's eyes. So you glance down at their mouth and it looks like you're looking at their mouth rather than... down at their chest or something, like it would without the adjustment. Finding eyes in an image is common/boring tech at this point, so I could see that being used to provide a relative adjustment.

You're looking wrong.

I was wondering why skype and any other video chat don't do this five years ago, or more.

Probably we needed deep learning to do this well.

Correct. Apple certainly "reused" the infrastructure they established with realtime facial recognition and manipulation, all encapsuled in ARKit (https://developer.apple.com/augmented-reality/arkit/). Furthermore, these compute intensive codes are accelerated with special chips ("AI chip", "Neural engine", etc.). That's actually quite novel stuff which wasn't there five years ago.

Some research achieved basic results in 2012 [0] and spun off into a plugin ("Catch-Eye") [1] which is no longer available (and the website was last updated in 2017).

[0] https://cgl.ethz.ch/publications/papers/paperKus12.php [1] https://www.catch-eye.com/index.html

You might also need the FaceID sensor as well. It turns your nose a little bit too apparently so I imagine that it's creating some sort of mask and mapping your facial movements onto it.

Sorry I'm not seeing it, can someone explain in bit more detail what is going on?

One of the drawbacks of video chat is that the camera on your device is offset from the face of the person that you're chatting with (typically your camera is on top of your screen, rather than embedded in the middle of it). This leaves people chatting with the impression that the person they're chatting with is staring at their neck.

This new iOS video chat software modifies your video stream to correct for this camera/screen offset by _redrawing your eyes_. The end effect is that the person you're chatting has the impression that you're looking into their eyes rather than at their neck.

The effect is subtle, but can be made more obvious by putting a straight wire in front of your eyes. With the software feature disabled, the wire appears straight. With the software enabled, the wire appears bent.

thanks, I get it now and see it now :)

The wire bends when it passes the eyes.

It's hard to see. Would have been better to use a straight wire instead of a bendy glasses arm.

This is such an elegant solution! Previous discussions about how to solve this issue revolved around putting a camera in the center of the phone behind the screen!

The Simpsons did it again: https://imgur.com/a/oGSkSgN

Nice to see this feature. A few months ago I made a demo app for my own personal use which did this exact thing, just to see if it would work. As I was still learning AR at the time, I didn’t fully understand the API well enough and I just had a point cloud with loads of holes in it, but it still showed it worked, and made me wonder why Apple hadn’t already done it.

I love this. What a casual use of futuristic technology to make what people want to express real. Great work from Apple.

I had a video call interview and I was quite impressed how that woman always seemed to look at the camera, because I really thought she was looking at me and not me on the screen.

I told her, but she answered that she was not really staring at the camera. I guess it makes sense now.

Off topic: anyone know why 100% of the time I load a Twitter post from HN it loads "something has gone wrong" and I must refresh the page once to see the tweet?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact