
Nvidia GPUs can break Chrome's incognito mode - charliehorse55
https://charliehorse55.wordpress.com/2016/01/09/how-nvidia-breaks-chrome-incognito/
======
brtmr
Previous discussion on the same subject, about a post written by me:
[https://news.ycombinator.com/item?id=9245980](https://news.ycombinator.com/item?id=9245980)

Basically, this issue is not restricted to NVidia GPUs or specific operating
systems - This can be reproduced on Windows, Linux and OSX. Basically the
concept of memory safety does not exist in the gpu space - which is the reason
why the webgl standard is so strict about always zeroing buffers. The issue of
breaking privacy and privilege boundaries on a multiuser system is very real,
and there is no workable solution. This seems to be one of those problems
where a lot of people are aware, but no one is sure how to fix it and so it
just stays how it is.

~~~
charliehorse55
Yeah, I remember reading that when it was first posted. Glad to see other
people are aware, but it's disappointing nothing has been done.

The fix is pretty simple, the GPU manufacturer just needs to update their
driver to zero the VRAM like an OS would with RAM.

~~~
cbd1984
Are device drivers _ever_ updated, much less for security issues?

Seems like an obvious way to increase the take, in any event: "Chip X has a
security flaw. Chip Y does not. Buy chip Y now or Evil People will own-zor all
your cash-zors."

~~~
taspeotis
Since we're talking about GPUs... search the NVIDIA archive [1], and look for
the WHQL drivers:

    
    
        Version: 359.00 - Release Date: Thu Nov 19, 2015
        Version: 358.91 - Release Date: Mon Nov 09, 2015
        Version: 358.87 - Release Date: Wed Nov 04, 2015
        Version: 358.50 - Release Date: Wed Oct 07, 2015
    

[1] [http://www.nvidia.com/drivers/beta](http://www.nvidia.com/drivers/beta)

~~~
cbd1984
So what about the BSODs that device drivers cause? Are those not device driver
issues, but OS issues, or are they unfixable?

~~~
Fr0styMatt88
They could be any of those.

In a sense, BSODs aren't anything special -- all a BSOD means is that some
code running in kernel mode has crashed or raised some exception that went
unhandled. The same thing, when it happens in a user-mode program, gets you
the error dialog box 'Program has stopped working'.

So the causes of BSODs and user application crashes are the same. The reason
Windows has BSODs is that it's dangerous to keep the system going when
something in kernel mode crashes. Things running in kernel mode have access to
everything (think - all memory) and are deemed important enough to the
operation of the whole system that a crash in one of those is a significant
event that's worthy of special logging and rebooting. You can't guarantee, for
example, that a display driver crash hasn't corrupted other parts of memory,
cuasing potential for data loss if the system were to continue operating.

So, back to the original point. Device-driver BSODs from the big vendors are
probably rare enough in general that you should suspect a hardware problem or
glitch if you suddenly see one out of the blue. Graphics drivers, given their
complexity, are a bit more prone to crashing though. Also, things running on
the system can interact and cause the driver to crash.

Windows has lots of infrastructure in place for making sure device drivers
behave safely. There's also good facilities for figuring out exactly what
caused a BSOD beyond the usually cryptic-looking error code you see on the
screen.

Resplendence WhoCrashed is handy:
[http://www.resplendence.com/whocrashed](http://www.resplendence.com/whocrashed)

Though if you really want to dig deep, the tools with the Windows SDK
(particularly WinDbg) can let you achieve the same thing; they are developer
tools though, so targeted more to that audience.

EDIT: Just to add in answer to your original comment, big-vendor graphics
drivers are VERY often updated. I'd bet they're the most often updated drivers
on a system. There are myriad reasons for this, both technical and
competitive. That doesn't mean that long-standing problems are necessarily
fixed, but both AMD and Nvidia have very regular releases with fixes and
performance improvements.

~~~
tracker1
Along a similar line, in my own experience, since around Windows 2000 (not ME)
it's extremely rare to see a BSOD that isn't related to either bad hardware or
drivers, more often than not hardware related to a driver than the driver
itself.

~~~
Fr0styMatt88
Another subtlety is that the term 'driver' on Windows tends to be used for any
loadable module that runs in kernel mode. So a driver often isn't actually
related to running a particular piece of hardware. Rather, it's a piece of
software that needs kernel-mode access to the system.

Two examples that demonstrate this point well:

\- There are various tools out there that you can use to perform a live memory
capture on a Windows system; not just doing a memory dump of a single process,
but doing a live memory dump of the whole system without having to halt or
reboot. I've used one of these and it works by loading a 'driver' component
when it is run that does the memory capture from kernel-mode (it requires
Admin elevation to run, obviously).

(For examples, see:
[http://www.forensicswiki.org/wiki/Tools:Memory_Imaging](http://www.forensicswiki.org/wiki/Tools:Memory_Imaging)

I don't remember if it was one off this list that I tried though).

Another example: A friend of mine had a system that would inexplicably BSOD if
he left it running for a long while, unattended (especially overnight). We
initially suspected perhaps a heating issue (it was a small Intel NUC). After
setting up for full memory dumps and then analyzing them after a BSOD occurred
using WinDbg, we actually found out that the BSOD was being caused by a
kernel-mode component of the anti-virus suite that he had installed -- I think
at the time it was BitDefender, but not sure. When he consulted the AV vendor
support website, I believe it turned out to be a known issue with a fix.

On my own systems, by far the largest cause of BSODs (of the few that I've
seen over the last couple of years) has been RAM going bad. These typically
manifest as BSODs out of the blue that seem to come from different modules
each time they happen, or they come from a module deep in the system that
'shouldn't' have crashes. My personal rule is, if I see one, be vigilant. If I
see another one, reboot and run MemTest86.

~~~
tracker1
In practice, other than the Windows Kernel, the only things running in kernel
space tend to be drivers, a/v software and malicious code.

------
rocky1138
Back in 2000, a very similar problem was my first lesson in frame buffers.

I was watching some adult material using Quicktime on Windows 98. A few hours
later, I wanted to show my mom something on my computer. As it loaded the new
video in Quicktime, the last frame of the porno sat there in inverted colours
until the new video began to play.

I had closed Quicktime hours ago... what was that still doing there in memory?

Needless to say it was very awkward.

~~~
mrsteveman1
Similar happened to me, except it was the entire family staring at a 50" TV
waiting for me to start a movie Christmas eve.

~~~
johnmaguire2013
I remember the first time I played porn on my TV. I felt like I had tainted in
an irreversible way. Moreover, I had a nagging fear that somehow it would show
up later when I didn't expect it (you know, like a file you forgot in your
Downloads directory). Thanks for letting me know my fears weren't entirely
unfounded...

~~~
chris_wot
From bash.org (which appears to be down, so linking to Google cache):

[http://webcache.googleusercontent.com/search?q=cache:_JGpv1r...](http://webcache.googleusercontent.com/search?q=cache:_JGpv1rzgSYJ:bash.org/%3F333409+&cd=1&hl=en&ct=clnk&gl=au&client=safari)

~~~
rocky1138
Ouch. The only thing I could think of to fix this is to burn a different image
overtop and hope the resultant burn is unrecognizable.

------
hn_123_throw
Serious kudos to the author for posting this even though he mentions viewing
pornography.

I had a similar problem on iOS. When I load Safari, there's usually a flash of
the previous screen (probably cached as a PNG), then the page loads. I think
it looks junky; I'd prefer a "loading" screen. It would flash the previous
screen whether I was in private mode or not. So porn would flash on my screen.
I didn't file a bug report or mention it on my twitter because I'm a little
afraid of the reception. So, again, thank you charliehorse55.

edit: i said "cached as a PNG" but that's just what I thought prior to reading
this article. it could be many things, including this bug.

~~~
mahouse
Believe it or not everybody watches porn.

~~~
cbd1984
It isn't that everyone watches porn, it's that watching porn is just as moral
as watching any other form of entertainment.

Anti-porn crusaders aren't necessarily hypocrites. They also don't need to be
hypocrites to be wrong.

~~~
bobthechef
Careful, you're showing your hand.

~~~
cbd1984
I also wear gray shirts and occasionally listen to ska.

Nothing makes people quite so alien as differing moral codes. I'm "showing my
hand" as regards something that simply _does not matter_ to the majority of
people on this website. Trying to make a big deal out of it simply makes you
look strange.

------
0x0
This sounds like fun, especially if webpages can use WebGL to read old buffers
back into javascript variables - and then AJAX them out silently in the
background. (preserveDrawingBuffer + canvas.toDataURL() + ajax ?)

Edit: Also, "google chrome incognito mode is apparently not designed to
protect you against other users on the same computer".. what? Isn't that the
_only_ thing it can and should protect against? It's not like it can protect
against non-local users (i.e. HTTP network interceptions)

~~~
TeMPOraL
RE your edit, there is a reason incognito mode is also known as "porn mode".
It's primary use case - that obviously can't be stated officially - is to let
you browse porn without fear that your mom / girlfriend / boss will find out
by checking your browser history or having the site's URL show up as a
suggestion when typing something in the address bar. It has never been a
serious security tool.

~~~
0x0
Obviously. It does have another use case too, for developers, it's an easy way
to run parallel login sessions on webapps without stomping over cookies. :)

~~~
shurcooL
Yeah. I wish all apps had an incognito mode for this reason.

Imagine being able to open an incognito terminal to type commands that won't
get saved to history or pollute what you already have.

~~~
aquadrop
In bash (and probably others) commands prefixed with space are removed from
history. (it's a setting - google HISTIGNORE )

------
byuu
> Google marked the bug as won’t fix because google chrome incognito mode is
> apparently not designed to protect you against other users on the same
> computer.

That's nonsense. For most users, that's _exactly_ what it's used for.

I really think Google is dropping the ball here. I know it's not their bug,
and they shouldn't _have_ to work around it in an ideal world, but this is a
pretty clear leak of data outside of private mode. It wouldn't impact
performance in any noticeable way (you're closing the window anyway at this
point), and would just be an extra safeguard.

Very short-sighted of them to ignore this bug. Perhaps we could ask distro
maintainers to add patches for this to their builds of Chromium.

------
TeMPOraL
Interesting. I've never thought of it as a security issue. But it's something
that's around since forever. I've seen old framebuffers containing stills from
games or videos showing up when resizing OpenGL applications 15 years ago.
Video cards don't clear memory for the same reason nothing is ever deleted by
default - it's a waste of time.

~~~
netheril96
> Video cards don't clear memory for the same reason nothing is ever deleted
> by default

Modern operating system zeroes memory pages all the time. It is a security
measure, and ensuring security is by no means a waste of time.

------
dogma1138
WDDM 2.0 gpummu is supposed to ensure that the memory has been zeroed between
different applications that use virtual GPU memory.

If this is the case there might be a compliance issue on nVidia side which
makes me wonder if webgl is vulnerable also.

WebGL was amended to request a zero when provisioning or disposing of a buffer
but it relies on the API which is handled by the driver if nVidia is taking
some shortcuts to save time it might be possible to leech stale memory this
way.

~~~
exDM69
> WDDM 2.0 gpummu is supposed to ensure that the memory has been zeroed
> between different applications that use virtual GPU memory.

Which Windows version introduced this WDDM version? Could it be that OP is
running an older version?

> WebGL was amended to request a zero when provisioning or disposing of a
> buffer but it relies on the API

This is indeed a tricky situation. All modern GPUs do "zero bandwidth clears"
which means that upon clearing, nothing gets written to the actual
framebuffer, the memory is just marked "cleared" (by writing some special bits
to the L2 cache, for example). This makes it difficult to reason whether
there's any sensitive content left in the framebuffer.

edit: nevermind, the OP seems to be using OSX, so it's not WDDM. Additionally,
the OSX GPU drivers are written by Apple.

~~~
dogma1138
Yeah this was confusing he said that it was an Nvidia issue which is why I
thought it was on Windows.

As far as WDDM goes 2.0 requires that for sure I'm pretty sure this was part
of the original WDDM GPUMMU spec also but I can't really find those details
anymore on MSDN since most of the pages refer to 2.0 atm.

------
13of40
Sometime around 1995 I had a brand new 80486 Dx-100 I was putting together,
and just for kicks I decided to pull the CPU out while it was running. I
expected some kind of epic C-64 style rainbow gibberish crash, but it actually
just froze, with the Windows 95 desktop still on the screen.

------
scurvy
Reminds me of when network card drivers would use random bits of memory to pad
out minimum Ethernet frames. Oh hey, there's your sensitive data going out in
an ICMP ping request.

~~~
maaaats
Do you have more info about this? Sounds crazy!

~~~
0x0
For example,
[http://www.securitytracker.com/id/1008910](http://www.securitytracker.com/id/1008910)

It's basically heartbleed in your ethernet driver in 2003.

------
logn
> Google marked the bug as won’t fix because google chrome incognito mode is
> apparently not designed to protect you against other users on the same
> computer (despite nearly everyone using it for that exact purpose).

What's the purpose of incognito mode then? It doesn't protect you from your
ISP, websites, or users on the same computer. I'm not sure what other use case
there is.

~~~
dogma1138
Saves you from clearing your history after browsing porn.

~~~
Strom
It could trivially also clear the framebuffer history. However they marked it
as wontfix.

~~~
dogma1138
I'm not so sure this shouldn't happen according to the wddm spec.

------
kevingadd
OP, it looks like this is on OS X (from the screenshots), in which case you
should probably report it to Apple as well. The driver stack on OS X is a mix
of Apple and NVIDIA code.

------
Animats
This should be easy to fix at the driver level. Window close and and GPU
resource release are not operations that occur often enough that memory
clearing would affect performance.

------
stordoff
> Of course, it doesn’t always work perfectly, sometimes the images are
> rearranged.

I've seen the same behaviour on OS X with an Intel GPU:
[https://i.imgur.com/3fagsYx.jpg](https://i.imgur.com/3fagsYx.jpg) (screenshot
of the contents of a browser tab - pretty sure it was Chrome. The Rooster
Teeth page you can see parts of had been closed hours prior)

~~~
rawnlq
Googling shows that this bug previously received a $1000 bounty in 2012 and
should already been fixed:
[https://code.google.com/p/chromium/issues/detail?id=152746](https://code.google.com/p/chromium/issues/detail?id=152746)

------
rubberstamp
This is a hack that is caused by hardware, due to some way in which its
currently designed. As such, I doubt if software patches would be able to fix
it. A while back I read an article on arstechnica that described stealing
encryption just by touching exposed metal parts of laptop.

[http://arstechnica.com/security/2014/08/stealing-
encryption-...](http://arstechnica.com/security/2014/08/stealing-encryption-
keys-through-the-power-of-touch/)

------
cmrx64
Check out this paper if you're interested in a solution to this and other
problems (somewhat amusingly, using the GPU):
[https://www.cs.utexas.edu/~sangmank/pubs/lacuna.pdf](https://www.cs.utexas.edu/~sangmank/pubs/lacuna.pdf)

In particular, they refute via counterexample the arguments that VMs or secure
deallocation alone are sufficient.

------
chris_wot
And this is why Linux folks hate proprietary drivers. At least the developers
of open source drivers would fix this. Probably very promptly!

There's a reason Linus Torvalds flipped the bird to NVidia. Here is a perfect
example of the reason why closed source drivers suck.

------
netheril96
It would be nice if a software exists that allocates many many GPU memory
pages and clears them, so we can be safe after viewing p-something.

------
ericlamb89
awesome -- also I'm glad you censored the porn but not the porn title ;)

------
vermilingua
Same issue with Samsung GS5... on Snapchat... with disastrous results...

------
netheril96
Is this issue specific to Google Chrome? Does Safari has the same problem? In
principle, I'd expect so, but if Safari surprises me, maybe I will jump ship.

------
Zekio
they could just fill the buffer with images of the nvidia logo free
commercials for themselves?

------
pstrateman
There's performance overhead to clearing memory.

If you were them would you take the performance hit?

~~~
Retr0spectrum
GPUs have loads of memory bandwidth. I can't imagine a framebuffer taking more
than a few microseconds to clear.

For example, Nvidia claims that the GTX 980 has a memory bandwidth of 223
GB/s. (1920 * 1080 * 3)/223e9 = 27us. Clearing all 4GB of VRAM would take
4/223 = 18ms. This would have a negligible impact on user experience in most
cases.

I guess the driver could also erase memory in the background as soon as it is
deallocated, with zero user impact.

~~~
exDM69
It's high bandwidth but also high latency. If a page was cleared every time it
was allocated, it would cause very unpredictable performance because the CPU
would have to tell the GPU to clear memory and then wait for the GPU to finish
before any other operations on the buffer could be done. This definitely isn't
something that should happen for every allocated page. It might be acceptable
if this happened only to pages previously used by other processes. But it
would still be unpredictable and could cause unwanted stalls in the middle of
a game session, for example.

Also note that memory bandwidth is typically the bottleneck in modern games.

The best place to do this would be in the browser, clearing out any textures
and buffers before deallocating them if the contents are deemed private.

~~~
Dylan16807
>because the CPU would have to tell the GPU to clear memory and then wait for
the GPU to finish before any other operations on the buffer could be done

If you're doing write-only operations, the CPU can queue them behind the
clear. If you do a read, then the CPU has to wait whether you clear or not.

Latency doesn't matter. Clearing can be slotted in with other operations, such
as first use.

------
bradhe
Uh, anyone else think this title is a bit sensationalist? I was expecting
something a bit more along the lines of actually leaking usable private data,
not just displaying a rastered frame.

Even further this has very little to do with chrome. The only way chrome could
actually fix this issue would be if it nuked the frame buffer when it released
it. This is a fine idea, but if I was a dev in that context I would assume the
OS would make stronger guarantees than that??

If anything, this is an edge case Chrome devs (and other developers) could
protect themselves against if they were so inclined, but I'm not surprised
they didn't assume they needed to protect against this.

~~~
djur
The primary use case for Incognito Mode, as far as I know, is so a user can
casually use a browser without leaving inadvertent artifacts of their usage on
the machine. Having a page you visited in Incognito Mode be visible in Chrome
after Incognito Mode is closed seems to be precisely the kind of thing users
expect the feature to prevent.

~~~
cookiecaper
If you associate the Chrome browser with your Google account (not just logging
in to a Google site, but going to settings and putting the information in
there), the history will be synced across several devices. Incognito can be
used to prevent pages viewed on your phone or laptop from going to your
desktop's history or vice-versa.

