Hacker News new | comments | show | ask | jobs | submit login
Protecting Google Cloud customers without impacting performance (blog.google)
465 points by theDoug 6 months ago | hide | past | web | favorite | 129 comments



> By December, all Google Cloud Platform (GCP) services had protections in place for all known variants of the vulnerability.

Could other major cloud providers boast this? It seems like Google's brand is benefiting tremendously from Project Zero in all of this. On the other hand, it feels nervous-makingly like a clear step towards running one's own mainstream hardware being too hard for the little guys.


> It seems like Google's brand is benefiting tremendously from Project Zero in all of this.

Is that not how it's meant to work? They're paying to run a top notch security team which is actively discovering vulnerabilities and creating fixes for them. Should they not benefit from this?


surely, tzar meant that in a positive way


I feel like even without this, there's a fair argument to be made that Google is better at running hardware than most little guys are.


And it's not necessarily a bad thing, either. Economies of scale means that everybody can win by putting all your chips in one basket. The obvious caveats apply when said basket stands to gain from serving its own interests once all the chips are there.

One example of that might include patching your own hardware while waiting to offer the same capabilities to others.


Absolutely.

My wife is an amazing and talented baker. But... any number of bakeries are better at rolling out a quality consistent product at scale.

Ditto with datacenters.


It's a very interesting trade off one faces. On the one hand I'm very concerned with hosting anything in the cloud, especially with U.S. based companies because +3-letter agency here+ (especially for ERP software). But on the other hand the realistic view has to be that my self hosted stack is likely way more insecure than any cloud stack even if I made a conscious afford to keep everything secure (and that's probably also true for companies with a decent number of dedicated IT security experts) so 3-letter agancy would have an easier time getting to the data if they wanted.


I agree it is impressive.

But if you are running your own hardware, you probably aren't sharing CPU time with strangers as GCP is.


Disclosure: I work on Google Cloud.

Even if you aren't sharing with "strangers", applications may be vulnerable to these attacks. For example, the JavaScript attack clearly applies to people's individual computers. If you take untrusted code and execute it, you might be vulnerable to these intra-"instance" information leaks.

It all comes down to your threat model though. Some people are rightly worried about insider risk. If a malicious employee can go run a binary on your shared computing infrastructure to get root credentials out of a machine, that's actually a real problem. Then again, there are lots of ways for a rogue employee to do bad things, so this is "just" another one. But don't take this as "only applicable to shared cloud environments", because it's not.


I am not too worried about javascript as it runs in a VM where the browser has full control on how code gets executed, and I trust browser vendors will come up with a mitigant.

Otherwise on a personal computer, if malware gets to execute in user mode, being able to access some machine private key is the least of our worries. The sensitive information is the documents stored on the machine, whether it is for them to be leaked or encrypted, if the code can access them in read/write, the party is already over.


Sure but when you are worried about insider risk (and even if you are not) you most likely have strong access controls, extensive logging, endpoint security solution that can mitigate or at least alert on this activity.


To manage insider risk, we do everything you mention (as well as much more I can’t mention). We treat insider risk the same as external threats: ever present and requiring constant vigilance.

The VM is not considered a security boundary any longer; tenancy controls are now in place, to prevent attacks similar to Meltdown and Spectre that are yet undiscovered.


well there are still people out there who do not have these problems, i.e. people who have just a "few" employees to manage infra. people who have VMs just to actually limit compute nodes.

Actually we have our Test Infrastructure seperated from the rest, so basically only this runs untrusted code.


That's a fair point, but somehow the nervous feeling isn't limited to sharing CPU time with strangers!

The amount of platform complexity abstracted by the GCP services is staggering. This is the job of taking a (hopefully) understandable piece of computer code that represents some real-world logic and installing it into reality, in a sense. It's clear that it's a very messy world out there for a program.


That is why Cloud companies are similar to insurance companies. Little guys won't stand a chance against them.


I don't think they're like insurance companies at all. Cloud companies don't generate a big investable float based on a very low initial cost of the service provisioning. Quite the opposite, cloud companies have substantial pre-existing costs related to being able to provide the cloud services to each customer (including being able to do it at immense scale as necessary). That cost remains considerable and on-going at all times during the service providing period generally speaking. For insurance companies, the cost for a given policy is extremely low until a claim arrives.

To be like insurance companies, this is how their cloud business would have to work: you pay them $20,000 over ten years while receiving absolutely nothing tangible in return, until one day, ten years in, you need to utilize $25,000 in cloud services in a short burst. Then afterward, your bill goes up significantly, while as an industry they reduce what you get for that.

Beside that, there are exceptionally few things the little guy stands a chance at when competing head-on with a giant. The giant benefits from bargaining power at scale that is usually rather dramatic on most everything.

Eg Diapers.com was a well funded, well run ecommerce start-up. Amazon was going to destroy their business through scale leverage - applying plausibly anti-competitive below-cost tactics on core products like diapers - to put them under if they didn't agree to be acquired (Amazon had already begun applying that tactic in the lead-up to the acquisition, scaring the hell out of Diapers.com).


> Could other major cloud providers boast this?

Not sure. AWS sent out alerts in early December than many instances needed a reboot, and would be force rebooted on 1/8 iirc. I'm guessing it was to handle patches that were put in place for these issues.


Disclosure: I work for Google Cloud.

We were naturally first because we took action internally, but also because we didn't need to reboot guests to ship new kernels to hosts. We update kernels on our hosts frequently, and so nobody would have particularly noticed that their VMs underwent migrations (we do it all the time!).


AWS rebooted PV instances to implement the Vixen shim ( https://lists.xenproject.org/archives/html/xen-devel/2018-01... ) from my understanding.

I do not believe that HVM instances (which would be closer to your KVM infrastructure) needed to be rebooted.


I had an HVM instance that had forced downtime.


I had 35 HVM instances that didn’t have forced downtime. There are other reasons a machine could need to be brought out of service.


Seconded, I have about 320 HVM instances and have received only a handful of reboot requests.


Any public docs on how you update the kernel without rebooting the guests?



Amazing. I'm jealous (I use Linode).


Thank you!


Live migration of VM’s is a well solved issue.


Why? If you run on your own hardware how would this impact you?


It basically means that all of your programs and users have root access. If a buffer overflow happens on an exposed non-root service, it now has a way to get root access. It takes away a layer of security.


Well for a traditional web service your only exposed service is something like NGINX. So yes it makes the vuln. in say NGINX more severe but the attack surface is pretty small. Also the only users that have access are SRE or equivalent who have root access anyway. (I am not arguing that there is no downside just that it's fairly limited in a traditional scenario)


Think they deserve to. But not just just vulnerabilities but they also discovered Broadpwn, Cloudbleed, Heartbleed and a bunch of other ones.

Sometimes do not think they get the credit they deserve.


Whether they get credit or not, they certainly have the advantage of knowing about the vulnerabilities before they're published.


I'm kind of okay with that as long as they're not holding out to give themselves an advantage. If somebody is going to discover these, why not a cloud provider who can protect huge numbers of clients early?

If it was a little guy who discovered it, then everyone except maybe a fraction of a percent of the world (or nobody if the discoverer runs no servers) gets the same late start.


Disclosure: I work on Google Cloud.

Project Zero disclosed this to Intel, AMD and ARM on June 1 as stated in the blogpost [1]. I believe an independent researcher working somewhere else (or even at home) would have done the same, as indeed the parallel independent discoveries ultimately did.

[1] https://googleprojectzero.blogspot.com/2018/01/reading-privi...


By the way, can someone tell a reason why these parallel independent discoveries came during the same year as Project Zero? The vulnerability has been there for 20 years, and it is somehow strange that over the past two decades nobody thought of this, and now there are several who do and disclose it independently.


The Project Zero staff along with other researchers are almost certainly keeping up with the latest papers in the field. I think it's not unlikely that these papers would lead them all towards the same set of interesting related unsolved problems.


From the Meltdown paper:

Jann disclosed the issue to Intel in June. The subsequent activity around the KAISER patch was the reason we started investigating this issue.


This is probably the real answer



Thanks, that was a good read.

Still, as they say in this article, without a tinfoil hat: "There’s no reason someone couldn’t have found this years ago instead of today."


It's probably related to this phenomenon - https://en.wikipedia.org/wiki/Multiple_discovery

I think it's also less statistically unlikely than you think. Thousands of security issues are discovered every year, it makes sense that some small number will be discovered in parallel. It looks like a big coincidence because you are focusing on the one parallel discovery, but if you consider it in the broad landscape of all discoveries then it's just statistically expected that there would be at least one parallel discovery.


Tinfoil hat: Because we didn’t need it any more, or because the other side figured it out, so either way we needed to make sure it got disclosed and fixed.

No tinfoil hat: Because the series of research to date connected enough dots it crossed a threshold and lightbulbs went off all over.


Public research has hinted at the problem starting August 2016.

https://en.wikipedia.org/wiki/Meltdown_(security_vulnerabili...


Yeah, I'm not saying anything was withheld. I'm just pointing out that Google probably had a fix in mind, tested, and deployed much sooner than most. Not a bad thing, rather a good thing that's just a benefit of having been the ones to discover it. You can patch your own machines with a lot lower global risk than sharing outside the company.


I doesn't look like any of the *BSD people were let in on this issue until very recently. Apple, Microsoft and the Linux Kernel devs seem to have been given priority. It kinda shows where Linux is right now in the global landscape compared to two decades ago.

Funding and time/people for the kernel comes from the big players (IBM, Intel, et. al.) and distribution decisions have created a monoculture, making the big Linux distros feel like they're part of the corporate/national tech culture and less community centered.


I read, don’t have a citation though, that they were excluded because they refused to respect the embargo, not because of who they were.


That's more relating to OpenBSD than the other BSDs. FreeBSD was informed during the embargo (someone on another threads said at the behest of Netflix) but in late December.


Didn't the *BSD folks have history of unilaterally breaking embargoes on security issues? That could further explain why they weren't notified ahead of time.


The problem is sometimes they have dumped some exploits onto the public without giving their competitors enough time to fix the problem...they've done this multiple times to Microsoft. It's good in so long as they don't abuse it and it's treated as a public good rather than a corporate middle finger to its competitors.


What do you consider enough time? Google maintains a strict 90 day grace period so that the other party can figure out how to address the issue. If not 90 days, what should it be? In a world of complicated software, we need to have a discussion of what vulnerabilities users are subjected to by using a product, and whether/when users should be notified that those vulnerabilities exist


How much time did they give all the *BSD developers?

DragonFly had a patch recently .. in master. OpenBSD just had a rant on their mailing list.

Somebody correct me if I'm wrong, but it looks like zero.


Disclosure: I work at Google, but not related to this.

Its my understanding that Google disclosed this vulnerability to ARM, Intel, and AMD, and no one else, and left the hardware manufacturers the job of disclosing to the various software vendors. So, they gave 6 months to the people they disclosed the vulnerability to. Those people did or did not disclose it to others.


I was under the impression Google publicized these vulnerabilities before the period was over. Like 4 or 5 days earlier, not much out of 6 months, but still (I believe MS haven't released patches yet).


They did, because other people figured out something was up and the speculation was considered more problematic than holding out a few days.

As you say, the vulnerability was disclosed to others on the 1st of June, which puts us at 6 months for the disclosure deadline. If 6 months in vendors aren't ready with patches I doubt those few days would have made a significant difference.


Specifically, there were PoCs floating around on Twitter


Not so strict in this case!


I think your parent meant strict as in "it gets disclosed after 90 days, no extensions".


The article said it made a special exception in this case. That being said, unusual circumstances call for unusual procedures.


My impression is it's always "90 days or whenever it's made public by other means, whatever happens first".


Hmm, I was under the impression that they've known about Meltdown/Spectre since June and gave themselves an extension. But I'm not sure where I read that.


I've been impressed with how long it took for this information to be "leaked"/"declassified"/released given the sheer number of people who knew.

The fact that hundreds maybe thousands of people knew and worked on this ahead of the press reports/rumors and subsequent information release speaks to how seriously everyone involved took this.


I don't think it was a complete success. An AMD engineer mentioned the nature of the vulnerability on lkml before the end of the embargo, even though AMD was a party to the embargo.

Of course, it was probably unwise to discuss the code changes implementing mitigations in public anyways, but I don't have first-hand knowledge of how these things work in the Linux world.


Wasn't that also a reply to some part of this which had already surfaced to the public (but the impact maybe not known to the public).


There was an email thread discussing page table isolation mitigations for some bug, but Tom Lendacky mentioned it was about speculation. This let many others finally connect the dots, despite the earlier discussion taking place in the open for a while. Until the connection with speculation was revealed, it could've just been another KASLR info leak defense.


Really it never leaked at all. It wasn't until several weeks after the linux known-now-to-be-meltdown-defense patches entered review that non-involved people started putting the pieces together and realized it was a security fix. Only at that point Google pulled the trigger on the revelation.


Well we can’t really know if those have not leaked before. I imagine that this vulnerability was worth a lot of money, even if is was clear that it would be fixed one or two months later. There were too many people involved to think that nobody took the opportunity to sell it to the highest bidder. I personally would have preferred if google kept it in house as long as possible and made sure even at Intel, etc, a very limited amount of people could have seen it. This vulnerability could do a lot of damage if it was given to dangerous people.

But you are right it is very impressive that there were almost non public leaks (and the only leak was not more like a slip up which nobody took too serious until the patches were made public). I would not have surprised me if someone posted them and they got picked up. It would have made things a lot worse.


We can safely assume that black hats have moles on these lists.


I don't know why this was downvoted, it's a very reasonable assumption given the scale and spread of people who knew. Sure there's no evidence of any actual attacks exploiting this vulnerability, but that doesn't mean we can discount the chance that it was known to malicious parties before the wider public.


Upvoted, because you are 100% correct, and I need to counteract the downvotes that you for some reason received.

Edit: ... which is why I now received downvotes...


yeah the idea you can keep something like this away from malicious parties across dozens of huge companies is a ridiculous farce.


I've worked to mitigate other vulnerabilities within Google. at least with the vulnerability I helped on, I'm surprised at how good the information containment was.

Based on my area of work and job title, I was notified of the software that was vulnerable and to contact them with further info if we used it. I wasn't allowed to talk to others without explicit consent, so I talked to my central contact to find the right PoC in my neighbor teams to know who I was allowed to pull into a room for any joint strategies.

To everyone else, I simply said "sorry. On call is busy this week. I'm working on mitigating an undisclosed vulnerability." No questions asked. To this day I'm still not 100% sure the vulnerability is public so none of my peers know anything more than the time window in which I was distracted.


Direct link to a description of the fix: https://support.google.com/faqs/answer/7625886

Someone on SO trying to explain it another way: https://stackoverflow.com/a/48099456

Some interesting discussion about how this patch isn't a 100% fix for Skylake processors (at least that's my understanding): https://www.mail-archive.com/linux-kernel@vger.kernel.org/ms...


I don't understand how this fix can possibly be so performant. The speculative executions must be doing something positive, right? It seems like it should never hurt (ignoring security, obv). So how can disabling it not hurt?

Thanks for the stackover flow link!


Good for them, giving the credit to Paul Turner. Unusual for an enterprise to allow cracks in the corporate "We".


Google is better than most in this regard. For one, it allows its employees to publish papers with their names on it. It also frequently lists names in press releases of key members.


Every company that publishes research allows employees to put names on it. It's not really an indicator of a forward thinking company so much as one that wants prestige in the academic world.


> Every company that publishes research allows employees to put names on it.

Except Apple: Apples papers are authored by the "Apple ___ team".


Apple.


Paul is a boss. This is just one of his many amazing accomplishments. He's kinda the guy you go to if things are slow and you need some help to make them really ridiculously fast.


They usually do for researchers bringing new ideas, like on ai research papers


According to this page retpoline is "insufficient on Skylake and newer CPUs, where even ret may predict from the indirect branch predictor as a fallback; those need IBRS".

https://github.com/marcan/speculation-bugs/blob/master/READM...

What gives?


The way I read that, Skylake and newer don't see a performance hit from ibrs. So it's retpoline on older CPUs = no performance hit. Ibrs on newer CPUs + microcode update = no hit.

>Requires microcode update on current CPUs. Perf hit vs. retpolines on older CPUs


That's close, but I think I summarized it more accurately in this sibling comment: https://news.ycombinator.com/item?id=16130871


You're missing the point. It says "insufficient on Skylake and newer CPUs".

The Google Cloud blog post does not mention that retpolines are insufficient on newer CPUs. That omission is dangerous because readers will believe they are protected by recompiling with retpolines when in fact they aren't.

Perhaps the blog post can be updated to clarify the limits of retpolines so people don't get the wrong impression and end up with vulnerable systems.


That's incorrect (in our opinion). See my comment at https://news.ycombinator.com/item?id=16130871


Okay, the link I posted is just a writeup summarizing publicly available information, some of which may be misleading or incorrect.

On the other hand, "opinion" isn't enough. Everything depends on the microarchitecture and only Intel can give assurance on whether retpoline actually works in Skylake or not. I hope they will release information about it.


"Retpoline ... modifies programs to ensure that execution cannot be influenced by an attacker. With Retpoline, we could protect our infrastructure at compile-time, with no source-code modifications"

I am confused, doesn't this mean that Retpoline needs to sit in the compiler and won't protect from already-compiled binaries?


Yes.

We deploy new code binaries all the time so it's not a big deal.


That was my question also. How could they guarantee that nobody else using a VM on the same physical machine isn't using binaries compiled elsewhere?


They don't, but it doesn't matter.

Something compiled with retpoline is resilient from it's execution being impacted. Once KVM is recompiled with retpoline, guests cannot attack it via spectre variant 2, and as such, cannot attack other guests.

The hypervisor being compiled with retpoline, however, does nothing to protect from intra-guest attacks - if you have untrusted code running in your VM, and you don't have IBRS and the other microcode features on, or your sensitive apps compiled with retpoline, you are still vulnerable internally. Just not from other guests


Effectively both Linux and Windows Server need to be fully recompiled. Hope someone hasn’t lost some source code.

I am not super familiar with google’s offering but I suspect they don’t just offer VMs. Anything that runs in a shared infrastructure (serverless design/websites hosting) runs on top of a Linux box I presume. Google would need to get those Linux binaries recompiled too, not just the hypervisor.


I'm pretty sure that Microsoft can easily recompile Windows. Probably even Windows 95 with some prep time.


A hack they used in a recent update (I believe for the equation editor) suggests that's probably not the case.


I believe that the equation editor was third party coffee to which Ms never had the source code.


The point is they can’t recompile it.


Disclosure: I work on Google Cloud.

We've already patched the host (for a long while now). You can also recompile your binaries in your guest, and/or get an updated kernel. retpoline performs much better than the brute force hammers you've seen reported.


Google doesn't run VMs. Or, rather, even VMs that make up GCE are actually running in containers. From the Borg paper:

https://pdos.csail.mit.edu/6.824/papers/borg.pdf

VMs and security sandboxing techniques are used to run external software by Google’s AppEngine (GAE) [38] and Google Compute Engine (GCE). We run each hosted VM in a KVM process [54] that runs as a Borg task.

There might be a few machines here and there running actual VMs on bare metal, but those are going to be very special cases. Using Infrastore, it's easy to find which teams are running which containers whose binaries were built before the new compiler defaults.

https://github.com/kubernetes/kubernetes/issues/44095


I don't understand how you think that answers his question, as being in a container wouldn't magically add any more protections against these exploits than being in just a KVM VM. I'm also not sure that "Borg task" means it's in a container, vs. just being something scheduled by Borg, but it isn't particularly important either way.

If the question is 'How do they guarantee their customers aren't using bad binaries', the answer is they don't, and has nothing to do with VMs vs VMs in containers, etc.


I should have been clearer: their external customers can and will use bad binaries inside their VMs, but everything else, especially anything at the boundaries between VMs, will not: it will have the appropriate mitigations in place.

Running a VM in a Borg task (container) does not add extra security in this case, but it does help with auditing and catching the snowflakes and stragglers that invariably end up taking a disproportionate amount of time with changes like this. That kind of auditing can be easily done organization-wide, instead of having every team reinvent the wheel their own way.


It's my understanding that Spectre needs to share some address space with the victim, which isn't the case on containers, other than the host address space?


You just need a shared cache.


I'm not understanding, isn't a shared cache exactly the problem here?


You can share a cache on a physical CPU without having shared address space.

KVM virtual machines making use of Extended Page Tables or Nested Page Tables do not share an address space with their host or other guests, however, they are susceptible to Spectre v2, for example. (This was the PZ PoC for variant 2)


yes, I meant that to suffer from spectre you just need a shared cache; it's not necessary to share address space.

It follows from what a cache is. You have a large set of numbers (memory addresses) mapping into a smaller set of numbers (cache lines); my necessity there will be a collision.

The mapping is deterministic and you can recover information by looking at these clashes.

Some intel high end CPUs have a feature called CAT which was designed to avoid that different workloads interfere too much by stomping on each other's cache lines, but I think it's meant mostly as a performance feature not a security feature (although there is some research about how to use it to defend from cache-based side channels, see http://palms.ee.princeton.edu/system/files/CATalyst_vfinal_c... and https://arxiv.org/pdf/1708.09538.pdf).

Another possible approach is to make it harder to predict the mapping between a physical address (or an address delta) and a cache line. I don't know if using a cryptographic-grade mapping for hardware cache would be even remotely feasible nor whether it would actually solve the problem of information leak.


What does "retpoline" stand for? ret for the ret instruction, I suppose, but the rest?


The name “retpoline” is a portmanteau of “return” and “trampoline.” It is a trampoline construct constructed using return operations which also figuratively ensures that any associated speculative execution will “bounce” endlessly.

(If it brings you any amusement: imagine speculative execution as an overly energetic 7-year old that we must now build a warehouse of trampolines around.) [1]

[1] https://support.google.com/faqs/answer/7625886


"Trampoline". The term actually shows up with some frequency among compiler people.


I actually know that word in the compiler meaning, I think I just never got the connection, because I was pronouncing it in my head as ret-poh-line, not ret-poh-leen.

Thanks!


It's good to see Google crediting Retpoline to Paul Turner. As Senior Staff Engineer, Technical Infrastructure I wonder whether he was actually tasked with working on mitigation for these vulnerabilities or he came up with this in his free time.


As the article notes, lots of people worked basically round the clock on this as a top priority. That isn't to say they did nothing else, but Paul et al. definitely didn't just happen to do this in their spare time :).

The relevant quote from the article:

> For months, hundreds of engineers across Google and other companies worked continuously to understand these new vulnerabilities and find mitigations for them.


A bit off topic but the amount of real estate taken up by the header, side nav and "related articles" footer on this is just obnoxious. Obnoxious to the point of making reading this a really rotten experience.

I fear this is the medium.com effect of content on the web now. Simply having content for content's sake is now seen as a missed "growth hacking" opportunity.


Sad that this was collapsed-by-default — I realize it's not relevant to the article, but it likely never will be. If we complain enough hopefully someone can actually address it.


Also the low contrast, hard (for me atleast) to read font.


A very good reason why people are going to start moving to GCP over AWS.... Project Zero is a big win here.


Wait... doesn't Reptoline have some irritating performance penalties?


We use automatic feedback directed optimization[1], which lowers the cost quite significantly in practice

[1] https://research.google.com/pubs/pub45290.html

Performance gains are even higher now than they were then.


Disclosure: I work on Google Cloud.

Not particularly (if you read Paul's post, the branch to the retpoline predicts perfectly for obvious reasons), and especially not compared to the brute force flushes as an alternative.

Edit: I phrased that backwards. The return predicts, so that the whole thing is about as bad as an unpredicted indirect call:

> This has the particularly nice property that the RSB entry and on-stack target installed by (1) is both valid and used. This allows the return to be correctly predicted so that our simulated indirect jump is the only introduced overhead.


Edit: My post was prior to the parent edit, and now is largely unnecessary. Keeping for posterity, I suppose!

I must be misunderstanding Paul's post.

Isn't it specifically preventing any sort of prediction?

"Naturally, protecting an indirect branch means that no prediction can occur. This is intentional, as we are “isolating” the prediction above to prevent its abuse."

Of course, you can then go and manually add direct branch hints, as is noted in the post, but unless I'm misunderstanding things, there's not an obvious reason why these branches predict perfectly.

Not that it means performance is impacted in a significant way, since that same section also says "Microbenchmarking on Intel x86 architectures shows that our converted sequences are within cycles of an native indirect branch (with branch prediction hardware explicitly disabled)."

(which also confuses this issue - how is it predicting perfectly if prediction hardware is disabled?)


Does anybody know if Retpoline will make it to the compiler that the Linux Kernel is compiled with ? It doesn't specifically mention in this paper, so I'm not able to figure out what was actually compiled using Retpoline - userland or the Linux kernel itself ?


Disclosure: I work on Google Cloud.

Patches for both LLVM (the infrastructure behind clang) and gcc are available. You choose what you compile your kernel and applications with, and others are actively looking at retpoline and retpoline-inspired techniques for other code generators (e.g., various JIT compilers). That's why Paul and the folks made this public.


thanks for your reply. in Google Cloud's specific case, what was compiled using Repoline ? Was it the kernel or userland ?

Because you are kind of claiming that the patches that the linux kernel used to fix this (with a 10% drop in performance) are no longer needed. I am kind of wondering if this was submitted to the core linux kernel to be mainlined ? If yes, why was this not used there.

If no, then it means it is being used in some Google Cloud-specific way that is not mainlineable.

EDIT - found this comment which seems to suggest that retpoline is not bulletproof and the kernel's performance-killing patches are still needed

https://github.com/marcan/speculation-bugs/blob/master/READM...

http://lkml.iu.edu/hypermail/linux/kernel/1801.0/03137.html

Does it mean that Google Cloud is doing this only on non-Skylake CPU instances ? It is a very interesting stand - it will mean that it will suddenly be more cost effective to use Google's OLDER machines than the newer Skylakes... because the newer machines have a performance degradation that the older machines will not suffer.


Disclaimer: I'm not Paul :).

As mentioned elsewhere, we recompile everything at Google all the time. I'm not sure which things we've rebuilt with retpoline enabled. As Paul mentions in the article, the point is for software you believe needs to be protected, which may not be everything we build.

That thread on retpoline on Skylake has a lot of confusion. For some folks, they aren't 100% certain it works (it relies on understanding internal details of Intel CPUs) and they argue that IBRS on Skylake is cheap enough so "why not just always use IBRS and not bother?". That's the gist of this comment:

> personally I am comfortable with retpoline on Skylake, but I would like to have IBRS as an opt in for the paranoid.

I want to highlight that Paul and the team have had a lot more time to think about this issue than the folks just joining the discussion. Could our folks be missing something? Sure, and that's the point of public discussion and code review. We hope that over the coming weeks and months it's decided one way or another, but we believe retpoline to be correct and a good optimization (especially for older hardware).


> I'm not sure which things we've rebuilt with retpoline enabled. As Paul mentions in the article, the point is for software you believe needs to be protected, which may not be everything we build.

With all due respect, i'm not sure about the answer you are giving me. I dont see why we are using the word "belief". If there is some secret sauce that Google is unwilling to talk about, then it would be good to have that declared. Because it is extremely weird that you would claim to have a compiler tech that fixes this issue (which needed kernel level fixes otherwise) and not be able to categorically declare what was compiled by Retpoline that gives this benefit.

Possibly its the hypervisor you are using... which explains the zero downtime migration.

P.S. I'm a Google Cloud customer as well.


I meant that only in the “I’m not sure all code at Google has been rebuilt and had retpoline enabled” sense. We said in the article:

> By December, all Google Cloud Platform (GCP) services had protections in place for all known variants of the vulnerability.

which means all the bits of the stacks across all our GCP products (i.e., not just our host kernels and GCE’s hypervisor but also say App Engine sandboxes).

KPTI for kernels plus recompiling any sensitive thing (kernel, hypervisor, etc.) with a retpoline capable compiler. We’re not holding back secrets. Just trying to get the patches out while keeping blog posts relatively parseable by the general public.

Note that no matter how disruptive kernel or other changes are, live migration always makes them “zero downtime migration”. The key here is that using retpoline to protect systems software against Variant 2 results in much lower overhead than other proposed mitigation strategies, particularly on older hardware but even a bit on Skylake.


Has this at all caused google to reconsider the heterogeneous nature of the cloud in terms of hardware? It seems like Google the company is constantly fixing/redoing various intel problems such as ME and now this. Google is part of openpower after all, it would be interesting to see another architecture being pushed.


The post seems to suggest not all CPUs are affected by Variant 2. Is it Haswell and earlier only?


[censored]


That's not unusual; if a story gets one or two upvotes very soon after being submitted the HN ranking algorithm will put it on the front page. The rank then drops very quickly unless it gets more votes.


Interesting google has time, money etc for this.

But actually showing search results on page 4 of google search or youtube, when it said there were 22 million results for my search seems too hard for them.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: