Hacker News new | past | comments | ask | show | jobs | submit login
The new microcode from Intel and AMD adds three new features (lkml.org)
421 points by DoreenMichele on Jan 24, 2018 | hide | past | web | favorite | 158 comments

That link doesn't seem to show the continuation of the thread, but it continues to be rather interesting:


It looks like Ingo Molnar, Andi Kleen, and others are coming up with a pure-software alternative to IBRS, which could be much faster (rather similar in spirit to the retpoline trick)


Leverage CONFIG_DYNAMIC_FTRACE. This is an OS only fix resulting in effectively zero overhead for chips other than Skylake.


  There's another possible method to avoid deep stacks on Skylake, without 
  compiler support:

  - Use the existing mcount based function tracing live patching machinery
    (CONFIG_FUNCTION_TRACER=y) to install a _very_ fast and simple stack depth 
    tracking tracer which would issue a retpoline when stack depth crosses 
    boundaries of ~16 entries.

  The overhead of that would _still_ very likely be much cheaper than a hundreds 
  (thousands) of cycle expensive MSR write at every kernel entry 
  (syscall entry, IRQ entry, etc.).
Pretty damn cool if it works.

I think there's some misunderstanding in this discussion thread on what Ingo is working on.

This isn't a general fix or "pure-software alternative to IBRS" - it is specific to handling the 16 stack depth ret being vulnerable on skylake+ processors, as it falls back to an indirect branch prediction. The zero overhead for non-skylake+ chips isn't that it is a zero-overhead general IBRS solution, it's that it does not incur additional overhead on non-skylake+ chips as it is not doing any work to mitigate the issue on non-skylake chips.

This is because all of the major distros already compile their kernels with the CONFIG_FUNCTION_TRACER=y option set, so no additional work is being done unless the conditions appear and it is on a skylake+ chip.

What do we want for post-Kaby Lake systems? Should Intel return future hardware to pre-Skylake behavior?

Is AMD similar to Intel pre-Skylake, for the purpose of Spectre mitigation?

Could Intel and OEMs restart sales of Broadwell hardware, until they can reverse the Skylake/KabyLake/KabyLakeRefresh changes (in Cannon Lake? Ice Lake)?

I couldn't find much about this 16 entries stack depth issue. Does anyone have more information?

It's explained in the thread. Basically, the RET instruction has its own tiny branch predictor, the return address stack (also known as RSB) which is only 16 or 32 entries deep. If your call stack is deeper than that, sooner or later the stack becomes empty.

In this case, old processors simply does not predict the jump and stalls. But Skylake instead uses the generic indirect branch predictor, since after all RET is an indirect branch! Except the idea behind retpolines was to avoid the generic branch predictor, and now every single RET can use it! So, if you don't want to use IBRS you need to find a different way to protect the kernel from someone forcing undesired speculative execution after the RET.

RSB! Thank you, now I know what to focus on.

as a skylake user I hope I'll be able to turn those protections off

It is linked at the left of the screen where it says "Messages in this thread"

For me, when I click the link in TFA, that message is shown as the last message in the tree, with no children.

I just tried again, still didn't show any children. Then I did a force-refresh. Now it shows them ...

So due to past terrible performance of lkml.org, it had aggressive long-term caching. So since I happened to see this message a couple days ago, it wasn't showing a newer version of the reply tree for me, I only found them in the top-messages list on the lkml.org main page.

Semi off-topic: What is up with the lkml.org threading interface and why is it so ridiculously inconsistent?

I've always found it really frustrating that lkml.org seems to be the canonical web link that comes up for kernel mailing list discussions because of this, as I really find the threading there frustrating; it's nearly impossible to determine whether you've read the entire (sub-)thread.

Back when Gmane was alive I usually gravitated there as an alternative.

lkml.org owner here; let me explain the current behaviour: If available, it shows you the grandparent message and all its descendants for up to four levels (so n-2 to n+2), and if you navigate to another message it will just do that again - which means that the whole tree will change.

Suggestions (or working links to what you like better) are welcome!

A simple enhancement would be to show (for instance with "...") if there are more levels, that is, if there's a n-3 or n+3, put a ... where it would be. That would make it at least a little bit easier to navigate, by making it explicit when you're seeing an incomplete view of the thread.

Personally I would like some way to navigate through the whole thread in some sort of reasonable order. My preference would be depth-first with the leaves sorted by the most recent post /in that subtree/. This matches how tools like newsreaders and Mutt will sort threads, and I believe how Gmane worked as well. (Though I will admit that for something like a web site that does not keep the state of whether posts are read or not, reordering leaves like that may be confusing. In a tool like Mutt it is useful behavior as when new posts come in they will naturally gravitate to the bottom [or top, depending on your sort order] of your message list).

In particular I'd like to be able to, starting with a particular post, repeatedly hit "next" and be assured that once I hit the next sibling post to the original I've read that entire subthread. Gmane let you do this just by whacking the space bar; if you weren't at the bottom of a post it would page down, if you were it would go to the next one.

In other words I'd like it to behave as much like a good NNTP newsreader as possible.

> like a good NNTP newsreader

That's lost technology from the golden age of our ancestors, who were so much greater than any of us now ;-)

I can't tell if you are being sarcastic, but because the format and transport were fixed, it enabled the creation of many clients, some of which had some incredible useful innovations.

Today this innovation has moved to the myriad of proprietary alternatives (like Slack or our very own HN) but they are closed worlds so not only can nobody create a new client, we also loose all the contents if the company dies (which isn't rare).

--grumpy greybeard

it's not lost - it's just not in the spotlight due to these young person's webs.. facebook, twatter or whatever .)

there is plenty of active usenet groups, comp.arch being one of the best places where the greybeards meet.

How about just show the whole thread? If it is a huge thread, so be it, we want to know.

As I just noted above, my problem was apparently with caching. I saw the message linked in TFA a couple days ago, when it had no replies yet. Multiple days later, I still saw it as having no replies, until I force-refreshed.

Something like google groups will be great. You can expand/see all replies though not in threaded mode (just linear).

Sounds like you didn't read Andi's reply.

> But wait, why did I say "mostly"? Well, not everyone has a retpoline compiler yet... but OK, screw them; they need to update.

That got me interested, who has reptoline support, and from when?

GCC added support in 7.2 [0], which means a lot of people don't have access yet.

LLVM added it in D41723 [1], but I don't think that has landed in a release yet.

I can't find any talk around TCC, but that isn't exactly surprising.

Haven't heard anything about MSVC, and can't find anything, but that's not surprising as I don't think Windows is using reptoline in their mitigations, but I really have no idea.

[0] https://www.phoronix.com/scan.php?page=news_item&px=Clear-Li...

[1] https://reviews.llvm.org/D41723

I'm not sure how close this is to the retpoline mitigation but ... https://blogs.msdn.microsoft.com/vcblog/2018/01/15/spectre-m...

Our tests show the performance impact of /Qspectre to be negligible. We have built all of Windows with /Qspectre enabled and did not notice any performance regressions of concern. Performance gains from speculative execution are lost where the mitigation is applied but the mitigation was needed in a relatively small number of instances across the large codebases that we recompiled.

Thanks! I think that and the inner-linked post tell us what we need to know.

From Terry's post:

> For context, on newer CPUs such as on Skylake and beyond, Intel has refined the instructions used to disable branch speculation to be more specific to indirect branches, reducing the overall performance penalty of the Spectre mitigation.

And your link:

> /d2guardspecload, that is currently equivalent to /Qspectre.

It seems equivalent to reptoline, at least from a high level perspective.

So MSVC is probably protected, but only optionally. They're leaving it up to developers to use the /Qspectre flag, it isn't something automatically enabled when you use an /O(x) optimise flag.

Edit: I should add that /Qspectre is currently being added to a lot of Windows, but they're assessing where before doing so, because not everywhere should need it.

From what I understood from that link, /Qspectre is for variant 1, while retpoline is for variant 2, so it's not equivalent.

> GCC added support in 7.2

No, GCC adds support in 7.3, which is scheduled to be released tomorrow.

You're right.

A few distributions like Clear Linux and FreeBSD patched 7.2, but everyone else needs to wait til 7.3.

Solus' kernel is also compiled with retpoline since a few days ago. They "borrowed" the compiler from Clear Linux.

That "not everyone has a retpoline compiler yet" was in the context of the Linux kernel, which so far compiles with GCC only (IIRC it also compiled with ICC in a distant past; and there's an effort to make it possible for LLVM/clang to compile the kernel, which is nearing completion). So in the context of this discussion, he's talking (and thinking) about GCC.

Sure, but it still got me curious what compilers had added support.

Can somebody here explain why Intel and kernel devs are still debating about solutions in late January? I haven't followed this stuff closely, but I was under the impression that Intel and others had been informed about these vulnerabilities 6 months ago and everybody had fixes ready by early January.

IIUC, a limited number of people from kernel community knew about this and NDAs didn't exactly allow all the parties work together. The first solutions/mitigations exist (might not be nice), but now embargo is lifter and the communities (incl. companies) can work on better solutions together.

I would hope people from the Linux and BSD kernel communities arent signing NDAs. I would expect them to refuse on principal. Do they not?

I don't sign them (in general), but my employer has agreements and I like my job enough to accept that secrecy and embargoes are on occasion a necessary evil.

In this case, however, it was handled horribly. During the embargo period, we kept telling Intel and AMD what they were doing wrong, and they wouldn't listen (or alternatively, they made clear enough that they wouldn't listen that we didn't even try). But really there's little more than I can do than hope that the next time they listen to us.

If only one of them had listened, it might have been an impetus to stick with that manufacturer.

It is reasonable to avoid disclosure of security issues till the bugs have been patched. Absolute principles mean nothing in the context of flawed realities.

wouldn't the NDA in a case like this just be about the embargo? If this is the case, what strong argument against an NDA is there given that agreement not to disclose the bug is necessary until a good mitigation strategy is in place?

In the Spectre case, the 6-month NDA did not result in a good mitigation strategy. A few weeks of public discussion has identified better mitigations for some customers.

First, it did. Retpolines on pre-Skylake + IBRS on Skylake is pretty good actually. However, it turns out that (as is expected when you add new people to the group which are smart and bring a fresh mind) you might be able to do even better. It's okay, and it's expected. Since these are mitigations, not fixes, incremental improvements are the right thing to do. The "garbage patches" are more than enough for distros that want to provide a mitigation to their users, Linus just doesn't consider them a good idea in the long term. What was not okay is that no one knew who exactly knew what and hence it was not even possible to discuss anything---which is the reason why all these things are being discussed _now_, after the embargo has been lifted. But even if this wasn't the case...

... for Linux distributions the actual embargo time was a little less than two months. That is actually a very small time to do the amount of work that was needed to mitigate Meltdown and Spectre. No Linux distribution was able to ship retpolines on the date the embargo was lifted (heck, only RHEL and SuSE shipped anything for Spectre at all), and the extra week would have bought us nothing. We would have needed to be notified a month or two earlier.

No, it was also about who you'd discuss the issue with, even if you knew/suspected that they were disclosed.

Botched disclosure process.

More details at the end here: http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-...

I am sorry but after 6 months of Intel/AMD doing nothing I don't expect the Linux community or anyone else to keep their mouth shut.

Also because of 5 days? Embargo was for the 9th of Jan. and it leaked on the 5th after 6 months! What are you going to do in 5 days that you didn't in the last 6 months.

The botched embargo destroyed their chances of managing the PR like they thought they would be able to do.

Intel thought they could roll Spector and Meltdown into a single massive bug, that all CPUs were vulnerable to. Instead, AMD leaked the fact that AMD CPUs didn't need the KPTI patch, which meant the very first thing people knew was "it's an Intel bug". This very much put the Intel PR effort on a back foot.

Then Intel thought they could just release their microcode update (hell, they even coordinated with AMD to release the same microcode patches), and tell linux "here are the patches to use it, you must merge it". I assume the plan involved disclosing on the 9th and essentially bully Linus into merging the patches by the 10th.

They wanted people to thank them for spending millions to develop the new microcode and once-and-for-all solving this problem. They wanted the headlines "Intel releases microcode which fixes Spectre/Meltdown flaw" so that people would stop demanding intel replaces their broken silicon.

Then google releases details about their retpoline idea (I'm assuming it was developed not long before the embargo ended). It achieves the same thing, but with less performance penalty and no microcode update needed.

So now the Linux mailing list has something to debate. The Intel engineers are very motivated to get their patches merged, because they spent millions developing that microcode and they want the PR wins for "fixing the problem". Watch how the immediately segway into "oh, but retpolines don't work on skylake"

The Linux engineers are not happy with this, they are angry at Intel for being the source of the problem and probably see through Intel's little PR plan. They are triple checking intel's work and seeing if they can come up with an even better solution that works on skylake too.

According to https://plus.google.com/+jwildeboer/posts/jj6a9JUaovP Google was deploying a retpoline-based mitigation in production in September. Seems likely that existed _before_ any thought of microcode fixes. I wonder if, based on working with Skylake and newer chips only at the time, Intel thought the microcode version was going to be a performance alternative to retpoline instead of it being the other way around.

Is Microsoft relying on Intel & AMD microcode to mitigate Spectre, or could they also use the approach now being proposed by the Linux kernel developers? If they don’t, will Windows performance be worse than Linux on the same hardware, for a comparable workload?

Yes, RHEL too (and probably apple).

The Intel engineer in that email seems to be bragging about it:

    I think we also want to expose IBRS to VM guests, even if we don't use
    it ourselves. Because Windows guests (and RHEL guests; yay!) do use it.

RHEL is going to switch to retpolines too. But it will take some time.

It seems pretty obvious that they will be following what happens at the very least, and won't allow themselves to fall into irrelevance if better classes of solutions come along.

Since Microsoft is in the Linux Foundation and runs Linux guests on Windows, they could contribute to the public discussion on Spectre mitigation in Linux.

I hope so. Their first round of bug fixes caused pretty significant slowdown on 2015 and older CPUs.

> With Windows 10 on older silicon (2015-era PCs with Haswell or older CPU), some benchmarks show more significant slowdowns, and we expect that some users will notice a decrease in system performance.

> With Windows 8 and Windows 7 on older silicon (2015-era PCs with Haswell or older CPU), we expect most users to notice a decrease in system performance.

> Windows Server on any silicon, especially in any IO-intensive application, shows a more significant performance impact when you enable the mitigations to isolate untrusted code within a Windows Server instance. This is why you want to be careful to evaluate the risk of untrusted code for each Windows Server instance, and balance the security versus performance tradeoff for your environment.


Retpolines certainly existed in November, perhaps before. But it's somewhat harder to modify compiler+kernel than shipping microcode you get from Intel and modifying the kernel to adopt a (no matter how ill-designed) new CPU feature. Also, it's not easy to make sure that retpolines cover all the corner cases. There was little time.

Intel engineers are a nice and cool bunch. It's not their fault.

Perhaps a naive question but can we expect something like retpolines for Windows or is it already done that way, or does Windows rely on the slower Intel microcode? Any idea?

EDIT: I just saw replies to walterbell asking the same question as me, so please ignore this one.

nit: segue, not segway

Whether it was botched or not (from the position of this article) is based on whether you think it was important that FreeBSD have been notified ahead of time. Obviously the FreeBSD devs think they should have been notified. Since this basically affects everyone that runs x86 code, and not everyone could be notified ahead of time and still keep any form of secrecy, some groups will need to be notified and others not.

I think a case can be made that as operating systems go, FreeBSD is at a much lower threat level than Linux, MacOS/iOS, Windows (i.e. desktops that run unvetted code) and the VPS platforms that run host operating systems that need to worry about guests breaking isolation (Google, Amazon, others).

Given all that, while it wasn't done perfectly (it was leaked a few days early and some cloud providers had little warning, especially Joyent which also runs a different OS), I'm not sure I would call it botched entirely since it is also likely the largest exploit in history. Others disagree with some or all of this.

I read somewhere (didn't verify, wasn't obviously substantiated) that the OpenBSD folks were beside themselves with anger, and that in turn others shot back that they had demonstrated a very poor track record with respecting embargoes in the past and had let slip “seven of the last five” major vulnerabilities they had been pre-warned of.

EDIT: For reference, consider some of the comments in this thread: https://news.ycombinator.com/item?id=16110750&ref=hvper.com

Seven of the last five? Sounds like they have some out-of-order execution issues of their own.

It’s a common British idiom, to opine against eternal doomsayers. :)

First time seeing this usage, can you tell me what is this called exactly, in grammatical terms? Tried to Google and can't seem to reach anywhere.

How do you mean? The formal disclosure was supposed to be earlier this month.

[Edit] I see, link implies the problem is that the 6 month disclosure process lacked urgency, and many affected parties were notified late in the cycle. Perhaps this is an argument against agreements to keep vulnerabilities secret for a long time, as an attitude of, "there's enough time to address this," can be counterproductive.

Echoing the sentiments from OpenBSD's Theo de Raadt as well:


4,000 cycles for a speculative execution barrier is just horrible. Since everyone needs a patched kernel, might as well take the time to recompile the kernel with retpoline compiler and remove all offending instructions in the first place.

Except, thanks to modern speculative execution, skylake and later also speculates through retpoline.

So you still need some solution there.

I'm pretty sure it's not that it speculates through retpoline, IIRC retpoline is coded with trickery to avoid that, but it only replaces indirect branches. It's that it speculates through normal `ret` instructions in functions, in a way that can be exploited.

It's that prior to Skylake, ret instruction prediction only ever used the "return stack buffer" (RSB), but since Skylake the indirect predictor is used as a backup. The RSB is a prediction mechanism specifically for call and ret pairs: each ret is predicted to return the instruction following the corresponding call.

This works great for the common pattern of matched pairs, but the stack used to track outstanding calls has a limited size (32 in Skylake), let's call it N. If you have N + M calls followed by N + M rets (or any other pattern where the call chain gets that deep), you will predict correctly the first N rets, but then the stack is exhausted and the last M rets won't be predicted by the RSB.

Prior to Skylake, those last M rets just wouldn't get predicted at all (probably instruction fetch would just fall through to the instructions following the ret), but in Skylake the indirect branch-predictor, which is usually used to predict jmp or call instructions to variable locations, is used as a fallback instead.

So the concern is that people could train the indirect predictor prior to a kernel call, and if the 32-deep RSB was ever exhausted, the indirect predictor could kick in causing a Spectre-like vulnerability.

On ARM (A64, A32, T32) we get CSDB barrier, but it's a hint instruction instead of going via MSR registers.

From whitepaper CSDB is 1101_0101_0000_0011_0010_0010_100_11111

Here some snippets from ARM manuals:

HINT instruction:

1101 0101 0000 0011 0010 0010 100 11111 CRm = 0010 op2 = 100

Some encodings described here are not allocated in this revision of the architecture, and behave as NOPs. (This is important)

Hints 18 to 23 variant Applies when CRm == 0010 && op2 != 00x. HINT #<imm>

Hint is encoded in CRm:op2 pair, existing similar:

0010:000 ESB // Error Synchronization Barrier 0010:001 PSB CSYNC // Profiling Synchronization Barrier

Thus in assembler this is written:

hint #0x14

Which is a NOP if SOC does not understand this hint. It's being used here: http://lkml.iu.edu/hypermail/linux/kernel/1801.0/04191.html

and also here: https://github.com/ARM-software/speculation-barrier/blob/mas... (which is being upstreamed to compilers in cross-platform generic form IIRC)

ARM whitepaper states that conditional selection/conditional move is enough on most ARM implementations. If it's not the case then the new CSDB solves the problem. On older CPUs it's still a NOP.

X-Gene disables branch prediction: http://lkml.iu.edu/hypermail/linux/kernel/1801.2/06482.html

ThunderX2 branch prediction hardening: https://patchwork.kernel.org/patch/10151975/

CSDB is for Spectre variant 1. The post is about variant 2.

Thanks for the nice writeup!

This might be fundamentally irrelevant to this thread, but it got me wondering and I would like some assistance/guidance:

Whenever I read the mailing list, or threads related to it, I don't understand 99% of the stuff. Do I need to know C _very well_? Do I need to be familiar with kernel? If so, how do I go about doing that? At the moment I'm reading "Code: The Hidden Language of Computer Hardware and Software".

Generally, you need to know the operating systems and hardware work together. And then depending on the details, you might need to know C and assembly. In this case, you need to have good understanding on how modern processors do performance optimizations.

There probably isn't any good single resource for learning all this, but perhaps getting some good textbook on computer hardware architecture might be good next step after reading the "Code". Unfortunately I don't have any immediate suggestions on the current crop - I remember learning from William Stallings' books some 15-20 years back - not sure if they are the best choice nowadays.

The course "Computer Systems Architecture" at Queen's University uses "Computer Organization and Design: The Hardware/Software Interface" by Patterson and Hennessy. It's a 400-level course in the computer and electrical engineering department. I can recommend it. Most chapters in it have a "Real Stuff" section where they look at a real world CPU and compare it with the contents of the book to see how the theory actually ends up in practice.

I'd also recommend the text Structured Computer Organization by Tannenbaum[1].

[1] https://www.pearson.com/us/higher-education/program/Tanenbau...

Thanks. A simple search returned that it is still being used by some universities. I should take a closer look after I'm done reading "Code".

At this level you need to start understanding details of the hardware and how actual execution differs from the model presented to the programmer. Here's the MIT course Computer System Architecture which gives an introduction to what you need to know.


After that, I don't know, read some deep dives into a modern architecture and look up any terms you don't understand.


When you're working in userspace, the kernel shields you from much of the hardware complexity. When you're working in kernelspace, the whole hardware complexity is visible; it has to worry about details no userspace code has to care about. It doesn't help that the Linux kernel is highly optimized, using tricks that are uncommon in user space (like self-modifying code; the "alternatives" mechanism overwrites instructions with a variant depending on the hardware, at boot or hotplug time), or highly complex lock-free code (like the "read-copy-update" mechanism).

My personal recommendation is to read the Linux Weekly News (https://lwn.net/) every week (if you don't want to subscribe, the previous week editions are available for free). It always has great explanatory articles about the Linux kernel internals, written by Linux kernel developers. For instance, this explanatory article about the "read-copy-update" mechanism, from the author of the Linux kernel implementation of that mechanism: "What is RCU, Fundamentally?" (https://lwn.net/Articles/262464/).

Well, you’ll only understand 100% of it if you’re a kernel developer (and a good one at that). What’s your target? Knowing C and fundamentals of computer architecture + OS design probably gets you to 20-30%. Having spent a bit of time around the Linux kernel itself (writing extensions, drivers, read random blog posts, etc) might get you 60-70%, and so on.

The book you’re reading is very good, your time probably isn’t wasted reading it.

Thanks! My object is actually merely to understand it myself. I'll stick to reading the book, and then finding a book on OS design etc. as the other commentators suggested.

A lot of the discussion where really isn’t about C. It’s about the mechanics of how the CPU works, particularly the functionality it exposes to the kernel. I recommend reading the Intel Architecture Reference Manual, Volume III.

Having a very good grasp on C (things like pointer arithmetic, bitwise operations, ...) is definitely a prerequisite

1. Basic Computer Science: You need to be intimately familiar with hexadecimal and binary notation as well as boolean logic and similar fundamental knowledge, like data structures, their construction and manipulation, searching and sorting algorithms, abstract programming concepts, etc. etc...

2. Language and Vocabulary: You need to be able to read and write (technical) English at a competent level. Virtually all technical documentation is in English, and most of the resources you could find on the web (like this Wiki and the forum) are, also. Using incorrect terminology will make you look foolish and confuse the people willing to help you.

3. Language and Vocabulary, pt. 2: Most operating systems featured on this site, as well as most of the code snippets and examples, are written in C (or C++). Even if you choose to use another language (like FreeBASIC or Pascal), C is the lingua franca of programming, and you should be competent in making heads and tails of it.

4. Assembly: You should have knowledge about the low-level language Assembly. Read a book. Take a course at school. Write some user-space code to familiarize yourself with it. You will need it, even if you plan to write most of your operating system in a higher-level language.

5. Programming experience: Learning about programming with an OS project is considered a bad idea. Not only should you know the language in which you will be developing inside out, you should also be familiar with version control, debugging, etc. - in short, you should have written quite a few user-space programs in that language successfully before trying OS development.

6. Programming Practices: You should know how to write both code and user documentation, and be prepared to carefully document all aspects of your code and design, even if the project is purely for your personal use. Also, you should learn and use a suitable Code Management practices, including setting up and using an off-site repository (e.g., Savannah, GitHub, GitLab, Heroku). These two practices alone will - not may, but will - save you a great deal of trouble in the future, if you follow them.

7. UNIX experience: You will soon notice that many of the tools used in OS development are developed for Unix, and later ported over to Windows. The Linux kernel is often used as reference or example for how things can be done, and many of the hobby operating systems have some resemblance to Unix. Having experience with the Unix command line (preferably Bash or ksh) is a very important requirement. (Cygwin provides an easy-to-install Unix command line for Windows.). If you haven't, go ahead and use Linux or a BSD for a while - for Windows users, this can be accomplished quite easily by using a virtualizer (see below) to host a virtual system, without the need to re-partition your home system. For macOS users, you can simply use the Terminal - macOS is built on a Unix variant kernel (a combination of the Mach and BSD kernels), so as long as you have Xcode and an emulator or virtualizer installed, no further tools are necessary (though you will need to set up a cross-compiler); the default Terminal shell is the Bourne-Again Shell, but C shell and KornShell are available as well.

8. Toolchain: You must know the behavioral details of your compiler, assembler, linker, and make utility. You should know what do the emitted warnings and errors mean. You should have the documentation of the tools you use at hand, and refer to them before asking the community. Rest assured that any possible beginner's question about GCC, GNU as, NASM, GNU ld, Visual Studio and GRUB has been answered twice over. Search before you ask.

9. Emulators and Virtualizers: Familiarity with tools such as Bochs, VirtualBox, QEMU, or Virtual PC is key to having a reasonable turn-around in development, and provide a buffer between your real hardware and your test system. While these can be learned for the specific purpose of OS dev, you will certainly want to be aware of what they are and where to get them before beginning an OS project.

10. Executable Formats: Kernel space programming has many additional requirements unknown to application development. Being able to parse executable binaries is one of them (you do want your OS to load and execute applications, do you?) Make yourself familiar with Executable File Types, their internal structure, and how a linker generates them.

11. The Platform: You should have studied the manuals for the processor you will be programming for. They contain the information you need to design your kernel in the first place. Asking for information that can easily be obtained from these documents will only give cause to "Read The (...) Manual" responses, or simply RTFM.

12. The Concept: You should have an understanding of how existent operating systems function at the technical level (e.g. by having read some Books), what is good or bad about them, and how you want to go about your own OS. Having finished some tutorial and then asking "what now?" in one of the forums will just make you look silly.


Read Tanenbaum, take it from there

To GP: Modern Operating Systems[1] is taught in almost every CS program in the world. It covers a lot of the topics needed. It might be a bit dated but has a solid foundation.

[1] https://www.goodreads.com/book/show/166195.Modern_Operating_...

Thanks! I'll take a look at his book. You're talking about "MOS", right?

Yep, it might not be exactly how it works in the current Linux kernel but the underlying principles are solid.

That and Structured Computer Organization. MOS is software-focused, SCO is hardware-focused. Both have some overlap, of course. Meltdown/Spectre are hardware bugs at their core, so having a more thorough understanding of hardware is helpful with them.

Check out /r/kernel for some links on learning more about kernel hacking.

Inside the Machine by Jon Stokes will cover this kind of stuff.

Since all these CPUs are turning out to be so inherently insecure, maybe we should work harder to run only trusted/verified code.

There’s a limit at which improving the car compensates for the degradation of the road. At some point, repaving is the only correct fix.

Unfortunately, there is only so much trust we can put in hardware that is "known" to be secure. What other exploits are possible that we don't know about?

While I wouldn't want to compel people to run only verified binaries, I think it's important that we make it an option. Right now, it really isn't, especially with proprietary software.

Is retpoline the equivalent of picking another road in this analogy?

no, it's the equivalent of slowing down because you're hallucinating behind the wheel and can't trust what you see.

More like picking special tires that can run on the shitty road. The roads are the hardware itself and their vendors (I would assume).

I can see that making sense. Still see the retpoline reducing the forks in the road, forcing the "proper route".

No you still misunderstand. The road in the analogy is the hardware itself. And the car is the OS. The OP's point was that we have to replace the hardware, because at some point you are hacking up your car to absurd degree just to be able to drive on the road of the hardware. It has nothing to do about forks in the road and the analogy to branching.

He could be arguing for anywhere from, we need to replace all our hardware now, to we need to start over with how we design chips.

Trusted and verified are separate things. Trust is by the user. Verification is by user or trusted third party.

Do you trust random JavaScript from the network? No? I didn't think so. So you trust RandomThirdParty verification?

Even banking code (ran on Secure Elements in credit cards) has levels of verification (by certified bodies) and trust (by banks and card processors). Where did you draw the line?

How does a user decide which third-party to trust for code verification?

E.g. do all/some “certified bodies” accept financial liability if their verification is later found to be incomplete? Since Spectre/Meltdown were not widely known, what happens if previously-certified code was exploiting this vulnerability?

I could act like Richard Stallman, and refuse to run any JavaScript. That doesn't sound very appealing, though. I would argue, however, that we need a culture shift away from JavaScript.

As far as verification goes, if authors use a GPG private key to sign their code, and distributors (package maintainers, etc.) do the same, they can be "verified" by the web of trust. That verification could also be hardened by a third party certificate body, although I'm personally wary of letting certificate authorities decide which code I can run on my computer.

One problem with that is Javascript. Users download and run dozens of untrusted programs a day. I have JS mostly disabled in my browser, but that leaves websites partially broken.

Is there a point to running verified code on insecure hardware?

If you verify that it does no funny business with the hardware bugs it should be no problem, right?

I figure comprimising alien/userland code could well be an issue, something like Javascript in a browser.

I assumed that the GP included userland, and the browser, when they advised only to run trusted code.

The new microcode from Intel and AMD adds three new features.

The second (STIBP) protects a hyperthread sibling from following branch predictions which were learned on another sibling.

Is this about threads in the context of SMT?

I guess what I'm asking is "hyperthread" a thing outside of Intel marketing speak that warrants mentioning AMD?

Hyperthreading is Intel's implementation of SMT. However, since Intel was the only one doing SMT on x86 until very recently, hyperthreading has become the de facto generic name for it.

Ah that makes sense. My last exposure to this space was IBM Power stuff and we were calling them sibling threads or side threads.

In new AMD CPUs (Ryzen, Threadripper and EPYC) there are simultaneous multithreaded engines - core siblings, one per core, issuing is 2 extra instructions compared to Intel SMT 4.

> You might want this when running unrelated processes in userspace, for example.

When are you not? Why have security and process isolation if all processes are considered related?

You might perhaps have a setup where trusted users are running trusted code, can guarantee* a lack of malicious intent, and primarily want the user-and-process-segregation system to prevent a single failure of a single process from affecting anyone else.

You might conceivably have this in a corporate environment (I do).

For the vast majority of systems that do run untrusted code, including web browsers, this mitigation is probably appropriate.

Perhaps the word “might” here could be upgraded to “probably”?

*for some arbitrary value of “guarantee”

So you run everything as root, or your setup would work just as well if you did?

How is that comparable? Running everything as root is a) asking for accidents; b) a different class of problem.

You still want to run as a regular non-root user, possibly different users. The key difference to the mainstream use case is that you're looking to minimise the impact of bugs and failures, rather than manage security and combat malicious actors/code.

For this particular use-case, you might well consider that you don't need to patch for meltdown and spectre given the perceived risk/performance tradeoff for the current patches.

I think you misunderstand. The statement "when running unrelated processes in userspace" probably means, when running multiple threads under the same process id. Threads are "lightweight processes" in linux parlance (e.g. linux treats each "thread" as process, and then deals with the fact that some of them share memory). What they are saying is that this instruction is - for example - for a browser (a userspace process) to clear the branch prediction cache after running untrusted javascript on another thread.

At work we use the same account on some remote boxes to run trusted code. This works as we trust each other and follow the agreement not to mess with co-worker files unless previously agreed. Yet process isolation allows to kill run-away processes without affecting other processes or shells.

That is the same mindset as users who say they have nothing to hide so they don’t need to use a strong password.

Unless your computer is air-gapped you can’t trust it to run only the code you expect.

The decision on how much effort you expend to secure your particular system(s) often comes down your perception of risk and reward (and it's worth emphasising that word perception too).

There's usually a cost associated with greater security[0], and generally a balance that is struck. You can argue whether a particular system should lean further towards security, or further towards lower-cost, but ultimately there'll be a judgement call.

I'd suggest there isn't a single correct answer for this.

[0]whether in actual dollars, ease-of-use for users, time-to-market or possible reduction in feature set

Indeed, many people make wrong decisions based on incorrect assumptions.

At present time using separated user account gets very little from a security point of view judging by number of local privilege escalation bugs that are constantly found for common Linux setups.

In our case if the malicious code will run on those shared boxes, it implies either a bug in ssh or an Ubuntu software update bringing malicious code or an attack through a hacked computer of one of the users. In either case protecting other users from the effect of malicious code on those boxes will be the least of our worries. Yet the separated accounts do have substantial usability overhead.

So, the interesting thing here, to me, is this bit:

>So now we mostly don't need IBRS. We build with retpoline, use IBPB on context switches/vmexit (which is in the first part of this patch series before IBRS is added), and we're safe. We even refactored the patch series to put retpoline first.

Unless I'm reading this wrong, it looks like retpoline is not enough, and you still need the microcode updates from Intel or AMD that offer IBPB.

If this is the case, why has Google been saying that retpoline is all we need?

From what I can see, the retpoline works on pre Skylake processors. I am guessing that Google might not be using Skylake+ processors and so their comments are not wrong for their equipment.

No, it's pretty specifically not about the Skylake issue - The retpoline not working on Skylake is fixed by IBRS, and is due to ret calls falling back to indirect prediction in deep stacks.

vmexits and context switches are a very different thing than the deep stack ret calls.

I have always felt it, but presumed it was "cross talk" in the optical nerves being interpreted as "movement" in the outer ear.

I think you might’ve commented on the wrong post somehow there.

What's very odd is that it's a copy of the first sentence of this comment: https://news.ycombinator.com/item?id=16218608

Accidental copy/paste through an interface for the visually impaired?

> about: Visually Handicapped but can handle computer online work

Wow. You made me smile. Took the effort to investigate, identify a good reason and defend.


Huh, that's a good point. Apologies if my comment came off as accusatory.

I think your comment was fine. I would have had the same kind of thought -- possible bot trying to age an account with content lifted from other legit posts.

That’s a good analogy for Spectre

I swore that Intel had recalled the microcode updates because of issues (I've not had any of note)?


My 2 unasked for pennies - the kernel is already too bloated with support. These are hacks and should not be mainlined. Let them fix their hardware and give arduous instructions to patch...they deserve it. Don't further pollute the kernel. Imagine if a tiny hardware vendor tried the same...ubiquity is not an excuse.

> These are hacks and should not be mainlined

Of course they should. Linux needs to work on hardware that exists not just on hardware that may, one day, exist.

> Don't further pollute the kernel.

One of the primary jobs of kernels, drivers and compilers is to abstract some of the underlying hardware complexity away. Kernels, drivers and compilers have always dealt with bugs and limitations in hardware.

Things like retpoline and CPU barriers aren't especially unusual. Emergency fixing such a pervasive issue is unusual though, hence the attention.

> ubiquity is not an excuse

No, but it's a really, really good reason

If I were inclined to facetiousness (which I'm not), I would point out that there is a large number of other kernels out there that aren't especially performant and don't work well on most hardware. If you really object to this, you can always use one of those.

”Let them fix their hardware”

That could (and, I guess, would) mean a much, much, larger performance hit than a combined microcode/kernel fix would give. I doubt people would be happy about that.

Given that we don’t want to give up all speculated instruction execution, it will be difficult to find the spot where we get safety at minimal performance impact.

Right, but why should OS devs and vendors scramble to save Intel’s share price? The right outcome here is for the devs to do nothing and the world switch to another CPU vendor on the next hardware refresh cycle.

>Right, but why should OS devs and vendors scramble to save Intel’s share price?*

In what universe this is what the try to do?

That might be a second order effect of what they do (although even the idea that Intel's share price will crumble and people will go to another vendor for their next hardware refresh is bogus -- Intel is going nowhere, not to mention that AMD also has their issues with Spectre, and IIRC, part of Meltdown) but it's nowhere near the target, which obviously is to ensure that a billion PCs operating today continue to operate safe tomorrow -- and not just when people refresh their hardware (!!!).

Which CPU vendor would you switch to? Although Meltdown is (mostly) an Intel bug, all the main CPU vendors, including AMD and ARM, are affected by Spectre, which is what this particular discussion is about.

Agreed, but it almost doesn’t matter: what matters is sending the signal that shenanigans will be punished by the market.

If devs do anything it should be learning to write software that doesn’t depend on speculative execution for performance

The problem is that spectulative execution is fundamental to performance in systems that have conditionals. With memory latencies being ~100ns, and a 3GHz core, you waste 300 cycles every time you need to load the conditional from RAM, and maybe 50 cycles if you need to load it from L3 cache. Branch prediction is generally pretty accurate, so avoiding that penalty is crucial if you want to go fast.

Now, maybe it is possible to reduce use of indirect branches . C++ for example, is pretty gratuitous in that respect. But in the end, most software is going to perform a lot slower without speculative execution. I don't think that's a price worth paying.

What will happen is the CPU vendors will fix meltdown (which should never have happened) and Spectre 2. Sharing the branch history buffer between contexts is something that fundamentally can be avoided, in the same way we can tag TLB entries with the PCID on recent hardware.

In the long run, the real issue is Spectre 1. Using software to contain untrusted software in the same address space is fundamentally hard with speculative execution. After all the fuss has died down, that's where we'll see innovative software techniques. It's already starting to happen, if you look at the webkit changes (scroll down to Branchless Security Checks):


>Agreed, but it almost doesn’t matter: what matters is sending the signal that shenanigans will be punished by the market.

No, what it matters is understanding the issue. And the issue has no "shenanigans". All vendors suffer from Spectre (and some non-Intel models from Meltdown too), but more importantly, none of these was done "on purpose" which is the definition of a shenanigan.

>If devs do anything it should be learning to write software that doesn’t depend on speculative execution for performance


none of these was done "on purpose" which is the definition of a shenanigan.

The “shenanigans” are that it has become acceptable practice to cut corners on security to improve benchmarks.

Maybe speculative execution is just a thing that shouldn’t be done? People computed perfectly well before it. Itanium isn’t impacted by Meltdown so Intel is capable of it.

>The “shenanigans” are that it has become acceptable practice to cut corners on security to improve benchmarks.

Nobody "cut corners". Those were unexpected bugs that took more than a decade for someone to even discover, not some deliberate decision to sacrifice security for performance.

>Maybe speculative execution is just a thing that shouldn’t be done?

No, it should 100% be done.

>People computed perfectly well before it.

Perfectly slowly too.

>Itanium isn’t impacted by Meltdown so Intel is capable of it.

Itanium tanked.

Nobody "cut corners".

What do you call accessing memory before checking permissions then? AMD do the check!

Perfectly slowly too.

A simple predictor that assumes backwards branches will be taken (loops) and forward branches (most likely exceptions) will not is really all you need, if you insist.

You appear to be confusing Meltdown and Spectre. This is about Spectre Variant 2. You're describing Meltdown. Also, the fault is queued and not discarded. If the branch ends up being taken, you'll get a fault for accessing a forbidden part of memory. If not, the CPU "undoes" the speculatively executed part as if it never happened. The Meltdown issue is that it doesn't undo the cache effects, which nobody thought to be a big deal, and which are very hard to undo.

Also, you severely underestimate the speedups gained by a better branch predictor. There's a nice writeup at https://danluu.com/branch-prediction/ if you're interested.

Intel is discontinuing Itanium. One of the reasons it didn't go well was probably due to the way it was designed.

So, how do you propose we punish them with the market power? Abandon all products with CPUs in?

What you're advocating amounts to exposing users to possible harm, hoping that users that are harmed then act to punish Intel.

That is not just a convoluted scheme with no chance of effect, it is also morally reprehensible. You're willfully harming people for reasons almost entirely outside their control and expertise.

Imagine the coast guard letting people drown "because that's what happens if you buy a cheap boat". Except these people didn't buy cheap boats: Intel is quite obviously the default option, and usually more expensive in case where there actually are alternative.

And we haven't even gotten into the difficulty in making moral judgements about Intel's past behavior that led to these problems. Considering these bugs have existed for decades without being exposed, I would tend to think the pitchforks might just be an overreaction. There's a scale that goes intent to harm > accepting harm > accepting the possibility of harm (gross negligence) > not anticipating the risk of harm (negligence). One should calibrate one's outrage to it.

That is not just a convoluted scheme with no chance of effect, it is also morally reprehensible

Oh please. If company X sells cars with defective brakes, we don’t change the entire country’s speed limit just in case - it is 100% on the manufacturer to make it right.

Because the vulnerability exists today and may be executed today and may create crapload problems today.

The "not my problem - let someone else fix it" thinking is no different from the thinking that caused issues at Equifax.

Are you willing to wait half a decade for the fix?

> My 2 unasked for pennies - the kernel is already too bloated with support.

Is this not the ultimate fate for any monolithic kernel? Because by design you need to add everything to the kernel.

While these patches might apply to micro kernels as well. I wonder if there is kernels that abstract the cpu into a device driver to even keep microcode bloat out of your kernel?

As a side thought; If you abstract your cpu like this; will it allow you to hot-swap your os between devices and architectures?

It is. I'm very much against mono kernels, but it is what we have. Let Intel side patch like grsec.

That is one of the most mobile unfriendly sites I have ever seen

"since the peanut gallery is paying lots of attention it's probably worth explaining it a little more for their benefit."

what a ridiculous sentiment to defend a patch from Intel thats effectively disabled by default for one of the most egregious bugs in microcode since F00F.

"The new microcode from Intel and AMD adds three new features." No ones talking about AMD, David, not Linus and certainly not hackernews or other outlets. Spectre patching will happen regardless of this fiasco but This NOT SPECTRE. this is a taylor made shit sandwich thats been warned about by researchers and lurking on Intels plate for more than a decade. intel chose speed over security, and intel lost. Now they're trying to find a way --any way-- to avoid having to eat an 8 generation recall or worse.

its been said on HN before, but this is now more of an 'if not when' of intels loss of dominance in cloud. customers no longer care about the fastest chip in the west if its going to get neutered to half speed due to wreckless design that allows for a complete security disaster. Theyll buy twice as many AMD, at half the price of intel, and make up the difference in sheer volume.

David Woodhouse works for AWS, not Intel. So maybe take it down a notch?

While the op may have mistaken the author, the point is valid on numerous levels, including attempting to slander AMD because Intel has bad designs

Umm, no. This is about Spectre variant 2, which AMD is 100% exposed to.

"GPZ Variant 2 (Branch Target Injection or Spectre) is applicable to AMD processors."


Maybe I'm sensitive but "peanut gallery" felt derogatory. A quick google gave me "the top gallery in a theatre where the cheaper seats are located." which I feel is quite fitting...

As we sit here and watch how this dystopia future plays of around us and how corporations want to dodge responsibility.


It is derogatory. It generally means people who are inconsequential and ill informed, but they are going to loudly give you their opinion anyway as if you should care when you probably don't ("you" being someone whose opinion actually matters).

I can't figure out how to link to it, but this definition was my top search result:

a group of people who criticize someone, often by focusing on insignificant details

Tone it down there. This is about Spectre, which is a non-Intel chip problem too.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact