It's an SMTP server, started just two or three years ago, written by people who care about security and hope for it to be included in OpenBSD, written in... C.
(Edit: Corrected by vezzy-fnord, started in 2009.)
The only currently-extant memory-unsafe language.
I... just... argh... when we will finally be rid of this albatross? What hope is there for the world of computer security if we can't even greenfield a solution to what has already proved to be one of the most dangerous security domains there is (thanks, sendmail) in something other than C? Why are the words "buffer overflow" appearing in relation to such young code? What am I doing with my life? Why am I even trying to write secure code in a world where this is even remotely considered an option?
"The OpenBSD project produces a FREE, multi-platform 4.4BSD-based UNIX-like operating system."
You can't really divorce UNIX from C.
The only other language option for an smtpd server in OpenBSD base would have been Perl, and I bet people would complain about that just the same.
And perhaps we wouldn't be able to review it properly if it was written in Perl. Perhaps Marc Espie could. But OpenBSD has plenty of people on the team who are well trained at auditing C.
I'll be doing my part and read some smtpd code during an 8h train ride tomorrow. I already read one file and found one bug in it and told gilles about it. That's how this works. Changing the language won't magically fix anything.
> I'll be doing my part and read some smtpd code during an 8h train ride tomorrow. I already read one file and found one bug in it and told gilles about it.
Wonderful! This is exactly what I (and many others) were hoping for. Thanks so much for your willingness to jump in like that.
Did you try to tell gilles, eric, and some other committers about your concerns in private, before posting on a public mailing list and then submitting your own post to HN?
As much as he'd like to make it like we don't fix bugs, our public bug tracker is one click away and shows that out of the 25 open issues and the 493 closed issues, none mentions even remotely a pending crash that he is experiencing. He does have some feature requests and bugs though, with many handled already, so it seems that he does knows where to find it when he needs to.
As for his concerns, yes he did tell me, he sent me 16 mails yesterday, including the one with the trace. Unfortunately, he sent it past midnight, at about the same time he made it public because he just couldn't wait until morning to disclose.
It's a conversation that's been going on for a long time now over various channels but has/had mostly stagnated into a disappointing lull of complacency. The Quayls results were a big wake up to that, and my own finding yesterday was the final straw: fresh ideas and voices on how to carry the project toward maturation are quite necessary, as well as smart folks like yourself becoming interested in reviewing the source code.
This is a typical HN response: write system code in a hip, modern language - this will solve all of the problems. Which language? I haven't seen a single successful, commercial OS written in something else than C or C++. Part of this is the amount of legacy code, part of it is compiler and stdlib maturity, performance and predictability. You can probably get maturity with some of the existing "safe" languages, but rarely would you get any of the other guarantees. If you can throw HW on the problem, you can probably get away with SMTP server in, say, Python. But you typically can't do just that.
Haven't you been paying attention? You're supposed to write your security-critical code in a language that was released 5 months ago and has no major deployments yet.
I am hoping that Rust will make in-roads into the BSD world, since it interoperates well with C and is much safer by default while not imposing overhead like a managed language+runtime would. It also has very liberal licensing (GCC being GPL has always been a tough pill for some to swallow, FreeBSD at least has already switched to Clang) https://github.com/rust-lang/rust#license
The OpenBSD folks have done great work through auditing and writing safer code in C, and I agree that using a language that ameliorated at least some of the common mistakes programmers make would be a net positive.
Related - this blog from @tedunangst is worth reading, regarding the limit to which even a language like Rust can protect against specific exploits (the OpenSSL "Heartbleed" vulnerability, in this case) http://www.tedunangst.com/flak/post/heartbleed-in-rust
Note I said "BSD World" :) FreeBSD is already using LLVM (Clang) instead of GCC, and support fewer platforms - I think this would be a more pragmatic place to push something new like Rust.
If it works out there, OpenBSD could audit and import things at their leisure, at least in theory.
I have hopes for Rust, too, in this space. They're making a lot of good design decisions to balance safety & performance. Far as Heartbleed, that sort of thing can be detected using techniques from "information flow control security." Essentially, the programmer labels resources in terms of confidentiality and integrity. The type system, compiler, whatever checks them against a security policy. So, a variable containing confidential data shouldn't be accessed by functionality sending data to public sources.
Here's an example from one of the leading groups in the field:
That was turned into SIF, Swift, Fabric, and I think Civitas voting system. Swift even auto-partitions the system between client and server side while maintaining confidentiality/integrity policies. Fabric kicks butt on all kinds of issues in the hardest space of them all. So, given the above, I'm sure a problem like Heartbleed in a mere SSL protocol would be trivial to prevent at language level if software and compiler developers merely chose to adopt info-flow security at least for important projects.
Note: One can use stuff like that even if not supported in main tool set by creating a mock-up of software in supporting toolset. The abstraction gap presents issues but a problem in one might mean problem in other. Especially for what amounts to covert channel analysis of software.
What tedunangst says is true enough in the logical sense, but it makes me nervous in the human sense. Humans seem prone to the following logic failure:
1. Object X frequently removes limbs from the user.
2. Object Y does the same thing, but more safely.
3. Wait, Object Y causes hangnails if you misuse it hard enough.
4. It's OK to keep using Object X!
Like many fallacies, this doesn't survive being clearly written out, but I still see it an awful lot.
I am saying tedunangst's treatment is fair. What I don't like is what people in general tend to do with the argument. I am not claiming tedunangst is doing that with the argument.
I agree. Further, these problems have been known since the 1960's with solutions available then. Should've been adopted once the cheap hardware caught up in performance. The thing that surprises me is how many people think C's design decisions were well-thought or something vs competition. No, they just stripped every good protection or feature from ALGOL, etc so it would fit on the PDP-11. Did same for UNIX. That made sense given what they had on hand but keeping it all the way into 2015? See main article's gripes for entirely-predictable result.
Gave details here on C and UNIX history with counterexamples from other systems:
Abandoning C would mean abandoing one of the very cornerstones of UNIX, or even the heart and soul of UNIX. While I won't be shedding many tears over that, I do expect OpenBSD to remain one of the last fortresses of C instead of being in the forefront of abandoning it. It it their way of doing things and they have showed little interest for appeasing the masses by changing themselves.
So while I don't like C (or UNIX for that matter) I still do celebrate the fact that these people are actually doing something and even more importantly they are doing what they want despite the naysayers and doing it well.
Starting in 2009 mitigates my criticism by, oh, let's say, 40%. Which I mean as an acknowledgment of your correction, even as I'm not generally backing down from the core point.
I want to respect OpenBSD. To a great extent, I do. And there are reasons to prefer tested C codebases to untested code bases in new languages, sure. But to the extent I can't give my full, unadulterated respect to OpenBSD, it is because I don't see that they consider everything being in C a problem to be solved.
I mean, I get engineering considerations and that we can't rewrite everything at the drop of a hat and performance matters, etc. But it is my opinion that any engineer that cares about security really ought to see "written in C" as starting out with a huge entry on the debit side, and take appropriate actions. And every year that goes by that's more true.
I'm conflicted on the issue. As a general principle, writing security-conscious code in C should be avoided.
On the other hand, the issue is a tad more subtle. There seems to be an almost Pavlovian revilement and panic against C code, and this usually comes from people who haven't dealt with the language in a long time or at all in any depth.
For one thing, the blanket assertion that secure C code is a Herculean task, doesn't quite pan out. djb showed one way, based on a few general principles espoused in qmail, that has since successfully been reapplied by people like Wayne Marshall, Laurent Bercot, Bruce Guenter, Paul Jarc, JdeBP and so on. Much of it comes from abandoning the C standard library in favor of more robust alternatives like libdjb variants, clearly architecting process/module boundaries, using simple formats like netstrings and text0 that need not be "parsed" in the conventional sense, etc.
Further along the line, we have the SEI CERT C Coding Standard that comprehensively identifies many pitfalls. [1]
Deeper still, cryptography is one area where abstraction can be an enemy. You actually have to deeply worry about time complexity, branch prediction, precise timing, compiler optimizations and so on [2]. A tool like C becomes desirable here.
Lastly, but crucially - you're on Unix, for better or for worse. POSIX, SUS, all of those standards? They're deeply written with the conventions and API/ABI semantics of C in mind. Such is life. When you primarily write code at that level, the easiest way not to have any impedance mismatch is to just write it in C.
I don't consider my revulsion Pavlovian. Even if I have had that particular connection repeatedly made, the point is that if it requires super-mega-experts to code in a totally specialized manner, using libraries other than the easiest to use, using levels of care attainable only by the top 1% of programmers, it's not a suitable language for the vast bulk of people to pick up. And even the top 1% of programmers would probably be well suited to do something else with their cognitive firepower. The fact that somebody once proved capable of walking on a tightrope between World Trade towers does not prove tightrope walking above vast chasms to be a safe mode of transport.
For crypto, at this point, C is now too abstracted from the hardware to be useful. You need assembler, and that's getting dangerously abstract too, since it's actually an abstract language over the microcode that actually execute, the entire point of which is generally to provide lovely, wonderful things... that trash timing guarantees that crypto would like.
"I had to write code that allows remote attackers to own the machine because it was the only way to avoid impedence mismatch."
Incidentally, since it hasn't come up, note that I consider "C + useful static analysis tools" to be a different language. "C + useful static analysis tools" can be a sensible choice. I still at least have questions about the cognitive load being loaded on to the developer vs. a language that removes the problems entirely, but it must be acknowledged as having radically different cost/benefit profiles. However, I observe that very few people seem to be using useful static analysis tools beyond -Wall.
Yeah, there are other memory-unsafe languages. His overall point remains. Matter of fact, thanks to you, I can now drop bombs on concurrency angle thanks to Concurrent Pascal, Solo, etc. Relevant given all the C & BSD apps juggling threads. To think mainstream is just now solving these problems in past 3-5 years despite proven solutions available... (rolls eyes)
I don't see what concurrency has to do with any of this.
Concurrent Pascal was an interesting project for its time, heavily dated by now. The greatest victory for concurrency was having architectures with thread-level parallelism instead of the dominant instruction-level parallelism, and one specifically optimized for CSP in the occam language. This was, of course, the transputer. So much for that.
Erlang/OTP leads in the modern era.
But I don't see any relation to our current topic of discussion.
Server apps, including mail, implement forms of concurrency to handle incoming requests. Smart ones use either multi-threading or event model for performance. OpenBSD and others have all kinds of such apps they built on C. There have been numerous concurrency-related errors in major server apps due to this. So, just like with C itself, ignoring the safe, concurrency approaches from languages like Concurrent Pascal (or SCOOP/Eiffel today) is a failure to learn from the past with a direct, proven cost.
Think all the race conditions, deadlocks, etc that have hit server apps would've happened on Solo, Ada tasking, or SCOOP? Probably fewer to none. So, it's still relevant and Ada/Eiffel are actually building on it. Especially SCOOP R&D community.
Yeah, specifically unsafe handling of it. Forkexec seems to be a heavyweight (and less portable) approach that even DJB defended in his paper. My gripe should apply to any using threads.
"Event loops seem to have been the norm for C projects for a while."
I'll take you word for it as I've seen that in the few, top-tier ones I scrutinized. There's a similar trail of research dealing with that alternate model. Have a feeling the meme might apply there, too, but won't accuse them until I see for sure. ;)
To a first approximation, every single one of those wouldn't happen in a memory-safe language.
Of course, plenty of other things will. But it's not like choosing memory-unsafety makes those other things less like (memory unsafety does not, for instance, prevent incorrect ACL specification), so that's not very compelling.
Think of it this way. Suppose C didn't exist, and somebody proposed a new language today that was memory unsafe. Would it have any success, or be laughed out of existence?
In fact, go ahead and suppose C did exist, because it would still not be taken seriously.
If it's so wonderful that C is memory-unsafe, where's all the new languages that are also memory-unsafe?
(Note having specially-called out "unsafe" operators does not make a language "unsafe" the way C is.)
A few big projects written in C proves nothing about them or C. Need the larger picture with many instances to prove a claim. Certain, preventable vulnerabilities have occurred mostly in C software. Another commenter gave an example. Many of these similarly occurred in the C-lang projects you reference. So, using C instead of a language that prevents them was the first reason for their existence.
It doesn't, it's a fallacy. A lot of code that we think of today as buggy was written in the era in which people did not understanding threats and consequences of certain decisions. The reason we have nice things today is that we were lucky enough to have languages like C to experiment on and with and learn. "Safe" languages of today have the benefit of hindsight that good old C didn't have. Comparing legacy code in C to modern code in whatever you deem worthy is not only unhelpful but also very deceptive.
Compare new to new and tell me: how many web, smtp, imap, dns servers in Python, C# or Rust do we run today? There were some attempts but very few successful ones, despite of all the goodness that comes with better languages, libs and tools. I bet there's a reason behind it, other than just "people didn't try hard enough". There's old stuff out there being used today that we should definitely phase out. But we'll replace system software with new one written in system languages. Most likely C, perhaps Rust. But definitely now with ones written in "safe" languages that should deliver (if they are so awesome) but somehow didn't.
"that we were lucky enough to have languages like C to experiment on and with and learn. "Safe" languages of today have the benefit of hindsight that good old C didn't have. "
I've already detailed some of what was known about avoiding reliability and security problems before C and UNIX in the link below.
I know at least MULTIC's tricks were known at time of UNIX because it's name is a pun on MULTICS. Anyway, the better parts were largely eliminated to make C and UNIX efficient enough to run on a PDP-11. So, it's a fact that good stuff was known, they stripped it out due to constraints, and they never fixed those decisions as constraints lightened. That language, toolset, and coding style were adopted by many developers that continued to do things that way.
Meanwhile, in parallel, other trends began running. The evidence by the 1970's suggested that building secure systems require specs of behavior, a security policy, analysis one possesses the other, careful choice of language, pentesting, minimal TCB's, interface protection, and so on. Starting with MULTICS security evaluation, which talked about "insufficient argument validation" (ex: buffer overflows), that crowd demonstrated in one project and paper after another how to build secure systems. Plus, where most bang for buck was in terms of efforts.
In parallel, people like Margaret Hamilton, Dijkstra, Wirth, Hansen, etc determined how to make software that was correct by construction. Hamilton did that in 60's on Apollo. Dijkstra's method of layering of simple components with careful interfaces made nearly flawless software. Wirth & Jurg did a whole PC, safe (but efficient) language, compiler and OS in 2 years that performed fine. Kept building on that. Hansen and Dijkstra built things like Concurrent Pascal that eliminated concurrency errors on top of Wirth-style work that reduced sequential errors. One group after another built on such approaches with better results than UNIX and C crowds in terms of functionality vs labor available vs defects found.
So, this stuff was well-known at repeated points in the implementation of UNIX, server software, compilers, etc. Better approaches are finally mainstreaming with likes of Go (Wirth-inspired) and Rust (Ada redoux?). Other safe languages were plenty mature before they showed up. Yet, many projects focused on security still use C. So what I'm talking about not only happened then: it's still happening now.
"Compare new to new and tell me: how many web, smtp, imap, dns servers in Python, C# or Rust do we run today?"
Let me just apply your words to another problem. How much enterprise transaction processing uses C and UNIX today vs IBM and COBOL? There were some attempts to switch but very few successful ones despite all the goodness that comes with better languages, libs, and tools. I bet there's a reason behind it. Yep, but it's not that IBM and COBOL were best. I'd venture people not willing to put that work in, fear of breaking things, inertia, not knowing about better approaches, etc. The usual. Currently happening to C, C++, and Java stacks. :)
"But definitely now with ones written in "safe" languages that should deliver (if they are so awesome) but somehow didn't."
They did. Wirth made a whole series of them that delivered in many ways. Borland Delphi, Oberon, Component Pascal, and Modula-3 saw use in industry. GEMSOS used Pascal w/ coding guidelines to hit high security in the 1980's. Ada improved defect count in every empirical study I saw. A tasking profile for Ada and Eiffel both knocked out plenty concurrency errors. MULTICS used PL/0 to avoid issues that plagued C. Academics working on large projects like compilers and sometimes OS's had positive experiences with the LISP's, ML's, and Haskell. And so on.
They proved their worth every time they were used. People's gripes could've been fixed with less effort than what's been put into making C code maintainable, reliable, secure, etc. Instead, they kept trying to build modern stuff on a language & OS stripped of its best features to run on a PDP-11. And stayed on that bandwagon to the end.
I ran my own mail server using OpenSMTPD and Dovecot for a little over a year, and found OpenSMTPD to be very easy to set up and configure. I needed something light, that I could easily tweak to fit my needs. I mostly got things working reading the man files, and asking for help in the IRC channel (thanks Gilles). I also looked into Postfix, and I'm sure it's great, but the amount of configuration options were a bit daunting.
I think the project deserves more contributors and more eyes to look at the code. A fantastic foundation has been laid, but we can't expect Gilles to be able to do it all alone. This is a diamond in the making!
"we can't expect Gilles to be able to do it all alone"
Just like with GPG. In FOSS circles, there's all kinds of people that want to take the code but few to create it. So, people gripe about the state it's in while putting no effort in to improve it. At least the author contributes significantly. However, gotta wonder how many users it has vs how many are contributing code/audit.
Bet the difference has an impact on the situation. It's why I spend time here and there investigating proprietary, open-source models that require money or contributions to use the software. Doesn't even have be a lot: at least enough for a significant product to support at least one developer. That way the users are always investing into it and esp bug fixes happen.
gilles operates in a community where problems like this are taken very seriously, and where enough developer resources exist to sweep the smtp code base several times over.
I expect a lot of fixes coming up over the next few weeks.
The real problem is that the OpenBSD community didn't realise the state of things internally, because gilles and eric were trusted to not fuck up. This happens, since not everybody has time to follow every corner of the tree all the time. But when focus on a particular area is needed we can do it.
It's good that Jason pointed out these problems, however the manner in which this was done creates huge public exposure which puts pressure on all of us (especially gilles and eric) to produce results very quickly while we prefer moving slowly and being careful because that produces better results in the long term.
To be clear, I wasn't smearing any developers or the project relative to others. Just noting a common problem in OSS development that probably hurts their project to some degree.
Appreciate you adding detail on the project's situation. Also on how you all view the post and its effects. As HN gets exposure, do you a specific, alternative approach for anyone reading that might end up Jason's shoes to get the OpenBSD team focused on a particular project and digging into it? I'm guessing he didn't make an attempt outside the public disclosure or made an ineffective one.
Find out who's been involved with the area of the code base recently, and mail all their @openbsd.org addresses directly in one mail.
If they don't reply after a few attempts, or give unsatisfactory answers, try mailing Theo. He acts as arbitrator in such situations.
If even that doesn't help, your last option is to write good fixes yourself, and keep submitting them as diffs to tech@ until you're either banned from posting to the list (at which point you should just give up) or your diffs end up being commited.
That's crystal clear: now anyone reading knows the best approach. Makes sense, too. Except that last part.
I agree it's the best option to rollup the sleeves and just fix the problem you're griping about. However, a write-up like Jason's is a valid option, too, if person doesn't have time/resources/skill for that option. Because, at that point, known issues would've been ignored by everyone up to Theo despite it being in default install and under OpenBSD's quality image. A write-up drawing attention to the issue would be justified to prevent users from placing unjustified trust in that component. Being informed lets them take action either on improving the component themselves, limiting damage it could do, or just avoiding it altogether.
Still agree that the gripe should go to the developers and Theo first.
But Jason did send diffs, and we are discussing his situation as an example.
Anyway, like it or not, in the OpenBSD community, working diffs, especially ones that fix a security issue, are the best way of winning an argument. You can try to blog about it but you probably won't be taken as seriously. So if you really care to convince the community from the outside and everything else has failed, and you can't code C youself, find a friend who can and ask for help.
I tried OpenSMTPD / Dovecot for an environment with about 120 e-mail accounts (even promised to write it up afterwards). It just didn't quite work out. Filtering was a pain and it crashed a couple of times. I switched back to Postfix since I know its configuration pretty well.
On the other hand, as shipped, I have had no problems with it on non-mail server. I will probably try again next time I need to revise my mail server.
I like the concept and some attributes of it. Some good ideas might come out of those projects. The thing I can't overemphasize here, though, is that this kind of application might be attacked or just fail in many ways. So, whatever people do, they need to keep the main functionality simple for human and/or machine analysis. Python helps with readable logic but obscures how it's actually implemented (attacker's focus). A similar project in Go might be a better idea as it's easier to see what assembler comes out of it and Go is often the replacement for Python in server apps.
Regardless, the main worry areas are parsing, protocol handling, formats, and limiting effects of any given external action on system's state. Making sure those are correct should knock out most issues. Not that it will be simple or easy. ;)
What worries me is the lack of updates for OpenBSD. Is those issues he's talking about are for real? I run "pkg_add -u" frequently and I don't see any updates. I'm running "stable 5.7". I understand that OpenBSD doesn't provide updates or bugfixes after release, but security updates should be pushed.
I'm casual OpenBSD user and may be I don't understand something.
"The packages and ports collection does NOT go through the same thorough security audit that is performed on the OpenBSD base system. Although we strive to keep the quality of the packages collection high, we just do not have enough human resources to ensure the same level of robustness and security. Of course security updates for various applications are committed to the ports tree as soon as possible, and corresponding package security updates are made available as snapshots for -current."
Exactly. You're safe so long as you don't run 3rd party apps. Past that, let's hope the exploit mitigation technology works. Least they have that tech, though.
Yes, but that implicit (I hope) with security upgrades. I do the compiles myself. It is pretty easy to build a set for distribution after compiling once.
Do you have (or know of) a good set of practices to follow on this?
I've tried this in the past but some ports (typically big ones like web browsers) will run out of some resource or other during the build.
Also it seems that if you build from ports all the dependencies are built from ports too, even if they're already installed as packages? It ends up taking hours in some cases.
Assuming you're tracking -current sources, it's usually possible to just run the published binary packages from mirrors on top of your own base builds.
If library bumps happened, the packages might take a while to catch up, though.
What I do is I build a release with patches I'm interested in (my own or other people's). After upgrading my machines to that release I try `pkg_add -u`. If that complains about libraries, I just wait a day or two for a new set of packages to come out and try again.
Sometimes I cheat and compile the missing library in a slightly backdated src checkout, or fetch it from a system that happens to have a copy of the right .so file.
If all that is too complicated, or if you want -stable instead of -current, try mtier.
> I'm casual OpenBSD user and may be I don't understand something.
Yikes.
"pkg_add -u" is just for ports/packages. Security updates to OpenBSD itself are distributed via patch and are listed here for your version here: http://www.openbsd.org/errata57.html. Explanation on how to apply them is found in the patch file itself.
It's a lot simpler to setup and configure than Postfix or qmail. It promises all the nice privilege separation of those daemons too. The appeal for me initially was that it would be small, simple, and very secure.
And "a lot simpler" meaning A LOT simpler. It is the first mail server that I as a non-professional sysadmin could understand and configure correctly the first time using just the man page.
Both postfix and qmail have quite thorough privilege separation.
Most people run Postfix out of the box (apt-get install, answer which domains you want to route mail for, done) and it's completely solid. Wietse Venema had a name for himself in security circles long before he started writing it. DJB too. Neither is likely to be the vulnerable link in your system.
The qmail code and configuration are both quite unlike what many people are used to. It definitely wasn't being imported with the config/service model it uses, and reworking it to be like every other daemon would involve a lot of code changes.
Do you think they could apply any of that to current project? Particularly in sections 3-5. I defer to you here as your experience is directly relevant. Would any of his approach, on top of looking for top 10 C issues, have improved the security of this app? Or was none of that applicable?
I think the approach is quite similar, but the devil's in the details. From looking at a couple bugs, the efforts to reduce TCB were not as effective as hoped.
I suppose that could be done, but email itself is a very complicated thing to deal with with a lot of moving parts. OpenSMTP was supposed to have a much simpler scope of operation.
Think of the configuration method as using duct tape to permanently set the switches/levers of an f1 race car instead of just getting an automatic Toyota car that is simpler for almost all people to drive.
A lot of the security and simplicity is supposed to come from not having so many options/knobs/levers of operation.
Good points. I'd say in that case I'd try to draw on the code or architecture of qmail that was beneficial while ignoring anything unnecessary. Better than trying to get the hard stuff right twice.
I think that qmail's design is based off of being super modular. So much that most people don't actually run the qmail-smtpd because it had some security issues (It's been a while but this was around the time they started work on the OpenSMTPd).
I assume 5 years later it's much improved but at the time forking the code of an SMTPd that had security vulnerabilities didn't seem like a worthwile investment and everything else has a OpenBSD prohibitive license.
Edit: It was all pretty thoroughly discussed on various mailing lists and whatnot if you're curious
Far as their decision, that makes sense. I still think they should've leveraged proven design principles or tactics in the other software. I've just seen enough of these situations where I believe what you say about how their evaluation and decision process went. I figured licensing would factor into it, too.
"it threatens the OpenBSD.org frontpage security claim"
This misinformation is my only problem with the article. That claim has probably never been true. If it was, they wouldn't be regularly fixing shit that might be exploitable. Still, I came up with a way to test it years ago on Schneier's blog for anyone that wanted to try:
"Watch mailing list for bug announcements. Quickly determine if it's exploitable. If so, turn it into a payload. Hit target system. Profit." So, it's special in terms of quality focus but still can be hacked if anyone bothered to try. (Few bother.) Especially at layers below kernel that they don't care about, privilege escalation because they refuse M.A.C., in a VM because Theo doesn't worry about those issues, and covert channels because they probably never heard of them.
Note: It's reputation adds more risk as it's often used for "fire-and-forget" appliances. This can be reinterpreted as "they don't patch or they patch very rarely." The more they trust it, the more likely the bad guys get in. ;)"
Real quick, as I'm clearly not an OpenBSD user. Does the "default install" include the apps a deployment will almost certainly use? That is to make the claim meaningful. And have there been any software defects in OpenBSD or its critical apps over past "8 or so years" that could've been exploited by attackers?
That would be a start. My guess was that OpenBSD team didn't track (and update web site claim for) the number of defects that might have been exploitable: just fixed all defects as they found them instead. If so, that makes for great QA practice but inaccurate claim on web site. Otherwise, there's a tracker somewhere listing all bugs found in privileged software, their exploitability, and time it took to fix them. That tracker will have a huge list of issues with only 2 marked as security issues.
So, the claim is meaningless for most users unless they restrict themselves to the tiniest subset of software. That answers the first question.
Ok, so the second is, "Have there been defects found in any parts of the default install after those parts were distributed? And were they just fixed or analyzed for exploitability with an update on the website if necessary?"
The claim is quite old, and comes from an era where a fresh install of an operating system would have web servers and DNS servers and file servers globally exposed. Many projects have adjusted since then, but there was definitely an era of Apache worms that could spread juat by pwning the default "it's working!" Web site that many operators didn't even know they were running.
Best explanation I've heard. Still supports my concern more than the claim's truth. Nonetheless, I can totally understand that given I was around back then and remember what that was like. It would certainly have been a differentiator back then before sane defaults were a thing.
It's generally a security by obscurity claim. Which is strange coming from the OpenBSD guys. I don't think you can even log in with their own OpenSSHd from a default install.
That's what I'm thinking and suggesting. Because, if we get down to it, anyone reading the claim would have to believe something along these lines:
1. There's never been a defect that could affect security in any OpenBSD software, from kernel to protocols, after it was released.
2. For meaningfulness, the default install supports real-world usage (and usefulness) of the system.
The good news for No 1 means that they never had to issue a patch in a protocol or software due to defects with potential to leak data, allow hijacking, or experience DOS. Knowing that, once released, every OpenBSD system was unhackable and needs no patches past functionality fixes would greatly reduce administrator overhead. Truly fire-and-forget system. If the website claim is true, that is. :)
Further, if No 2 is true, users won't have to use anything but the base install to handle day-to-day tasks on their network. Anything critical is included, usable, and secure-by-default. Users would only need to rely on 3rd party stuff for non-obvious use cases or stuff that truly doesn't belong in a baseline. So, the ports collection wouldn't even be necessary in most deployments (esp infrastructure, console desktop) as the functionality and what counts in security claims were batteries included like the competition.
I'll leave it to users and people following their bug trackers to determine if either is the case. Especially No 1 which is my main point. The mere existence of security patches would kill the web site claim. It would instead become, "Of all the defects found, only two were analyzed and shown to be exploitable over X years." Makes for a totally different impression, eh? ;)
I think most of your posts are very informative. But your lack of knowledge about OpenBSD surprises me. Perhaps it would save you and the reader some time if you spent a bit more reading about OpenBSD instead of reasoning about all theoretical directions OpenBSD could have taken.
Appreciate it. Far as lack of knowledge, a brain injury in an accident cost me most of my hands-on IT skills and many details in INFOSEC. Among other things... However, the stuff I did repeatedly, like high-level security engineering and assessments, stays with me. Barely relevant here even though it makes plenty of sense that you'd think that. The short-cut around it is the concept of burden of proof in INFOSEC: one is expected to back up their big, security claims with evidence rather than everyone else expected to disprove them.
I certainly could dig back into OpenBSD (and C) to fully understand its current situation, its code, its mitigations, every source change from vulnerability 2, and so on. Those details could benefit OpenBSD in form of contributions I attempt or the reader following along to get a better picture of insides. However, it's not necessary for evaluating my concern about the 2 vulnerability claim. We can cleverly cheat around all that knowledge by going straight to fundamentals and claim itself. Then, only acquiring (me) and/or applying (others) that knowledge if the specialist skill turns out to be required. Others and I did this with success when drowning in product and project security claims with finite time to review them. Let's apply that.
"instead of reasoning about all theoretical directions OpenBSD could have taken."
My main question is a simple, concrete one that I shouldn't have to investigate at all: what evidence is there that there have only been 2 exploitable flaws in all OpenBSD releases? To claim this, there would have to be a tracker of every defect fixed in OpenBSD since its inception, an analysis of each one's exploitability, and a result saying only two were exploitable. There's trackers like that for many projects. Where's that record for OpenBSD or, to save you time, do they even analyze every bug for exploitability?
If that doesn't exist or that analysis wasn't done, then the only 2 vulnerability claim can be rejected immediately because it wasn't proven. Instead, the claim becomes "Of all X defects found, people only showed two were exploitable remotely. (Optionally) That's a defect ratio of X per Kloc with vulnerability ratio of Y per Kloc. (Not optional) The rest either weren't assessed for severity or were shown harmless." Alternatively, the claim is totally false if even one security fix for a remote attack was done after the 2nd vulnerability was fixed. It would have to be changed to a maybe pending further, specialist analysis. This isn't theoretical: it's a direct implication of the claim they're making, an extraordinary implication, and one which deserves concrete evidence to back it.
So, I encourage people to reject it until they have that evidence and in a way they can share easily. If I have time or resources, I might attempt a thorough assessment of OpenBSD in terms of design or code. A past study I did comparing it to prior secure UNIX attempts showed some things it's missing where there's surely issues. It has some other attributes, esp code quality & protections, that help it more than others (including old ones). Nonetheless, the issue I have is an extraordinary claim that pops up repeatedly in discussions over OpenBSD vs whatever with no evidence to back it and high chance of failure. See the problem? Also, do you see how I don't need to be an OpenBSD developer to spot it?
> The short-cut around it is the concept of burden of proof in INFOSEC: one is expected to back up their big, security claims with evidence rather than everyone else expected to disprove them.
Totally agree with that.
> do they even analyze every bug for exploitability?
"Our security auditing team typically has between six and twelve members who continue to search for and fix new security holes. We have been auditing since the summer of 1996. The process we follow to increase security is simply a comprehensive file-by-file analysis of every critical software component. We are not so much looking for security holes, as we are looking for basic software bugs, and if years later someone discovers the problem used to be a security issue, and we fixed it because it was just a bug, well, all the better. Flaws have been found in just about every area of the system. Entire new classes of security problems have been found during our audit, and often source code which had been audited earlier needs re-auditing with these new flaws in mind. Code often gets audited multiple times, and by multiple people with different auditing skills."
and
"Another facet of our security auditing process is its proactiveness. In most cases we have found that the determination of exploitability is not an issue. During our ongoing auditing process we find many bugs, and endeavor to fix them even though exploitability is not proven. We fix the bug, and we move on to find other bugs to fix. We have fixed many simple and obvious careless programming errors in code and only months later discovered that the problems were in fact exploitable. (Or, more likely someone on BUGTRAQ would report that other operating systems were vulnerable to a `newly discovered problem', and then it would be discovered that OpenBSD had been fixed in a previous release). In other cases we have been saved from full exploitability of complex step-by-step attacks because we had fixed one of the intermediate steps."
> This isn't theoretical: it's a direct implication of the claim they're making, an extraordinary implication, and one which deserves concrete evidence to back it.
Fair enough. Maybe they should change the claim to something like "Only two remote holes (known at the time of the fix) in the default install, in a heck of a long time!" Then again, isn't this what most readers would expect?
Appreciate it. That covers it. I'll keep it archived. It proves the claim is false.
" Maybe they should change the claim to something like "Only two remote holes (known at the time of the fix) in the default install, in a heck of a long time!" Then again, isn't this what most readers would expect?"
As your quote says, they actually discovered and fixed a ton of them. Users thinking that claim is true might believe their system doesn't need patching if no new features are added. So, I'd ditch the claim altogether if I were them in favor of something reflecting what you quoted. Those quotes show the real argument which is lots of prevention and constant improvement at a higher rate than competing OS's for code quality. That mindset dictates one continue to patch and upgrade: no "fire and forget."
My reading is that, if I followed the default install instructions (that is included on the CD Liner), that OpenBSD has not been remotely exploitable in the last 16+ versions.
By default OpenBSD has not opened up and ports, daemons, sockets, or other resources that would allow a remote attacker to exploit it.
If I wish to start making changes to the default install, that is, modifying /etc/rc.local to start enabling daemons/services, then I need to take responsibility for those changes, and ensure that I stay current with the security issues associated with those daemons/services.
That's how I read it, too. That's why I'm wondering if it has had a security-related patch or change since the 2nd flaw was found. If it had, then the claim should be modified. If it hasn't, claim might be true.
Most making a claim like that... one that's usually wrong.... have data to back it up. Most people's position has been that I should accept it by default, accept reversed burden of proof, and disprove the claim by countless hours of inspection. Whereas, from high-security to mainstream stuff, the status quo has been something is insecure until proven otherwise. I see no reason to change it.
That conclusion doesn't follow from the links unless they attempted to produce exploits from each potentially, exploitable defect to confirm or reject whether it was a vulnerability. Like what happens with disclosures in many projects. The truth is that they've fixed many problems in the kernel in the last 8 years which may or may not have been exploitable. They're only sure about two of them with a policy to not check the rest as their time is better spent fixing everything and moving on. I agree with that approach but it makes their 2 vulnerability claim impossible it wasn't tested on defect data.
So, I'm dismissing their claim entirely and will counter it with their claims linked above along with my argument. However, even without that claim, the OpenBSD team can still make a great argument based on their QA processes and track record in fixing things as you linked. It will certainly have fewer 0-days than anything else which is why I told another OpenBSD developer I recommended Xen projects use it for Dom0 wherever possible.
Btw, thanks for the links. It answers the question about their defect tracking in general. Lots of detail in there and evidence of their QA working. As I expected. I'm keeping it to share if the topic comes up in another discussion.
(Edit: Corrected by vezzy-fnord, started in 2009.)
The only currently-extant memory-unsafe language.
I... just... argh... when we will finally be rid of this albatross? What hope is there for the world of computer security if we can't even greenfield a solution to what has already proved to be one of the most dangerous security domains there is (thanks, sendmail) in something other than C? Why are the words "buffer overflow" appearing in relation to such young code? What am I doing with my life? Why am I even trying to write secure code in a world where this is even remotely considered an option?