
Was the iOS SSL Flaw Deliberate? - sp8
https://www.schneier.com/blog/archives/2014/02/was_the_ios_ssl.html
======
tptacek
Another sad indicator of the level Schneier is playing at today, in the same
vein as "avoid elliptic curves, we don't trust the math".

Once again: the only reason this bug got so much attention and press is that
it's easy for laypeople to get their heads around. All you have to understand
is how "goto" works. The bug is vivid, and so (paradoxically) seems scarier.

Significantly worse bugs are found every week. Within a few days of the
announcement of this TLS bug, a Flash bug was announced, after being detected
in exploits in the wild, that enabled reliable drive-by hijackings of browsers
--- multiple browsers. It was off the HN front page within an hour.

TLS bugs aren't even unusual. We get a new one every few years ago. Firefox
managed a PKCS1v15 parsing bug that allowed anyone with a Python script and 30
milliseconds to generate a certificate for any domain. Other browsers have
screwed up certificate chaining, so that any domain could sign any other
domain. But nobody understands PKCS1v15 padding, nobody understands
certificate chaining, and so nobody writes stories about these bugs. But their
impact is identical to this one.

~~~
josu
>Once again: the only reason this bug got so much attention and press is that
it's easy for laypeople to get their heads around. All you have to understand
is how "goto" works.

I don't agree. Most people don't even have a basic understanding of coding. I
think that it has more to do with the fact that there are about 500 million
vulnerable devices out there, and that we are talking about Apple. Remember
Antennagate, it wasn't even that big of a deal and look at what the press made
of it.

~~~
aidenn0
Most people on HN know what a "goto" is. We're talking about attention on the
HN front-page after all.

[edit]

Also I bet most readers of Schneier's bug know what goto is as well.

~~~
josu
He said laypeople though, not HN readers.

------
nicholassmith
Often when things like this happen there's a large conspiracy at play in some
peoples minds, "Apple _deliberately_ left a security vulnerability". But it
falls apart pretty quickly, it's in an open sourced package, so the assumption
is _someone_ is eventually going to see it, so it's not going to remain a
secret and thus is useless as a stealth backdoor.

The likelihood is pretty simple, someone fucked up. On a potentially _huge_
level, but a fuck up none the less. These things do unfortunately happen, and
no doubt it'll prompt an internal review of their change management process,
and their build chain, and what they can do to isolate issues like this in the
future.

~~~
corresation
While I believe that it was just a screw-up that wasn't caught because of gaps
of process (the code essentially repeats blocks with minor changes, so I can
envision someone doing some refactoring -- removing the passing of the context
-- and trying to save time by copying the first changed block to the second,
failing to overwrite that single additional line), considering the possibility
that it was intentional in no universe suggests that it was Apple the
organization that chose and instituted the vulnerability.

A single employee receiving a second paycheck from a three-letter organization
could have been responsible. A vulnerability could have been used to plant it.
And so on. Remember when everyone was railing about Google giving NSA a
backdoor (cue a thousand "don't be evil" cries) when in reality the NSA and
friends were simply exploiting a weakness of Google's network. The net effect
was the same.

It could be intentional and still entirely unintentional as far as Apple the
company is concerned.

And to the open source thing -- it sat there for 18 months. Indeed, it was
caught [EDIT: Actually I don't know how it was caught. I faintly recall
someone posting an issue someone submitted to Apple detailing some weird
behavior with invalid private keys, but can't find it now]

~~~
ybaumes
"And to the open source thing -- it sat there for 18 months." !!! Just .. wow.
If it was the first time we heard such stories. But it happends many time with
the linux kernel source code and all. I definitively conclude that source code
begin open to everyone is not a mark of higher quality.

------
gmac
As noted in the article, _plausible deniability_ is a key criterion for
thinking of this as a possible NSA insertion.

In other words, it's only possible that it's an NSA job if it's also possible
that it isn't. Something tells me therefore that we're unlikely ever to know
for sure.

~~~
maxerickson
Doesn't it strain credulity that the NSA would (bother to) insert a flaw that
would be detected by many commercially available analysis tools (or the right
compiler flags)?

(I certainly shouldn't pretend to know much about C, but the above matches my
understanding of the situation)

~~~
smackfu
Who's to say the NSA doesn't know what analysis tools Apple runs as part of
their builds?

------
mikeash
If I were at the NSA and wanted to introduce a bug like this, I'd get access
to Apple's build servers (either through an exploit or by just talking to an
employee who has access) and arrange for a binary patch to be applied to the
generated object files at build time. It would be basically undetectable, as
no amount of source code auditing would reveal it. Could probably make it look
like a compiler bug without too much difficulty.

Of course, this presumes that the NSA _needs_ to introduce bugs like this. I
imagine they do just fine for now merely taking advantage of naturally
occurring bugs.

~~~
mzs
Again I think this was just a mistake, but there are other groups that could
want to add something like this other than the NSA where it would be more
plausible for them to have such a need. It shows that Apple needs to
reconsider its processes though, and I guess they are now, which should make
the likelihood of something like gotofail for nefarious means working very
small from now on.

~~~
mikeash
I'm hoping that their security folks are now going through and writing
extensive automated tests for all of this code, and anything similar, and
turning on lots of automated static analysis to go along with it.

I fear that it may end up as, "we already found the bug, we don't need tests
now". But that's probably pessimistic of me.

------
mukundmr
Well,
[http://www.opensource.apple.com/release/os-x-109/](http://www.opensource.apple.com/release/os-x-109/)
has the list of open sourced code from Mavericks. Let the code analysis begin,
it would help everyone out.

~~~
hengheng
Honest question, what are decent tools to find bugs like this in C code
nowadays? I was saddened to read that gcc cannot check for unreachable code.

~~~
anon1385
The -Wunreachable-code flag does work in clang, to some extent at least (I
have a rather old version here so YMMV). Other tools that may help:

The clang static analyzer: [http://clang-analyzer.llvm.org](http://clang-
analyzer.llvm.org)

Valgrind: [http://www.valgrind.org](http://www.valgrind.org)

Address Sanitizer:
[http://clang.llvm.org/docs/AddressSanitizer.html](http://clang.llvm.org/docs/AddressSanitizer.html)

The various -fsanitize options in clang (undefined behaviours, integer
overflows etc):
[http://clang.llvm.org/docs/UsersManual.html#id28](http://clang.llvm.org/docs/UsersManual.html#id28)

~~~
mzs
There is a list here:

[http://en.wikipedia.org/wiki/List_of_tools_for_static_code_a...](http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis)

Also there is oclint not listed there, which is pretty amazing, but it has a
strange notation for ignoring wqrnings. splint is pretty good too though and
still supports the legacy lint stuff like /* NOTREACHED _/

BSD lint seems to work still (though it looks like the lintlib is no longer
built on FreeBSD and stdlib is not lint clean!) but even it would have caught
this gotofail:

    
    
      01: #include <stdlib.h>
      02: 
      03: int
      04: main(int argc, char *argv[])
      05: {
      06:   if (argc == 0)
      07:           goto fail;
      08:           goto fail;
      09: 
      10:   exit(0);
      11: 
      12: 
      13: fail:
      14:   exit(1);
      15: }
      $ lint a.c 
      a.c:
      stdlib.h(286): warning: ANSI C does not support 'long  long' [265]
      stdlib.h(287): warning: ANSI C does not support 'long long' [265]
      stdlib.h(287): warning: ANSI C does not support 'long long' [265]
      a.c(10): warning: statement not reached [193]
      a.c(15): warning: function main falls off bottom without returning value [217]
      a.c(4): warning: argument argv unused in function main [231]
      _types.h(61): warning: struct __timer never defined [233]
      _types.h(62): warning: struct __mq never defined [233]
      lint: cannot find llib-lc.ln
      Lint pass2:
      exit used( a.c(10) ), but not defined
      $ cc a.c 
      $ ./a.out 
      $ echo $?
      1
    

edit: grrr, sorry small copy/paste mistakes before.

------
gwu78
It's not just an "iOS" flaw. It's a "latest, greatest Apple OS" flaw.

And anyone who given a choice between SSL and TLS relies on TLS is not putting
security as a top priority.

And why would anyone who cares even the slightest about security use Flash?
Are you serious?

I guess this is why security consulting could be easy money... clients want to
use Flash and "stay secure". Yeah, sure, we can handle that for you.

Well, now you cannot even use a Mac without the potential for HTTPS
authentication not working. Better make sure the OS is updated. Sounds a lot
like Microsoft. Maybe you could start a business updating Mac OS's.

"Does anyone know what's going on inside Apple?"

If they did they couldn't say. All employees are sworn to secrecy.

I blackhole all traffic from Apple devices to *.apple.com

You would not believe (or maybe you would, if you are a "security consultant"
or some such)... you would not believe the amount of "phoning home" that these
devices do.

I agree you can't trust "security consultants" who do their marketing via
blogs and forums.

But you surely cannot trust Apple either.

The flaw was one line of code.

I'm curious. What is the size on the update?

Imagine if you could make the change yourself, recompile and dd an image to
your device.

------
ancarda
I'd like to believe Apple simply as no code review but it's odd they removed a
specific check -- was it not suppose to be there? Does anyone know if the
patch added the line back in or if it removed the duplicate goto?

The one thing that points to it not being a backdoor is I doubt Apple would
open source the code. Surely they'd maintain a separate branch or something?

~~~
rikkus
They could have made it open source because they knew they were under pressure
to backdoor it, so wanted people to see what happened without explicitly
telling the world... Or not. It's a fun theory though.

~~~
interstitial
You think a billion dollar, cash-rich company known for software excellence
would understand testing and hardening? A company known for draconic quality
assurance, would skimp? Silly me. HN must be right.

------
camillomiller
I really hope Apple will disclose its findings on this matter, in name of the
transparency they're advocating. Imagine if Apple itself states something
along the lines of "we found out that the line of code responsible for the bug
was planted on purpose". Should they make such a statement? Better: COULD they
make it?

~~~
antsar
I fear they'd be too concerned about the potential impact on their business.

------
pieter_mj
Why is is so important to know wether this bug was left deliberately or not,
or even sneaked in by the NSA?

To the more important questions 'Did the NSA almost immediately discover this
bug' and 'Did they exploit it', I answer with a resounding yes.

------
garethadams
I think I'll apply Betteridge's law[1] to this one.

[1]:
[http://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines](http://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines)

~~~
SigmundA
Also:
[http://en.wikipedia.org/wiki/Hanlon's_razor](http://en.wikipedia.org/wiki/Hanlon's_razor)

~~~
ronaldx
Hanlon's razor is just the worst. Systematic incompetence _is_ malice.

Here's my alternative but equally reasonable platitude: "Never attribute to
incompetence what could be attributed to malice with plausible deniability."

------
fit2rule
I honestly think it was deliberate. What class of Operating System developer
ships their OS releases without 100% CODE COVERAGE? Apple do code coverage
testing, surely? I mean, more than the "-warn-dead-code" args that get flung
around. I can't understand how this would have gotten released into the wild
if they were doing industry-standard code coverage tests. And .. if they're
not doing industrial-strength code-coverage testing on their iOS/OSX release
builds, thats the real news here ..

------
Patient0
It looks like the sort of bug that can be introduced by an erroneous merge in
source control.

------
cl8ton
I don't think the bug was deliberate, could of been just an honest mistake.

But on the other hand, why didn't the compiler generate an "Unreachable Code"
warning during build?

We have explicitly set this warning to "Treat as Error" during our builds.

~~~
Eiwatah4
Gcc doesn't have such a warning. Clang has it, but it has to be explicitly
enabled. (It's not even in -Wall or -Wextra.)

~~~
throwaway2048
It appears that -Wunreachable-code was removed from gcc due to optimizer
issues

[http://gcc.gnu.org/ml/gcc-
help/2011-05/msg00360.html](http://gcc.gnu.org/ml/gcc-
help/2011-05/msg00360.html)

------
rimantas
Where am I mistaken that having this bug deliberate is meaningful only if you
control most of the networks? In which case there are a lot scarier things to
worry about.

------
ape4
I prefer always using brace brackets for if's...

if (cond) { goto fail; }

instead of:

if (cond) goto fail;

~~~
xroche
Yes, and this is actually a sane coding rule. Without it, badly written macros
(using several ;-separated lines for example) will cause subtle bugs like this
one.

~~~
simias
I would argue that in this case the problem lies in the macro, not the lack of
braces. You can't program defensively against broken macros.

I've seen quite a lot of discussion surrounding the coding style around this
"goto fail" vulnerability, but IMO that's missing the forest for the tree. If
this coding error was truly a mistake (which I tend to believe) it was
probably the result of a bogus copy/paste. Shit happens, no matter the coding
style.

In my opinion the real problem is: why wasn't this code properly tested and
audited? For crypto code it borders on gross negligence.

In particular, I disagree with TFA that "The flaw is subtle, and hard to spot
while scanning the code". When you're used to read properly indented C code
the 2nd "goto fail" with its weird indentation jumps to the eye IMO. Also the
fact that the same statement is repeated twice in exactly the same way. I'm
used to reading a lot of kernel code using the same kind of error handling and
it really stands out IMO.

------
zacinbusiness
I don't think the NSA is nearly as technologically capable as people think
they are. If they want info from someone all they have to do is detain them
and bust open their kneecaps with a hammer. No one would ever find out. So
there's really no reason to go through all the shadow games.

If this is an intentional bug, I think it's likely a hacker or just a
disgruntled employee. But I'd be willing to wager that it was a tiredness
error, rushed to implementation by an overworked and likely underpaid engineer
deep within Apple.

~~~
fit2rule
You're absolutely wrong on this. The NSA are far, far more technologically
competent than we've been giving them credit - and that is _precisely why the
NSA releases have been so imporant, and chilling, to the computer industry and
hacker community_. The NSA are way ahead of the pack. To think of them as a
stupid bungling agency, the way most people associate government TLA's, is
doing the world a disservice. Their competence is terrifying - as in it
literally _induces terror_.

~~~
zacinbusiness
I think they are paranoid and have a blank check budget. But I don't think
they are competent. If they were competent then they would know that no
serious threats will be discussed in world of Warcraft or over yahoo
messenger. The supposed enemy in the shadow is at least smart enough to know
better. If they aren't, then they aren't smart enough to warrant the invasion
if basically everyone's privacy. It just doesn't add up to me. If we are
fighting super technical enemies, we must assume they are smart enough to not
use basic, consumer grade chat. If they are not technically capable enough to
know better than to use something like yahoo, then just send them an email
with Sub7 in it. Either way, spending billions of dollars of my money to watch
some millionaire banker sext with his secretary is a waste of money and of
time.

~~~
pessimizer
> If they were competent then they would know that no serious threats will be
> discussed in world of Warcraft or over yahoo messenger.

No, that would be a sign that they were incompetent, because secretly
communicating groups do use unusual chats, such as graphical children's chat
applets, in-game communications, shoutboxes and such. Why wouldn't they?

------
kalleboo
Wouldn't it be way safer and easier for any attacker (NSA or otherwise) to
compromise a trusted CA than to compromise the Apple SSL code?

~~~
cperciva
It might be easier, but I wouldn't say that it's safer. Fake certificates
leave a paper trail -- that's how hacked CAs have been discovered in the past.

~~~
kalleboo
Good point!

------
blueskin_
I was wondering if this was going to be one of those pages that just says
"Probably." or similar in large text.

------
tehwalrus
...There are goto statements in the code that is running on my laptop and
phone? I feel dirtier now.

~~~
deletes
Did you ever read assembly?

~~~
tehwalrus
I know that function calls are fundamentally just jmp calls underneath, and
that's exactly what a goto is too, but we built high level language compilers
(including C compilers) so that we didn't have to deal with goto-centric
spaghetti any more.

As with everything, gotos work just fine if you restrict yourself to a few
specific uses. At that point, however, you've basically just invented function
calls, switch statements, and maybe exceptions if your rules aren't strict
enough, which can all be managed by a compiler which _always_ generates the
correct number of jmp's to do the job.

EDIT: you could more accurately replace each instance of "function call" in
the above with "subroutine" as FORTRAN defines them, which is closer to what
you get for free with gotos than a function call with an actual stack pointer.

~~~
lutusp
> I know that function calls are fundamentally just jmp calls underneath, and
> that's exactly what a goto is too ...

No, this is false, both at the high-level-language and assembly-language
level:

[http://en.wikipedia.org/wiki/X86_calling_conventions](http://en.wikipedia.org/wiki/X86_calling_conventions)

A function call transparently returns to the point at which the call
originated and resumes execution. A GOTO departs from the original location,
never to return. That is a crucial difference.

> EDIT: you could more accurately replace each instance of "function call" in
> the above with "subroutine" as FORTRAN defines them, which is closer to what
> you get for free with gotos than a function call with an actual stack
> pointer.

Still wrong -- you're confusing two very different things. A function call, in
both the assembly and high-level sense, is completely different from a GOTO
instruction.

~~~
tehwalrus
I feel that your answer proves the point my original comment - that there is
literally Silicon level support for completely avoiding gotos. I stand by my
disgust at finding that they exist in any non-legacy code, especially in
something I generally think of as high quality (iOS[1]/OS X).

(combined with a global pointer to a stack, a few instructions and a jmp
should be enough to reimplement functions, including storing the return jmp
address somewhere, obviously. As I think I made clear, I believe we invented
compilers so that we don't have to know how this stuff is actually
implemented. I do realise that I'm never going to know more about assembly
than the set of HN readers who will pounce on this kind of comment - I was
just trying to show that I have heard of jmp, and that I know it is spat out
by compilers to implement high level language features, and that any
nontrivial compiled program will contain many jmp instructions. A bug from
using goto where a switch or if/else statement would have worked is
inexcusable, and also nothing to do with assembly instructions.)

[1] actually, iOS 7? I take that back.

~~~
lutusp
> I feel that your answer proves the point my original comment - that there is
> literally Silicon level support for completely avoiding gotos.

Yes, and there's a reason -- chip design should reflect how computers are
programmed at a higher level, and function calls produce code that's easier to
understand, create and maintain. The reason is that function calls obey a
logical hierarchy that often reflects the meaning of the algorithm being
executed.

To see the point, try creating a recursive-descent parser that uses GOTO and
avoids function calls.

This understanding isn't limited to an HN gaggle of nattering bystanders:

[http://www.u.arizona.edu/~rubinson/copyright_violations/Go_T...](http://www.u.arizona.edu/~rubinson/copyright_violations/Go_To_Considered_Harmful.html)

> I was just trying to show that I have heard of jmp, and that I know it is
> spat out by compilers to implement high level language features, and that
> any nontrivial compiled program will contain many jmp instructions.

Yes, but what a compiler does, and what a high-level language does, are often
completely different. As one example, many recursive algorithms are compiled
into functionally equivalent loops, to make them more efficient and to keep
from blowing up the stack.

My point is that a compiler's output doesn't have to be read by a human, so it
can break every rule. Not so for a high-level code listing that might have to
be understood and maintained over decades.

~~~
tehwalrus
Then I think we agree on everything, my ignorance of the details of x86
assembly language included! :)

~~~
lutusp
Not if you think using GOTO in modern code represents acceptable programming.
Remember? The topic?

~~~
tehwalrus
...I don't. I was never arguing that. See my first comment.

------
pierre_a
> The flaw is subtle, and hard to spot while scanning the code

The author can't be serious. This particular bug has made the rounds and is
understood even by non-programmers.

~~~
mikeash
The ability to understand it when it's pointed out is unrelated to the ease of
spotting it among a thousand lines of other code when you don't know what
you're looking for.

