
Helping to Build Cloudflare, Part 2: The Worst Two Weeks - jgrahamc
https://blog.cloudflare.com/helping-to-build-cloudflare-part-2/
======
tyingq
The video linked in another comment is terrific.
([https://www.rsaconference.com/videos/inside-
cloudbleed](https://www.rsaconference.com/videos/inside-cloudbleed))

Particularly the value of getting recurring core dumps to zero, such that they
stand out enough to find and fix. Reducing log/stats noise has been a
traditional go-to for me.

Moving sshd to a non-default port, for example, initially sounds like useless
security through obscurity. But, it then makes unusual ssh access stand out
and get noted.

------
tyingq
The openness is laudable, but I'd seen this phrase from them before around
cloudbleed: _" We didn’t find evidence of exploitation"_

It just doesn't mean much. They had something like 2 weeks worth of logs. And
further, they never explicitly stated that the logs had enough context to show
if it was being exploited. That is, are there any fields in the log files that
could distinguish normal uri accesses from malicious ones?

I'm curious if that's all deliberately careful wording because they know, that
they "don't know".

~~~
lioeters
Indeed, "Absence of evidence is not evidence of absence."

~~~
lkbm
A popular quote that's usually wrong.

It's true iff you haven't made any effort to find evidence whatsoever OR
there's a zero percent chance that you would find evidence even if it existed.
This is a very rare circumstance, and one that doesn't apply in this case.

There were some logs that. had there been exploitation, had a non-zero change
of providing evidence of said exploitation. They looked in the logs, and
there's a non-zero chance that, if that evidence were in those logs, they
would have seen it.

P(exploitation) < P(exploitation|they had and looked at the logs)

That's the definition of evidence.

------
manigandham
Cloudflare is the kind of technical company I always wanted to build. Glad to
see their success as a happy customer.

------
matt2000
I really enjoy reading these war stories, even though this sounds like a super
difficult time for the author. On a technical note though, aren't these kinds
of breaches a powerful reason to use memory safe languages? I know performance
used to be a major driving factor, but I doubt that's the case anymore. Even
then, does it seem reasonable to say "I'm risking it all for a 20% performance
gain"?

Admittedly, this isn't the kind of stuff I normally write, so I might be
missing something. But since the ramifications of these memory leaks are so
severe, I would think the industry would be adapting.

~~~
jgrahamc
Yes. And tomorrow's installment of this series of posts begins: _After
Cloudbleed, lots of things changed. We started to move away from memory-unsafe
languages like C and C++ (there’s a lot more Go and Rust now)._

~~~
matt2000
Oh! Well there's my answer then, looking forward to reading that.

------
ttul
Reading the first part of this post reminds me that Cloudflare should really
get into email security as well ;)

