
Patching is hard; so what? - runesoerensen
https://blog.cryptographyengineering.com/2017/09/15/patching-is-hard-so-what/
======
chronid
> I don’t operate production systems, but I have helped to design a couple of
> them. So I understand something about the assumptions you make when building
> them.

Start by operating production systems. You will rapidly discover that patching
is _not_ a technical issue (as I already said last time there was a patching
discussion on HN). Patching is technically "easy".

But if the business does not prioritize (and allocate time towards) patching
(and testing that systems work with after) patching won't get done. There are
features to develop and deadlines to meet and new systems to turn on and old
systems to retire and oh-my-god don't touch those servers otherwise the vendor
will not support the prod environment anymore (no longer certified, yay!) and
it costs us 30000$ to get certified again.

Too many times I see HN assuming that everyone is running on a public cloud
with B/G deployments and immutable infrastructure you can rebuild and redeploy
easily. Unfortunately that's not the case.

~~~
xorcist
> with B/G deployments

Why do we call it that now, instead of A/B deployment like we used to?

~~~
_asummers
To me, blue/green is for infrastructure, and A/B is for the user side of it.
That is, you should be able to do blue/green with the content not changing,
but when you're A/B testing, you're collecting metrics and other usage data
about the users to make future product decisions. You can also do A/B testing
without having a B/G setup, using things like user gates. This distinction may
be incorrect, but that's how I've used them.

------
sqldba
Patching at an OS level (not what the author is talking about, probably) is
extremely extremely hard - from an enterprise Microsoft platform production
point of view

1) Because there's no budget for tools. It's mostly WSUS and SCCM if you're
lucky.

2) Because staff don't even know how to use those tools properly. Can't blame
them because there's very poor documentation from MS and very little
accessible training on design patterns that integrate them into large corps.

3) It's outsourced. So it's even worse than above.

4) Because management fights it every step of the way. They hate patching
because they're stuck in the 80s where they think every patch is deeply
documented and you should only apply what's necessary. And you should only
apply security - ignoring updates which give critical stability fixes.

5) Because the enterprise as a whole is so poorly documented it's unclear who
owns what or if a rollback is possible and what it would affect. So everything
is in CYA overdrive.

6) And then nobody working there wants to own it because it's not sexy to
patch. (As a geek I think patching is sexy and very fulfilling work, but I'm
in the extreme minority).

None of this excuses anything. But most largish companies are still in the
80s, with IT managers in the 80s. I can't wait for them all to die.

------
chasb
For teams that use a DevOps model, fast, predictable deploys that can be
safely rolled back are important for security, for this reason.

If deploys are like playing Jenga on a sailboat, you're not going to be able
to patch fast or safely.

That said, even becoming aware a CVE exists in the first place is still a
problem for many teams. There are plenty of good options, it's just
underinvested in early on.

~~~
yeukhon
It isn't necessary DevOps or system admin not capable. You've vendor software
not up-to-date that is beyond your control, and often users do not know until
vendor sends a notification or a hack headline comes online then "oh fuck"
moment.

Just to be fair too: I have seen a number of enterprises run their "legacy"
infrastructure (mostly refereing to on-premise servers and old applications)
doing a good job keeping up with updates such as Microsoft and whatever Linux
flavor they use. But the new system/DevOps havr trouble getting through the
gate, mostly IMO because we adding too many layers to the stack.

~~~
sverhagen
Your first paragraph, do you find any of that a valid excuse for a company
like Equifax, who handle so much sensitive data? _I_ consider them responsible
for the entire chain. I understand it's hard, makes no difference with those
stakes.

~~~
yeukhon
No, not at all. My point was DevOps is not a magical group of wizards, and
often DevOps team can be unproductive compared to whatever enterprise IT
operation team/client service team. If DevOps is unable to carry out its core
values, yes, entite chain is to be critized.

------
guelo
Patching isn't that hard. Once Equifax _finally_ noticed the intrusion
(intrusion detection system?), they took the system down and patched it within
a day.

~~~
hyperpape
I'm working with a fifteen year old app that's slowly modernizing. There's two
types of updates, the normal kind that's slow, and "everything is on fire, get
the fix into production ASAP, then pray". The latter kind is super quick, and
it's what Equifax did here. But that doesn't mean patching is easy for them.
It means patching without doing whatever laborious QA/manual testing they
normally do is easy.

(Btw: it might sound like I'm defending Equifax. I'm not. I fully agree with
the article. They have a responsibility to figure out how to secure their
systems. "We can't patch quickly" may be true for them, but they needed to
figure out how to change that).

~~~
sqldba
Yes. You've hit the nail on the head. They'll have such a tight bureaucracy on
patching that it's impossible (which is an IT problem in their eyes) but the
second a senior manager says just do it - its quick and painless.

I see this over and over again everywhere.

~~~
hyperpape
It's quick, but it's risky, because testing is manual, you don't have blue-
green deploys, etc. So it's not that it becomes painless, it's just that their
are circumstances where you accept the risk.

------
empath75
I know equifax is probably dealing with a hairy legacy system, but really you
should design all your systems to be able to push out a fully tested
blue/green deploy in _hours_ not months. There's no excuse not to with
Jenkins/aws/docker/etc.

You should be testing and pushing updates regularly as a matter of course, not
as a red alert only when security vulnerabilities are published.

~~~
ams6110
That's not the mindset of enterprise IT though. The mindset of enterprise IT
is "if it works, don't touch it." This mindset developed because of decades of
dealing with arcane, complicated software that was at the same time critical
to business operations. I'm aware of a major university that still has a
three-week "change freeze" at the start of every semester because of this
attiude and I'm sure they are not the only one.

~~~
topkeker
I've worked in enterprise security teams for the last decade. This is spot on.
It's retarded and will never change because there is no accountability.
Corporate IT don't give no fucks.

~~~
toast0
Corporate IT is doing what software developers have taught them through
experience and pavlovian condition:

Don't touch anything, because it will break if you touch it.

IFF people who release software learn how to release security patches and
bugfixes that don't include new features or break existing parts (except when
breaking changes are actually needed), then, after several years of good
results from pushing changes, Corporate IT will be conditioned to just push
security fixes, it's not a big deal, plus it's important.

------
rdtsc
> Specifically, these folks point out that patching is hard. The gist of these
> points is that you can’t expect a major corporation to rapidly deploy
> something as complex

Then they should be taken to court and be forced to pay for it. It's like
saying about a company which runs trains and didn't do maintenance which
caused an accident to happen "Well, you see maintaining the brakes on the
trains is too hard so don't blame them too much...".

> The gist of these points is that you can’t expect a major corporation to
> rapidly deploy something as complex as a major framework patch across their
> production systems.

So those corporations will be hacked, and then they should be dragged to
court, made to pay, some might even go bankrupt and disappear altogether.
Eventually only the companies that manage to build better infrastructure will
survive.

Oh well, one can dream...

~~~
sillysaurus3
The fact that Equifax reported the breach at all represents a good faith
understanding: they tell the world, and in return the world does not make them
cease to exist.

If you kill companies over security incidents, then magically you will no
longer hear about security incidents.

~~~
mirashii
It's not a good faith gesture, it's required by law almost everywhere they
operate that they notify of a security breach of PII.

[http://www.ncsl.org/research/telecommunications-and-
informat...](http://www.ncsl.org/research/telecommunications-and-information-
technology/security-breach-notification-laws.aspx)

~~~
sillysaurus3
And yet, as if by magic, you will no longer hear about security incidents if
you kill companies.

~~~
Sevii
If there is no penalty that hurts shareholders and executives breaches aren't
going to stop.

~~~
pizza
Penalizing shareholders and executives to the extent that reporting breaches
becomes enterprise suicide is only going to make breaches unannounced and
prevent the collective from learning from individual mistakes, so therefore
more common.

The way to reduce breaches is to encourage open admission of security
failures, so that preventative knowledge spreads quickly. It is worth some
positive amount of respect to follow through in one's obligation to warn one's
dependents, and moreso if one tries to warn society as a whole.

~~~
viraptor
> so that preventative knowledge spreads quickly

It doesn't need to spread. It's not original research. People do this every
day in other places. This shouldn't be talked about as some black magic
skills.

~~~
pizza
If security best practices weren't made irrelevant by ignorance in the first
place then people wouldn't cause these preventable mishaps in the first place.

It's not rigorous enough to say "I am certain some people can RTFM and
understand without resorting to glorified mimicry, therefore the benefit to
the group of encouraging people to share mistakes is marginal".

There is just too much at stake in the outcomes of security breaches. It is
far far better to be have a high _variance of variance_ of protective +
relayed benefit gained from analyzing someone else's failure, than it is to
overestimate your certainty of understanding while at the same time
underestimating the lessons of someone else's failures.

Security is too important to put the focus on people learning in rigorous and
logical ways over the protection of users. That said, due to the informational
nature of security, to protect users necessarily contains as a sub-goal wholly
understanding what your security _is_. But it's disingenuous at best to say
that incident disclosure isn't actually empirically sound but instead gives
security the appearance of black magic.

------
lucb1e
TL;DR: patching is hard, so use defense in depth to minimize/mitigate the
impact of individual components becoming insecure from time to time.

------
BrandoElFollito
The whole problem boils down to a choice :security or availability.

If you prefer to have systems 100% up (no downtime for patches and fixing
potential issues after the patch) then it is fine. The actual risk (described
beyond "having security") has to be accepted by the owner of the data /
service and the problem is over. I have yet to see a CFO who will accept in
written that his systems are prone to hacking, theft, manipulation.

Until the CISO does not force the company to sign off such risks, he or she
will be the one who fucked up.

------
rythie
Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt
it.

Systems need to be designed to be patched regularly from the outset. Yes,
defense in depth can help, but it's not defense in depth if you one of the
layers is pretty much always broken.

If they are relying on legacy systems that can't be proactively and regularly
be patched, they shouldn't be holding that data.

I think part of the problem here, is that we can't see the risks companies are
running with our data and goverment doesn't regulate it.

P.S. I do manage systems, though not on this scale.

~~~
hn_throwaway_99
> Is it so hard though? Do Facebook, Google etc. have this problem too? I
> doubt it.

Companies with monopolistic pricing power like Facebook and Google are a poor
comparison for almost any other company. Many problems become very easy when
you have a near infinite supply of money. Other businesses need to work with
much more difficult constraints.

~~~
genmud
Average IT spend runs between 2 and 5% of _revenue_ for fortune 500s. For
reference... GM makes 166 billion/year in revenue, and on the low end their IT
spend is probably more than 2 billion/year. In 2004 they were making the news
because they went from 4B/year in 1998 to 3B/year. In 2006 they were making
190B/year, so I doubt their IT budget has changed significantly and is on the
low end compared to other companies in the fortune 500 list.

Facebook and other technology companies might spend a lot more on IT, but
that's part of them doing business.

Don't confuse the fact that technology companies do technology better with the
fact that any company in the fortune 500 list have HUGE IT budgets. At a large
company that I used to work for we had an IT budget of more than 1 billion USD
and they didn't patch very well either. Our budget was actually smaller than
many of our peers.

If you have a IT infrastructure/application budget of 1B or more and you can't
patch, your doing it wrong and there is no excuse. For a full baked/managed
solution, you can find folks to do an entire program for less than 12m/year
for a 200k node network.

------
alkonaut
It's pretty simple to patch if you designed the system to be patchable in the
first place _and_ assign enormous cost to not patching. If you assume running
an outdated version of struts for 3months is a $1B expense then you'll patch
even if it means disruption to business operations, or will breach some
conract.

And if the system is properly designed then anything can be rolled back if it
turns out there were unforeseen consequences.

------
shawkinaw
Maybe patching is hard. But the real question is why one vuln in their web
framework allowed unfettered access to basically all of their data. There are
much deeper problems here than slow patching.

~~~
Johnny555
_the real question is why one vuln in their web framework allowed unfettered
access to basically all of their data_

The hack involved "primarily names, Social Security numbers, birth dates,
addresses and, in some instances, driver's license numbers", that's a very
limited subset of the data they hold on people.

I'm not defending them, but this was not "basically all of their data".

------
nickpsecurity
"I don’t dispute this point. It’s absolutely valid."

I do dispute it. Patching is _not hard_. It's incredibly easy to the point
that some places and projects have the process automated from notification to
application to testing for breakage to deployment. For reducing downtime, one
can use clusters with rolling deployments of patches. Patching is only hard
for things such as web applications when the company's application or
processes are done in a way that makes patching hard. As in, they have to be
incompetent or just not care about IT. Far as competence, your claim about
fragile systems is a good example that might have happened.

" then you’ve implicitly made the decision that you’re never ever going to
allow those vulnerabilities to fester."

You're thinking like an engineer that cares about quality. You instead should
think like an Equifax CIO or something. To start with, this is a company that
collects PII against people's will to sell to third parties for their main
goal of huge profits. Politics plays more a role than engineering talent in
people getting the senior, management positions in such companies. They also
tend to chase whatever is popular among Fortune 500, esp with cheap labor or
ecosystems available. Java was one of the fads that financial sector was all
over. Combine all this to have a company whose fad-chasing CIO keeps costs
down and profits up applying the thing he or she read in a computer magazine
with the cheapest talent available on a tight budget. The result of their work
is a pile of garbage they have trouble patching. If you doubt this, just look
at the security of the web site they deployed for credit monitoring and
apparently to help hackers get at people interested in credit monitoring. Or
they just made mistakes so easily avoided that they're either inexperienced
beginners or don't care at all.

"So what would those systems look like?"

Well, they would have built it some time ago. So, let's work our way from old,
high-assurance security toward something commercial and affordable from at
least 2000-2005 era. The original work in securing data involved security
kernels:

[https://www.acsac.org/secshelf/book001/19.pdf](https://www.acsac.org/secshelf/book001/19.pdf)

Several of those are still available but expensive. Both security kernels and
databases such as Trusted Rubix built for them. Today, those look more like
the next link with companies such as Sirrix and Green Hills selling them
commercially:

[https://os.inf.tu-dresden.de/papers_ps/nizza.pdf](https://os.inf.tu-
dresden.de/papers_ps/nizza.pdf)

In any case, we'd need a robust combination of OS, database, and application
code. Nothing hits the database without going through the app server first.
So, we embed our security policy into app server. How to implement it? Ever
since Dijkstra's THE OS (1960's), we knew to specify correct behavior with
preconditions, invariants, and post conditions. Then use anything from formal
analysis to testing to runtime checks to ensure that behavior is enforced. The
security kernels did the former where Design-by-Contract in Eiffel used tests
and runtime checks. Got to pick method with best bang for buck. Two links to
illustrate safer languages of past with Ada 2005 since we're looking at
earlier stuff.

[http://www.adacore.com/knowledge/technical-papers/safe-
secur...](http://www.adacore.com/knowledge/technical-papers/safe-secure/)

[https://www.eiffel.com/values/design-by-
contract/introductio...](https://www.eiffel.com/values/design-by-
contract/introduction/)

Given labor situation, we use DbC in a safe language. All prior work showed
simplicity was necessary for security. So, it would be an app server done in
Eiffel, Ada, or (if absolutely necessary) Java/C++ using Design-by-Contract
methodology plus lots of testing. The web component would be a middleware done
similarly that basically translates web actions to simple protocol the server
uses for requests/responses. Any new types of problems with web apps are
mitigated in the simple framework a la Airship CMS, caught with some kind of
monitoring a la Spectre, or both.

This would have to run on something robust. You'd have the database/storage,
app server, web component, management, and monitoring which is behind one-way
link (data diode) w/ logs copied to write-only media. A company from likely
mainframe background now run by wise technologists might first make the
software target IBM but portable. Balancing reliability, security, and cost,
the best targets would be AS/400 or AIX. The former is a capability-oriented
architecture with few 0-days whereas other is a rock-solid UNIX. Portability
means they'd migrate part or all of this to new OS's as they showed up.
Eventually they'd notice OpenBSD's security advantages or buy one of those
certified-secure kernels w/ POSIX layer. Depends on their viewpoint and
budget. Probably throw some clustering software with rolling releases in there
since OpenVMS and NonStop had high-availability in the 1980's. It was well-
known strategy.

So, there was a straight-forward way to make a robust stack with a web front
end which gets more robust over time. This could be done in high availability
or just easy patches. They instead end up with fad-driven crud possibly
running on other crud that's easy to hack but hard to patch. Just bad
engineering that's typical in the market versus methods such as I described.
Also note that there were (are) large institutions using some of what I
recommended for those benefits. The tools and people just cost a bit more
which doesn't align with incentives of companies as greedy and IT hating as
Equifax. Sure turned out extra profitable, didn't it? ;)

