
The Colossal, Monumental Screw Up That Is Marriott Security - danielmunro
https://danmunro.com/posts/the-colossal-monumental-screw-up-that-is-marriott-security/
======
ejcx
I'm sure Marriot had an IDS that created 10000+ alerts per day.

I'm sure they also had a credential rotation policy, hired 3rd party
pentesters, had a vulnerability management program, etc

Securing systems is really hard. A lot of the old school recommendations
create more issues than they solve, like rotating every database login
password every 90 days or so.

~~~
joncrane
>A lot of the old school recommendations create more issues than they solve,
like rotating every database login password every 90 days or so.

This is one of the ones that drives me crazy. You can maybe make it work if
you have a really good secrets management system, especially if it's hooked
into AWS EC2 roles. But having to manually log into servers to change config
files/passwords every 90 days is ridiculously disruptive.

~~~
inetknght
> _having to manually log into servers to change config files /passwords every
> 90 days is ridiculously disruptive_

Then make it so you don't have to manually log into servers to change
files/passwords.

------
a2tech
This article is useless. It says nothing, has no information on what or how
the breach happened, and is basically security word salad with a heaping of
'these people are idiots'.

~~~
sol_invictus
Yeah, I'm fairly sure one of the first news pieces on the attack literally
said the security team discovered the breach when "alerted by suspicious
activity on their customer database" \- sounds a lot like an IDS
functionality.

------
fefe23
This article proposes buzzword-level security theater. IDS! Rotate
certificates and credentials! Have pentests!!

What it fails to mention is: Do not collect data you do not need. You do not
need my email. Forcing me to give it to you is bad. You do not need to know my
home address. Forcing me to give it to you is bad. You do not need to know my
birthday. Forcing me to give it to you is bad. etc pp. The mind boggles why
they thought they need to collect passport data!

IDS and certificate rotation are snake oil and security theater. Sure, they
usually don't hurt. But here is some good advice:

1\. Don't collect the data. If you don't have it, it can't be stolen.

2\. Apply all the patches. Immediately. No you don't know better than the
vendor. Install all of them. Always. Immediately.

3\. No unnecessary dependencies. Yes that means don't go in the cloud.

4\. Have an architecture that segregates stuff by security level. Don't put
all your things in the same basket unless you are prepared to have the highest
security level for all of them. No "this is just a chat server, it is less
important than the database" unless those are properly isolated.

5\. Minimize your TCB. The less things you have to trust, the better.

And THEN, after all this is done, can we talk about IDS and certificate
rotation.

~~~
ramses0
The same way you can throw snark at a company for their (lack of) security
knowledge, they can do the same for your lack of industry knowledge.

Generally, gathering passport data for hoteliers is a legal requirement (see
here: [https://www.quora.com/Why-do-some-countries-require-a-
passpo...](https://www.quora.com/Why-do-some-countries-require-a-passport-to-
stay-in-a-hotel) ).

Now. Agreed. Required to collect v. having available online and hackable for
all guests ever is not a best practice, but it's easy to see how a hotel
(quite physical-space-intensive, labor-intensive, capital-intensive business)
may not have viewed or understood the risks of having this data around.

The last time I stayed in a hotel, there was a car whose window was broken in
the parking lot (unfortunately). A crime was committed, on hotel property!

The question businesses are struggling with is: how can they focus on their
business and either government or industry can focus on crime-prevention?

~~~
fefe23
I am aware of legal requirements for hotels.

Here's my technical view:

That does not mean you have to _have_ the data. Either forward the customer to
a government system where they enter the data, then it's the government's
fault. Or do escrow: For example, you could store the data encrypted with a
public key of the government. Then only they could decrypt it. If someone
stole it, there would be no problem. And the government could still view the
data.

My political view is that the government has no business asking hotels to
collect passport data, or indeed any data on their customers. This is a
blatant privacy and data protection violation. The government does not need to
know my location at all times. It's deplorable that things have deteriorated
this far already.

------
coldcode
Pays to read the original article, it wasn't Marriott it was the company they
bought, long before the purchase. Marriott's system was not compromised.

~~~
wavefunction
When they bought that company it became their system. You can't just hand wave
away PCI compliance hehe.

~~~
WillPostForFood
It just might be a more accurate if it was something like, “Marriott bought
itself a security nightmare with the Starwood acquisition.” It is certainly
Marriott’s problem to deal with now, but there security team might not be as
bad as Starwood’s.

------
watertom
I'm an Information Security and Privacy professional, and until there are real
penalties for a lack of security nothing will change.

Go see the Ford Pinto case, cheaper to pay lawsuits from deaths than fix the
problem, then don't fix the problem.

The other problem is an utter and total lack of technical knowledge by Sr.
Management, they hire charming idiots who tell Sr. Management what they want
to hear. I've been to conferences and I've listened to discussions from
"security professionals" and I'd swear I was at my local supermarket asking
people about Information Security and Privacy.

~~~
russdpale
The appearance of security is much more important than actual security. I
would gander that is precisely because there is no real penalty outside of
anything that would be considered the cost of doing business.

How to enforce punitive action upon a company with such international reach is
the real question.

------
jmount
It is only anecdotal: but I had my identity stolen within days of joining the
Marriott rewards program in December 2016. I think they may have been getting
credit reports on members (which generates a lot more personal data than you
gave them) and leaking for quite some time (not just a small number of data
breaches).

------
jcrawfordor
The way this article talks about IDS sounds, to me, like someone who has never
worked with IDS professionally or on any large scale. This goes for other
points in the article as well, but that seemed particularly glaring.

I don't intend to defend Marriot, from other coverage it sounds like someone
did a very poor job (although not necessarily Marriot itself). But this
article also makes things sound far simpler than they are.

My best guess is that the attacker gained access to a database server, and
let's say they dumped the contents to a file and exfil'd the file (not always
the best way to go, but often the best way to go). Assuming they stole
database creds from somewhere else (e.g. some application), that might
generate around a half dozen auditable log items on the database server. The
retrieval of a large file would be a good opportunity for detection by SIEM
content, but without further knowledge of the application it might not be -
large file transfers from that machine might be normal as part of e.g. batch
processing.

For me, it's hard to say at this point that this would have been easy to catch
at all. Perhaps it would have been, but if the attacker was some combination
of competent and lucky (combined with the lack of measures like limiting
database access rate for applications, which are quite rare in practice), they
may have been in and out with very little detectable activity.

~~~
tetha
> The retrieval of a large file would be a good opportunity for detection by
> SIEM content, but without further knowledge of the application it might not
> be - large file transfers from that machine might be normal as part of e.g.
> batch processing.

Or an eccentric and occult edge case like "backups", especially if it's a
database system. Sorry for the snark, but I've had to tell some people the
importance of backups for production persistence like a broken record for a
week or two.

And sure, you could have IDS rules / firewalls setup to flag or block traffic
except to the backup storage hosts and the replication servers and the batch
processing servers and the monitoring andso on and so on, flag files, ...

But that stuff is hard, requires a lot of maintenance and adds risk to a lot
of critical / stress-powered processes. Change your backup storage at 3 am due
to hardware failures? Whoops, the firewall of database host #13 wasn't
updated, and now you have no more backups from that host.

------
danielmunro
OP here, thanks everyone for the interest and discussion in the topic.
Awareness and open discussion is going to be the disinfectant our industry
needs to improve security hygiene. I have only recently taken blogging
seriously and am still working to find my voice and balance between too little
information and information overload. I took the feedback here to heart and
tried to improve and clarify my ideas and recommendations. Sorry if there's
still not much specific information provided, I wanted to keep it at a high
level, maybe that was not the best call -- anyway the feedback is very
helpful.

------
yoaviram
Send Marriott an Erasure Request now and maybe next time it will not be your
data: [https://opt-out.eu/?company=marriott.com#nav](https://opt-
out.eu/?company=marriott.com#nav)

------
jrochkind1
What's "M & M Security"? Linked article never defines the "M"s, I had no luck
googling.

~~~
tetha
It's linked, and it's perimeter based security. I've also known it as egg-
based security - once the shell breaks, you've got a big mess on your hands.

~~~
jrochkind1
Ah, I get it now, thanks. Yes, it was linked, but the linked article didn't
explain the metaphor either. I wasn't thinking of the candy, now I get it.

------
jeanvaljean2463
Disclaimer: Not defending Marriott, as their Starwood Rewards/Marriott Rewards
merger has been demonstrably one of the most epic, public IT integration
failures that I've ever personally witnessed as a consumer bystander.

BLUF: I am a huge advocate of companies being fined on the basis of number of
people affected and types of data leaked. This incentive to not be fined will
be built into the formal or informal risk matrix that a company utilizes for
decision making and these types of breaches will decline in number and
severity from boneheaded mistakes. In the current model, the only incentive
that exists is public embarrassment, but is quickly forgotten despite the
incredible disclosures. ( See Equifax )

I know literally nothing about the internal state of their IT department but I
suspect a great deal of it is likely outsourced and probably "least cost".
From being a long time traveler ( over 1500 nights in Marriotts over the years
) I've seen their payment processing system go down, people remoting into
public kiosks and typing plaintext passwords early in the morning, and (not
so) hidden pages on their website that were intended for special promotions.
As an example, their system that allows one to log into their "internet TV"
account in-room to watch netflix will not purge account information at the end
of a stay. I've checked in and seen other folks' Netflix splash pages when
using the app. ( I always log them out as a courtesy, but suspect that others
might not. )

All that being said, it's easy to point fingers and point out failures in
hindsight. Every large company/government organization that I've served has
similar failures, often not as public, but usually much more serious. In my
own experience there is a usually a core contingent of competent tech
workers/developers who are aware of the technical debt and attempt to bring it
up to management to solve, but get shut down as "there is no reason to spend
money on something that isn't driving revenue/mission". The easiest way to
solve this would be to introduce fines tiered for the number, type of data,
and period of non-disclosure for companies. ( i.e. Equifax breach should have
been a historically large fine in this thought. This, while widespread, is not
on quite the same plane, sans the passport numbers. ) I'm not a big believer
that the federal government is an effective information technology provider,
but this falls in the realm of public good, making it a better fit. Structure
the organization in a similar fashion to NTSB or the FTA, where case officers
lead investigations with teams who have no axe to grind with any particular
organization and are screened for non-bias. ( Just the facts, ma'am ) This is
currently a role being filled by industry security companies, but I would
argue that there has been sufficient bias demonstrated that it should be
removed from private industry and put in a public forum. Similar to how the
NTSB operates, if an American company has global presence, regardless of the
location or nature of the disclosure, the disclosure would be investigated in
a similar fashion forensically. ( NTSB investigates airline crashes of
American manufactured aircraft regardless of location in the world. ) With the
ubiquitous use of syslog data and packet captures that most companies retain,
these investigations should be fairly easy to handle; recognizing that in most
cases, like airlines crashes, large scale IT failures such as breaches are
usually a culmination of a series of failures and bad decisions over time
rather than technically sophisticated attackers.

I hope that we start taking the current problems that face our burgeoning
technical society a little more seriously rather than engaging in idle
political artillery with little outcome for the public good. You know, public
good, the thing that government is supposed to ensure through consent of the
governed?

~~~
number-sequence
I agree with the idea that fines based on the number of users affected makes a
lot of sense. One question I have is how would you propose that number be
calculated? In truth, I think the company whose data has been leaked should
know exactly how many records have been leaked, but per-individual based fines
create an incentive for them to underreport this number. Do you think that’s a
problem, and if so, is there a good answer for how society could get an honest
answer as to how many individuals are affected in a breach?

~~~
jacques_chester
> _One question I have is how would you propose that number be calculated?_

As a percentage of worldwide revenue on a sliding scale.

> _In truth, I think the company whose data has been leaked should know
> exactly how many records have been leaked, but per-individual based fines
> create an incentive for them to underreport this number._

Very true, so triple damages for wilful underreporting and/or criminal
sanctions for individuals.

