
StockX was hacked, exposing millions of customers’ data - rmason
https://techcrunch.com/2019/08/03/stockx-hacked-millions-records/
======
rvz
> ...The company “robbed their users of the chance to evaluate their exposure”
> by not informing customers of the breach when it happened...

StockX is valued at $1B and aside from their cataclysmic choice of using MD5 +
salt as their way of hashing passwords (They obviously don't take security
seriously) the company failed to inform their customers of this security
breach as soon as it happened and left it very late for the customers to
change their credentials. I would expect any unicorn valued company to have
some form of incident-response system to immediately inform staff of the
breach and to instantly reset all user credentials and to notify their users.

Instead they didn't inform their customers after the breach and now someone is
calculating all those MD5 collisions and attacking all accounts with common
passwords.

In the case of StockX handling this security breach, this is un-
professionalism at its finest.

~~~
barbecue_sauce
Jeez. Even coding bootcamps teach developers to use at least bcrypt for
passwords.

~~~
xkcd-sucks
Then they get into I dustry and get told that getting stuck in the weeds of
security isn't something which generates value

~~~
mdip
You nailed it.

We used to joke that working in security was always chasing "zero". If you've
done your job _perfectly_ , an attack isn't successful, just like every other
day. If they had stored passwords properly and had the various other layers
worked out, the attempt on the environment might not even have been seen[0].
It's an _easy_ thing to put off when all you're looking at is today's balance
sheet. For months/years[1], the site hummed along without a breach[2].

Now you're the head of security, you make a whole bunch of points about the
risks discovered within all of the systems. The person who holds the purse
strings reads each risk (in a vacuum, of course) and sees each _individual_
risk as minor, plus that new feature will bring in revenue while spending
money on security only causes a fuzzy category of "cost avoidance". Forget
about the fact that convincing the low-level non-technical purchasing manager
is going to be a whole lot easier than when he/she takes that same
justification to a C-Level executive.

When the problem isn't understood, it looks a lot like a choice between
spending money to reduce a risk (that the audience is going to under-estimate)
versus spending money to give customers a new feature. Do customers _want_
security? Sure, they say they do, but most of your users use the same password
everywhere despite years of being told not to. They think that breaches are
routine and that if their account is breached that it's probably not going to
matter[3]. It all speaks to an _expectation_ of security (a baseline, a
'zero'). It's somewhat ironic that users take security as seriously as typical
developers/maintainers.

And then there's the core problem of securing a system. It's a problem who's
solution has a variable half-life. To prevent a hack, you have to be right
100% of the time, to be a successful hacker, you have to be right once. There
is _no limit_ to the amount of money (good money after bad) that you can spend
securing your systems and you can spend _all_ of it on that and still fail.
It's been my experience that when defining success is difficult, and the
amount of money that is required to be spent to achieve success has a large
range, the amount of money spent will be the lowest amount suggested by the
first person who can convince management that their solution is "good
enough"[4]. It's also easy to look at logs, see a bunch of dropped
packets/thwarted attacks and jump to the wrong conclusion that "our defenses
are working just fine as they are[5]" rather than "wow, we're under constant
attack, unrelenting attack!" Of course, if your physical home was attacked as
much as your web application, you're more likely to put up a stone wall/fence
rather than be thankful that nobody has figured out the out to breach the
5-tumbler dead bolt that anyone with an internet connection can learn how to
breach.

All of that said, it hurts to write this. I'm from the metro Detroit area.
StockX recruits like _crazy_ over here and as a result I have a number of
friends who work at the company. At least of the people I know, they've got
some great developers over there[6] -- we've all had more than a few hundred
conversations about best practices around password handling (frequently
centered around "don't if you don't have to").

[0] Sure, they could be logging every dropped packet, but even then.

[1] Not sure how long StockX has been around but they employ a lot of my
friends.

[2] As far as anyone knows. It sounds like the breach was discovered not by
internal monitoring but by the existence of credentials for sale.

[3] I've had my account credentials published several times, I've had my SSN
published publicly on a web site (in 1998). I'm actually surprised I have had
little in the way of attempts on my credit.

[4] We salted our hash and have appropriate ACLs set up on the database. Our
application firewall prevents all but our corporate IP and the web host from
attaching to the database. Sure, there's a lot of IP addresses that exit that
proxy. We also use MD5, but the salt protects us from rainbow tables and the
other protections should add enough layers to the onion. I mean, after all, an
attacker will just move on to an easier target when they hit (pick one of the
three defenses).

[5] That ranks right up there with "That's what we have business insurance
for!"

[6] Can't pick on them too much; all of them are recent hires and would have
been unlikely to have the authority to do much about it (or even the knowledge
of the code-base required to identify that anything had to be done)

------
maxdata
I very nearly worked there in their engineering department, but once I got
through the initial HR interview into the technical stuff, there were so many
red flags that I got outta there as soon as I could.

A few higher level people who were all let go with me ended up going there,
and having met up with them a few times, I've heard some absolute horror
stories about everything ranging from dev workload, to security, to extremely
unqualified devs being hired to fill seats.

I'm not surprised by this in the least, and frankly, I'm surprised it's not
worse.

~~~
PopeDotNinja
I had a similar experience, but I made the mistake of taking the job. I spent
several months in denial about how smart people who act so... not smart. At
one point, I asked the CTO for guidance on how to work with the team architect
whose feelings I kept hurting. For example, I wrote a constructor for a class,
and the architect asked me what "def initialize" was for, and got upset when I
asked if they knew how OOP in the language worked. Another time they asked me
for help on a weekend to figure a null pointer error on a machine that wasn't
running the software they were trying to debug, and got upset when I pointed
that out. I brought up both examples to the CTO. The CTO pointed out that the
Architect had 10 more years of experience than I did, and while I might be
right, I probably wasn't. Then the CTO said (direct quote) "you have delusions
of grandeur about your technical abilities" to my face. I started looking for
a new job immediately after that conversation. After I left, I heard the
project got canceled, the architect got promoted, and they were hiring devs
straight out of bootcamps because they couldn't attract anyone with
experience.

~~~
pgm8705
I used to work closely with Quicken Loans and other FoCs and can attest that
this behavior is commonplace. There is this strange culture within the Family
of Companies where non-tech leaders think that tenured Quicken engineers and
tech people are these sort of super-geniuses. Many years back I was a part of
a company in the Quicken led start-up space. We were often "encouraged" to
meet with Quicken or FatHead senior engineers for advice. One time I
reluctantly agreed and met with "the best programmer in Michigan" who's first
piece of advice was:

"Delete your app and start over. Ruby on Rails is trash. Real programmers use
C#. With C# you can create libraries that you can reuse across all your apps."

~~~
jnbiche
While I don't agree with the way he spoke, was C# one of the de facto or
explicit in-house languages of the company, and Ruby was not? If so, he may
have been referring to the fact that the company already had many libraries in
C# that you could use. Plus, if C# was one of their areas of expertise, it's
typically best to use that as opposed to a new, unfamiliar language unless
you're explicitly testing out a new approach.

Also, in general, for programming in the large, many experienced programmers,
having worked on multiple large-scale software projects, tend to prefer
statically-typed languages. We've found by experience that in such large-scale
systems, major refactorings are much, much smoother and feasible in
statically-typed languages (although unit and integration tests are certainly
still needed). And a whole class of errors are eliminated.

I don't think RoR is trash, but if you were starting a large program that
would be used across multiple departments whose in-house language was C#, it
was probably the right call to suggest switching to C#.

Please note: I'm not a C# programmer, and have never done any work in C#
(although I've worked many dynamically-typed and statically-typed languages,
and have a clear preference for the latter for large-scale software projects).
So this isn't something I have any personal investment in.

Finally, I do get it, working for these types of companies is misery for most
programmers. I've worked in such companies. But in this case, the senior may
have had a good point.

~~~
gruez
>While I don't agree with the way he spoke, was C# one of the de facto or
explicit in-house languages of the company, and Ruby was not? If so, he may
have been referring to the fact that the company already had many libraries in
C# that you could use. Plus, if C# was one of their areas of expertise, it's
typically best to use that as opposed to a new, unfamiliar language unless
you're explicitly testing out a new approach.

[...]

>I don't think RoR is trash, but if you were starting a large program that
would be used across multiple departments whose in-house language was C#, it
was probably the right call to suggest switching to C#.

According to the parent comment, he was working at a startup that was in the
"Quicken led start-up space". My interpretation is that Quicken was acting
like an incubator, and he isn't working in quicken, and so the engineering
teams are separate. Therefore I don't think organizational inertia applies
here.

~~~
pgm8705
This is correct. Our company was under the Quicken "Family of Companies"
umbrella, but we were a 5 person start-up building a web app unrelated to
Quicken.

------
harrygallagher4
I got an email 3 days ago asking me to reset my password due to "system
updates" and initially assumed it was just a phishing email. Since gmail is
pretty aggressive about filtering those I looked into it more and realized it
was genuine which made me even more confused because I couldn't imagine what
sort of "system updates" they could've done that would require a password
reset for all users. I kind of assumed they were covering up a breach so I
wasn't at all surprised to see this headline. How scummy.

~~~
aarbor989
I thought the same thing when I received it. However, at the end I gave them
the benefit of the doubt that perhaps they were migrating from one hashing
algorithm to another and decided to just reset everyone's password in the
process. Lying about the breach entirely is so shady

------
hacker_9
> The stolen data contained names, email addresses, scrambled password
> (believed to be hashed with the MD5 algorithm and salted), and other profile
> information — such as shoe size and trading currency. The data also included
> the user’s device type, such as Android or iPhone, and the software version.

The serious tone of this article made me double check if this was April 1st
when I read this paragraph. The stolen data is shoe sizes? MD5 hashing for
passwords isn't ideal, especially combined with email addresses - that could
lead to some email accounts being accessed if people use the same password for
everything. The article seems to not really give much attention to this
though, not clear if the author even realises this is the main problem.

~~~
adrr
We need to start charging companies with criminal negligence if they are not
using secure password hashing algorithms. People reuse passwords and this leak
puts other companies at risk.

~~~
whatshisface
That would be a civil case prosecuted by the other companies, not a criminal
case.

~~~
kokowawa393
It should be criminal.

~~~
gruez
Good luck proving that a "reasonable person" shouldn't have done it. Most
people on HN probably do, but I wouldn't be surprised if everyone coming out
of a 3 month coding bootcamp only knew to hash passwords and nothing else. The
other comments in this thread seems to suggest that the company is filled with
bootcamp programmers.

~~~
shakna
As a government body actually publishes advice on this, NIST [0], it may well
be possible to argue for what is reasonable in a court of law.

[0] [https://csrc.nist.gov/projects/hash-functions/nist-policy-
on...](https://csrc.nist.gov/projects/hash-functions/nist-policy-on-hash-
functions)

------
txcwpalpha
> The stolen data contained names, email addresses, scrambled password
> (believed to be hashed with the MD5 algorithm and salted)

This is absolutely atrocious if this is the case. MD5, even with a salt, can
be cracked in a matter of seconds even with the most basic hardware. MD5
hasn't been an acceptable password hashing algorithm for at least a decade
now, and StockX was created in 2015, long after the creators should have known
better (sadly, despite this, an absurdly high number of companies still use
MD5 for pass hashing).

These passwords, hashed with MD5, might as well be considered to have been
stored in plaintext.

~~~
xwdv
Sounds alarming, but not true. If you don’t know the salt, you are not
cracking an MD5 password on _basic hardware_. You are probably not cracking
the password in any reasonable time, period.

And when you have a unique salt per user, that’s basically game over.

~~~
txcwpalpha
Look up benchmarks for cracking MD5 hashes. You can crack an MD5 hash of an
average length password (and StockX only requires an 8 character password),
_even with a salt_ , within seconds with a single consumer grade GPU. A hacker
group with any serious setup for hash cracking has almost certainly already
cracked most, if not all, of these hashes.

~~~
ryanlol
Unless you presume that you know the salt your comment is utter nonsense.

The fact that the article says “believed to be” strongly suggests that things
are not as simple as they’re “believed to be”, because if the passwords were
easy to crack that’d be _trivial_ to prove.

~~~
alasdair_
>Unless you presume that you know the salt your comment is utter nonsense.

Usually the attacker will also know the salts in a breach of this type, unless
the company did something clever with the salts (doubtful since they used
MD5).

~~~
ryanlol
The fact that the article says “believed to be” strongly suggests that things
are not as simple as they’re “believed to be”, because if the passwords were
easy to crack that’d be trivial to prove.

We don’t know how the passwords are hashed. All we have is a journo
_guessing_.

------
throwawayfoc
I used to work at a Dan Gilbert Quicken Family of Companies company, and I am
not the least bit surprised by this, especially the abysmal choice to use MD5.
Let’s just say, the engineering chops across the family of companies is...
mediocre, at best.

------
40acres
Discussions regarding use of MD5 hashing is missing the point, Capital One and
Equifax had plain text data exposed. The hashing strategy is irrelevant.

~~~
txcwpalpha
"Discussing one company's bad security is irrelevant because a totally
separate company had even worse security"?

I don't see where you're coming from at all.

~~~
UncleMeat
There are two topics that are HUGELY overrepresented in discussions of
computer security: password complexity and password hashing.

MD5 hashes is not good. But it also isn't catastrophe level security. If you
aren't reusing passwords then the hashing choice doesn't matter since the
system has already been breached. If you are reusing passwords you don't
exactly want to rely on bcrypt hardness to keep you safe.

If I could make the web services I use switch to MD5 hashes and spend more
time on other relevant security posture, I'd very seriously consider that.

~~~
toyg
This is bad advice and you should feel bad.

We _know_ people reuse passwords. This is a non-negotiable threat model for
any user-facing system. Given this, one should make password-decryption as
hard as possible. MD5 is just not good enough by any standard, in 2019.

~~~
UncleMeat
We do know this. And we can also choose how we spend our energy on education
and outreach. IMO, we should be pushing password managers and 2fa as hard as
humanly possible. People can only integrate so much security advice. If I'm
providing guidance for a business that wants to improve the security posture
of its users, I'd tell them to make 2fa integration as easy as possible and to
encourage their users to use password managers before spending time on the
particulars of the cryptographic storage of passwords.

------
bertil
This is every day now, right? Is anyone tracking those regularly? Less the
passwords (haveibeenpwd does that well) than the executives? Does anyone loose
their job over this? What does the email asking the engineer who rung the
alarm to get lost read like?

~~~
rolltiide
Well the reporting of the breaches is more strange than the fact they
happened.

A platform like StockX should be a continual breach, because the information
will let you make advantageous trades and time series against the customers.

Its pretty dumb to even announce a past tense on this as if it was a single
event.

~~~
joshu
a time series of... shoes?

~~~
EpicEng
Most people who use StockX buy limited release sneakers that you can no longer
get via retail. Prices go up and down depending on demand and scarcity (think
eBay.) It may seem silly, but certain designs can go for a lot of money
($1,000+).

------
alephnan
I expect a lot of downvotes for this post from people who have not had
experience working with people in fashion.

Investors should be weary of people from the fashion industry. I say this as
someone who has both a computer science degree and a fashion design degree,
and 90% of my friends were in the fashion industry at some point. Coming from
tech, you'll find people here are much flakier and just unreliable. In the NYC
fashion scene in particular, people have huge egos and they don't always act
out of pragmatism or logic. The tendency to keep up appearances manifests
itself in many ways. Look at Barney's, it appears great on the outside, but
recently considered bankruptcy before receiving a capital injection.

Recently, I went into one of the top streetwear brands in the world, a staff
member tried to start a fist fight with me after a piece of paper fell out of
a hat, and I refused to pick it up and told them to screw off after the guy
tried to disrespect me in front of my girlfriend. I've never been to a retail
store where a staff member told a customer "meet me outside p___y", but that
is the nature of streetwear culture in NYC. In case you aren't aware, these
streetwear stores in NYC have BOUNCERS. Let that soak in. They're just
accustomed to bullying customers because people are so desperate to buy
clothing that they are willing to put up with the nonsense. They particularly
like to single out mainland Chinese who don't realize (or maybe don't care)
when these staff are disrespecting them, and, since I'm Asian, the guy who
picked the fight thought I would not stand up for myself. If you want more
examples of ridiculousness of streetwear, search "ym bape compilation" on
YouTube. This dude loves the clothing brand called "Bape" and goes around and
assaults everyone he sees wearing the brand Supreme.

There's economic demand and money to be made here, but just know the
demographic you're dealing with. The customers and investees are of the same
thread. I'm looking to start a fashion tech company myself, and aren't
intimidated by potential competitors considering how disjoint these two worlds
are, both network-wise and culturally. The typical engineer won't see the
value of all this vain-ness, and people in fashion business aren't always the
most reasonable people. A UX designer I work with recently told me a story of
how they were redesigning a website for a top fashion brand, and the brand
requested they make the shopping experience as UN-usable as possible and
difficult for people to actually make a purchase (but it works I guess). At
any rate Investors, find a leader who can bridge that gap while still being
able to attract engineering talent.

Don't get me wrong, you can fund leaders in the FashionTech who are completely
unreasonable, and even incompetent, but the company will still do well since
product-market fit and demand will outstrip all other factors, until something
like this happens. One of my professors owns a set of retail stores in NYC
that was acquired by one of the largest clothing manufacturers in the US, but
they did not know a single thing about accounting, business operations,
engineering, and non-artistic things. Super unreliable, super unprofessional
professor but they had a really good intuition for branding.

 _Luxury brands, corporations (Adidas, Nike), and the LVMH conglomerate are a
slightly different story_

~~~
omarchowdhury
Whose hat did the paper fall out of?

~~~
alephnan
The store's. I put the hat on while I continued shopping. I had every
intention of buying the hat, but apparently the paper fell out. The employee
came to me fist-clenched, with a fighting tone. I'm sure you're skeptical and
think there's two sides to every story, but I don't want to expand too much on
the story for privacy and retaliatory concerns. I invite you to come to SoHo,
NYC and visit a few stores if you have any doubts.

~~~
omarchowdhury
I've lived in NYC all my life and am familiar with the scene.

Did they ask you nicely (initially) to pick it up? What was the tone of your
response? These are two missing clues on how this might have gotten started.

~~~
alephnan
> Did they ask you nicely (initially) to pick it up?

Nope. It was "Are you going to buy that?" and other rhetorical heckling
questions. I said "yes" and didn't give them the attention they want.
Eventually, looks over to my girlfriend, turned back to me, and commands me
"go pick that up". If they simply asked "did you drop that?" I would have
immediately said "oh, sorry, I didn't see that" and done so.

~~~
omarchowdhury
I see. Typical territorial domineering of an immature ego.

~~~
alephnan
In every other sane retail environment, this should have never happened at
all, regardless of how rude the customer is, unless they're being racist or
intentionally degrading or belittling the staff.

------
conroydave
arent they a quicken family company?

~~~
werber
Yep

------
albertshin
Can't believe something like this (md5) wasn't flagged during tech diligence
by any of their investors (GV, General Atlantic, Battery, DST Global, etc)...

------
CPLX
The interesting part to me was that this happened in May, and they closed
$110MM in funding a month ago. That could explain quite a bit.

