
When a bad day gets worse – getting hacked twice in one day - killwhitey
http://chrishateswriting.com/post/84931829578/when-a-bad-day-gets-worse-getting-hacked-twice-in-one
======
knowtheory
Having been through open sourcing an app platform repo
([https://github.com/documentcloud/documentcloud](https://github.com/documentcloud/documentcloud)
) that wasn't explicitly created w/ open sourcing in mind, moot has my deepest
sympathies.

Moot left out the most sensible response to Mistake #5:

I went through a rushed 8hr rebase to remove all of our keys, certificates,
cookie secrets and the like before DocumentCloud's platform was opened. Two
days after, a friend asked me why I hadn't simply revoked and replaced all of
our secrets to the sound of me facepalming repeatedly.

It is always much easier just to revoke and replace your secrets. Doing so
should always be relatively easy for you to execute, and having a chance to
practice that is also always good.

~~~
moot
Good point -- frankly I'd suggest doing both (clean repo and revoke/replace
the secrets). The developer in question simply overlooked the issue, and since
we were busy spinning down the company, nobody else caught it.

~~~
stouset
Also, just swear of ever committing secrets to a repo to begin with. It's a
bad idea, period.

~~~
StavrosK
There are many times when that isn't doable. Fortunately, you can swear off
ever committing _plaintext_ secrets to a repo. I use git-crypt:

[https://github.com/AGWA/git-crypt](https://github.com/AGWA/git-crypt)

------
patio11
Of related interest to HNers: Admin consoles getting owned up is a fairly
frequent way that even security-conscious shops lose their web apps /
databases / networks, because one often does not code defensively against
malicious actions executed by "trusted" users. You can mitigate this quite a
bit by putting them on a private network interface and requiring the admins
VPN in to use them.

~~~
StavrosK
As an easier alternative to set up, you can require SSL client certificates.

~~~
selmnoo
Speaking as someone who got owned even when using SSL client certificates
(made mistakes setting things up, long story), I think patio11's suggestion is
better. Put it on a private network, seriously it isn't that hard.

~~~
StavrosK
That's too bad. It's not hard to set up, it's just harder to use.

------
elithrar
> "Mistake #3: Unescaped SQL query..."

This should be "failure to use parameterised queries." The term 'unescaped'
may lead some to think that simply escaping characters is sufficient.

~~~
userbinator
Are you implying that there's a way to get past escaping...? That's a new one
to me. I thought escaping was precisely so you can put _any_ value in a string
and have it still work.

~~~
mnw21cam
Yes, but you had better be 100% sure that your escaping function is completely
reliable, and the server hasn't introduced some new syntax since you wrote it
that you aren't escaping properly.

I would trust parameters much more (although I have used proper escaping in
the past).

~~~
throwaway0010
A few popular database drivers use escaping under the hood for parameterized
query arguments. mysql2 ruby gem (and any rails stack on top of it) for
example.

~~~
astrodust
They probably do a better job of it than you do, so let them handle it.

------
danso
The disclosure of these mistakes is both kind of relieving, in that the errors
were obvious in _nature_ , and terrifying, in that even though they seem
"basic", they were still committed by a dev as experienced as moot. This would
be a good post to create a checklist from, if nothing else.

I wonder how old the code was for "Mistake 2"? SQL injection is something most
sites have patterns/frameworks to prevent, but unless the site started out
with such practices...the code that was written when the site was just a fun
side-project might go unchecked even as the site becomes a well-run project in
its later years.

~~~
moot
> they were still committed by a dev as experienced as moot

I'm not a terribly good programmer, and have been very hands-off with 4chan's
code for quite some time. I still direct development and am responsible for
the servers/sysadmin tasks, but there are far more talented developers out
there than I. In the case of Canvas/DrawQuest, I was 100% uninvolved on the
tech side.

But again, in both cases I accept full responsibility for the breaches since
ultimately it's up to the project leader to ensure these things don't happen
-- even if not active on the technical side.

> I wonder how old the code was for "Mistake 2"?

Very new. It was in a once-off file that we used to quickly pull stats about
reported posts, which a) shouldn't have been on a domain without HTTP auth, b)
should have been deleted long ago, c) shouldn't have had a bugged auth check
or injection vuln to begin with.

~~~
noir_lord
one-off files have a habit of hanging around and temporary often becomes
permanent.

This isn't to say that you should treat all one-off's and temporary solutions
_as_ permanenent but it is a good idea to audit them periodically.

Storing that kind of metadata about code is something I've often pondered we
could do better, putting it in comments is a nasty hack, storing it away from
the code means it instantly gets out of date, commit messages are not a good
place to put that stuff either.

I've never come up with an elegant solution even in my head but it would be
something I'd love to have for my own uses.

~~~
jeffasinger
Agreed.

Someone once came into my office and asked why the email export feature had
stopped working. Once they described going to test.php, I realized that about
a month ago, I had migrated our version control system to a new deployment
system, and hadn't included test.php, what I thought to be an insecure relic
left hanging around by a predecessor.

Things that end up on a live web server are one offs much less than the people
who make them think.

~~~
noir_lord
nice and eerily familiar.

Codebase I once worked on, I found a /csv route that dropped the _entire
customer database in CSV format_ and /route_csv that enumerated all the routes
the application had including admin and cron routes :| (denial of service by
spamming the cron routes that did no access checking was the _least of it_ ).

When I checked the commit date it was 19 months ago..and in production for 17
months :|

The midden and the windmill fully hit each other that day.

------
LukeB_UK
Thanks for this Moot, I love the fact that people are being more and more open
with how they were hacked. These post-mortems help others identify areas where
they may be vulnerable which in turn helps the community as a whole.

Sorry about you getting hacked twice in one day though. I hope things have
been better since then!

------
matthuggins
I'm not familiar with DrawQuest, but why not sell it "as is" instead of
shutting it down completely? Especially when you have 100k monthly actives?

~~~
adventured
If I'm not mistaken, they took on $3.6 million in venture capital, and that
might be the complication in deciding exactly what to do with the product
(that is, their partners in it may disagree with selling it for $500k or
whatever it might fetch).

~~~
moot
Our investors have been incredibly supportive -- especially these past few
[rough] months.

The biggest concern I have is not so much the price it would fetch (which is
probably not much given recent events), but rather finding a suitable steward
for the community.

~~~
johnnymonster
I'm pretty sure there was someone who could step up. Product owners usually
over value their involvement and thus never give up the reigns, going down
with the ship instead of letting someone else take over.

~~~
jmathai
I don't know. Community members usually undervalue the product owner's
involvement as well.

Stewarding a community is a difficult job. You need more than just passion and
I imagine finding one and feeling confident that they'll do a good job is
quite difficult.

------
dkarapetyan
Security is a usability nightmare. All the instinctive and usually lazy things
people do are just bombs waiting to explode. I don't know why this continues
to be the case. Most platforms for web development should have taint checking
turned on to the highest setting by default. SQL access points should scream
in your face every time you use a bare query. With a bit more you could even
get developers to do the right thing in terms of salts, passwords, and hashes.
I think it is clear that pointing people to OWASP and asking them to read
their security articles before bedtime is not as useful as it could be.
Library writers also need to be involved and use safe options by default
instead of opt-in.

------
coldcode
Shows you that strict controls over your AWS credentials are worth whatever
hassle they might be. 100 huge instances spun up can cost you big time.

~~~
vacri
AWS also lets you put in billing alerts, letting you know when you hit the
threshold.

~~~
willvarfar
(Not an AWS user, so don't know how this works)

Can someone with the leaked credentials turn off the billing alerts?

------
driverdan
The list of vulnerabilities in 4chan read like a standard web hack CTF. I'm
surprised a site as big as 4chan has multiple OWASP top 10 issues, especially
after the source was leaked. Storing credentials in client side cookies? Who
wrote that shit?

~~~
redthrowaway
4chan has always operated on a shoestring budget. When you have a single dev,
there's nobody around to catch stupid mistakes. I just spent two days trying
to track down a bug only to realize I'd been adding elements to an array in a
braindead manner. Stupid happens.

------
riquito
Nobody mentioned the fact that _you don't put secrets in a versioning system_.
.gitignore the configuration files and have example files available, or use
configuration files that can import the missing parts (the secrets) from an
ignored file. Another way is to put the secrets in environment variables
outside the project. In any way, if they're missing provide a meaningful
message to the developer. I do understand that you must keep some of these
secrets available somewhere: just don't version them.

~~~
ansible
I can understand the desire to version control your secrets.

The bad idea was not to maintain the source in a separate repo that could
easily be released.

If you want to keep the checkout process simple for your developers, maybe use
git subtree for the overall project.

------
primitivesuave
Some security bad practices, but it happens to the best of us. Glad you got
that figured out though.

I always tell my devs on the first day, "leaving AWS keys in a public
repository is as bad as showing up drunk to work". Mainly because the last guy
to leave our AWS keys in the open was drunk at work.

~~~
johnnymonster
its really sad to see how many AWS Keys are public in github right now. A
quick github search reveals many, many key pairs that were checking as
recently as this week.

~~~
tommu
Amazon are apparently scanning themselves. If they are doing this then they
might figure that the quantity of exposed keys will undermine their
reputation. That is quite something in itself.

~~~
jffry
Amazon AWS support have (at least on here) a reputation for refunding
fraudulent usage that stemmed from compromised keys. If that is in fact a
policy they follow, it's in their best interests to cut down on leaked AWS
credentials.

------
ufmace
This makes me wonder what kind of practice change would be best to help avoid
this sort of thing. Maybe anytime you put in something that you know is hacky
and insecure as a one-off, you should put a note in wherever you would look
regularly, and maybe some kind of reminder too, to make sure that your one-off
is either removed or properly secured reasonably quickly.

Or maybe just a rant on PHP and/or frameworks that make it easy to do the
wrong thing, and hard or time-consuming to do the right thing.

~~~
jaryd
Looks like a good opportunity for an open source tool that scans a git
repository for interesting/private information. Obviously it could be used for
nefarious purposes, but it would be helpful for anyone looking to open source
an existing repo.

Another alternative is to store credentials in environment variables :)

~~~
moot
> Looks like a good opportunity for an open source tool that scans a git
> repository for interesting/private information.

These already exist in droves, which is exactly how our Amazon credentials
were found.

[http://www.itnews.com.au/News/375785,aws-urges-developers-
to...](http://www.itnews.com.au/News/375785,aws-urges-developers-to-scrub-
github-of-secret-keys.aspx)

~~~
jdhendrickson
I wasn't cognizant of the need to include a step in which you scan your own
repo, just as a fail safe. So thanks for posting this.

------
nicpottier
I actually think this is something GitHub should help with. When you flip a
repo from private to public, it should prompt you asking if you want to wipe
out all the version history first.(essentially create a brand new repo in the
background)

Seems like far too common a mistake and something they could help fix once and
for all.

------
Kiro
So Mistake #4 is something you hear over and over again is bad but I don't
understand why it's so bad. If you have the password you can just login anyway
and I don't see how it affected this particular case either. Please enlighten
me.

~~~
mpeg
Because he didn't have the password, he had a hash of the password.

So if the auth system wouldn't have let him login directly with the hash by
changing his cookie, he wouldn't have been able to login as an admin.

~~~
userbinator
Am I right to say this would've been easily prevented by putting the hash(salt
+ password + client IP) in the cookie, instead of just hash(password)?

~~~
jrochkind1
Normally, if I understand right, the point of hashed passwords, is what is
sent from client to server to auth is the ACTUAL password.

Then the actual password is hashed, and the hash of the input password is
compared to the previously stored hash, to see if it matches.

This way, you need the actual password to auth, but the actual password is
stored nowhere by the app, only the hash is -- so the actual password can't be
stolen by getting access to the db or the file system or anything else,
because it's simply not there. But it's what you need to auth.

This is really the whole point of storing hashes rather than original
passwords. Right?

Now, I guess that would mean you'd store the _actual_ password in the cookie.
But that does sound risky, even if the cookies are https only. Which is why
usually you store a _session id_ in the cookie, and auth the session with a
single auth action, not store the password in a cookie.

~~~
mnw21cam
If I could upvote you multiple times I would. I see a lot of confusion on this
matter, even on HN.

When authenticating, _the server_ needs to do a hashing step to compare
against the password database. Otherwise, it isn't a hashed password database
- it's a plaintext password database.

------
pearjuice
moot, why do you refuse to speak to us on 4chan while go happy hand-in-hand
with all these web 3.5 cloud entrepreneurs here? Want to find some investor
friends? Thanks for ignoring your actual friends.

~~~
nsmartt
I strongly suspect moot posts anonymously on 4chan. Posting under moot would
result in his dominating the conversation.

------
SDGT
> Boneheaded cookie auth—we simply stored the bcrypted password from the
> database in a cookie, which was all that was required to pass PHP auth.

Wh... Why... Who...

How was that ever OK?

~~~
oldmanjay
Okay, so I'm guessing this is an example of "feigned surprise" that is
supposed to be considered bad?

I've been on a few sites (very recently) that just stick the password right in
an auth cookie, served via HTTP no less. Luckily they're sweet enough to
base64 encode it for extra security.

~~~
SDGT
Not even feigned.

I would have expected a site like 4chan, the source of all sorts of script
kiddie idiocy, would know better by this time than to stick a password, even
hashed, in a cookie.

But sometimes you piss off the HN bees nest, and downvotes come streaming in.
Reminds me of Reddit.

~~~
lnanek2
I think the original script it was based on didn't even have login per se. You
just occasionally stuck in an admin password hard coded in the PHP to do a one
time post delete or whatnot. It's mostly an anonymous forum after all. As more
moderation functions grew a real login was needed and grafted on. Users still
didn't have a login for a long time and instead had a special part of their
username that was hashed when displayed to prove identity.

------
rushonerok
Putting the hashed password directly in the cookie? That's shameful. A hack
was well deserved.

~~~
jebblue
>> A hack was well deserved.

That's a despicable thing to say. No one deserves to have bad things thrust
upon them.

------
loceng
So simple. Could have been much more painful.

~~~
loceng
I think people who are downvoting this comment are projecting what I meant by
"so simple." There was no judgement on that statement. It was more an
observation relating to that it could happen to anyone, and that they were
"lucky" the person who got into their system wasn't really malicious.

~~~
pbhjpbhj
IMO they more likely consider your initial comment to add nothing to the
thread.

~~~
loceng
So I suppose that's worth the effort of downvoting vs. upvoting comments that
are worthy..

~~~
jtheory
Yes; upvoting just puts the better comments nearer to the top. Downvoting the
fluff comments (with the minimal "effort" of a click while you read past)
actually greys them out, which both discourages people from posting fluff in
the future (with a minor karma hit), and makes it obvious to other readers
that they've made it to the end of the real comments.

