
Dreamhost asking users to change passwords after database breach - hopeless
http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/
======
RossDM
I'm a Dreamhost customer and haven't received any emails regarding this yet.

~~~
andfarm
We're not quite done investigating the issue and resetting passwords; once
that's done, though, we will have a mass email going out.

~~~
shill
FYI: The forgot password feature does not seem to be working at this time.

~~~
dredmorbius
If you log in with your Web Panel password, you can change an accounts'
password without needing to know the old one.

The DreamHost engineer who's been commenting here says the web panel passwords
haven't been compromised (I changed ours anyhow).

------
djenryte
This is pretty bad. I've received password reset emails from dreamhost in the
past and the passwords are in plain text...I just renewed two days ago too.

~~~
ehsanu1
Surely they don't send you your old password, but a freshly generated one?
Then they could still be hashing them after emailing you.

~~~
andfarm
Embarrassingly... no. Our login/authentication system was written in 1999, and
it shows -- we store panel login passwords using symmetric encryption, and
send out the decrypted password when you request it.

Getting this fixed was already on our to-do list. This incident has moved it
up to near the top of the list (competing with a few other security-related
tasks).

~~~
milkmiruku
I have always been bothered about cpanel passwords coming through in plain
text. To confirm, is this the same storage system with mail passwords also?

Shell passwords - they're hashed, but are they salted? If not, can they be in
future?

Thanks for your time.

------
nabilt
I have limited web development experience so maybe someone can fill me in. Why
is securing a web server so difficult? I understand they are complicated
systems with many technologies, but large companies with lots of smart
engineers can't seem to solve this problem. Can most of these hacks be
attributed to user error?

~~~
krobertson
This is based on my own biased opinion...

It can happen for a number of reasons and isn't necessarily with securing a
webserver.

Sometimes security issues get that "well, it isn't an issue right now" view
and get put in a backlog and never resolved. Other cases it is a lack of good
code reviews or security reviews. Or inexperienced developers (at least
inexperienced with that kind of security).

From a Dreamhost employee (<http://news.ycombinator.com/item?id=3491643>),
that code was from 1999. Could have been any one of the above.

The truly sad part is the number of examples of this kind of thing blowing up
yet companies are still making the same mistake. At what point does someone
think of checking their own apps password practices to ensure they aren't
next?

~~~
dredmorbius
One practice that's starting to help, a bit, and it pains me to say it, is
security audits. There are a few standards emerging for SAAS services,
especially at the B2B level. And as startups (or non-startups) start
intersecting with large, regulated, publicly-held businesses, these
requirements propagate, and with audits, receive multiple reviews.

The process is far from perfect, and many of the standards are laughably lax
(and yet ... they still aren't met). Because they're standards, the requests
are fairly uniform. A number of issues regarding Dreamhost have been on my own
back burner for months, and I'm finally getting that Round Tuit this
afternoon.

Pushing for good, solid standards would be a net benefit. That's the silver
lining here.

~~~
krobertson
Mentioning standards and security audits makes me cringe in their current
state. Certainly agree with you that they'll help, but many are draconian.
Things like physical access lists when you use cloud services... or policies
on tape backups. :)

~~~
dredmorbius
There's a lot of dumbth in them, to be sure.

I'm hoping that they'll evolve toward slightly more sanity than insanity.

Some institutional laws suggest that this is wildly optimistic on my part.
Security theater exists because bureaucrats and legislators want to appear to
be doing something. Vendor-based solutions are mandated because vendors have
political clout. Effective tools are difficult to deploy in real-world
scenarios -- filled with failing devices, intermittent communications, poor
training, and worse end-user understanding.

But give me my dream. Just for this one night.

------
hopeless
From the comments, it seems the passwords were hashed but they're (rightly)
still concerned.

Unfortunately, there still seems to be some confusion about when/if the
password reset has taken place, and whether users should change them now, wait
for Dreamhost, or both (and then update them again after the reset!)

~~~
andfarm
We're currently in the middle of resetting passwords, machine by machine. If
your old password has stopped working, that means we've gotten to the machine
that you're on, and you're safe to reset it through the panel.

Panel passwords are safe, BTW. The only passwords that we believe may have
been compromised are UNIX user passwords (e.g, FTP/SFTP/SSH users).

~~~
Kambria
Unix passwords are _Never_ safe and never will be.

I don't get you guys...

------
muraiki
I just got the email from Dreamhost at about 3:30AM EST. I checked out their
status blog and found something interesting: over the past month a good number
of people have had malicious .htaccess files and scripts inserted into their
Wordpress installs. That would include me.

About a month ago one of my Dreamhost sites was hacked. An .htaccess file was
modified to insert some PHP scripts that provided a backdoor to the site and
that also embedded ads. I see now that I'm not the only one who has had this
problem. When I reported this at the time I submitted as much information as
possible but only got a canned, autogenerated response. Dreamhost ran an
automatic scan on my site and IIRC it did pick up the offending file, but by
the time that had been done I had already found it. The particular backdoor
was c99shell.I. As I've read more comments, people have had .htaccess files
modified who weren't using Wordpress, so it's not a Wordpress problem.

This leads me to ask: Is this compromise related to the hacks that I and other
users have experienced? If so, how long has the password database been
compromised?

Thankfully I use GMail for email, so at least that level of personal
information wasn't stored on their servers. But this makes me wonder just how
long someone has had free reign to access many Dreamhost accounts.

Needless to say, I will be switching hosts and will never recommend Dreamhost
again. I guess you get what you pay for, and if I had been able to speak with
an actual human being about my site being hacked maybe they could have
connected the dots. Furthermore, they've had this weak password system since
1999?

Edit: Here's a link to the status blog post:
[http://www.dreamhoststatus.com/2012/01/20/changing-
ftpshell-...](http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-
passwords-due-to-security-issue)

Edit2: I just remembered that I got a strange call from a friend a few months
ago who said that when he tried to access my site his corporate firewall had
blocked it because of the presence of pornography. This is a church's web site
we're talking about, so of course there was no porn on it. This is the same
site that was later more overtly hacked in December. Apparently the initial
hack only inserted text when a crawler hit the site, so it was an SEO hack. In
December, the text became visible to regular users of the site. This post has
more detail: [http://www.dreamhoststatus.com/2012/01/20/changing-
ftpshell-...](http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-
passwords-due-to-security-issue/#comment-265213)

~~~
muraiki
After a few emails with DH support, I discovered that there was a file created
by another user in two of my accounts. Somehow the permissions for two
directories had been set to 777, and a compromised user put a trojan in them.
I think that I jumped into conspiracy theory territory above, and
misrepresented Dreamhost's security and response as a result of that. The
combination of my emotions and my inexperience with dealing with hacking on
web hosts led to an incorrect judgment.

------
malandrew
Do only shell and ftp passwords need to be reset or does the main account web
panel password need to be reset as well?

Ask because, we don't use shell/ftp passwords and I'm not sure if we are
affected or not.

~~~
andfarm
Just shell/FTP passwords. As noted above, panel passwords are fine.

------
mbrzuzy
I am absolutely not shocked at all. I got off their service years ago when
they had all their FTP accounts breached.

------
joering2
Hostgator is no better. Talked with "technical support" couple weeks ago;
decided to save the chat:

Welcome to GatorChat! You are being connected to a representative in our
Technical Support department right now.

(01:33:17 PM) Joe: hi there.

(01:33:26 PM) Nathan: Welcome to Hostgator LiveChat. My name is Nathan. How
may I assist you today?

[...]

(01:38:30 PM) Joe: never got password

(01:39:20 PM) Nathan: Its the same as your billing password.

(01:39:46 PM) Joe: YEAH

(01:39:48 PM) Joe: I JUST GOT IT. WHICH MEANS YOU GUYS KEEP PASSWORDS IN PLAIN
TEXT

(01:41:01 PM) Nathan: Correct.

(01:41:26 PM) Joe: it would be good to have more security in place :)

(01:42:44 PM) Nathan: What is not secure about what we do?

(01:42:58 PM) Joe: the password. kept in plain text

(01:43:46 PM) Nathan: Its a very secure password, dont give it to people and
you will be secured.

(01:44:16 PM) Joe: no. you dont understand. i'm saying: keeping password not
encrypted or hashed on registryrocket.com part is not secure

(01:45:21 PM) Nathan: No one should know your billing password Joe. We see no
reason to create a new password for that.

(01:45:39 PM) Nathan: I apologize.

(01:46:08 PM) Joe: no problem, but you still dont get it. if hackers hack into
registryrocket, they have a database of domains and passwords for each domain.
in plain text!

(01:46:31 PM) Nathan: Okay, you can change the password if you want.

(01:46:58 PM) Joe: but what if I change it today, and hackers hack into
registryrocket 2 hours later?

(01:47:20 PM) Nathan: Then we will change the password. I dont understand how
hackers would randomly guess your password Joe.

(01:47:59 PM) Joe: they dont need to randomly guess my password. my password
is stored in registryrocket database in PLAIN TEXT

(01:48:45 PM) Nathan: Please show me where you can see it.

(01:49:10 PM) Joe: I cant!! I did not hack into registryrocket

(01:49:36 PM) Nathan: Okay, no hackers are getting into our database Joe. I do
not understand how they would get your password.

(01:49:44 PM) Joe: but if someone WILL hack into registryrocket AND copy the
entire database of users/passwords, then those passwords wont be encrypted or
hashed, they will be in plain text

(01:52:20 PM) Nathan: Joe, no our passwords are encrypted with MD5.

(01:52:29 PM) Joe: impossible!!

(01:52:34 PM) Nathan: No one is going to hack into registryrocket.

(01:52:35 PM) Nathan: Okay.

(01:53:01 PM) Joe: ?? how can you encrypt password with MD5 and within 10
seconds from clicking "remind password", I got my password in my mailbox??

(01:53:54 PM) Nathan: Joe, our passwords are secured in a database. Please do
not worry about us getting hacked. We have it under control.

(01:54:16 PM) Joe: you misleading me.

(01:54:40 PM) Joe: passwords CANNOT be properly encrypted in the database AND
the same time you sending me password 5 seconds after I clicked "remind
password"

(01:56:07 PM) Joe: even for very powerful computer it would take much more
than 5 seconds to "decrypt" (or should rather say: crack) MD5-encrypted
password

(02:00:35 PM) Joe: any news, Nathan?

(02:01:28 PM) Nathan: Our passwords are not MD5 encrypted, but they are
encrypted and its done by Enom.

(02:03:06 PM) Joe: (01:52:20 PM) Nathan: Joe, no our passwords are encrypted
with MD5.

(02:03:15 PM) Joe: whats "Enom" ?

(02:03:20 PM) Nathan: I apologize I made a mistake. Enom is who we sell our
domains through.

(02:03:45 PM) Joe: ok, what do they use to encrypt passwords??

(02:03:55 PM) Nathan: I would not know.

(02:04:11 PM) Joe: I tell you: NOTHING

(02:04:54 PM) Joe: thats why most websites you CANNOT retrieve your old
password, you can only reset it.

(02:07:22 PM) Nathan: What can I help you with Joe? Do you need the password
to login?

(02:07:48 PM) Joe: I was just wondering how secure my vulnerable data is.

(02:09:22 PM) Nathan: We have a secure database, we have not been hacked in
the past and I don't think we will be hacked anytime soon.

(02:09:41 PM) Joe: ok thanks.

Current Rating: 12345

~~~
guywithabike
I'm not sure what point you're trying to get across by posting that entire
chat log aside from the fact that you're kind of a dick to the support person.

~~~
wizard_2
He wasn't super polite and didn't try to educate him, but I wouldn't call him
a dick. There probably isn't any other way to communicate this message except
via support.

~~~
aspensmonster
I have worked as a "Chat Tech"/"Junior Administrator" --tier one support-- for
a little over a year now for HostGator. I typically do phone calls rather than
chats but the experience across each is similar. That said, none of this
reflects the official endorsement or opinions of HostGator.com LLC.

The type of concern "joering2" has is obviously not something we handle at a
first encounter level. It's not even a "support" issue in the normal sense.
It's an infrastructure issue. Support chats and phone calls aren't going to
end 15 minutes later with me having: designed, developed, tested, and deployed
an entirely new password storing, reset, and re-send mechanism. This is true
for any company that one has a widespread concern with regarding their
policies and infrastructure. I may not approve of Microsoft's policy of
forcing me through a "Customer Retention Specialist" when I wish to cancel my
excess Xbox live subscription, but haranguing the poor girl I'm speaking to
about company infrastructure and policy issues is a waste of her and my time.

That said, for HostGator, if you want to quickly reach someone regarding this
type of concern, here's an email address: feedback@hostgator.com.

With regards to "registryrocket.com" I can assure you that it is and has been
the bane of every tech's existence here at HostGator. Some explanation:
HostGator used to, and still does, resell domains through eNom. When you visit
"registryrocket.com" -- visit <http://hostgator.com/domains> in order to get a
working session -- what you are seeing is eNom's idea of good default site
design and then their idea of user experience. The user experience with this
tool is... lacking. To put it this way: I've never had a call where a customer
told me how easy and great purchasing a new domain through
hostgator.com/domains was.

Moving on, this is not a system that we directly administer and control, so
far as I know. It is a service hosted on and developed by eNom, as stated on
their own page describing the product: "We take care of the website hosting,
merchant processing and other eNom services."
(<http://www.enom.com/help/faq_rr.asp>). This, the poor user interface for
managing domains, and the general frustration of having to work with an
external registrar rather than being in complete control, has led to us
forming our own registrar: "LaunchPad." We're already registering new domains
under this tool and getting a live web interface for it has been a huge
request both from our customers and us lowly tech and sales staff. Here's to
hoping the hiring push for development talent pays off :)

------
ahi
This was a nightmare. I was trying to get some urgent changes committed for a
client that needed it this afternoon. Just about to commit around 14:00 est
and Comcast takes a dive. Comcast support says they'll have it fixed by 5. Not
great, I wanted to be done with work early on a Friday, but client is on the
west coast so it will still be okay. Comcast comes up and ... shit, can't log
in to the server. Try a few dozen variations of my password, then scour the
web for an hour looking for hints if I'm locked out or something. Finally
decide to check Dreamhoststatus.com. Fuckwits. Only sysadmin has access to
panel to reset my password. He's sick and only checking IM sporadically.
Finally get logged in and commit at 23:45pm est. Comcast+Dreamhost fucked me
and my client real good today. Not much I can do about Comcast, but our
migration off of dreamhost is getting pushed up to the top of the list.

~~~
dredmorbius
Sounds like it's time for some process review.

Backup systems admin.

Key passwords available in a shared (but encrypted) repo.

Multiple data links -- tethering through your cell phone or heading to the
local cafe or library in a pinch.

~~~
davebanham
<http://passpack.com/>

Best password sharing/collaboration tool we've found.

