We're doing some tough client love right now on security practices, and having a nice hard wall to push off of, namely, Dreamhost being assholes, in a good way, would be a very Good Thing[tm].
I won't be the only person doing this.
Or do I need to go through all 20 right this moment and change them from their old value to a new value?
Basically do the hackers possibly have access to my ftp accounts or have you already switched my passwords to random strings?
As it was I found out about 4 hours after your first blog post via HN.
We're still hashing out what we're going to do with folks who, last time we instituted a password/process change, wanted a 3-weeks heads-up.
The DreamHost engineer who's been commenting here says the web panel passwords haven't been compromised (I changed ours anyhow).
Also, the fact that you can see your user passwords in the panel has always irked me, and that has gone unchanged even after past breaches. I have been quite happy with their service, but maybe this my queue to leave... :(
EDIT: To their credit, I see this when logging in to the panel: "Due to some unauthorized activity we detected within one of our databases, we have forced a reset of all FTP/SFTP and shell passwords as a precaution."
Getting this fixed was already on our to-do list. This incident has moved it up to near the top of the list (competing with a few other security-related tasks).
Shell passwords - they're hashed, but are they salted? If not, can they be in future?
Thanks for your time.
If you could forward these articles to whoever's working on security, I'd appreciate it (and they're a good read):
At least Dreamhost says the Shell passwords are hashed, which makes sense.
I didn't know that about the plain text dreamhost web-panel passwords though.
For shared hosts it's worse: a flaw that punches through can affect many customers at once. And there's often a larger attack surface because of the complexity of hosting multiple customers in a single system.
Take, for example, Wordpress. It uses a single login into a relevant MySQL database. Plugin code has the same level of access to MySQL as the core Wordpress code. If a there's a defect in either, your MySQL database is wide open to receiving SQL statements.
If you didn't configure the database properly, that wide open pathway now leads to other databases, or to shell access with interesting privileges, and so on.
So the design of wordpress (and in fairness, pretty much all web apps follow a similar pattern) both increases the attack surface and decreases the depth of defence.
It can happen for a number of reasons and isn't necessarily with securing a webserver.
Sometimes security issues get that "well, it isn't an issue right now" view and get put in a backlog and never resolved. Other cases it is a lack of good code reviews or security reviews. Or inexperienced developers (at least inexperienced with that kind of security).
From a Dreamhost employee (http://news.ycombinator.com/item?id=3491643), that code was from 1999. Could have been any one of the above.
The truly sad part is the number of examples of this kind of thing blowing up yet companies are still making the same mistake. At what point does someone think of checking their own apps password practices to ensure they aren't next?
The process is far from perfect, and many of the standards are laughably lax (and yet ... they still aren't met). Because they're standards, the requests are fairly uniform. A number of issues regarding Dreamhost have been on my own back burner for months, and I'm finally getting that Round Tuit this afternoon.
Pushing for good, solid standards would be a net benefit. That's the silver lining here.
I'm hoping that they'll evolve toward slightly more sanity than insanity.
Some institutional laws suggest that this is wildly optimistic on my part. Security theater exists because bureaucrats and legislators want to appear to be doing something. Vendor-based solutions are mandated because vendors have political clout. Effective tools are difficult to deploy in real-world scenarios -- filled with failing devices, intermittent communications, poor training, and worse end-user understanding.
But give me my dream. Just for this one night.
Unfortunately, there still seems to be some confusion about when/if the password reset has taken place, and whether users should change them now, wait for Dreamhost, or both (and then update them again after the reset!)
Panel passwords are safe, BTW. The only passwords that we believe may have been compromised are UNIX user passwords (e.g, FTP/SFTP/SSH users).
I don't get you guys...
About a month ago one of my Dreamhost sites was hacked. An .htaccess file was modified to insert some PHP scripts that provided a backdoor to the site and that also embedded ads. I see now that I'm not the only one who has had this problem. When I reported this at the time I submitted as much information as possible but only got a canned, autogenerated response. Dreamhost ran an automatic scan on my site and IIRC it did pick up the offending file, but by the time that had been done I had already found it. The particular backdoor was c99shell.I. As I've read more comments, people have had .htaccess files modified who weren't using Wordpress, so it's not a Wordpress problem.
This leads me to ask: Is this compromise related to the hacks that I and other users have experienced? If so, how long has the password database been compromised?
Thankfully I use GMail for email, so at least that level of personal information wasn't stored on their servers. But this makes me wonder just how long someone has had free reign to access many Dreamhost accounts.
Needless to say, I will be switching hosts and will never recommend Dreamhost again. I guess you get what you pay for, and if I had been able to speak with an actual human being about my site being hacked maybe they could have connected the dots. Furthermore, they've had this weak password system since 1999?
Edit: Here's a link to the status blog post: http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-...
Edit2: I just remembered that I got a strange call from a friend a few months ago who said that when he tried to access my site his corporate firewall had blocked it because of the presence of pornography. This is a church's web site we're talking about, so of course there was no porn on it. This is the same site that was later more overtly hacked in December. Apparently the initial hack only inserted text when a crawler hit the site, so it was an SEO hack. In December, the text became visible to regular users of the site. This post has more detail: http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-...
Ask because, we don't use shell/ftp passwords and I'm not sure if we are affected or not.
Welcome to GatorChat!
You are being connected to a representative in our Technical Support department right now.
(01:33:17 PM) Joe: hi there.
(01:33:26 PM) Nathan: Welcome to Hostgator LiveChat. My name is Nathan. How may I assist you today?
(01:38:30 PM) Joe: never got password
(01:39:20 PM) Nathan: Its the same as your billing password.
(01:39:46 PM) Joe: YEAH
(01:39:48 PM) Joe: I JUST GOT IT. WHICH MEANS YOU GUYS KEEP PASSWORDS IN PLAIN TEXT
(01:41:01 PM) Nathan: Correct.
(01:41:26 PM) Joe: it would be good to have more security in place :)
(01:42:44 PM) Nathan: What is not secure about what we do?
(01:42:58 PM) Joe: the password. kept in plain text
(01:43:46 PM) Nathan: Its a very secure password, dont give it to people and you will be secured.
(01:44:16 PM) Joe: no. you dont understand. i'm saying: keeping password not encrypted or hashed on registryrocket.com part is not secure
(01:45:21 PM) Nathan: No one should know your billing password Joe. We see no reason to create a new password for that.
(01:45:39 PM) Nathan: I apologize.
(01:46:08 PM) Joe: no problem, but you still dont get it. if hackers hack into registryrocket, they have a database of domains and passwords for each domain. in plain text!
(01:46:31 PM) Nathan: Okay, you can change the password if you want.
(01:46:58 PM) Joe: but what if I change it today, and hackers hack into registryrocket 2 hours later?
(01:47:20 PM) Nathan: Then we will change the password. I dont understand how hackers would randomly guess your password Joe.
(01:47:59 PM) Joe: they dont need to randomly guess my password. my password is stored in registryrocket database in PLAIN TEXT
(01:48:45 PM) Nathan: Please show me where you can see it.
(01:49:10 PM) Joe: I cant!! I did not hack into registryrocket
(01:49:36 PM) Nathan: Okay, no hackers are getting into our database Joe. I do not understand how they would get your password.
(01:49:44 PM) Joe: but if someone WILL hack into registryrocket AND copy the entire database of users/passwords, then those passwords wont be encrypted or hashed, they will be in plain text
(01:52:20 PM) Nathan: Joe, no our passwords are encrypted with MD5.
(01:52:29 PM) Joe: impossible!!
(01:52:34 PM) Nathan: No one is going to hack into registryrocket.
(01:52:35 PM) Nathan: Okay.
(01:53:01 PM) Joe: ?? how can you encrypt password with MD5 and within 10 seconds from clicking "remind password", I got my password in my mailbox??
(01:53:54 PM) Nathan: Joe, our passwords are secured in a database. Please do not worry about us getting hacked. We have it under control.
(01:54:16 PM) Joe: you misleading me.
(01:54:40 PM) Joe: passwords CANNOT be properly encrypted in the database AND the same time you sending me password 5 seconds after I clicked "remind password"
(01:56:07 PM) Joe: even for very powerful computer it would take much more than 5 seconds to "decrypt" (or should rather say: crack) MD5-encrypted password
(02:00:35 PM) Joe: any news, Nathan?
(02:01:28 PM) Nathan: Our passwords are not MD5 encrypted, but they are encrypted and its done by Enom.
(02:03:06 PM) Joe: (01:52:20 PM) Nathan: Joe, no our passwords are encrypted with MD5.
(02:03:15 PM) Joe: whats "Enom" ?
(02:03:20 PM) Nathan: I apologize I made a mistake. Enom is who we sell our domains through.
(02:03:45 PM) Joe: ok, what do they use to encrypt passwords??
(02:03:55 PM) Nathan: I would not know.
(02:04:11 PM) Joe: I tell you: NOTHING
(02:04:54 PM) Joe: thats why most websites you CANNOT retrieve your old password, you can only reset it.
(02:07:22 PM) Nathan: What can I help you with Joe? Do you need the password to login?
(02:07:48 PM) Joe: I was just wondering how secure my vulnerable data is.
(02:09:22 PM) Nathan: We have a secure database, we have not been hacked in the past and I don't think we will be hacked anytime soon.
(02:09:41 PM) Joe: ok thanks.
Current Rating: 12345
The type of concern "joering2" has is obviously not something we handle at a first encounter level. It's not even a "support" issue in the normal sense. It's an infrastructure issue. Support chats and phone calls aren't going to end 15 minutes later with me having: designed, developed, tested, and deployed an entirely new password storing, reset, and re-send mechanism. This is true for any company that one has a widespread concern with regarding their policies and infrastructure. I may not approve of Microsoft's policy of forcing me through a "Customer Retention Specialist" when I wish to cancel my excess Xbox live subscription, but haranguing the poor girl I'm speaking to about company infrastructure and policy issues is a waste of her and my time.
That said, for HostGator, if you want to quickly reach someone regarding this type of concern, here's an email address: firstname.lastname@example.org.
With regards to "registryrocket.com" I can assure you that it is and has been the bane of every tech's existence here at HostGator. Some explanation: HostGator used to, and still does, resell domains through eNom. When you visit "registryrocket.com" -- visit http://hostgator.com/domains in order to get a working session -- what you are seeing is eNom's idea of good default site design and then their idea of user experience. The user experience with this tool is... lacking. To put it this way: I've never had a call where a customer told me how easy and great purchasing a new domain through hostgator.com/domains was.
Moving on, this is not a system that we directly administer and control, so far as I know. It is a service hosted on and developed by eNom, as stated on their own page describing the product: "We take care of the website hosting, merchant processing and other eNom services." (http://www.enom.com/help/faq_rr.asp). This, the poor user interface for managing domains, and the general frustration of having to work with an external registrar rather than being in complete control, has led to us forming our own registrar: "LaunchPad." We're already registering new domains under this tool and getting a live web interface for it has been a huge request both from our customers and us lowly tech and sales staff. Here's to hoping the hiring push for development talent pays off :)
Tech support, especially tier 1 support, are there to follow procedures. No matter how fouled up those procedures are.
If the procedures need changing, you find the tier-1 support's management level. Going to the CTO/CEO if you must.
I agree that MD5 isn't what they're using, as it isn't reversible encryption, or even encryption at all, but that ignores the fact that it's very possible to store a salted, encrypted password in a database, the salt on a separate file system, and still be able to send you your password in plain text.
Backup systems admin.
Key passwords available in a shared (but encrypted) repo.
Multiple data links -- tethering through your cell phone or heading to the local cafe or library in a pinch.
Best password sharing/collaboration tool we've found.