
Namecheap live chat social engineering leads to loss of 2 VPS - Casseres
http://www.postphp.com/namecheap-livechat-social-engineering-leads-to-loss-of-2-vps/
======
r1ch
I had my 2FA at Singlehop bypassed by social engineering attack. They
helpfully changed the entire account contact info without any notice to me,
presumably from a phone call. The attacker didn't even have any information to
go off other than the IP address. I only found out when I saw the server
rebooting into rescue mode and luckily I still had an active management portal
cookie (changing the password doesn't log you out of the portal, another big
problem) so I immediately knew what was happening.

I wish there was a way to disable the "customer support backdoor" in all these
kinds of services. I've started to deploy full disk encryption to all my
servers now so if the attacker does manage to get into the management account
the server itself is still protected from single user / rescue mode / etc.

~~~
brianwawok
It's tricky because a lot of customers really DO lock themselves out of a
service, and forget their password reset code.

Fun story time. I use to play MTGO, the online Magic the Gathering game.
Played it from beta for a few years say 2002-2004. Wanted to check it out in
2014 to see how it changed. Failed password reset online, had to call in to
support.

The support guy was like _chortle_ what was your security passcode? I had no
idea. He tried giving some hints. I said it has literally been 10 years, I am
never going to remember. So he went ahead and rest my password. He told me for
the record my security code was "I am the nacho king", but I would be prompted
to change it at next login.

So could I have social engineered a 10 year old MTGO account? Yes. But without
it, I would have been locked out. I was NOT going to remember some stupid
passcode kid me set on a game account.

And apparently I am the nacho King.

~~~
rev_null
I once tried to log into a site only to discover that the security question I
left for myself was "What is blue?". I never figured it out.

~~~
oaktowner
AT&T has a security code which is "What is your favorite restaurant?" that we
set a decade ago when signing up for internet service.

My wife and I have made, I don't know, 10 guesses over the years and have
never been able to figure out what our response was back then.

Questions with fact-based answers are much better. But...I once had a site ask
me for my best man's first name (Good! This probably won't change over time!).
I filled in "Dave" ... and the site gave me an error "Your answer must be at
least 6 characters."

Doh!

~~~
mikeash
I'm always amazed at how little thought seems to go into these questions.

My wife filled one out a few weeks ago where both the questions _and answers_
were selected from popup menus. One of the questions was "What's your favorite
summer activity?" Her answer was, "Swimming." Yeah, that's going to add about
one bit of entropy to most people's accounts, you idiots.

Another favorite is "middle name of your youngest child." That answer can
change over time!

~~~
ymse
I can't reply to the sister comment for some reason, so I'll piggyback on the
parent.

I always fill these with awkward or absurd questions/anwers that would be
amusing if a human operator ever needs to verify them. E.g.

 _Would you like to go on a date with me?_

 _What color pants am I wearing?_

 _What is the square root of insanity?_

Obviously you need to store these in a password database in order to remember
them, which kind of defeats the purpose. If I have to choose from predefined
questions, it goes along these lines.

Q: What was your mothers maiden name?

A: Why, are you stalking her?

Q: Where were you born?

A: Oh I can't remember, it's been so long!

~~~
uxp
I've just started filling them with randomly generated strings that my
password manager helpfully creates for me. Though, apparently my bank uses
those answers for phone verification also, which makes answering questions
like "What's your Significant Other's nickname?" awkward when the answer is
"F9-#g7a2<qj"

~~~
raverbashing
Use a random but user friendly value

#$%&+@ is going to be hard to type or speak, just say their nickname is "the
frozen one", same entropy, easier to handle

~~~
kbenson
>> What's your Significant Other's nickname?

> Use a random but user friendly value

> "the frozen one"

I imagine that may make make your significant other who was previously
friendly, markedly less so, if they see your "friendly value". But that may
have been your point, as it may be quite a bit easier to remember. :)

~~~
raverbashing
My point is to use a term that is easier to type and speak rather than just a
sequence of symbols. Or just modify the "right" answer in a quirky way.

But yeah, what you said might happen.

------
matthewdrussell
Disclaimer: I'm CIO @ Namecheap

1\. The credentials were resent to an already compromised email account

2\. This is an isolated case

3\. Established procedure was not followed

4\. With thissaid, we've used this as a learning example and additional
training has been provided to the individual involved

5\. Anyone with any self-managed server with ANY provider should always keep
their own multiple backups

~~~
swanson
My hobby: role-playing how I would respond as the CEO if my company was
getting skewered on HN. Here is my version!

\---

Disclaimer: I'm [not] CIO @ Namecheap

We messed up, big time. While we handle 1000s of live chat sessions everyday
without issue, I realize that even one breakdown in security protocol can
cause huge problems and a loss of trust for our customers.

In response to this isolated case (in which our established procedure was not
followed), we will be creating additional training material for all our live
support staff. Additionally, we will be exploring technical solutions to try
to make this kind of breakdown much harder. Mistakes happen, but if we can
prevent them, it is worth doing.

We also would like to take this opportunity to remind folks that any self-
managed server (regardless of provider) should always be backed up in multiple
places. For information on how to do this with Namecheap, we've published a
guide here: <link>

I've reached out to author of the post already by email and we are working to
help them resolve any outstanding issues.

~~~
kelukelugames
That's impressive, can you teach me to write like you?

~~~
swanson
Formula:

* Actually apologize in a human way

* Show empathy by identifying the impact of what happened to customers (not your impact internally)

* State action items that you've created, even if they are just in 'evaluation' state

* Indicate that the specific incident in question is being handled outside of this forum

* Take responsibility for things even if you shouldn't "have to"

~~~
voltagex_
For a "what not to do", have a look how (the CEO of?) FTDI responded after
they were caught intentionally "bricking" chips that were detected as
counterfeit by the Windows drivers.

~~~
Namidairo
Last I heard, these tainted drivers from FTDI weasled their way through WHQL
and into Windows Update...

------
tyingq
Not trying to be snarky, but the biggest lesson here seems to be "don't
operate without off-host backups". Cheap VPS providers don't typically offer
that sort of thing as a standard feature. Even when they do, the backups would
be on the same infrastructure, and easily wiped from the same (compromised)
console.

You could have just as easily lost all the data in an accidental way, with no
malice or 3rd party involved.

That said, I do empathize, and it's disappointing that a major player like
namecheap would be so easily socially engineered.

~~~
matthewdrussell
We offer full backups with all managed servers/services

Self-managed a customer is responsible for their own backups. Just like with
DO and that full server loss a couple of months back.

~~~
joenathan
We who?

~~~
gist
Example of on HN expecting everyone (newbies and all) to know who you are.
This happens with DANG and SAMA comments as well. Back when PG used to comment
also happened. Look at their profiles, really no explanation of who they are
here:

[https://news.ycombinator.com/user?id=pg](https://news.ycombinator.com/user?id=pg)

[https://news.ycombinator.com/user?id=dang](https://news.ycombinator.com/user?id=dang)

[https://news.ycombinator.com/user?id=sama](https://news.ycombinator.com/user?id=sama)

Why is it so hard to put info in your profile or to put a footnote in your
comments for the newbies? Would you have your business act this way? Reply to
a person's inquiry and not say who you are and what you do? Of course not.

~~~
mikeyouse
He posted this down the thread which starts with:

> _Disclaimer: I 'm CIO @ Namecheap_

[https://news.ycombinator.com/item?id=11479810](https://news.ycombinator.com/item?id=11479810)

~~~
prawn
Still not outrageous to think that people might be able to give rookies some
context rather than check the full thread for other comments doing so.

------
pfarnsworth
Social engineering in tech has been around since before Kevin Mitnick
publicized it and went to jail (unjustly). Why do we keep making the same
mistakes over and over again as an industry? We NEED UNIFORM security
standards with ALL trusted companies with customer support, where we have
tiers of support, and 1st tier doesn't have any access that could compromised
security. Similar to ISO standards.

This means there can't be any "impedance" mismatching that can be used from
one service to another. For example, one company gives out the last 4 digits
of the credit card, and the other uses the last 4 as security info.

What we need is a uniform security standard and training for ALL customer
support personnel so that you can use Apple to break into Amazon, or Digital
Ocean or Namecheap. The staff need to be trained to never succumb to social
engineering ever, and in fact make it impossible for 1st line support to reset
anything. Have any security information get passed up to second tier support
who are extremely well-trained. Etc.

And have this standardized so that there is incentive for customers to look
for this certification so that we don't have to keep suffering the same
mistakes over and over again.

~~~
Someone1234
> Kevin Mitnick publicized it and went to jail (unjustly)

You're joking right? He even fully admits that he did what they accused him of
doing.

~~~
Negative1
Ever heard of a plea deal? He was obligated to 'admit wrongdoing' in order to
get a reduced sentence. Sure, he could have stuck to his guns, but, jail has
very bad internet.

~~~
Someone1234
I meant in his book and speeches he has since given he has admitted doing more
or less what they accused him of doing.

I'd definitely agree that the solitary confinement was cruel and unusual. I'd
also agree that the law wasn't mature enough when he was charged so he was
charged with proxy-laws.

But ultimately he did do what they said he did, and some of it was pretty
messed up. He would definitely be charged today with computer crimes (or
generic crimes) for many of his exploits at the time.

~~~
josefresco
I would imagine owning up to his "crimes" helps his business / brand. If he
disclaimed all credit, he would be seen less an an expert in his field of work
_.

_ [https://www.mitnicksecurity.com](https://www.mitnicksecurity.com)

------
willyyr
So he is using 2FA for all the important accounts but for the most important
one (the email which he used to register an account at all these services)
he's using a weak pw and no 2FA? Am i missing something here? Yes they did not
follow protocol but why would one not use 2FA for such an important email
addy?

~~~
matthewdrussell
Correct.

I'm surprised no one else has mentioned no 2FA for the email. The email being
compromised opened the door to this happening.

~~~
mbrameld
The email being compromised opened the door to his email being compromised.
The door to his Namecheap account being compromised was apparently already
wide open.

~~~
levemi
No, OP didn't have 2FA enabled on their namecheap account. It was namecheap's
fault for improper handling of the social engineering attack but OP could have
protected themselves by having 2FA

~~~
cname
The article pretty clearly states 2FA was enabled for the Namecheap account in
question. In fact, that is sort of the whole point of the article.

~~~
levemi
Oh this comment by CIO led me to think 2FA wasn't...

[https://news.ycombinator.com/item?id=11480221](https://news.ycombinator.com/item?id=11480221)

You should not be able to overcome 2FA with social engineering wtf!

------
0x0
I remember when NameCheap launched the "security notifications" feature where
it would email you whenever there was a login or activity on your account. I
noticed that logging in on the mobile site didn't trigger any emails. When
asked, they replied that the mobile site was just a beta version.

It doesn't help that the front door is securely locked when the back door is
not! :-/

~~~
mikeash
Crazy. I have trouble figuring out how you'd even program it that way. Is it
not obvious that security notifications belong in the authentication layer
that everything uses, and not in platform-specific front-end code?

------
johnnyfaehell
At the end of the post he seems quite snarky and bad mouthing namecheap's
security for things that aren't even their fault or even security issues.

> The VPS panel allows full serial console with only a login/password (no 2FA
> required or possible)

Yea that's because it's a serial console, if you want 2FA or something then
that's a matter for your operating serivce. A serial console is literally like
you're plugged directly into the machine.

> They send out your VPS panel login/password in plain text emails when you
> sign up, and when you reset the password. So if you ever failed to delete
> one of those emails completely and someone gets into your email…your totally
> screwed…

To be fair this is pretty standard. It's your job to secure your passwords
once they've given them to you. If they're storing it in plain text then you
can complain but this basically sounds like you're complaining that they're
not encrypting emails. Sure they could only show you it once when you boot it
up. But since this action was done via customer support they would have to
give you the password some how. To your email address the most secure other
than the chat which can be by an attacker like it was in this case.

> VPS can be irrevocably wiped within seconds without any prompts or
> confirmations just by the click of one button; whether the server is turn
> on/off it doesn’t matter.

This isn't a security issue. A UX issue yea, but it's not even that big of a
deal. It's in an area you won't be that often and where you know you're doing
admin related thing.

> They keep no backups, even to cover hardware or security failure.

This isn't a security issue. It's your job to back up your stuff not a VPS
provider.

> And of course the icing on the cake is that they ignore 2FA and are willing
> to send out your username/password to anyone that asks.

Yep. Pretty valid.

~~~
dougmany
>> They send out your VPS panel login/password in plain text emails >To be
fair this is pretty standard.

This practice has always bothered me. An expiring link to reset is much
better.

------
koolba
What's this crowd think of this idea for solving this problem?

1) Offer an option to opt-out of all automated account recovery. If set, no
more email resets, support PINs, or similar. This would be targeted at people
truly care about security and have no issue with "forgetting passwords" (i.e.
you use a password manager and you're not an idiot about backups).

2) Offer in-person, manual recovery. To participate in this you'd need to pre-
register with full contact details (name/address/etc) of the valid people who
could use this feature. The person would have to physically come to the office
of the company, present two (or more) forms of identification. To add further
security, you could add a mandatory wait period between initiating a reset and
it taking effect (ex: min 7 days). That way a combination of fake ids and
social engineering could (in theory) be stopped by getting an alert that "
_You initiated a manual reset of your XYZ account. Did you actually do this??_
"

EDIT: For #2 you could also add a non-trivial fee (say $500) that would need
to be charged and cleared in advance of the person showing up.

~~~
StavrosK
We do something similar at Silent Circle. In your recovery options, there's a
page with a high-entropy secret key and a QR code that you can print out to
use if you ever forget your password.

There's also a checkbox that says "don't ever recover this account" (i.e. the
"I have a password database on Dropbox") checkbox. Checking that box actually
disables password resets on the admin interface, so your account is pretty
much dead if you lose the password.

~~~
jontas
OT question about Silent Circle: I was just looking at your website, and I
noticed that you cannot ship to PO Boxes. Is this a security feature (eg, no
government knowledge of the recipient) or a logistics issue (eg, FedEx/UPS
can't deliver to PO Boxes). I would imagine it is pretty hard to get service
for your SIM card without revealing your identity to degree.

~~~
StavrosK
I'm not actually sure about that (I assume you're referring to shipping a
Blackphone?). The Blackphones are a semi-separate division that I don't have
much contact with, unfortunately.

------
tshtf
The most significant security problem with Namecheap is really this: It only
takes a 4 digit PIN to perform any action on an account through live chat
(which seems to be outsorced to Eastern Europe), even if the account is
protected with a 2FA... All you need is the PIN, and an attacker can do
_anything_ to the account.

Sometimes you get what you pay for.

~~~
castis
If I wanted more security on my account, is there a different service I should
be using?

~~~
dorfsmay
I once lost my gandi.net password. It took sending copies of 2 photo id, and
answering the phone listed in the who is database before they reset it.

I just wish that their DNS updates were push through faster.

~~~
ryanlol
That's not good verification. It takes a couple of minutes to produce
convincing fake ID scans, and they aren't going to have anything to verify
them against.

And presumably they wanted you to send those photos to them as an unencrypted
email attachment, right?

~~~
Xylakant
Faking ID scans adds a whole layer of law enforcement on top. I'm uncertain
about the situation in the US, but in germany the fake itself is punishable by
law (up 10 ten years). It also creates more traces to look at and creates
work. You'd also need much more information to create a convincing fake id
scan of your intended victim. It's all about increasing the amount of work for
the would be attacker.

~~~
ryanlol
>Faking ID scans adds a whole layer of law enforcement on top.

That's why _nobody_ has ever used a fake ID at a bar!

> but in germany the fake itself is punishable by law (up 10 ten years).

[https://dejure.org/gesetze/StGB/267.html](https://dejure.org/gesetze/StGB/267.html)
5 years.

But producing fake scans isn't covered by this law, scans aren't even an
official document. In fact, it is illegal for a german company to ask you to
send them scans of official documents.

> You'd also need much more information to create a convincing fake id scan of
> your intended victim

To fake a _good enough_ passport scan you'd need your victims name. That's all
the rep is going to have.

------
mrrsm
Does Namecheap claim to take backups? Even if they do you should be taking
backups as well if you care about your data.

I do agree with more login forms needing to support 2FA. At this point I wish
almost everything did. It is a bit more hassle but is easy to manage for me at
least.

~~~
tamar
This was a self-managed server. Managed services at Namecheap have backups.

------
danielsamuels
> but on the way out decided to click the conveniently located “Re-install”
> button next to each VPS. This instantly wipes everything and installs a new
> OS. Again this action requires no 2FA authentication or any other form of
> confirmation

This is the same for DigitalOcean. I'm always amazed that clicking "Rebuild"
or "Delete + Scrub Data" doesn't require _any_ confirmation at all.

------
matthewdrussell
Also let me reiterate this is an isolated event. We handle over 10,000 chat
sessions every day without a glitch. I invite people to use our live chat
service and see what is and what is not possible, as well as the security
precautions we have in place.

~~~
onion2k
_Also let me reiterate this is an isolated event. We handle over 10,000 chat
sessions every day without a glitch._

What do you use to tell whether a chat session is a genuine user or someone
successfully using a social engineering attack against your chat operatives?
If the answer is "nothing" then you can't know if this is an isolated event or
how many of your chat sessions go without a glitch.

~~~
tamar
There are identification methods requested via chat. Matt invited you to try
it. Go for it.

~~~
vehementi
Parent's point was, how do you tell whether or not your rep was socially
engineered? Only some mistakes get complained about. If you don't have such a
method then your "10000 sessions a day without a problem" number is fantasy.

------
site-dot-onion
So full of failure, this thread.

No, it is never okay to compromise security just because "it's good when you
forget your password". There should be no way around this for any reason; a
bypass via SE or any other mechanism is a failure of the company, end of. If
you forget your password or you do not have your 2FA/security questions
available, tough, you should lose access.

For the sake of "workarounds for legitimate users", that just translates to "a
security hole".

Special pleading is a logical fallacy and if it works on a support rep, that
rep has failed at their job.

~~~
HemanHeartYou
>No, it is never okay to compromise security just because "it's good when you
forget your password".

No one is arguing that.

------
tamar
Namecheap made a response here: [https://blog.namecheap.com/social-
engineering-issue/](https://blog.namecheap.com/social-engineering-issue/)

------
po1nter
Since the CIO (and another employee) are here. Why are you not offering
support for Google Authenticator? Last time someone asked for it was 2 years
ago[1] and still no sign of the feature. Cheap prices are good to have but
combining that with more security can only add value.

[1]:
[https://www.namecheap.com/support/knowledgebase/article.aspx...](https://www.namecheap.com/support/knowledgebase/article.aspx/9253/45/how-
to-two-factor-authentication#comment-1103776931)

~~~
tamar
It's in the works. We're aware of the request.

~~~
DiabloD3
You should integrate Authy instead.

------
kayoone
Don't get the backup complaints, thats bad luck but you can't blame the hoster
for not having backups.

------
joering2
STOP. USING. NAMECHEAP.

Its been months since I wanted to write a detailed summary, but the notion
that "namecheap is hackers best domain registrar" is not valid anymore!

About year ago I noticed DNS changes on many of my there-parked domains. Upon
reaching via Chat (no phone support so that angry customers cannot vent off) I
was told that they cannot help me cause Im not the owner of the account! Upon
full verification even with CC on file and telling them purchase history going
back to 2009, I was still denied the access. As it turned out, all hacker
needed to know is my public WHOIS info to take over my account!! That was
insane! Only continuance of threats from my side that I will plaster it all
over the net made them change their mind, which again is a breach of trust -
what if I was actually the hacker??

What really made me start moving domains to NameSilo (Im not affiliated) is
that upon doing a thorough research, I found many cases where Namecheap gives
up on fighting for peoples domain! I seen names like nanotmz where company was
building some sort of magnetic devices and TMZ came in and threat to sue
Namecheap if they dont shut the domain down. That's where I found similar
cases for NameSilo and learnt that they stand their ground and would not give
up on your domains, even if are threatened with legal action.

I'm out of Namecheap completely as of last month with last SSL expiring.

~~~
tamar
Namecheap has had two factor authentication and was the first provider to have
it. Knowing public whois would not grant someone access to anyone's account at
Namecheap. They'd still need to know your Namecheap username, your password,
and your PIN, and if you had 2FA, that would need to be provided as well.

~~~
joering2
... unless their Ukraine-based customer support that sometimes doesn't speak
English is intimidated enough.

~~~
tamar
If you have specific information, you are welcome to contact us with full
details. Policies and procedures are in place to ensure no one falls victim to
social engineering and as you call it, "intimidation."

~~~
skj
With respect, "policies and procedures" do not protect against social
engineering. You need a technical barrier.

~~~
tamar
skj, there's a lot more to policy and procedure than just human intervention.
We're fully aware of this valid concern and are committed to security on both
human and technical sides.

------
jqueryin
As someone in the domain registrar industry, are there any features beyond 2FA
that you would like to see implemented by registrars?

More bluntly, what is it that you think your current registrar is lacking?

I read a few comments on Gandi and support of GPG keys. I'm guessing this is
what you're referring to:
[https://wiki.gandi.net/en/gandi/documents](https://wiki.gandi.net/en/gandi/documents)

~~~
extrapickles
I would like to see something where a postcard is mailed and a phone call,
each with half of the code needed for a reset. Postcard should not be sent
using a method that supports forwarding so an attacker cannot setup a mail
forward.

Customer support should not be able to see anything about these accounts
except for a reset button. I do expect to be charge a fee for a reset if I
need to use it. This would need to be rate limited to prevent people from
dosing an account through it.

~~~
PhantomGremlin
_a postcard is mailed_

If you want them to go thru the trouble of mailing something, at least require
it to be a letter. Inside an opaque envelope.

A postcard is the exact opposite of something you want to use to send
sensitive information.

------
gk1
In 2014 I was able to bypass .htaccess restrictions of a site hosted on
Namecheap using social engineering through the live chat... Twice.

All I had to do was tell the chat operator I was having some issue with
.htaccess and .htpasswd until they offered to delete it temporarily. In the
few minutes between them deleting it and reuploading it, you're free to do
whatever you want.

I reported this and was told it was resolved (I guess with new policies).

------
mikestew
I'd give money to a VPS, or what have you, that had a stated policy along the
lines of, "here is a recovery key, here is the 2FA setup, here's how you
recover your password with those items if you forget. If you call about
account recovery and you do not have $REQUIRED_ITEMS, our service reps have
been instructed to hang up on you. If you lose access to your account without
$REQUIRED_ITEMS, you have lost access to your account permanently because we
have set procedures from which we do not, under any circumstances, deviate.
With a name like Ft. Knox VPS, our customers value the security of their
accounts. As security is our priority, we have chosen to trade convenience and
possible loss of data for the elimination of a large threat surface. If you
regularly lose your car keys, don't use a password manager, or get angry when
companies won't just email you your password instead of making you reset it,
we're probably not the right choice for your VPS needs."

~~~
matheweis
I don't know... Apple did this for a while, and it backfired on them. The
average user (even technically savvy user) simply wasn't competent enough to
understand that they really truly would be locked out forever.

From 2014: [http://thenextweb.com/apple/2014/12/08/lost-apple-id-
learnt-...](http://thenextweb.com/apple/2014/12/08/lost-apple-id-learnt-hard-
way-careful-two-factor-authentication/)

They no longer do this: [https://support.apple.com/en-
us/HT204921](https://support.apple.com/en-us/HT204921)

~~~
mikestew
I by no means intend such a system be sold to average consumers. As you point
out, it would be doomed to failure. More for the hard-core "I'd rather lose my
customer DB than watch it walk off in the hands of someone with a good sob
story" crowd. And I'm not 100% sure that there is large enough, bold enough
type to forestall some customer saying, "but I didn't think you _meant_ it!"

------
adamowen
The lesson here is you should always keep your own off-site backups -
especially if you don't pay for a 'managed' server.

There will always be rare occasions such as this, but considering how many
customers Namecheap handle, I don't think we should be seriously concerned.
I'm pretty confident lessons will be learned.

------
danielweber
I'm leaning more and more towards treating the email address that you use to
register for business-critical services as secret.

It's a level of security-through-obscurity, yes, but that doesn't mean it's
wrong. It means you can keep that address monitored well. You could make any
activity on it send a page, for example.

~~~
matthewdrussell
\+ Better email security with 2FA on your email address.

------
moviuro
[https://twitter.com/NameCheapCEO/with_replies](https://twitter.com/NameCheapCEO/with_replies)

"This was an isolated case and procedure was not followed. I can assure you we
have addressed it so it won't happen again."

~~~
tamar
matthewdrussell is also commenting in this HN thread @movluro - he's
Namecheap's CIO. I, too, represent Namecheap.

~~~
moviuro
Sorry I missed that. I have catch-up to do, now

------
turbohz
"On April 9, 2016 I had an email address compromised, with the attacker brute-
forcing a weak password."

Namecheap is obviously to blame for the compromise of the VPS, but failing to
secure an email account which can be used for password resets is even a bigger
fail, IMO.

------
stevepaulo
If you only use 2FA on ONE THING online, make it your email! How could you not
consider your email the most important service of all? It is your identity!

~~~
r3bl
I'm going to go ahead and say that he used an email service which does not
offer 2FA since every email service that does support 2FA probably has a good
brute force protection.

------
devereaux
That's bad, really bad. No 2auth can save you from humans who do support.

I also had one of my VPS attacked recently, and I feel for you.

But the name namecheap says "cheap". Maybe they are indeed cheap? I'm not sure
the same would have happened with say HE. You pay, but you know what you pay
for and get in return.

Personally, I am thinking about moving from a "manually setup" distribution to
a "no ssh but deploy", so as to ease reimaging in the future. This way, if a
server is compromised, all I have to do it to start the install of a new one.

Any suggestion for tools to do that with Debian distro? (yeah I could write a
shell script, but I think there must be better tools out there)

~~~
stephenr
> Any suggestion for tools to do that with Debian distort?

If you write apps, package them as Debs. If you need to configure other Debs,
make config packages with config-package-dev [1] from the DebAthena project.

Create a metapackage that depends on your software + config packages, and your
setup process just needs to be "add private apt repo, apt update, apt install
<metapackage>".

[1] [https://packages.debian.org/jessie/config-package-
dev](https://packages.debian.org/jessie/config-package-dev)

~~~
voltagex_
This is fantastic, I wish I'd known about this earlier. Now I just need a way
of testing Debian preseed faster than spinning up VMs, and I'll be set.

------
yAnonymous
1\. using a weak password for an important e-mail address

2\. not deleting the mail with the login information

3\. not having a backup

Yeah. This was just as much your fault.

~~~
vehementi
You misread the article at a very basic level if you think 2 is related to
anything

------
sverige
The comments remind me of the time I tried to get a copy of my credit report
from one of the big 3 agencies back in '09\. One of the authentication
questions was, "What is the name of your mortgage company?" My house had been
foreclosed during a divorce 5 years earlier, and of course the mortgage had
been sliced and diced about 15 times by different companies during the heyday
of mortgage-based derivatives before the '08 crash. I finally gave up trying
to get a copy from those guys.

~~~
PhantomGremlin
_I finally gave up trying to get a copy from those guys_

I had a similar experience. IIRC TransUnion was the problem one. They wanted
something like 3 credit card numbers as part of the identification. But I only
had two active credit cards. Fortunately I was able to find an old cancelled
one in a drawer and they accepted the number!???

The problem is that you're not their customer, you're just an irritant that
the federal govt demands that they give "free" information to.

They treat their real customers much better. E.g., go to a used car dealer.
Give them your SSN and they'll have your life story in front of them in about
10 seconds. They get good service because they're paying for it.

------
Pxtl
I'd love to have an option on services where I define a X-hour wait period for
manual password resets. That is, "oh, I've lost my email account and I need to
reset a password so I have to access my account through pleading over Live
Chat... they can do that but there's an X-hour wait period before you will
gain access to the account."

~~~
ryanlol
Which is great until the customer actually needs to access the account.

$CUSTOMER calls in, their nameservers are down and nobody has the account
password. Do you think the management at $CUSTOMER is going to accept "hey we
need to wait 6 hours to get our site back up because namecheap wont allow us
in"?

~~~
Pxtl
Let the customer set it during sign-up as part of the password reset process.
You set up your email address and password, let them choose $HOURS for last-
line-of-defense password reset.

I don't see any perfect answer here. Ultimately you need a way to recover your
account when you've lost all of the "somethings you have" and you've lost your
"something you know", but then that allows a social engineer access to do the
same. So let the user decide during sign-up.

------
aeturnum
Incidents like this remind me of Blizzard's policies vis-a-vie their 2-factor
auth system.

I don't know if it's changed since this happened, but in the early-ish days my
friend's phone broke and he lost his ability to generate 2FA codes. Blizzard
was happy to remove his 2FA once he had made a photocopy of at least one
(maybe two?) forms of ID and (I think?) some evidence he owned the credit card
paying for the account. Once he mailed that in to Blizzard hq, some human
confirmed the info and they removed the 2FA. If Blizzard can do that, and
they're protecting MMO characters, certainly other providers can do so as well
(maybe for an increased fee as I realize it's more expensive than online
chat).

IIRC, people hacking into Blizzard games resorted to compromising users'
computers and capturing the 2FA codes in flight - then logging in and changing
the credentials before the user could react. That's a much higher bar to clear
than the one here.

------
jiiam
The whole point of social engineering is that, no matter how good the protocol
you use, if there is a point of failure involving a person (like a tech
support guy with access to a console), then it will fail.

The only solution is to remove these sort of powers from your general tech
support guy and let them in the hands of a few highly technical, well trained,
well paid staff members (presumably managers?). Of course, I made the blind
assumption that your average support staff member is not paranoid enough, but
based on my acquaintance with a few guys in the business it seems to me that
the salary is not high enough to expect well trained technical stuff doing
support.

I might be wrong, and in that case I would gladly know which companies employ
such well trained staff, so that I can move my servers there.

------
moviuro
Okay, so just for fun, I tried to do the same with my Google Account: Login:
my email address Password: forgot it

"Please note that without your phone, the recovery procedure will take 3 to 5
days". hopefully, this means that there will be many many checks to avoid
social engineering

------
ryanlm
On another note, there 2FA seems to broken as well. I once received a text
about resetting my Instagram account from the same number that they send me to
authenticate my login session. I've never had an Instagram account.

~~~
tamar
Namecheap uses the same 2FA provider as Tumblr, Yahoo, Microsoft, and a slew
of other services.

~~~
stephenr
What do you mean by "provider"?

I find it hard to believe Microsoft of all companies has outsourced two factor
authentication.

~~~
tamar
It's true :) We hear about it on Twitter all the time that the same number we
use for 2FA is also texting 2FA codes for the other services I mentioned.

(disclosure: obviously I work for Namecheap.)

------
voiper1
Whoa. I turned on my 2FA at Linode. Their policy to disable is to require a
copy of credit card + government ID so hopefully that's followed...
[https://www.linode.com/docs/security/linode-manager-
security...](https://www.linode.com/docs/security/linode-manager-security-
controls#recovery-procedure)

Also checked: if you pay for their backup service, nobody can delete the
backups, so those backups are even safe if compromised.

------
sn
The real problem here is customer support not following established
procedures. Maybe simulated social engineering attacks to test compliance
should be part of SOP, like some companies do simulated phishing attacks.

~~~
tamar
Yes, and this matter has been appropriately handled. We'll be doing a lot more
to ensure it doesn't recur.

------
majcherek128
I agree with most people that calling to reset password should be a service
that can be entirely disabled, or at least require 2FA. But think about this.
All of this could have been avoided if they had the simple policy of calling
you back. The only thing you would have to do as a user, would be to keep your
number up-to-date.

Of course this also requires that you should never be able to add a number
using the phone though, but this makes sense, since they can just say: "To do
that you just have to sign in and click on..."

------
pcl
This refers to the loss of two virtual private servers -- two VPSs.

I read through this looking for the part where two Vice Presidents from
Namecheap were fired over the incident.

------
Kephael
I transferred all my domains away from namecheap several months ago when the
Ukrainian based live chat support was adamant they couldn't send an "ACK"
message and have my domain transfer out automatically, and instead had to wait
nearly a week for the "AUTO-ACK" to process. They lied to me and insisted
ICANN require a five day wait even when I linked the ICANN documentation
stating otherwise.

------
neotek
I've been battling with Namecheap for over a month now just to try and get the
registrant details updated on a handful of .com.au domains I own. Their
"support" is hands down the worst I've ever experienced, and the moment I'm
able to move every domain I have away from Namecheap to a registrar that
actually gives a shit, I'm going to do just that.

------
awinter-py
Is password recovery a bug or a feature? cloud providers seem on the fence
about this. AWS has certainly had similar problems in the past.

~~~
ToastyMallows
It depends on how its implemented.

If it's "Whats your mother's maiden name?" and they let you reset it in the
browser, it's a bug.

But if they send you an email (in my case to Gmail, that has 2FA turned on),
then it is a feature, because then you'd be required to either 1) intercept
the recovery email (and get the password reset URL) or 2) know the format of
the password reset URL and just happen to guess mine after brute-forcing every
possible link (assuming there is no timeout for the URL or anything else like
that).

~~~
stephenr
I had an interesting thought (literally as I was reading your comment) about
improving "forgot password" emails, albeit only likely useful for the
technically minded:

Have the customer provide an SSH/GPG public key, and store it with the
account.

When a password reset is requested, encrypt a random string using said public
key, and email it to the email for the account.

An attacker who may have breached your webmail is then reasonably unlikely to
also have your private key to decrypt the string.

Follow the link (which didn't necessarily need to be encrypted) and enter the
string you decrypted to reset the password.

On a related note: do any/many sites with 2FA, require the 2FA code to do a
password reset?

~~~
infinite8s
That's basically what TOTP/HOTP authentication tokens are, which many sites
(including Google, AWS, Github) etc use for 2FA -
[https://en.wikipedia.org/wiki/Google_Authenticator](https://en.wikipedia.org/wiki/Google_Authenticator).
When you set it up, the service provider creates an 80 bit secret key, which
you enter into your local device (or some implementations create a QR code)
and then whenever you log in you need to provide a 1-time password from the
app.

~~~
stephenr
I'm aware of 2FA using (T|H)OTP, my thought was that a GPG/SSH key can be
stored in a secure and yet reasonably easy to use way, effectively offline
(i.e. add a passphrase and store it on a USB key or similar).

With a 2FA code, you either a) use the same code they use for regular logins,
or b) require them to find a way to securely store the (T|H)OTP secret and
then add that information to a 2FA app when they want to do a password reset.

I realise the pubkey concept is more than most people would bother with (or
even be able to get through on their own), and I think the first 2FA option is
definitely better than no extra security at all on password resets, but my
thought was about _increased_ security for those who are particularly
paranoid/security conscious.

------
mpnordland
Moved my domains off Namecheap because I could find a better deal elsewhere,
but man I'm glad I don't have anything there now.

~~~
tamar
It wouldn't happen to you. As you may see by other comments here (written by
@matthewdrussell), this was an isolated incident that occurred specifically to
an _already-compromised_ email account. Still, we could always do better, and
there have already been many meetings and policy improvements that have
resulted from this single incident. We always take these opportunities to
improve.

Yes, I work for Namecheap, but that's probably implied by my comment.

------
ryanlol
I for one haven't seen very many services that don't allow you to reset your
2fa if you control the attached email.

Would anyone here _seriously_ expect that someone in control of their email
wouldn't be able to take control of associated accounts?

~~~
ajmurmann
I think this is a very good point that I also overlooked when I first read the
article. If someone hacked my gmail account, I honestly am not sure if there
would be any account of mine that would be safe. Anyone using the Internet
today has to put utmost care into protecting their email address and most
email providers enable you to do that fairly easily. There was an article here
a few months ago promoting logging in via email token as the only way to log
in instead using passwords. Because as you said, 99% of websites allow you to
reset the password anyway if you control email, so why bother having insecure
passwords? If I remember correctly then that article was fairly well received.

~~~
tamar
And this is an argument for making sure your email provider has two factor
authentication to avoid having an external breach that could give someone
access to accounts that do not support 2FA.

------
greggman
So I just was about the enable 2FA on namecheap but at least from the
description it only supports SMS

I'm traveling constantly. I always have a different country's SIM in my phone
meaning I can't receive SMSs to a static number.

Is SMS only 2FA acceptable?

~~~
jaredsohn
A few ideas:

* If they happen to be using authy for 2FA and you have the Authy app on your phone, it will use that instead of sending an SMS. You could also just have it send to Authy's Chrome extension.

* Consider setting up a Google Voice number to receive the SMS.

~~~
greggman
no, no authy option

Google voice ok, but given google hasn't updated google voice in like 3 years
I expect they'll announce it being discontinued soon.

Any other options?

~~~
jaredsohn
Perhaps you could set up a Twilio number to receive the texts.

------
smoyer
Hmm ... I have my VPS' set up so that I can only log into them using a private
key, but I hadn't thought about the possible security flaws in my provider's
control panel. Time for another audit!

------
rebootthesystem
Don't think anyone asked the obvious question:

Which VPS providers do it right?

Which are safest from social engineering attacks?

------
z3t4
Don't call them hackers! A better word would be thief or vandal.

------
RawInfoSec
If you can't secure your email, why would you be surprised when your servers
dissapop?

I understand that there should have been more layers beyond this and all, but
really, what is the point if you're vulnerable across several OpSec levels?

------
kelukelugames
Can you get your stuff back from namecheap?

------
donald123
I guess you can't blame them too much, they are a domain seller after all.

------
joe_momma
the first error was buying server space from namecheap. good domain server but
they get hit frequently being a midsize provider of services, they have enough
bait and not enough people to protect it.

~~~
matthewdrussell
That doesn't really make sense. We're a great domain and hosting provider.

~~~
newjersey
This isn't your part of the business but there is room for improvement when it
comes to kb articles in domain
[https://www.namecheap.com/support/knowledgebase/article.aspx...](https://www.namecheap.com/support/knowledgebase/article.aspx/473/2/demo-
changing-host-record-settings)

The domain registration process isn't automated from my experience but it
works well. However, I think the meta is (I think many people will agree) to
not mix domain and hosting with the same provider. For example, if you get
your domain from namecheap, you should not do hosting at namecheap. Therefore,
the argument is that if you want to buy a domain name from Namecheap (as
they're pretty decent), you shouldn't do hosting there.

Sorry if I sound like a prick.

~~~
matthewdrussell
You're right. And this sounds like constructive feedback which we love and are
always open to.

Our KB platform is getting some attention as articles are improved and then
the UX will be overhauled. We have work to do here and we're doing it.

------
justusw
What legal consequences could there be for Namecheap, if any at all?

~~~
ryanlol
Why would there be any? Would there be legal consequences to YC if I hacked
your email and reset your HN account password?

~~~
bdcravens
The hacker wiped the VPSs, and there was no backup.

------
rootw0rm
hashbackup + backblaze. you're welcome.

------
suresh70
namecheap PR is going all guns

~~~
tamar
We're always keeping tabs on what's going on - it's called taking care of our
customers and making sure we learn from our mistakes.

This isn't new behavior for us. I've been at this since 2009.

