
Linode hacked, CCs and passwords leaked - DiabloD3
http://slashdot.org/firehose.pl?op=view&type=submission&id=2603667
======
AlexMuir
Ah this is so shit. I want to support Linode, I've had nothing but a good
experience. But I just had to check my credit card to be sure they hadn't lost
my details. I've NEVER had to do that before with anyone - they've got to
respond fast here because if I don't trust them with my CC then I can't leave
five-figure contracts at jeopardy hosted on their servers.

I've been living comfortably on Linode servers for over three years. This is
like suddenly being evicted and having to pack my stuff up and find another
apartment.

I have to wait for some sort of verification for this but if true then I have
to leave Linode. I have client sites hosted here - not for cost reasons, just
because I like Linode.

For the sake of $5 a month I can't even take the slightest risk of being
criticised for using Linode. And this lack of transparency could be a nail in
the coffin here.

I don't want to waste a couple of days on this but that's what's going to be
involved if this is true.

~~~
clicks
I've now heard from a number of people using Linode that have suspicious
activities on the cc which they used with Linode.

I just called up my bank to tell them to 'block' it as a precaution (I will
now have to give them a visit later today to get a new card). I encourage all
other Linode customers to do the same, _because it'll be easier to just spend
half an hour doing this instead of spending hours upon hours disputing
specific transactions_.

Linode customer support keeps saying they have "no comment" on this issue
(which I suppose does make sense -- I'm assuming they've been ordered by law
enforcement persons to not share details), so as we're not being given much
information to work with... just treat this as a worst-case scenario (all
names, addresses, credit card numbers, etc. have been compromised). Do operate
now with the assumption that all of this data has been compromised and may
very well be public soon.

~~~
ereckers
"because it'll be easier to just spend half an hour doing this instead of
spending hours upon hours disputing specific transactions."

I live on the internet. Put my credit card out on many services. Over the last
5 to 8 years I've had my credit card numbers taken I believe 4 times.

Never had to dispute it once. These Credit Card companies and Banks have a
stake in not allowing your account to be drained.

I think it would be a waste of time to go out and cancel our CC until hearing
from Linode that yes, CC information was taken.

~~~
NewAccnt
Wow, four times? You should probably be more careful about who you give your
number to. Personally, I usually get a new card every 3-5 months. If someone
ever sat on my card number, it's useless to them now. Never had any issues
either.

~~~
joewee
Even old card numbers can be used for transactions in some cases.

~~~
Evbn
Sure, but that isn't the former owners problem.

Are you thinking of _expired_ cards? That is different.

~~~
runeb
I can be in some cases. I got mugged and my card was used to pay for parking
garages for 1.5 years until it expired even though it was canceled and blocked
by the issuing bank. They said that for some transactions, the blocking
mechanisms are so expensive its more economically sane to them to refund
whatever was drawn.

~~~
Evbn
Ugh. Did that require you to protest each charge? Or did you get a charge and
credit on every bill?

------
robomartin
How about you guys cool it and stop organizing a lynching mob devoid of any
real data? It's embarrassing. HN is supposed to be populated with lots of very
smart, data-driven analytical folks. Yet, every time something like this
happens out of the woodwork come people who would ran you and your children
down in the event of an emergency rather than turn around, carefully evaluate
the situation, and help you. Don't be a moron. Stop it. For all you know
there's a serious law enforcement effort under way that prevents Linode from
talking.

For the record, I am a Linode customer and just got a new server to migrate a
couple of sites into. My plans have not been altered at all by this. I have no
data to suggest I should.

~~~
Legion
I am a customer too, and I wield no torch or pitchfork, but I grow
increasingly frustrated at the lack of response from Linode.

I understand your point about the idea that they may be unable to speak to the
issue due to law enforcement efforts, but for the moment, acknowledgement
would be satisfactory. I would be happy with, "We're aware of the rumors
regarding the intrusion at Linode this past week. We are working with law
enforcement and cannot comment on details at this time. However, we will
provide a full postmortem once we are able to do so."

The problem is when the explanation _never_ comes. It's OK if it's not this
second, but tell us it's coming, and then follow through. Complete silence is
frustrating.

~~~
zevyoura
Here's the response I got from my support ticket:

>Thank you for your inquiry, and I certainly understand your concern. We are
still conducting an active investigation and unable to disclose most
information at this time. This being said, we do not yet have any evidence
that any payment information of any customers have been compromised. We will
be releasing further information regarding the incident soon, so please keep
watch of our website and blog for said information. If you have any further
questions, please feel free to ask.

That actually sounds pretty close to what you're asking for, although I have
to say it didn't make me feel much better. It would be nice if they would make
a public statement too.

------
danso
From a purported abridged chatlog with the alleged hacker:

> _05:42 < ryan||> credit cards were encrypted, sadly both the private and
> public keys were stored on the webserver so that provides 0 additional
> security_

> _06:00 < ryann> They did try to encrypt them, but using public key
> encryption doesn't work if you have the public and private key in the same
> directory_

[http://turtle.dereferenced.org/~nenolod/linode/linode-
abridg...](http://turtle.dereferenced.org/~nenolod/linode/linode-abridged.txt)

~~~
dom96
Here is what Linode replied to me when I asked them about that chat log in a
support ticket:

    
    
      Hello,
      
      Thank you for reaching out. We appreciate and understand your concerns. At this time the evidence suggest that this activity was targeting a specific customer. We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.
      
      We have no comment regarding ryan*'s comments in #linode. You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence.
      
      I am sorry that we cannot provide more information at this time. As always feel free to contact us at any time with any future concerns.
    
      Regards,
      Quintin

~~~
gburt
These guys are looking totally incompetent at this point.

If you believe this Ryan guy, credit cards stored on the same server as the
key to decrypt them, Lish passwords stored in plain text, they've known for
some time and lied about what actually happened and now they're saying "we
won't do anything about it" via email?

"You are of course free to take any steps you deem prudent or necessary to
ensure the integrity of your online presence."

Unbelievable.

Edit: not to mention they "made a deal" with the hacker not to tell anyone?
What the hell?

~~~
tmcneal
To be fair the hacker didn't say the keys were stored on the same server as
the credit card numbers, he said they were stored on the web server. It's most
likely the database containing the CC numbers resides on a separate set of
boxes than the web servers.

~~~
danielweber
The Cigital-recommended way to hash your passwords is to use an HMAC/scrypt
combo, with the HMAC key stored on the app server (not the database).

What Linode did may, or may not, be dumb. They are being tight-lipped so we
can only guess.

~~~
lawnchair_larry
Why would you use an HMAC for password storage? It's not like length extension
attacks are relevant in that application.

------
citricsquid
My Visa card that I used with Linode was stolen and used on an Amazon order I
didn't authorise last week, my bank successfully blocked the charge. Someone
else reported their Visa had also been compromised in the thread 2 days ago,
looks like that confirms the suspicions:
<https://news.ycombinator.com/item?id=5542015>

Poor show Linode. (edit: worth noting I use the card with other things too, I
have no confirmation it was leaked through Linode other than the compromise
happening at the same time these supposed leaks happened).

~~~
nivla
Great, now I am feeling paranoid although I don't see any unauthorized charges
on my card. Does anyone know if debit cards are legally protected the same way
as credit cards with 0% liability.

~~~
UnoriginalGuy
Debit cards have less protection.

Wouldn't hurt just to ask your bank to re-authorise it anyway? It will change
the three digits on the back.

~~~
spkthed
They don't, protections are exactly the same.

~~~
UnoriginalGuy
They really aren't.

I know someone who got their debit card cloned. While the bank eventually
repaid him, that did nothing to repay him the additional fees he owned his
normal debtors (e.g. rent, utilities, etc).

With a credit card you aren't losing "actual" money. You are losing the bank's
borrowed money which the bank pays back. With a debit card you're losing cash
which you won't be able to replace yourself and which the bank might take days
to weeks to replace.

Even if you NEED to borrow while your credit card is out of commission you can
either use the overdraft facility on your debit card or other quick sources of
credit. Hard to get quick cash without going to a pawn shop.

------
ConceitedCode
If this is true then all the trust that Linode has built up over the years was
just thrown out the window. According to the hacker they've known for 2 weeks
and made a deal with the hackers. Ultimately, they were as far from
transparent as it gets and on top of that they did a horrible job with their
security.

Hopefully, they own up and start being transparent.

If this is true then what alternative hosts should I look at, besides AWS?

~~~
clarkdave
Two alternatives often mentioned on here are DigitalOcean and RamNode.

I've only used DigitalOcean. My anecdotal experience from running a Chef
Server on a 1GB instance has been pretty mixed. The price is good, but network
and CPU performance feels very variable to me. A month ago their Amsterdam
servers were unable to be resized, and there was nothing about it on their
status page. I tweeted and was told they'd be working "some time later today".
Doesn't fill me with much confidence in general.

I'd still choose Linode for anything of importance - their long reputation is
well earned in my opinion. But, if this breach is true, I hope they handle it
well.

~~~
sk3tch
>But, if this breach is true, I hope they handle it well.

That is unfortunately the problem though. If this _is_ true, they have already
handled it terribly as it has already been 2 weeks since the attack.

------
carbocation
Just like I can have application-specific passwords for my Google account, I
wish I could have application-specific credit card numbers from my CC issuer.

If I had these, I would immediately cancel my Linode-specific CC# and reissue
a new one. I would not have to worry that my other recurring bills will go
unpaid, or spend hours dealing with tracking them down and changing them.

~~~
clauretano
Bank of America provides this [1], as does Citibank [2] and likely others.

Paypal at one time provided this service as well, but it doesn't seem to
anymore [3]

1: [https://www.bankofamerica.com/privacy/accounts-
cards/shopsaf...](https://www.bankofamerica.com/privacy/accounts-
cards/shopsafe.go)

2: [https://www.citibank.com/us/cards/gen-
content/messages/van/i...](https://www.citibank.com/us/cards/gen-
content/messages/van/index.htm)

3: [https://www.paypal.com/va/webapps/mpp/security/general-
freet...](https://www.paypal.com/va/webapps/mpp/security/general-freetools)

~~~
GuyCall
Nice. I wonder if any UK providers do this

~~~
kalleboo
My bank in Sweden does (Swedbank), so it's not a US-only thing. They even have
an iPhone app for it.

------
jyap
Remember when the Linode customer service portal was compromised which exposed
everyone's VPS?: <http://julianyap.com/2012/03/01/compromised-linode-vps.html>

In that case, specific Bitcoin users were targeted.

It's pretty much why I don't trust Linode.

You can't trust a company which puts random AMI BIOS files on the main index
directory on the main web site. You can't trust a company that can't even lock
down their own Linode customer service portal (which could lead to a breach of
each and every customer's VPS).

Perhaps history is fuzzy for people when new announcements come out or low
prices are around.

------
xhrpost
Off topic but still relevant, but doesn't it seem a bit primitive that
companies have to store you CC# for recurring payments? The one number that
uniquely identifies your account and everyone you want to re-use it has to
keep a copy. Couldn't the credit card company issue some unique ID to each
vendor for recurrent payments? Ex. the vendor issues your CC# to the CC
Company for charge and recurring process. The CC Co. responds by replying back
with authorization and an ID unique to that vendor that says "Use this number
for charging this customer again, but it will only work coming from you, so if
you lose it, it can't be used elsewhere". The vendor then discards your real
CC number.

~~~
msbarnett
> Off topic but still relevant, but doesn't it seem a bit primitive that
> companies have to store you CC# for recurring payments?

You really really don't have to. Any payment processor that isn't horribly
incompetent does the unique token authorization scheme.

Storing CC #s for recurring payments is solely the domain of incompetents who
have no business accepting payments from anyone.

~~~
kintamanimatt
What if you want to change processors? If you weren't storing the CC details,
wouldn't you have to have customers enter all their details again? I imagine
this could cause a drop in revenues due to people either forgetting,
procrastinating, or just not bothering.

It's never cool to be actually- or quasi-locked into a vendor.

~~~
tomchuk
Most decent processors have processes in place for the transfer of CC numbers.
I was involved with this process at a decent-sized magazine, it involved armed
security, an encrypted hard drive in a locked container and millions of
dollars of insurance. It's not an easy process, but is possible.

~~~
kintamanimatt
Sounds ... expensive.

------
pibefision
I had a VPS on linode.

I think that Linode did a big mistake here. Let's wait for a formal
communication.

But this is the moment to support them. Yes, maybe sounds crazy.

When you host on any third party datacenter, you take risks that something
like this could happen. So, deal with it. Check your credit card, if your
receive something wrong, call to your card and that's all. But we need to
support also the good work, and this guys do great work in the hosting
business. Just my opinion.

~~~
eridius
Well said. It's a fact of life that companies get hacked. So it's no surprise
that it eventually happened to Linode. If you flee somewhere else, all you're
doing is hoping that the other company you run to won't get hacked rather than
using any logical thought.

I can think of two good reasons why you should flee Linode. It remains to be
seen if either are actually true, and until indications say yes, then panic is
unwarranted:

1\. If it becomes apparent that Linode is far more vulnerable to hacking than
other hosting providers. But one hack alone does not prove this.

2\. If Linode grossly mishandles the situation. There have been a couple of
allegations to that effect so far, but nothing substantial. I don't see any
reason to claim that they've done this yet.

~~~
tomjen3
Linode has already grossly mishandled the situation by not coming out with a
complete statement about what exactly happened. I only read this news because
it was posed here -- no email notification, no update on their homepage, no
twitter, no nothing.

The alleged hacker has made serious and specific claims, and Linode has done
jack shit; without more information, how should I proceed? I don't want to
call my bank and waste time getting a new credit card (not to mention
replacing a million and two services) without a confirmation and I can't get a
confirmation because Linodes people are having a circle jerk (or whatever the
hell they do).

~~~
mitchellhislop
> I only read this news because it was posed here -- no email notification

There was an email notification a few days ago.

------
sergiotapia
So what happens now to all the goodwill Linode has amassed through the years?
Does it all turn to shite, almost overnight?

This sounds very very bad, and as a customer it's very off-putting.

~~~
biot
[Warning: imperfect analogy follows.] It's one thing if Linode is like someone
who gets drunk and crashes their vehicle. That's 100% their fault and they've
burned any goodwill. In this case, however, Linode is like someone who was
carjacked. Perhaps Linode shouldn't have been driving that type of vehicle in
an area known to have people attempting to carjack every single vehicle that
drives by. Perhaps they should have installed thicker bullet-proof glass. Or
even have taken measures not to trust any locks that the manufacturer insists
are secure but have zero-day exploits. Regardless, Linode is still the victim
of unscrupulous criminals. Maybe they could and should have done more but the
bigger question is now that they've been carjacked, what are they doing to
ensure that the carjackers haven't installed anything malicious that still
remains in the vehicle?

~~~
brandon272
I think a more accurate analogy is to say that Linode is moving your important
files from one location to another in their moving van. They park at a 7-11 to
run inside and grab a snack, leaving the van unlocked. An intruder comes
along, opens the unlocked doors, makes a few copies of your files and leaves.
Linode gets back in the van, notices the intrusion, does nothing except tell
you that "you have nothing to worry about, but you may as well change your
locks" and then when the truth comes to light, they basically stop returning
your phone calls.

"No comment."

------
agwa
FYI, I just learned from Linode support that accounts have both a default LISH
password AND a default API key, which means that _even if you've never set a
LISH password or generated an API key before_ , you still should still go and
reset them. This is not what I would consider expected or desirable behavior.

~~~
eevee
I've never generated an API key, and Linode showed it as blank. I generated a
new one anyway, but I can't imagine how they'd have a default key _and_
somehow not show it in the UI.

~~~
agwa
I know. I've asked for further clarification, especially since their email on
Friday said API keys should be reset "if applicable."

Edit: _Groan_ , here's their clarification. It's starting to look like they
don't know what the heck they're talking about:

"Thanks for getting back to us. To be extra cautious it would not hurt to
regenerate your Linode API key. You can do that in your user profile. Please
let us know if you have any other concerns we can address."

~~~
dhess
After seeing your original post here, I also asked for clarification, and
received a similar reply from support:

    
    
      The Lish password is set to a random string by default, however we would still recommend resetting this password even if you had not set one manually previously.
    

I had expected that if the password was not set, then password auth was
disabled. I've told them that's what I want and have asked when it will be
implemented.

~~~
potater
I'm kind of upset they didn't clarify this in the initial email/blog entry.
The way it was worded ("if applicable") implies that resetting the API might
not be necessary in some cases. I think it is reasonable to assume that those
who never generated an API key in the first place would've fallen under such a
bucket.

Now it sounds like basically everyone should have reset their API key. Bleh.

------
kaolinite
Just rang my bank to cancel my debit card. Hate doing that. Now I have a week
or two of failing payments, bills, etc to look forward to.

I will probably be moving away from Linode after this. The poor response to
this and lack of full disclosure, plus reading that they're using ColdFusion
(wtf?), means I don't feel I'll be able to trust them any longer. It's a shame
because their UI and service is generally fantastic.

~~~
eddieroger
There's nothing wrong with ColdFusion, especially if you've had it around for
a while. It's not as glitzy as Rails, but it works and it's still supported
and modern. Besides, this isn't ColdFusion's fault. Leave because Linode
violated your trust, but not because of the programming language they wrote
their site in.

~~~
kaolinite
It's closed-source, made by Adobe and seems to have a bad security record -
there are 3 things wrong with it.

Besides, it's not the reason I'm leaving - it just makes me question them. I'm
not after glitzy. If anything, I'd have expected Linode to have been written
in Perl or something.

~~~
caf
(made by Adobe) ⇒ (closed-source) ∧ (has a bad security record)

------
akennberg
Love the whole side discussion about bitcoin from the supposed attacker:

06:07 < ryannn> They say there's no 'central weak point'

06:07 < ryannn> Yeah there is, there's the developers

06:08 < ryannn> There's been bugs in the client that have allowed the
blockchain to split previously

06:08 < ryannn> One could just backdoor the bitcoin client binaries, not the
source.

06:08 < ryannn> Nobody would figure it out until it's too late

[http://turtle.dereferenced.org/~nenolod/linode/linode-
abridg...](http://turtle.dereferenced.org/~nenolod/linode/linode-abridged.txt)

~~~
drivebyacct2
I'm curious how he's going to backdoor the binaries running on my computer
built by the launchpad servers without anyone noticing. Granted, I'm not
checking the commit logs every time apt gives me a new version, but all the
same.

------
lawnchair_larry
Well I'll wait for a response from linode, but it certainly looks like they
were very dishonest. I think I will close my account.

~~~
hosay123
So you're unfortunate enough to be a customer who had their CC leaked. So you
spend 5 minutes changing your password (you use unique, non-formulaic
passwords, right?) and 15 minutes on the phone to CC company to ask for a new
card. Then you use your backup card for 2 weeks (you have a backup card,
right?)

A month later, spend 30 minutes on the phone with CC company only if strange
transactions appeared.

Not the end of the world. The CC industry is set up well to handle this kind
of thing.

~~~
brandon272
To dismiss this breach seems odd to me. The tech community in general has
placed a lot of trust and faith in Linode over the years. The shareowners at
Linode have surely been great beneficiaries to that. Part of that "unspoken
agreement", if you will, is that Linode be competent at what they do and that
means keeping your data and information secure.

If even an iota of what I read in the abridged IRC log is true, Linode doesn't
seem to care much about security or protecting Linode customer data. I mean,
storing "encrypted" card numbers alongside private/public keys? Really.

~~~
hosay123
Sigh, really? Ok, you typed your credit card number into a web browser at some
point. If your sole reason for doing so was "I absolutely trust the people on
the other end of this socket not to do what 99% of all people handling credit
card data do whether they pretend otherwise or not", instead of something like
"hmm that reminds me, I haven't scanned last month's statement yet", then the
problem lies squarely with you, the uninformed consumer.

I will happily dismiss this breach, not because they didn't make some amateur
crypto mistake, or because they weren't using freaking ColdFusion, or because
they were storing data in some nice compartmentalized form, I reject because
_this happens every single day and has done for decades, and there is an
entire sub-industry built around its after-effects_. If you don't understand
this _you shouldn't own a credit card_.

If you type a credit card number in online not expecting to recuperate any
damage caused from your card company, _call them up now for clarification or
cancel the damn card_. That's equivalent to stuffing cash in an envelope and
posting it to Nigeria because some prince promises he'll keep it in a safe for
you. It's 90% the reason you should be using credit cards _in the first
place_. _Think_.

Linode should not be rubbished here. They've got one of the largest VPS
installs around, so they most likely know their shit. They make an ultra-
common CC mistake that has happened daily for almost 20 years now, by
companies large and small, got pwned due to a bug in _someone else's software_
, and you think I'm going to play along with the righteous indignation
bullshit here? GTFO.

Let he without sin cast the first stone. Despite 20+ years' experience I
_still_ cannot cast that first stone. I make bullshit mistakes like this every
day, and despite your grandiose delusions you probably do too.

As for whiners complaining about their data suddenly being insecure, well,
data security 101: you're making the same bullshit mistake Linode are making,
and despite that you're complaining about it. If you care about data security
in the "cloud", hosting it on a freaking VPS is not the way to do things.

~~~
culturestate
So because companies A through X are irresponsible with data, customers should
regard that as acceptable and give company Y a free pass to do the same? I
don't understand how a reasonable analysis of the situation can come to that
conclusion.

~~~
VLM
You don't know the names of companies A through X, or supposedly safe Z for
that matter. All you'll be doing is an enormous amount of work and bother to
move from Y to, lets say, A, because you think you'll be more secure but
unfortunately if anything its probably the other way around, its just that A
hasn't been hacked... yet... so far as they know...

none of them get a free pass they all suck, but the one that just got busted
is probably going to be a little more security focused in the near future.

Hmm stay at a place that just got burned, or expend lots of effort to move to
a place that hasn't been burned yet...

------
Tomdarkness
Found it interesting that Linode uses Coldfusion. Wonder if Adobe has anything
to say about the apparent 0-day.

If the hacker's claims are true (Would appear so, the directory listing checks
out) then Linode really need to address this ASAP. Passwords are one thing but
to have CC details leaked is even worse. I'm not familiar with CC processing
but it seems like bad practice to store the encryption keys on the web server.

~~~
tptacek
It wouldn't take a zero-day flaw in the Coldfusion stack for a CF application
to have an undocumented vulnerability; in fact, it's much more likely that the
vulnerability is in the application code than in the stack itself.

~~~
127001brewer
A patch has recently been issued (09 APR 2013) by Adobe for the various
versions of ColdFusion:

 _"This hotfix resolves a vulnerability that could be exploited to impersonate
an authenticated user (CVE-2013-1387).

"This hotfix resolves a vulnerability that could be exploited by an
unauthorized user to gain access to the ColdFusion administrator console
(CVE-2013-1388)."_

[http://www.adobe.com/support/security/bulletins/apsb13-10.ht...](http://www.adobe.com/support/security/bulletins/apsb13-10.html)

~~~
tptacek
Yep that's bad.

------
thaumaturgy
Linode hasn't been very forthcoming in the past where security "incidents" are
concerned:

[http://arstechnica.com/business/2012/03/bitcoins-
worth-22800...](http://arstechnica.com/business/2012/03/bitcoins-
worth-228000-stolen-from-customers-of-hacked-webhost/)

[http://forum.linode.com/viewtopic.php?f=20&t=8509](http://forum.linode.com/viewtopic.php?f=20&t=8509)

I had really hoped that they had changed their stance on incident management.
_If_ it's true that they suppressed information about a possible wide-scale
compromise where customer data could have been affected, then despite
everything else about their service that's so great, there's no way anyone
should want to continue to be a customer there.

Given Linode's past behavior and the information provided in the IRC chat, I
think there's reasonable suspicion that customers' password hashes were stolen
and Linode wasn't completely honest in their recent email to customers.

~~~
taylorbuley
Did you see <http://www.linode.com/linode4.css> and
<http://www.linode.com/linode3.css>?

Nevermind the security concerns. These guys aren't using version control!

~~~
douglasheriot
Renaming CSS files with numbers or whatever is one legitimate way to handle
cache invalidation (not evidence of not using version control)

~~~
laxk
It is not the best way to handle cache invalidation. Why not
/linode.css?v=<file_timestamp> ?

~~~
thaumaturgy
This is inconsequential.

------
brandon272
Is it just me or is each passing minute without an acknowledgment of this
issue bad for Linode? There's tens of thousands of customers right now who
would kind of like to know if they need to request new credit cards or not, or
don't know about this and deserve to know that their VPS provider's credit
card database has been compromised.

------
thehermit
The chatlog does provide some evidence that it is indeed the hacker, but does
little to convince me that he got CC info and Linode is not telling us the
whole truth. The evidence he provides is just simple source code snips and the
directory listing, which would be expected based on what Linode has told us.

This could very well be the hackers own submission to /. trying to get more
attention for his hack by claiming he has CC numbers which I doubt he has.

~~~
lawnchair_larry
Doubting the claims of a hacker after it's _known_ that they compromised a
system is really bad OpSec. Consider everything burned.

~~~
thehermit
That's a fair point. I'm speaking more as a judge/jury view on the situation
though. I don't think Linode users should panic and run for the hills just yet
based off this alone.

------
whalesalad
I'm looking forward to seeing an official response from Linode on this.
Hopefully they are fast and honest about it. I've been a happy customer for
quite a while, but this is definitely a concern.

~~~
UnoriginalGuy
They weren't exactly fast and honest the last time a break-in happened. In
fact even to this day nobody is quite sure what went down aside from tons of
Bitcoins going missing!

------
ConceitedCode
I guess this is why they wanted everyone to reset their password 2 days ago.

<https://news.ycombinator.com/item?id=5541915>

~~~
Macuyiko
Not only that, it also makes me wonder about the free RAM upgrade from almost
a week ago. Some people are reporting their Linode credit cards being used for
fraudulent purchases as far as a week ago, so this might have been a move to
gain some pre-emptive goodwill.

I don't know though... will wait until more details are available but will be
keeping an eye on CC statements / VPS alternatives.

~~~
eridius
"Someone hacked us and stole customer details... quick, give everyone more
RAM!"

Doesn't sound very plausible to me.

~~~
nivla
Logically, they couldn't have planned it all in such a short period of time.
However if they did, from a business perspective it is very good plan. Without
the upgrade, today some customers would have had two reasons to leave Linode,
now they only have one.

------
gklein
That may explains the seclist hack too... <http://seclists.org/nmap-
dev/2013/q2/3>

~~~
Zancarius
If that's the same one, then that link implies Linode already fixed the issue
on or around the 13th. So, I'm wondering if the silence that has some folks
here up in arms is indeed because they've been instructed by law enforcement
to keep quiet pending the investigation.

Now if we could only stay the torches and pitchforks for a while before this
gets sorted out...

------
gee_totes
Is there any confirmation on this?

[edit] Just looked at twitter, this tweet doesn't look good:
<https://twitter.com/Jamiesingleton/status/322730588459114500>

But it may just be random coincidence.

[edit again]

Links from slashdot article:

IRC chat: [http://turtle.dereferenced.org/~nenolod/linode/linode-
abridg...](http://turtle.dereferenced.org/~nenolod/linode/linode-abridged.txt)

Link in IRC chat (i think it is of linode.com's web directory):
<https://bin.defuse.ca/hq0Ay8RzpKdR6vQwYxnmhc>

~~~
firemanx
That tweet would suggest to me that someone's credit card was stolen via
another method and used to purchase Linode services, rather than the other way
around. Ie, I don't think that was related to Linode breach.

------
tmslnz
So, they asked us to change the password for reasons they cannot yet tell. The
fact that they cannot lay it all out for our curiosity is driving most of
people here off the rails. I don't have any problem with letting them verify &
deal with the issue properly before releasing any public statement. Anyone
who's ever been with them has never received less than perfect customer
service. Can we give them a break, wait and hold our judging wands for a
moment? It's not like they are staying mum out of spite towards their
customers and the press…

------
neya
I am an ex-customer of Linode and I'm still worried about this incident - Do
they still store your card after you've quit the service? This is terrible :(

~~~
minib
I've asked them this question. Here is the answer:

Credit card information continues to be stored in our database in an encrypted
format, and the decryption key is not stored electronically. We are working on
a process on remove the credit card details of past customers on request and
can handle this for you soon if you would like. If you have any further
questions or concerns please let us know.

~~~
run4yourlives
So the answer really doesn't answer the question.

(It appears the answer is no, but given they haven't been forthright in the
actual details of the hack, possibly not)

------
DigitalSea
As someone who has been a very happy and loyal Linode customer for a long time
now, this whole situation paints an image of Linode I otherwise would never
have thought. The fact they apparently had both the private and public keys
for the credit card hashes in the same location as one another is beyond
belief. The very fact that Linode failed to mention they made a deal with the
attackers and then reneged on it all without telling anyone makes me sick. I
don't want to bash Linode purely because everyone else is, I am legitimately
concerned here that my personal details have been compromised.

I thought Linode was different but based on their lack of transparency in this
matter, I'm seriously considering just moving all of my sites to DigitalOcean,
Rackspace or even AWS instead. This makes me wonder who originally cleared
them for PCI compliance in the first place. This is a huge violation of trust
and now I've got to keep my eyes focused on my credit card statement for
fraudulent transactions, the bank I am with ANZ however has great fraud
detection systems and considering I'm in Australia any transaction should be
easily reversible, but the fact there is a possibility my card could be
fraudulently used saddens me.

Linode needs to come clean about this situation now.

------
perlgeek
Really off-topic, but still sad: This is a link to slashdot, but it's on HN's
frontpage before it's on slashdot's front-page (if it'll ever get there). (And
IMHO that's sad, because /. used to be top notch).

I've noticed before that stuff from the HN frontpage appears on /. one to
three days after, but I've never seen it for links to slashdot :-)

~~~
Zancarius
I rather wish this was a link instead to the original thread rather than to
Slashdot. It isn't a big deal, but it certainly would have saved an extra
click.

Also off-topic: I've noticed that as well with Slashdot, which is why I lurk
HN pretty regularly now. Plus, some of the front page material on /. does more
to insight angry discussion, and the community has become increasingly more
vitriolic.

At least here, even if someone's brash, they're fairly honest about it (in
general). I've even seen a number of disagreements that have been respectful
and cordial. It's sad to say, but that's a rare thing these days.

------
ebtalley
Just got a response from linode: somethings not adding up?

\----------------------

dportalatin 30 minutes ago Hello,

Thanks for getting into contact with us about this. Linode has found no
evidence that payment information of any customer was accessed. We have
implemented all appropriate measures to provide the maximum amount of
protection to our customers. If you have any other concerns we can address,
please let us know.

Regards, Dolores

~~~
treahauet
They're doing canned responses right now guys. Just got this exact same
message myself.

------
tomjen3
To those of you who have claimed that your CCs have been abused -- I checked
mine (which I used to pay for Linode) and it hasn't been used to do anything
funny.

~~~
kansface
I checked mine and nothing untoward has taken place. All the same, I put it on
hold. Everyone should assume their CC is compromised.

~~~
tomjen3
Why? It is a major waste of time to insert it into all the places that I have
used it and should it be abused, then I just dispute the charges.

------
EwanToo
Is there another way of learning the URLs like

<http://linode.com/googledebcc14d3c9f777a.html> and
<http://linode.com/y_key_57284cb2de704e02.html>

If not, it seems likely that he did have read access to the main website
filesystem at least

------
etjossem
The best part is at the very bottom of the log. A customer enters the IRC
channel to ask for support after ryann (the hacker) finishes explaining the
attack.

Customer: "hello, i forgot my password and linode's email reminder service
doesn't work. i checked spam box but there's no email from linode." Linode
Guy: "ryannn: can you give him the password?"

------
iriche
And right now we got confirmed on IRC that data is out, we do not know how
much, 4 last numbers of CCs are out but that doesn't mean that full CCs are
out.

E-mails and logins are also out it seems.

~~~
duskwuff
Last four digits of CC#s are often used to identify them to users ("Visa
ending in 1234"), and are specifically OK to store in cleartext. So that's not
necessarily a big deal.

------
jtokoph
OFTC has disabled Linode IRC channel:

 __* mode/#linode [+m] by tjfontaine

<tjfontaine> this is what I'm going to say, as a network representative

<tjfontaine> regardless of what has or has not happened with linode, OFTC
cannot tolerate release of sensitive information with itself as that mechanism

<tjfontaine> this channel is moderated until staff determines otherwise

~~~
regecks
Channel now unmoderated. Edit: HTP came in and trolled (pretending to be ex-
Linode), re-moderated.

------
segmondy
Is this why Linode doubled the RAM? To bribe us and make us stay. I'm pretty
pissed off about this and will be exploring other options. I'm not pissed off
they got hacked, I'm pissed off they are hiding and not being forth coming
about it. A simple, "We fucked up, we are going to take steps 1, 2 3 to fix it
and reduce the likely hood of this ever happening again" will make me happy. I
understand that any server can be hacked. I'm stunned that they are storing CC
details on the servers, there are ways to go about this without storing them
if you want recurring billing.

~~~
eridius
I'm stunned that you, and plenty of other people in this thread, are taking
the anonymous IRC person's word as the gospel truth.

~~~
cwb71
I am not sure that "gospel truth" is a fair characterization.

Anonymous IRC person has provided verifiable details that strongly suggest he
or she had access to Linode administrative systems. Fyodor's post to nmap-dev
supports the notion that customer nodes were accessed as well.

Linode has provided no details or evidence of anything.

I don't think one has to take that IRC log as gospel truth to be reasonably
concerned about the security of their data stored by Linode.

~~~
eridius
The only "verifiable detail" I saw in the chatlog was the output of `ls` in
the http root. And that's only verifiable because you can try to access that
weirdly-named HTML file and get a 200 back. Honestly, that doesn't tell me a
whole lot.

Everything else, such as the password hashes, don't seem at all verifiable
(even if someone were to crack any of the hashes, you can't verify that the
password worked at the time of the hack because Linode has presumably changed
them all anyway).

------
krakensden
A link to an unapproved /. submission. Fascinating choice.

------
api
That explains why my LLC's business debit card was turned off due to a fraud
alert.

------
jcoder
Well, here's a data point: <https://news.ycombinator.com/item?id=5554071>

Since Linode has proven in the past that they aren't the worst at
communication, I can only assume some entity really has them over a barrel,
considering the curt and callous responses there.

------
benmw333
I just requested a new CC. Can never be too safe. Not sure I'm going to stick
with Linode now... sucks because they seemed to be doing so many things right.

~~~
wesray
I have the same sentiments, new cc and might move off linode, who I have been
with for 4 years.

------
antsam
This is disappointing and scary. A friend on another forum posted that some
guy on IRC told him the last 4 digits of his CC and his e-mail address. I just
called my bank and cancelled my current CC and give me a new number. I really
liked Linode too :(

~~~
dhess
FYI, the last 4 and your e-mail address are both visible in plaintext from
your /account page in Linode Manager. Obviously, still disappointing and
scary, but it doesn't necessarily mean that whoever has that information also
has the full CC number.

------
oliwarner
I echo everything everybody has said. This sucks because _we don't know_ one
way or the other. This may be some asshat FUDing away, or this may be a
genuine hack.

It's hard to get angry at anybody but Linode needs better auditing around
sensitive data so they can tell people one way or the other.

I resigned to the fact that I'd find it easier to change my card details in
30-odd online shops than it is fight my bank to get my money back. Now I can't
make _any_ purchases for 7-10 days.

------
spydum
I don't know the skill level of the Linode folks, but could it be that they
left a honey-pot CC DB out for this hacker to discover on purpose? Keys
sitting right there?

------
adrianwaj
I wonder if they've deleted CC details of previous clients. Is Linode going to
contact all relevant customers? Seems like the right thing to do. Not everyone
reads HN.

~~~
regis
I was just told by a customer service rep at linode that I "shouldn't trust
everything I read on the internet" when I inquired about the possibility of
deleting my personal information from their server. This seems like an
extremely inappropriate way to handle this situation...

~~~
adrianwaj
By the sounds of it, they probably don't know the full extent of who and what
was taken, otherwise they could just email everyone involved and say: "there's
been a breach and it affects you" ... or they could contact CC companies.

------
jervisfm
Does any one know of other VPS providers who are about as good as Linode in
terms of quality and price but do not have a history of security issues ?

------
tlrobinson
Hmmm, Linode claims they emailed their customers about the password reset, but
I never got an email (nor in my spam folder)

~~~
buro9
Saturday morning:

    
    
        From: "Linode" <support@linode.com>
        Date: Sat, 13 Apr 2013 00:11:09 -0000
        Precedence: bulk
        Return-Path: 6723614.1706014@e2ma.net
        Message-ID: <knuab.c9dae.xxx@e2ma.net>
        List-Unsubscribe: <http://e2.ma/optout/c9dae/xxx>
        X-Test-Mailing: no
    
        Dear Linode customer,
    
        Linode administrators have discovered and blocked suspicious activity on th=
        e Linode network.=C2=A0 This activity appears to have been a coordinated at=
        tempt to access the account of one of our customers.=C2=A0 This customer is=
         aware of this activity and we have determined its extent and impact.=C2=A0=
         We have found no evidence that any Linode data of any other customer was a=
        ccessed.=C2=A0 In addition, we have found no evidence that payment informat=
        ion of any customer was accessed.
    
        We have been advised that law enforcement officials are aware of the intrus=
        ion into this customer=E2=80=99s systems. We have implemented all appropria=
        te measures to provide the maximum amount of protection to our customers. O=
        ut of an abundance of caution, however, we have decided to implement a Lino=
        de Manager password reset. In so doing, we have immediately expired all cur=
        rent passwords. You will be prompted to create a new password the next time=
         that you log into the Linode Manager. We also recommend changing your LISH=
         passwords and, if applicable, regenerating your API key.
    
        The following represent best practices in creating new passwords:
        -- Avoid using simple passwords based on dictionary words
        -- Never use the same password on multiple sites or services
        -- Never click on 'reset password' requests in unsolicited emails - instead=
         go directly to the service
    
        We apologize for the inconvenience. If you have any questions, please do no=
        t hesitate to contact our support team at support@linode.com.

------
bzalasky
So glad the CC I had on file expired this month. +1 for procrastination.
Here's to hoping Linode gets their shit together.

------
nazka
Again! I seriously start to lose trust in the startups' security of the
Silicon Valley...

~~~
u2328
They're based out of New Jersey, actually.

------
bradleyland
I'm most concerned about Lish access. I'm not liable for fraudulent charges to
my CC, so all I'll suffer is inconvenience. I'm most concerned about access to
my VMs, which can be a business terminating event if things go extremely
badly.

------
callahad
No suspicious charges on my end, yet, but I did just find out that my bank
(USAA) is finally offering cards with Chip-and-Pin, which causes the card
number to get re-provisioned. No time like the present, eh?

------
jkeesh
I recently sent an email to linode support, and got a very murky response. I
used my debit card on linode, and it was recently used on transactions I
didn't make in random parts of the world that I'm not in, so I had to cancel
it. My first guess was that it was linode, and all of the posts here make it
more likely.

Essentially: I am a linode customer. My cc details were somehow leaked. Adds a
data point here.

------
jimktrains2
Why would you store credit card numbers? Almost every processor has some
method to store them with the processor and all you need is a token.

------
epo
When did this breach occur? The implication is that it was a couple of weeks
ago. As it happens my CC expired at the end of March.

------
mikebailey2
I've seen no evidence of passwords or credit card numbers being accessed.

The purported "evidence" is a list of supposed entries in a public_html
directory.

<https://bin.defuse.ca/hq0Ay8RzpKdR6vQwYxnmhc>

Has anyone seen any evidence that this is not simply a hoax?

~~~
unnali
Well, yes: try some of those files out:

<http://linode.com/MyAddress.class>
<http://linode.com/y_key_57284cb2de704e02.html> <http://linode.com/ispcheck>
<http://linode.com/k5_3>

etc.

------
wildmXranat
the card with my linode account has a minimal amount total allowed to charge.
I'll go through my cc usage, but at worst they could've gotten a couple of
ebooks. as a long time customer though, Im conflicted and motivated to move
off linode to another provider.

------
cjh_
If this is indeed true I will be very disappointed, I have had an otherwise
great relationship with Linode.

I await for more data and hopefully an official response from Linode.

If the worst case is true, what are some good alternatives for Linux VPS
hosting?

------
ercu
this is why i always use my "virtual credit card" on web sites that store CCs.

------
level09
I'm just curious, is this in any how, related to the recent RAM increase ?

~~~
ZaneA
Doesn't look like it. If you believe the IRC logs, the attacker broke in using
a "0day" exploit of ColdFusion, which is being used on the site. Not because
of a flaw in RAM upgrades. Although I believe that Linode beginning to offer
free trial accounts is probably responsible for the increase in attacks
lately...

------
icn2
I dropped linode two months ago. Hope they have purged my cc information from
their system. Well I logged in just now with old credentials. Cc information
is still there, not so good...

------
yarou
Sad day to see that Linode is not immune to data breaches. That being said, I
don't necessarily support the witch hunt that seems to have taken place on its
reputation.

------
hevara
<http://blog.linode.com/2013/04/16/security-incident-update/>

------
plusbryan
When was this alleged hack supposed to have happened?

------
muloka
Fortunately my CC information on file is expired.

------
stefantalpalaru
A funny comment on IRC:

    
    
        "One bit of good news from all this hubbaloo: more space will be opened on the new E5 hosts from customers leaving \o/"

------
godgod
Being a former customer I decided to contact Linode support. I wanted to know
if we have anything to worry about as we used to have a credit card on file
with them. Notice how they say "Decrypted" which to me is not the same as NOT
COMPROMISED.

Here is their response:

Thank you for reaching out to us. We do archive customer credit card details.
At this point there is no evidence that customer credit cards have been
decrypted.

