I've been living comfortably on Linode servers for over three years. This is like suddenly being evicted and having to pack my stuff up and find another apartment.
I have to wait for some sort of verification for this but if true then I have to leave Linode. I have client sites hosted here - not for cost reasons, just because I like Linode.
For the sake of $5 a month I can't even take the slightest risk of being criticised for using Linode. And this lack of transparency could be a nail in the coffin here.
I don't want to waste a couple of days on this but that's what's going to be involved if this is true.
I just called up my bank to tell them to 'block' it as a precaution (I will now have to give them a visit later today to get a new card). I encourage all other Linode customers to do the same, because it'll be easier to just spend half an hour doing this instead of spending hours upon hours disputing specific transactions.
Linode customer support keeps saying they have "no comment" on this issue (which I suppose does make sense -- I'm assuming they've been ordered by law enforcement persons to not share details), so as we're not being given much information to work with... just treat this as a worst-case scenario (all names, addresses, credit card numbers, etc. have been compromised). Do operate now with the assumption that all of this data has been compromised and may very well be public soon.
I live on the internet. Put my credit card out on many services. Over the last 5 to 8 years I've had my credit card numbers taken I believe 4 times.
Never had to dispute it once. These Credit Card companies and Banks have a stake in not allowing your account to be drained.
I think it would be a waste of time to go out and cancel our CC until hearing from Linode that yes, CC information was taken.
With that said - almost everyone seems to feel comfortable handing out their credit card to random taxi drivers, waiters, sales staff - with no idea whether a copy of their information is being taken down. Heck - if you give them the Credit Card, they even get your CCV as well.
I even wrote a blog post about it. I probably don't know what I'm talking about, but these were my thoughts at the time:
Wishlist – A Method to Pre-Approve and Track Credit Card Transactions
A business using a credit card doing business with a relatively small number of vendors wanting to first avoid credit card fraud (stolen numbers) and secondly wanting to easily track down the offending business.
The business would like to approve particular vendors to use the credit card with number 0000-0000-0000-0000 with each individual business pre-approved to run the transaction with a 5th set of identifiable numbers, so something like 0000-0000-0000-0000-0001.
If the credit card is used to make a fraudulent transaction, then ideally, they would have had to have used the 5th set of identifying numbers. This 5th set of identifiable numbers would then allow for easy tracking of the offending vendor, which would allow the business to either re-think doing business with them, or to serve as a starting point discuss security issues with the vendor’s credit card transaction processes.
Basically, I believe there may be a need for a new or value added credit card type service. This transaction type would require a 5th set of numbers which have been assigned to pre-approved vendors. This 5 number set (ie. 0000-0000-0000-0000-0002) credit card transaction would most likely prevent theft right off (because the vendor is pre-approved and should provide their own private key (ie. CCV) to put through the transaction). Secondly, if and when the credit card number is stolen and used to make a fraudulent purchase, then, at least with the 5th number set a vendor can be identified and security policy with them can be re-evaluated.
My bank in Sweden requires MasterCard SecureCode for all online transactions on their debit cards. Stores that don't support it simply won't work with the card.
So, it's up to the bank how secure they want it to be. The technology is there.
My advice is different, though. I notice that I tend to get lucky in places where people can have very aggravating experiences. I'd say that if you've had problems before, then anticipate problems this time around too. If you haven't, then don't bother.
Also, great idea. But a pain, because most of my bills - cell, internet, insurance(s), etc all go through my credit cards. Is a gigantic pain to change the numbers.
Are you thinking of expired cards? That is different.
Unless the attacker dumped them all (semi) publicly, the more likely explanation is that the breakin caused people to check their accounts and a statistically normal percentage of them showed fraud from another origin. But anybody who sees it will be sure to get online and find others in a similar situation.
Everybody would be doing themselves a big favor if they stopped treating CC info as the #1 scary OMG data theft. The banks programmed you to care because congress made sure they're liable instead of you. Theoretically you might owe $50 due to fraud but practically you never pay a dime. Sure it's a bit of a pain in the ass to get resolved, but it's not worth stressing about until it happens.
I'd be way more concerned if my hoster lost my contact info, ip logs and identity challenge questions & answers.
I contacted Linode support and they've said in clear terms that they have no evidence that payment information of customers was accessed. I initially signed up for Linode because my friends spoke highly of the tech people working at Linode. Right now amidst all the commotions it's ryan's words (some anonymous dude who joined #linode/irc.oftc.net) vs. an established company's. I'm just going to now stop worrying and get back to my work.
On an interesting note, the big target who actually incurred identifiable damage was seclists.org: http://seclists.org/nmap-dev/2013/q2/3
Well that's a first. They were very evasive about it earlier.
With this lack of transparency, I feel like I had no choice but to block my card.
> In addition, we have found no evidence that payment information of any customer was accessed.
The question isn't transparency, but trustworthiness. Either Linode is telling the truth, and this anonymous IRC person with a pastebin is trolling everyone, or Linode is lying (or alternatively, Linode is incompetent and simply didn't detect the CC access). At the moment I'm going with Linode is telling the truth, because honestly, am I going to believe an anonymous person on IRC over a company I do business with?
Secondly, if they hold that data, it's possible one day someone will find a security breach and will access that data. The best solution is to never hold that data.
Since I don't trust most of the systems I use AND Linode has not denied it holds that data... I'm more inclined to believe in this anonymous IRC guy and err on the side of caution.
If Linode had come out and said "Look, we don't hold your CC number in our database" then I think there would be very little reason to be concerned. However...
Mine is not a CC or a debit card, it's a 'prepayed card' so liability is limited.
Still, I'm considering calling the lost/stolen line of the card.
Thank you for contacting us. We have no evidence at this time that any payment information was compromised.
"We appreciate the response, and we can assure you that we have implemented all appropriate measures to provide the maximum amount of protection to our customers."
So I guess the answer would be, if you ended up hooking your PayPal account up to Linode in the way I described, yeah follow the same steps as other cards, otherwise it's not even possible to have a problem.
Waiting for their own lawyers would be a particularly weak excuse. This is a priority, and they are responsible from conveying that urgency to their lawyers.
1. (offline) Boot new and old VM servers from live CD
2. old server: dd if=/dev/sda bs=8M | pbzip2 -c | netcat <newhost> <random high port>
3. new server: netcat -l <same port> | pbzip2 -cd | dd of=/dev/sda bs=8M
Yeah, I mentioned that at the bottom of my post. Using ssh or some other inline encryption would be a good idea if it is a system you care about. If you have a site to site VPN tunnel between your systems, you can skip adding the encryption.
It wasn't there when windsurfer replied to you (I was reading the thread earlier), hence his question.
let SSH handle compression for you instead.
You don't want to leave out the compression or block size however, so add those back in. Most WAN links are low bandwidth enough that compression will not slow things down (this is usually true even on 1 GBE LAN links for pigz) and in my experience the speed-up is substantial.
pigz is much faster than using ssh compression as it is multicore. apt-get it or http://zlib.net/pigz/
In most cases, compression at the raw block level will result in HUGE size savings, especially since a lot of that may be free space. It might even make this transfer tenable.
For the record, I am a Linode customer and just got a new server to migrate a couple of sites into. My plans have not been altered at all by this. I have no data to suggest I should.
I understand your point about the idea that they may be unable to speak to the issue due to law enforcement efforts, but for the moment, acknowledgement would be satisfactory. I would be happy with, "We're aware of the rumors regarding the intrusion at Linode this past week. We are working with law enforcement and cannot comment on details at this time. However, we will provide a full postmortem once we are able to do so."
The problem is when the explanation never comes. It's OK if it's not this second, but tell us it's coming, and then follow through. Complete silence is frustrating.
>Thank you for your inquiry, and I certainly understand your concern. We are still conducting an active investigation and unable to disclose most information at this time. This being said, we do not yet have any evidence that any payment information of any customers have been compromised. We will be releasing further information regarding the incident soon, so please keep watch of our website and blog for said information. If you have any further questions, please feel free to ask.
That actually sounds pretty close to what you're asking for, although I have to say it didn't make me feel much better. It would be nice if they would make a public statement too.
Before it was the widespread hacking that resulted in Bitcoins being stolen. And users were never told exactly what happened and what was done about it.
People have every right to expect the worse with a company with a track record as poor as Linodes.
This from a Terry Pratchett book. Can't remember which one, but it's one of the earlier Discworld books.
I mean, if I was a linode customer I'd definitely be on the phone to the bank, but this guy presented no evidence I'm aware of that he had the kind of access he claims.
> 05:42 < ryan||> credit cards were encrypted, sadly both the private and public keys were stored on the webserver so that provides 0 additional security
> 06:00 < ryann> They did try to encrypt them, but using public key encryption doesn't work if you have the public and private key in the same directory
Thank you for reaching out. We appreciate and understand your concerns. At this time the evidence suggest that this activity was targeting a specific customer. We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.
We have no comment regarding ryan*'s comments in #linode. You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence.
I am sorry that we cannot provide more information at this time. As always feel free to contact us at any time with any future concerns.
If you believe this Ryan guy, credit cards stored on the same server as the key to decrypt them, Lish passwords stored in plain text, they've known for some time and lied about what actually happened and now they're saying "we won't do anything about it" via email?
"You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence."
Edit: not to mention they "made a deal" with the hacker not to tell anyone? What the hell?
That's a rather key assumption. If you don't believe him, then all you have is a trolling (or at least self-aggrandizing) hacker whose credentials consist solely of logging into an IRC channel, refusing to identify who he was working with, and offering no tangible proof of having compromised any CC info.
On the other hand, it's conceivable that if ryan managed to get into the files a customer was hosting on Linode, and that customer was improperly storing CC info, then their customers' info would have been vulnerable, and ryan's claims would be sort of half-true. Even so, that wouldn't directly affect other Linode customers or put liability in Linode's lap.
there is a mixture of truth and lies on both sides, to be honest.
i am annoyed with it, because i reached out to several linode employees privately to given them an opportunity to explain what was going on -- they either said 'no comment' or said my linode was fine.
based on the irc log, that is clearly not the case. which is why i decided to raise my concerns publically.
luckily for me, my linode was not doing anything mission-critical, just some secondary monitoring and running an ircd for a network i like using, but there are others who are using linode for mission-critical work, and they deserve more transparency than this.
Sorry if you get offended that they didn't tell you much more. But seriously? you are not special. This whole thread is a lynch mob.
What Linode did may, or may not, be dumb. They are being tight-lipped so we can only guess.
> We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.
Edit: Oops, didn't refresh before replying. jmilloy got it first! Sorry about that.
But I just checked and my credit card haven't been used anywhere I didn't use it.
† We are not one of those.
Will add that just having gone through an ICANN registrar audit (which by the way were specified and supposed to be done literally 10 or 12 years ago but never requested by ICANN) with a third party company hired (accounting firm) it's total compliance theater.
Add: "hired by ICANN after a bidding process". Same happened with data escrow which was just implemented a few years ago and is operated by Iron Mountain.
The 'lower' your level, the easier the PCI audits are. If you are level 1 you have mandatory external audits. If you are level 2 you have a 'self assessment' which is basically a checklist which says "Yes, I promise I'm in compliance".
If you have a confirmed breach, you are upgraded to Level 1 merchant audit requirements. This is generally quite costly as the external audit is extensive and must be paid for.
"Where are the keys?"
"In the locks."
But if the things he claim in there is even half way true, nobody involved with linode should ever be allowed to be in business every again.
Sysadmins being lazy.
Somebody needs to get a file from a workstation to a remote machine. There's a firewall in the way somewhere that prevents SSH directly between them or one of them is a Windows box that isn't running an SSH server.
The "correct" solution is complicated and takes 5 minutes to setup. So the sysadmin just copies the file to the web server and downloads it with a browser on the other end. Because port 80 is always open.
Its also why we invoice and take wire payments rather than storing CC details. There's just so much to go wrong.
Also PKI is shit for this sort of thing. As demonstrated, the moment that public key is gone, then the whole system falls like a house of cards. For the non believers of this fact, why else would there be a certificate revocation list and root CA updates for windows periodically...
Card vaulting as a service: https://spreedly.com/ ($10/mo for up to 5000 cards)
Also you're fined by the credit card companies if you lose card information. I believe it's a per card fine, so it get expensive really quickly.
Actually I don't get why any company would choose to store credit card information, when most payment providers will do it for you.
But yes, significantly better than other situations.
But if that happens, it's not your responsibility (at least not 100%), it's theirs
Much like Linode are responsible for hosting my clients site but I sigh am accountable when something goes wrong.
Also I do not think, but I am not sure, that fraudulent wire payments/transfers are reversible.
Wire transfers often are not able to be undone once they happen (and are accepted by the other bank). This is the reason why there's so much verification that happens in wire transfers. (I helped develop 2nd factor authentication used for authenticating wire transfers for a financial company)
Credit card charges can be reversed.
Poor show Linode. (edit: worth noting I use the card with other things too, I have no confirmation it was leaked through Linode other than the compromise happening at the same time these supposed leaks happened).
My day job had some Google Apps account compromises last month, and this is making me paranoid that the database that contained hashed/salted passwords for our Intranet hosted on Linode was the culprit. The time frame doesn't seem to line up, but we didn't see any evidence of phishing or compromised desktops being involved.
(Don't want to spread fear - I haven't been able to find any evidence our Linodes were compromised either.)
I'm normally huge a Linode evangelist, but I'm severely disappointed with the lack of transparency on this. I'm debating right now whether to rebuild all our nodes from scratch as I'm not sure they can be trusted.
Linode has addressed the breach, but assured customers nothing of value had been compromised. This infers two thoughts. One: they knew of the breach and lied, thereby unveiling a unforthcoming and dishonest nature. Or two: they did not properly investigative the severity of the issue, thereby suggesting incompetence. Both equally reprehensible.
If I had a choice between a VPS provider who either:
- Only has large issues (eg. leaks credit card data) and goes weeks without reporting them to customers, or
- Has lots of small issues (eg. forgetting to clean the free space of LVM volumes) but fixes them the same day,
I'd much prefer the latter.
EDIT: My bad, apparently the problem was reported on March 27 and wasn't fixed until April 2.
Maybe you don't have anything of importance on your VMs, but plenty of people do. That data could contain credit card data, passwords, etc, etc, etc. It is very much a large issue.
According to DigitalOcean, they stated that this impacts 3% of all machines, only the largest and most expensive servers. None of the smaller plans were leaking data.
I don't know how many credit card numbers were leaked from linode, but I'd guess more than 3%.
Second, if security is important to you, you can use 'dd' to clear the machine yourself before shutting it off. (In fact, good data destruction policies mandate the use of 'shred' et al anyway). On Linode, affected users don't even have a workaround (like this) to avoid information compromise.
Personally I took precautionary measures and just called my bank to replace my credit card, which I think is the sane approach, as when it comes to hacking you have to assume the worst.
However, your statement on "has lots of small issues but fixes them the same day" is just stupidly childish. Linode's issues are bigger just because they are a bigger target.
also a twitter feed for the customers page ?
Visa has a zero liability program for debit card - http://usa.visa.com/personal/security/visa_security_program/...
These are the FTC's rules , I'm not sure if Visa or Mastercard can make them 'better' (give you a larger window). They have an interesting tidbit below their chart -
>If someone makes unauthorized transactions with your debit card number, but your card is not lost, you are not liable for those transactions if you report them within 60 days of your statement being sent to you.
Isn't it free to get a new card? That'd be the easier way than worrying.
Wouldn't hurt just to ask your bank to re-authorise it anyway? It will change the three digits on the back.
a good practice that bankers constantly tell me is to have a separate credit card for online purchases for the fact alone that it is one step removed from your checking account.
I know someone who got their debit card cloned. While the bank eventually repaid him, that did nothing to repay him the additional fees he owned his normal debtors (e.g. rent, utilities, etc).
With a credit card you aren't losing "actual" money. You are losing the bank's borrowed money which the bank pays back. With a debit card you're losing cash which you won't be able to replace yourself and which the bank might take days to weeks to replace.
Even if you NEED to borrow while your credit card is out of commission you can either use the overdraft facility on your debit card or other quick sources of credit. Hard to get quick cash without going to a pawn shop.
> [With a debit card,] "Until the bank provides provisional credit, you could temporarily be out of pocket for the amount in dispute," said Richard Foley, an FDIC attorney who specializes in consumer issues. "This would not typically happen with a credit card because consumers can withhold payment of the amount in dispute."
> Also, as discussed on the next page, consumers have better federal protections when they purchase faulty goods with credit cards.
Not that having your account drained doesn't suck, but your worst case scenario there isn't terrible unless you fail to check stuff and be responsible.
Hopefully, they own up and start being transparent.
If this is true then what alternative hosts should I look at, besides AWS?
I've only used DigitalOcean. My anecdotal experience from running a Chef Server on a 1GB instance has been pretty mixed. The price is good, but network and CPU performance feels very variable to me. A month ago their Amsterdam servers were unable to be resized, and there was nothing about it on their status page. I tweeted and was told they'd be working "some time later today". Doesn't fill me with much confidence in general.
I'd still choose Linode for anything of importance - their long reputation is well earned in my opinion. But, if this breach is true, I hope they handle it well.
That is unfortunately the problem though. If this is true, they have already handled it terribly as it has already been 2 weeks since the attack.
That's just poor security and 100% they're own fault. I accept that there are security issues with every platform, but basic security measures and being transparent is still expected. My biggest issue with them in all of this is not being transparent.
Looks like someone who likes attention on some random IRC channel who is apparently a hacker may have hacked our system and we don't know who/when/where/why/how or what they may have got. Nor are we sure we were even hacked???
It takes time for people to investigate stuff. It's not just a couple hours. Also some random guys words on IRC (who could very well own INSERT RANDOM HOSTING COMPANY for all we know and looking to scare people off Linode) should be taken with a grain of salt.
If I had these, I would immediately cancel my Linode-specific CC# and reissue a new one. I would not have to worry that my other recurring bills will go unpaid, or spend hours dealing with tracking them down and changing them.
Paypal at one time provided this service as well, but it doesn't seem to anymore 
> Set your Valid through date for up to 1 year in the future
I'd really like it to be up to the expiration date of my card. It's probably worth doing anyway, I suppose. Thanks for the heads up. Given the zero-liability status, I'm surprised that banks don't promote this feature more visibly.
I wonder how hard would it be to make a startup like this.
Do some research on "virtual account numbers". I haven't used them myself, so I can't verify how well they work.
Is it a legal requirement to prevent merchants without the CCV from using the credit card?
In that case, specific Bitcoin users were targeted.
It's pretty much why I don't trust Linode.
You can't trust a company which puts random AMI BIOS files on the main index directory on the main web site. You can't trust a company that can't even lock down their own Linode customer service portal (which could lead to a breach of each and every customer's VPS).
Perhaps history is fuzzy for people when new announcements come out or low prices are around.
You really really don't have to. Any payment processor that isn't horribly incompetent does the unique token authorization scheme.
Storing CC #s for recurring payments is solely the domain of incompetents who have no business accepting payments from anyone.
It's never cool to be actually- or quasi-locked into a vendor.
Otherwise you just ask customers to re-authenticate. Often you have a few months headway for a switch like that.
Stripe (and probably others) has functionality like this where the seller's server never sees the CC number, and developers can store a unique token to re-charge the customer at a later date.
You then put your trust in the gateway/processor to store the credit card. Which I assume is most likely behind the best possible security stuff money can afford. Since that's their entire business. One screw up and their gone.
e.g: Acme, Inc. sends Stripe your CC#, Stripe sends them some unique token, and they store that; correct?
So Stripe still has your CC#, and is at risk.
So this is really just risk mitigation; what I think TP is suggesting we need is unique authorizations at the banking level.
Something on the order of virtual credit cards, or temporary tokens, which are ultimately verified by your bank [or in other words: the lender(s) making anti-fraud guarantees, etc.]
(e.g: this token is authorized for 24 hours up to this limit; this token is authorized indefinitely up to $xx/mo.; this token is authorized for 1 year; etc.)
Edit: yes Stripe has your number, but since their sole business is about securing that information, they probably do a better job of it than your typical online merchant.
I think that Linode did a big mistake here. Let's wait for a formal communication.
But this is the moment to support them.
Yes, maybe sounds crazy.
When you host on any third party datacenter, you take risks that something like this could happen. So, deal with it. Check your credit card, if your receive something wrong, call to your card and that's all. But we need to support also the good work, and this guys do great work in the hosting business. Just my opinion.
I can think of two good reasons why you should flee Linode. It remains to be seen if either are actually true, and until indications say yes, then panic is unwarranted:
1. If it becomes apparent that Linode is far more vulnerable to hacking than other hosting providers. But one hack alone does not prove this.
2. If Linode grossly mishandles the situation. There have been a couple of allegations to that effect so far, but nothing substantial. I don't see any reason to claim that they've done this yet.
The alleged hacker has made serious and specific claims, and Linode has done jack shit; without more information, how should I proceed? I don't want to call my bank and waste time getting a new credit card (not to mention replacing a million and two services) without a confirmation and I can't get a confirmation because Linodes people are having a circle jerk (or whatever the hell they do).
There was an email notification a few days ago.
Linode's handling of the Bitcoin incident last year was sub-optimal. This too has been sub-optimal, given that credit cards were exposed but all we heard on Friday was to change our passwords, and even that was claimed to just be a super-careful precaution.
Linode needs to start giving us some frank talk ASAP. They've already burned through a very generous helping of benefit-of-the-doubt.
If that's even true.
This is not a mistake/technical issue, it's becoming an ethics/service one.
This sounds very very bad, and as a customer it's very off-putting.
If the allegations are true, then Linode was keeping encrypted CC numbers, with the decryption key in nearly the same place.
Trying to make the analogy more sufficient by incorporating this type of fact would only make the carjacking analogy more absurd. At the end of the day, an analogy is not needed.