Pro-Tip – when you are using a CoLo for your servers:
* Make sure you failover to servers in a separate cabinet.
* Better yet, make sure you failover to servers in a different data center.
* Best Possible – Configure a task to switch your primary and
failover environment from data center to data center
once a week at random times. Or eliminate the concept of primary
and secondary altogether.
Who knows what others servers are located in the cabinet you are sharing. The good news is your mitigation process also becomes a first class disaster recovery plan as well.
And if you're really serious about having the best possible failure-resistant system, install a chaos monkey[1] into your system and learn to live with it
America doesnt need freaky raids like those at this time. If i own a web based business i would seriously consider alternative hosting countries... because honestly between the Patriot act and the actual raids, no business should feel safe.
Or the FBI could just go through the proper channels like everyone else and contact the datacenter and request it's immediate takedown (after providing a warrant, of course). This isn't the first time the FBI busted into a data center all willy-nilly knocking innocent peoples hardware offline. Not only does this adversely effect the innocent users, it also almost destroys the datacenters reputation and it's a loss for nearly every party.
FBI seizures are only a tiny fraction of outages. Lots of things knock out servers: power failures, water leaks, A/C failures, theft, fire, explosions, zombie outbreaks, runaway forklifts, mice chewing on cables.
The only solution is backup hosting in a separate data center.
Honestly, the only legit reason for an outage is some natural disaster took down the entire building. I really wouldn't want my server hosted at a mice infested building anyway, or one without backup power, cheap A/C units, etc. To me, it looks much worse that your server went offline because the FBI busted into the datacenter and knocked shit around than say an earthquake took out our fiber lines.
Personally, if I heard the FBI raided a datacenter and knocked a bunch of servers offline which had nothing to do with what they were after, then I'd seriously question the security, legal team, and response team of that place. To me, it sounds like the FBI just showed up to one guy sitting in a chair watching TV and let the FBI roam free.
You're dreaming if you think a data center outage can't happen at any reasonably-priced data center. It doesn't matter what they promised or what high tech equipment they have. It happens at Google and at Amazon and it will happen to you too, eventually.
"The raid happened at 1:15 a.m. at a hosting facility in Reston, Va., used by DigitalOne, which is based in Switzerland, the company said... DigitalOne had no employees on-site when the raid took place. The data center operator, from which DigitalOne leases space, passed along the information about the raid three hours after it started with the name of the agent and a phone number to call."
I've tried to buy servers in good places, but it doesn't always work out. We're dropping a Hurricane Electric (he.net) server because despite all their high-end claims we've had three major outages in the past year.
If you care to try, I'd say give Softlayer a shot. You get tons of free stuff that other places charge for (KVM over IP, VPN, etc) and the only outage I've had so far was because Level 3 announced routes for Comcast IP's and then null routed them, lasted about 45 minutes and effected a lot more than Softlayer.
The impact of this clear case of incompetence by the FBI is minor compared to what it could have been. A 1U Dell R610 can fit 96G of memory and two very fast 64 bit processors. You can fit 48 of these servers in a standard rack. Running VM's, that configuration could easily support 20 virtual machines per server (4GB/machine + OS) which gives you 960 websites per rack. And that assumes you need an entire machine for each website (no virtual hosts).
So the next time the FBI decides to knock over someone's rack, the impact could be a lot wider than a handful of sites.
US security agencies are becoming increasingly cavalier when it comes to seizing domains and hardware. This is becoming a significant risk for online startups like my company and the companies of other HN members. The suggestions here to mitigate this risk are not cheap. Setting up a fast enough link between data centers to have real-time replication is prohibitively expensive for most small companies.
I'd like to see legislation that lets us know what our rights are and that lays out a standard procedure for these kinds of data center incursions. I'd hate to see a cloud provider's rack get taken down in one of these raids, and I don't think any of us are happy about the government using our tax dollars to settle costly lawsuits caused by their own incompetence.
This is becoming a significant risk for online startups like my company and the companies of other HN members.
It's like a meteor strike: well reported, flashy, but very very not likely to actually happen to your site. How many websites do HNers collectively operate? How many have ever gone down because the FBI took their hardware? How many have ever gone down because, oh, the hard drive crashed? Because they pushed bad code live? Because they misconfigured a firewall? Because the hosting company had a network or power issue? Those are real risks for your business. (Bonus points: many interventions for these and similar issues fixes your FBI problem, too!)
In the event of a totally freak incident like this, you're probably going to have downtime and a day's worth of data loss, but recovery for most folks here is likely "Reimage the VPS(s) from the latest backup, hit the on switch, and change DNS records."
Sure, there are companies who don't care about a day's downtime, having to restore from backups and their customers don't mind if they're offline for 24 hours. I'd like to think that our company and many of the startups on HN are not in that group. I'd also like to think that we give a damn if another startup gets taken offline like this.
Asking "how people many have ever X or Y happen to them?" is downplaying the actual risk when obviously the risk has been increasing in the past year, half-year or so.
Yes, if you look at the risk of a US government raid on servers, domains, or what-have-you in the past 5-10 years, you can fairly compare it to a meteor strike or a "freak incident".
But things are changing, incidents and "collateral" are increasing, and if you look at just the past half year, you'll find it's gone from a meteor-strike level probability to a very small but definitely not dismissable chance.
That's not at all what he said, is it? What he said was, it's very unlikely that an FBI action is going to disrupt your site at all (and, that's true), and also that the same measures you must use to protect yourself from the altogether more likely event of a hosting company failure also protect you from FBI raids.
According to the 4th Amendment a warrant is supposed to:
"...and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."
When they seize unrelated hardware they are clearly overstepping their bounds. I'd like to see the innocent parties take this to court and put a stop to this. We are far past the time when the FBI can claim ignorance as to how a colocation site operates.
I am the founder and current CEO of a dedicated server host. The FBI and other law enforcement agencies do often contact us regarding activity emanating from or related to our network. Its the same for any host and the bigger you are, the more often it happens. Rarely, a local law enforcement officer from within or outside of the US get cranky when you tell ask them to get a subpoena (makes work for them). However, in my experience the FBI and other three letter agencies are very professional, fair, and genuinely want to do it right. Getting a subpoena is no big deal at all for them. Usually the US Attorney is right down the hall from them, in the same building.
They do officially have policies and programs designed to befriend local businesses in the area of a field office. They just want to catch the bad guys and not mess with the innocent. They want people to like them and trust them. They count on the cooperation of businesses, especially hosting and access providers. There are civilized processes for everyone to get what they want.
There are the cases of people trying to get information who are not actually law enforcement, and that is one of the many reasons you must ask for a subpoena... To protect yourself and your customers privacy. The FBI can get one in a very short time.
Where this all goes bad is when you do not respond to subpoenas for subscriber information, when you don't hand over the disks, etc. If you do not comply, then what alternative does the FBI have but to come and get it?
My guess?... The host didn't play nice with or respond promptly to the FBI.
I lost my server in Estonia for over a week when the local police working with the FBI shut everything down looking for a command and control node for a bot net.
If the warrant is valid, I don’t see how anyone affected by the raid has a case. If the police serve a warrant on the house next door to you, and your street gets blocked off while they go over the building with a fine-toothed comb and carry off evidence, it will be damn inconvenient for you, but I don’t know of any law preventing them from doing it.
This is more like they have a warrant for one house so they seize the entire street or neighborhood simply because they can't make the distinction on the field between one and the other. In doing so they're causing harm to businesses, only because they happened to be in the same area code as some alleged criminal. They're not just intruding and looking around, they shut down the business, for days if not weeks.
Yes, but if in searching my neighbor's property they were poking in my living room that would be an illegal search of my property. I'm not sure which metaphor is stronger, but it's certainly not a clear case either wya.
If the warrant includes your living room (because the police have probable cause to believe that there is physical evidence in your living room that would link your neighbor to the crime), then they can search your living room.
In the case of a shared server, I don’t see how you can do a good forensic search of the affected server without having the physical box in your possession.
If the system in question has RAID disks, it should be sufficient to grab one drive (depending on the RAID configuration) from the array. The FBI could take that home and examine it, while everything actually running on the server remains uninterrupted.
One wonders whether their actual goal is not to gather evidence, but to actually interrupt the service.
>40 and primary world view formed working in fortune 100 corporate environment. Basically, every company I have ever worked for has had it's own multiple large data centers. I guess the point I didn't try very hard to make was, do you think a company the size of pinboard has the financial resources to engage in a prolonged legal battle with the FBI? I'm thinking not, but I could be wrong.
The US court system is not nearly as utterly corrupt as you seem to think. Even individual people often fight law enforcement agencies in court there, and win. Certainly a Fortune 100 company would stand a better chance. (Maciej might be at a bit of a disadvantage, being a foreigner, and one who's written publicly about a run-in with law enforcement in China, at that.)
There's a pretty large range between Pinboard (10 000 customers, maybe? Two employees? Annual revenue in the range of $100k? I'm guessing here) and Rite Aid (109000 "associates", maybe 20 million customers, $26 billion revenue). More than five orders of magnitude, actually.
Sorry, I was under the impression that random small businesses got damaged from the the government investigation into the inconveniencing of big multinational corporations.
My evaluation is that their priority is preservation of evidence, and they consider a few random websites being down for a couple of days to be relatively unimportant.
True, but unless they think someone at the colo is going to tip off the owners, taking 10 minutes to figure out which machine hosts which site shouldn't be an issue. With a warrant, I imagine the colo would be willing to look up the mapping from company -> rack/server being leased.
Keep in mind that the FBI agent-in-charge's perceived cost for pissing off some small web companies is pretty low; their perceived cost for tipping off whoever they're after, and blowing the entire operation, is much, much higher.
So let's say they go in, and power off all the servers. At this point, they ask the colo host who's server is who's. The non-involved servers get turned back on, and the correct ones get confiscated.
Where's the downside to this? The unrelated sites suffer maybe a few hours downtime max, and they don't have to worry about tipping anyone off.
Why are you downplaying the egregious police-stateness of this whole thing?
If the FBI is already there, it really shouldn't be a problem for them to locate the exact server that hosts their alleged offender and confiscate only that one.
How do you think anyone would be tipped off in that situation?
I wasn't addressing the egregious police-stateness at all, one way or the other (which I suppose is downplaying it, by not addressing it, but that wasn't my intention). I think that there should be incredibly strict bounds on what can be seized without a warrant, and still rather strict bounds on what can be seized with a warrant, but I don't know whether this case overstepped those bounds.
For example, I believe it's illegal to take information which was coincidentally seized along with legitimate evidence subject to a warrant, and use it in an unrelated case. I'm strongly in favor of such laws, to discourage "fishing expeditions", where law enforcement uses a legitimate warrant to seize a bunch of unrelated material that they're interested in using for other purposes.
However, I suspect if you walk into a data center where some malicious customer is doing something illegal, probably that customer has tried to make it harder to connect them to what they're doing.
Also, I don't know about this case, but there are lots of small hosting companies that lease servers from other companies, and the staff at the colo only know the lessor, not the lessee. They wouldn't have any access to the hosting company's customer database which might map customers to servers.
Besides, the FBI has to worry about low-probability cases like, what if one of the employees is a friend of, or paid off by, the bad guys? Or what if the bad guys are somehow monitoring the facility?
The FBI has a legitimate goal of seizing the evidence they need as quickly and with as little notice to the bad guys as legally possible.
Did the FBI act wrongly in this case? I don't know enough to tell.
if anyone associated with the colo tipped someone off (or there was cameras in the colo streaming) 10 minutes is more then enough to start deleting things.
they probably can only tell with precision the box, not a particular server. remember they are not the hosting company. So they just take it all to sort out later, makes sense.
I've been in the biz long enough to see several paradigms touted as the end-all solution. Today it's the cloud, but it's not too hard to envision something the size of your smart-phone, or maybe even your smart-phone, as the server. There are a lot of good reasons to keep the server in your physical possession. The network infrastructure to do that is just about there.
> The network infrastructure to do that is just about there.
There's a long way to go from having enough bandwidth to serve your personal blog or small-time web app out of your {apartment, house, office} to getting what you'd need for a large-scale web app (at a minimum, redundant 100 Mbps fiber connections with five-nines SLAs) routed there. That and getting a nominal 100 Mbps connection from your friendly neighborhood telco is a lot different from getting a 100 Mbps connection you're expecting to saturate 24/7. The telco won't put up with that for very long, because they overcommit their subscriber bandwidth in the (correct) expectation that most of their customers will not use all of it.
I'm surprised there aren't laws against this. Can a business sue the FBI for ruining them like this? Surely, they could put together a team which can go in, copy the hard drives they need, and be out in a couple of hours. Even then, they should evaluate and repay whatever damage they caused?
So you were sharing "virtual servers" with other companies and the FBI seized the actual servers? Or something to that effect?
That raises some interesting questions, the impression every seems to be operating on is that they stormed in and took 3 cabinets or full racks with computers owned by other companies. All things being equal, this sounds like the normal operating procedure for the FBI.
A little bit of devil advocate here. If the server rack was full of virtual hosting that would mean one server could ideally host multiple websites and if all the servers shared 1 or 2 SAN's it would be difficult from an outsider looking in to know which server hosted what website especially with Graveyard shift. Even if you knew node 1 server hosted the site it would still be shared on the SAN and could easily be started on another server.
If you dont keep redundant servers in other countries nowadays you never know when you're going to be left in the cold by some random government action.
A government official who declined to be named said earlier in the day that the F.B.I. was actively investigating the Lulz Security group and any affiliated hackers.
So these cracker groups have finally woken the sleeping giant? And the bleary-eyed half-awake giant is swinging his arms around wildly, demolishing everything in reach? Maybe after some coffee and donuts they'll be a bit more delicate.
If it's LulzSec, as the article assumes, I would hope they have good reasons to do so.
LulzSec is a group of hackers searching for recognition. By chasing them they are giving them more publicity.
How expensive is it to spin up a VPS and start poisoning data centers? Three racks to take down one site? It would be a nightmare of a whack-a-mole game on a global scale.
So, in thinking about how to mitigate unforeseen disasters while trying to maintain anonymity - I think the following solution to such a problem would be really easy (not free, but easy)
Host your site/content across multiple cloud providers. Then host VPS web servers with other providers and have them balance between the backend cloud servers.
Then put cloudflare on top of that.
Ideally, you would want to have multiple service instances with the cloud providers which are silo'd from each other - but exact duplicates (e.g. they are under fully different accounts with the cloud provider)
Your front-end web servers keep a cache of content from the cloud servers - and they are pushed to. For even lower-end content needs, you can pull from a drop box account or something.
Any of the front end gear gets seized - You can bring it up with a VPS image really quickly.
Then host VPS web servers with other providers and have them balance between the backend cloud servers.
If you don't know what the word latency means, this setup will teach you all about it.
Another problem is that your various providers will probably charge you for WAN bandwidth. So you will pay three times for every request: Twice from backend to balancer (one charge from provider A, one from provider B) plus another charge to send the same data back out from the balancer to the customer.
Plus you will be miserable trying to keep your site up 100% of the time across two cloud providers. Have a problem on either one, and 50% of your capacity will go offline.
Might be better to realize that "my servers were seized by the FBI" is a rare occurrence and you can probably afford a few hours' worth of downtime and/or data loss. Make offsite backups from your primary provider on a relatively long timescale (once per day, maybe once per hour if you're more sensitive; live database replication for the crazy-sensitive) and have a procedure for spinning those up at a secondary provider. Test that procedure every month or so. The beautiful thing about cloud services is that you can pay for your emergency-backup servers by the hour and only when you are using them, or testing them.
