
Pinboard.in service limited - FBI raided hosting company and pulled equpiment - rograndom
http://status.pinboard.in/
======
marceldegraaf
I love how Pinboard deals with this issue: site remains online (albeit with
limited abilities), all data is secured and backed up, users are encouraged to
use the export tool if they feel the need and there are status updates on
Twitter.

Keep it up Pinboard!

~~~
bgraves
_One thing that does work is our export page, and if you're feeling nervous
about your data I encourage you to use it. We have fresh backups (from one day
ago) safely stored on S3, but in these matters you can't be too careful._

+1 Maciej. I am glad I paid for an account.

------
idlewords
We're not down. Our main DB server is unreachable and there is limited service
(no API, search or feeds) while we run off a smaller backup server.

See <http://status.pinboard.in/> or @pinboard on Twitter for updates.

~~~
buro9
Thanks, one question though... I've just been making some more bookmarks... is
your backup server database now your master? So when you get your main
databases back (or a replacement), will my bookmarks remain when you switch
back?

~~~
idlewords
It's a master-master setup with only one accepting writes at any time. So it
should catch up first, and then I'll be able to switch back over to the
original DB without data loss.

~~~
jvehent
Good architecture. And nice of you to explain clearly and completely the
problem and to invite your users to backup. I'd like to see that more often,
but more companies are too scared to lose customers to do that.

~~~
agilo
But they'll win new ones when this story and the likes come to their
attention, and when they learn how the company handled it with honesty and
aplomb. I for one was unaware of this service and this story has made me
consider signing-up.

~~~
rufibarbatus
@idlewords: would you have stats at hand?

Maybe a temporary loss of functionality earned you a few users above the daily
average.

------
mikiem
I am the founder and current CEO of a dedicated server host. The FBI and other
law enforcement agencies do often contact us regarding activity emanating from
or related to our network. Its the same for any host and the bigger you are,
the more often it happens. Rarely, a local law enforcement officer from within
or outside of the US get cranky when you tell ask them to get a subpoena
(makes work for them). However, in my experience the FBI and other three
letter agencies are very professional, fair, and genuinely want to do it
right. Getting a subpoena is no big deal at all for them. Usually the US
Attorney is right down the hall from them, in the same building.

They do officially have policies and programs designed to befriend local
businesses in the area of a field office. They just want to catch the bad guys
and not mess with the innocent. They want people to like them and trust them.
They count on the cooperation of businesses, especially hosting and access
providers. There are civilized processes for everyone to get what they want.

There are the cases of people trying to get information who are not actually
law enforcement, and that is one of the many reasons you must ask for a
subpoena... To protect yourself and your customers privacy. The FBI can get
one in a very short time.

Where this all goes bad is when you do not respond to subpoenas for subscriber
information, when you don't hand over the disks, etc. If you do not comply,
then what alternative does the FBI have but to come and get it?

My guess?... The host didn't play nice with the FBI.

~~~
davidu
Or the FBI believed the host was aware of, sanctioning, in cahoots with, or
participating in the alleged activity and thus could not tip their cards.

This happens more often than you might think.

------
noonespecial
The FBI is just another in the list of bad things that can happen to your
servers. Flood, fire, theft, law-enforcement... its all the same thing to a
hosted service.

Off site backups, executed masterfully here saves the day.

There are a few services that I run on servers in separate countries. Failure
of reasonable rule of law within a single country is a failure mode we
consider.

~~~
sudonim
Is it possible to insure / sue for careless seizures by government agencies?
Pinboard is (probably?) not the one they are after. What recourse do they
have?

~~~
_delirium
I believe the current status is: no definitive resolution of whether there
should be any compensation required, but, paying compensation is not typically
done currently, and no strong precedent requires it, while some precedent
holds that it isn't required.

A 2008 post by a libertarian law professor
(<http://volokh.com/posts/1209706276.shtml>) argued that the Takings Clause
(which requires compensation if e.g. land is taken to build a highway or
military base) should also require the government to pay compensation to
innocent third parties inconvenienced or harmed during the course of a
criminal investigation. But, that post noted that no high courts had actually
held that, and at least one appeals court had just held to the contrary.

This is in the normal case, at least; things would probably be different if it
were deemed to be some sort of overt act, e.g. if a police chief orders a raid
designed to intentionally damage an innocent third party's interests (and you
have evidence to show that).

------
ldayley
The brave new world of overzealous law enforcement data warehouse searches
just continues to wreak havoc on "cloud" services and content. It's akin to
digging up the neighborhood to search somebody's house. What can the
Feds/providers do differently to prevent this from happening so often?

~~~
mapgrep
It seems ridiculous that this is affecting so many customers - pinboard,
Curbed, one Instapaper eval/test server, bunch of individuals - when there was
one FBI warrant.

Apparently the FBI took three racks. If those are whole racks, it seems
excessive. <https://twitter.com/#!/Pinboard/status/83256217174147072>

But another issue is that Digital One told Pinboard this was due to bad router
firmware -- how do you confuse a software issue and an FBI raid?
<https://twitter.com/#!/Pinboard/status/83257679023325186>

~~~
michaelf
Most likely, DigitalOne doesn't have support staff sleeping in hammocks at the
colo. If you were an admin, logged in remotely (from switzerland, say, where
DigitalOne is headquartered), and you saw a few racks of equipment go offline,
would you:

    
    
      1. suspect that someone had walked away with 3 racks, or
    
      2. guess that some network hardware had gone bad?
    

Usually it's a horse, but sometimes it's a zebra.

------
EwanToo
It's interesting to think through what would happen with AWS in the same
situation.

Would the FBI turn up and say "Where's the 100 servers for customer X", then
seize up to 100 different physical servers, depending on the distribution? Or
even 100 racks worth of physical servers...

~~~
jvandenbroeck
I think they just asked for a server on an IP and the datacenter staff (!= not
the DigitalOne staff) said the IP was in the range of DigitalOne - so the FBI
took them all. I think the datacenter will rent IP ranges to their clients.

I can imagine in an Amazone datacenter they would be able to point to the
exact server they're looking for.

~~~
Maxious
I think the OP's point is that Amazon EC2 can run 100s of instances for one
customer which could be on up to 100 different physical servers and when not
using EBS roots, any one of those servers might still contain the evidence the
FBI wants.

Mentioning the specifics of Amazon further, what about if your account for the
highly illegal operation was using/paying for EBS/S3/SimpleDB/RDS/SQS/SNS/SES
(or all at the same time). Any could contain forensic evidence of a crime as
they hold data in some sense. EBS especially is likely to run on some kind of
SAN; would they have to crack open the racks and take out individual drives?
would they have to take whole arrays of discs because of RAID-esque striping?

It's like a LEO denial of service (both on amazon and the forensic analysis).
Pragmatically, they might trust amazon enough to consolidate/quarantine the
data into the smallest surface area first.

------
buzzmunk
What I find awesome is that instead of being disgruntled or disappointed, the
way in which pinboard has handled this situation has reinforced my confidence
and appreciation for it's service. They are truly a case study in customer
awareness and communication for the tech industry.

------
thaumaturgy
Can a law-knowledgeable individual shed some light on what this means for
individuals who weren't a part of the warrant?

i.e., I'm hosted on the same server as Joe, who the FBI are investigating.
They seize the equipment we share. During the course of their investigation,
they naturally also examine my data.

Do I then have a viable lawsuit or claim towards unlawful search & seizure, or
invasion of privacy? Or, if I happened to also have illegal content on the
same system, would they be able to use the evidence they encountered there in
a case against me?

~~~
faboo
In general, law enforcement has to get very specific warrants accurately
indicating where they're going to search and what they hope to seize. This
includes both physical locations and, usually, computers. Additionally, case
law makes a distinction between property that you personally own and/or
control (e.g. your house that you live in), and property that is shared (e.g.
the apartment you rent with a buddy), and recognizes that the part of
something that is "yours" (your room in your shared apartment, for instance)
is separate from you buddy's (their room). (I imagine this is because the 4th
amendment protects your papers and effects from search and seizure, not just
your real estate.)

So, all things being equal, the FBI is supposed to take pains not to even look
at the data of people not named on the warrant (automated processes have been
exempted from this, IIRC, as they aren't actual people), even though your data
is housed on the save server as the person(s) named on the warrant. And if
they did by chance see something they weren't supposed to, it wouldn't be
admissible in court (in a case against _you_ \- they might be able to use it
against someone else).

IANAL, I just took a couple of law courses in college.

~~~
thaumaturgy
> _...they might be able to use it against someone else..._

That's an interesting point. So, if I'm running a hosting company, and they
come across data from one of my customers...?

Also, putting aside for a moment IT best practices and all that, they
potentially are crippling my business by seizing machines which I share with
someone else. Is there any recourse for that?

~~~
uxp
An article on HN a few months back about a guy who worked in a Three Letter
Agency specifically stated that when or if a data analyst came across data
that was not pertinent to their job, and viewing that data was potentially
infringing on someone's constitutional rights, there were forms filled out and
steps taken that stated such an occurrence happened. He also said the Patriot
Act screwed that all up.

The WP article below specifically talks about communications, but the skeptic
in me doubts there is much change between one form of data and another in a
highly bureaucratic environment such as the NSA.

<http://en.wikipedia.org/wiki/Thomas_Andrews_Drake>

[http://www.washingtonpost.com/wp-
dyn/content/article/2010/07...](http://www.washingtonpost.com/wp-
dyn/content/article/2010/07/13/AR2010071305992_pf.html)

~~~
thaumaturgy
Thank you.

------
rubergly
I just started using Pinboard a couple days ago (mostly because of integration
with Instapaper), and so far it looks great. The fact that, even with all this
chaos going on, the main functionality of the service is still up and running
is really great.

------
newobj
Really impressed by your ability to weather this storm. Definitely going to
take a closer look at the service and see if it's useful to me (I'm a kind of
'write-only' Instapaper user... could Pinboard help improve my read rate at
all?)

------
shrike
I've been a Pinboard customer for a long time, I think it is an example of a
well executed, very well designed service that just blends into the background
even when I use it a dozen times a day.

------
JeffDClark
I wonder if the law enforcement folks look at the "collateral damage", i.e.,
all the innocent servers (virtual or otherwise), as a windfall of extra data
to mine. If my neighbor gets his house raided by the SWAT team they do not
have access to my house only his, but with hosting this is not necessarily
true. Further, I at least know who my neighbors are and can choose to not live
in a bad neighborhood or even move away if I want, however with hosting this
is not the case.

~~~
falcolas
Suddenly, the PCI compliance requirement to encrypt all sensitive "data at
rest" makes a lot more sense. Probably not the case with Pinboard, but
interesting nontheless.

------
jarek
Can't wait to read the write-up of _that_.

------
ltamake
A site I visit frequently got its server raided by the FBI (the host was
rewiredHost; it was unrelated to the site I visit). The site had to start from
scratch except for their frontend code.

I wonder why the FBI is raiding all these servers. Another comment mentioned a
few hosts they've hit.

~~~
illumin8
A popular method used by hackers is to sign up for a virtual server with a
stolen credit card. If they are careful and only access it through a proxy,
their hacking attempts are virtually untraceable. With the amount of hacking
going on lately by Lulzsec and other groups, there is bound to be a lot of
collateral damage.

~~~
pbhjpbhj
Sounds like a very black hat way of taking out your unprepared competition.
I'm sure you can buy this sort of service somewhere - get a hit on a
datacentre/rack by security services in response to highly illegal activity
perpetrated in the open in order to cause the most possible disruption.

~~~
illumin8
I didn't mean to imply they would intentionally do that to take out
competition, although that is indeed a possibility. The main purpose of having
a stolen virtual server would be to launch attacks from.

The truth is even if you rented a virtual server in the same data center, your
chances of being in the same rack of hardware are pretty slim.

------
toddh
If the data was replicated in N places would they serve all N places? Just
curious how that works.

------
reustle
It seems civilian causalities will always exist.

------
chrisjsmith
Here's where hosting on EC2 might actually be worth it.

------
jvandenbroeck
Let this be a lesson to chose your hosting company wisely. You can't always
predict such things, but I think if you pick a big name, with a good business
reputation, you'll minimize the risk. They might do checks for abuse and have
plans in place to keep everything up when something goes wrong.

Eg. I can't imagine that if you host with <http://www.rackspace.com/> your
website would be down if you didn't do anything illegal.

DigitalOne's site is also down <http://www.digitalone.com/> so they don't even
have a backup server for their own website..

~~~
oscardelben
All big names were once small. I doubt pinboard didn't choose their hosting
company wisely, although I get what you're saying. Troubles and down times
happen to anyone, it's how good you can deal with them that makes a
difference, not the hosting company you choose.

~~~
jvandenbroeck
Ofcourse, there's no blame on pinboard, they handled it amazing! And the
webhosting company was around for 7 years if I googled it correctly - but I do
think there's a big difference between webhosting company's.

Suppose DigitalOne rents space in a datacenter and has no one on-site (exept
for staff from the datacenter). I can imagine that if the FBI enters there "we
need servers from IP x.x.x.x and y.y.y.y immediately - that the people from
the datacenter just point to the servers from DigitalOne "oh that's in their
range"..

Ofcourse that's just speculation if you don't know how it really went down -
never the less, I would chose webhosting carefully if your business depends on
it:)

