

Instapaper Server Update - psychotik
http://blog.instapaper.com/post/6854208028

======
patio11
I think you're more than justified in assuming no compromise happened. Someone
with legitimate authority had physical access to your server. That has
happened before and will happen again. Your server's cage opens several times
a month, probably, as techs and whatnot do their icky atom-touching sorcery.
Physical access to your server by the adversary means untraceable compromise,
sure, but if that is true, hasn't it happened before? (Sure, the techs might
be more trustworthy than the FBI... Unless they were the FBI, which is
paranoid squared, but if you think that agents executing a warrant routinely
grab servers in the vicinity because, hey, computers tend to have juicy stuff
on them and we have way too much free time, then assuming infiltration seems
almost reasonable by comparison.)

In a threat environment this bad, you would personally be scrutinizing sign-in
lists every month. You have better things to do, right? Right. Because this is
really not Instapaper's threat environment. You got mildly inconvenienced as a
side-effect of routine government operations. You can pick whether you want to
escalate that to A Full Crisis. That seems like a bad call, which you
recognize on the legal side. Do it on the security side, too.

You may think that vocal parties will disagree with "This was a non-event for
us.". They might. Screw them. You're a CEO; dealing with paranoid ramblings
does not generate business value like two months of normal operations does.

~~~
IgorPartola
Isn't there a way to get information from HDD's about how long they have
operated from when they were manufactured using SMART? Not that anyone keeps
those records on hand once the HDD is no longer in their possession, but if
you are paranoid, you could start offloading those records every minute to a
different server. That way if yours is seized, you could at least know if
someone had turned on your HDD's and for how long.

It'd be cool if HDD's/SSD's started compiling rough stats about numbers of
reads and writes for this purpose.

------
tswicegood
+1 to the attitude. It's all about getting the service back up and making it
better. We (collectively) tend to spend too much time trying to figure out who
to blame instead of just doing the cool shit that needs to be done. Kudos!

~~~
Gorbzel
Sadly, I simply cannot agree. I absolutely understand Marco's desire to focus
on developing Instapaper, but there's quite a bit more at stake here, and his
willingness to move on without even a bit of protest is concerning.

He claims that there's nothing that can be done, but there are definitely
legal actions one can take in this scenario. Sure, they might not ever fully
compensate him, nor convince the skeptics, but it will at least reveal the
details behind the seizure, and it will send a message to both the FBI and to
others like DigitalOne that could prevent further "seize first and ask
questions later" style operations.

~~~
statictype
So for a one-man software shop, given the choice between:

a) Spend money and time on legal action against the FBI for holding onto his
servers for a couple of days while in the process of an investigation, in
order to hopefully try and make a larger point.

and

b) Spending his time and no money by actually improving his bottom line by
enhancing the features of his software,

you expect any rational person to choose (a) ?

~~~
rickmb
Sorry, but the suggestion that fighting for justice is somehow the thing only
"irrational people" would choose totally pisses me off.

I totally understand someone making a rational decision in favor of other
priorities. It's what most people would do. But thank god for people who take
the road less traveled. It's to them we owe the rights and freedom the rest of
us take for granted. There's nothing "irrational" about that.

~~~
statictype
A major thrust of what I'm saying is in the fact that Marco is a single indie
developer. Not a large company with more resources.

Fighting against the federal government over this seems like picking the wrong
battle. He got is server back in a couple of days. Obviously its likely that
the feds have already cloned it but its not like they kept it indefinitely or
wiped it clean.

Does it suck? Yes. Is this something he should be fighting over? Not in my
book.

------
nanoanderson
Tangent alert! Marco makes me feel great.

One of my favorite things about Marco is that he gives me confidence that
there is not a gaping wide chasm between the skills of the best engineer-
entrpreneurs and me. This time, it was his frank admission that he was taught
by commenters and emailers than his current password encryption was unsafe and
that there were better options out there.

When you realize that someone as successful as Marco has the same learning to
do that "the rest of us" do, it makes me feel great.

~~~
sneak
News flash: Security breaches happen because almost everyone in the industry
sucks at this stuff, including (most importantly) those who SHOULD KNOW
BETTER, yourself and Mr. Arment included.

~~~
nanoanderson
And now we do. :-)

------
cheald
Out of curiosity, does anyone know how volume-level encryption (like dm-crypt)
holds up against this sort of thing? I find myself wondering what I'd tell
customers if my server were seized as collateral damage like Instapaper's
were. Will that sort of encryption serve as a plausible safeguard of customer
data, or is it more of a padlock (easily broken with the right tools)?

~~~
Tharkun
It would certainly make it much harder to steal the data -- and at the same
time it would make it much harder to boot the machine, because at some point
you'd have to enter a passphrase. It's possible to do so over SSH, but it's
still a bit tricky. Rebooting the server while you're away from an internet
connexion will leave it in a useless state.

That being said, even with volume encryption, the key is in memory somewhere.
RAM isn't wiped as quickly as you might think, and it's apparently possible to
extract keys from memory after the machine has been powered down. So be
warned, and be paranoid!

~~~
JoachimSchipper
Indeed, the encryption keys can be recovered from memory fairly easily.
[http://www.usenix.org/events/sec08/tech/full_papers/halderma...](http://www.usenix.org/events/sec08/tech/full_papers/halderman/halderman.pdf)
is very readable; <http://www.youtube.com/watch?v=JDaicPIgn9U> is a well-made
5-minute introduction by the authors.

~~~
mike-cardwell
I thought this issue had been partially addressed by overwriting the memory
containing the keys on unmount. Of course, that doesn't work if somebody just
pulls the plug on the machine.

There was an article I read recently about a patch for the Linux kernel to
store the key in CPU registers instead of memory as well...

~~~
shabble
A lot of servers I've seen have had 'chassis intrusion detection', which is a
fancy way of saying they had a microswitch hooked up to one of the SMBus lines
that could detect when the case is opened.

Take one of those, and a Google-style Lead-Acid internal UPS, and you'd have
an incredibly hard time getting access to keys held solely in memory.

For the extra paranoid, accelerometers or a simple mercury vibration/tilt
switch would make it even harder, at the risk of losing/hanging your server
every time someone worked on the rack, or drove a heavy pallet cart nearby.

I recall seeing somewhere a device designed for law enforcement, which was
essentially a cart-style UPS with probes designed to be clamped onto, and
penetrate, the power cable, allowing seizures without any power-loss, which I
assume is for this eventuality.

------
ck2
Assume all data on the drives has been cloned.

There's little chance they didn't.

~~~
m0nastic
He should absolutely assume that the data has been cloned, but there is about
a 99% chance that it wasn't.

If the data was cloned on-site, then we wouldn't have heard about the search
warrant (there'd have been a gag order).

And as this happened yesterday, there's almost no chance that the server could
have been taken to the lab, imaged, and then returned to be reconnected today.

I would bet 100 dollars that the actual target server of the raid is still
sitting on a dolly in the forensics cage while a tech is waiting for someone
to tell him what he's supposed to be looking for.

~~~
Skroob
You'd be surprised. I've worked with forensic law enforcement in the past, and
they were equipped to clone drives quickly and easily. The idea is to create a
clean image of the drive that they can use for evidence, and they don't boot
the machine so if there's any trapdoors or whatever they won't hit them. And
this was a local police force over 5 years ago, I can't imagine what the FBI
would be capable of these days.

~~~
amorphid
Some should build a hard drive that logs being powered up and powered down!

~~~
m0nastic
I wouldn't be surprised if SSD's somehow had information like that stored in
them (or at least the ability for it to be stored). Thankfully I got out of
the Forensics racket before they were mainstream.

------
cosgroveb
"I’m not convinced that they did everything they could to prevent the seizure
of non-targeted servers, and their lack of proactive communication with the
affected customers is beneath the level of service I expect from a host."

I can't speak to the level of service you did or did not receive but I
seriously doubt DigitalOne could have done more to prevent "non-targeted"
servers from being seized. If it was in the same rack as the "targeted" server
I'm sure that 100% of the time LE would take it and find out whether it was
related later. It's their job to take every step necessary to preserve
evidence first and foremost when executing a search warrant like this.

I agree with you that the data was almost certainly not compromised and I
think it's great that you're so proactive about protecting your customers by
moving to bcrypt.

------
smackfu
This is all just assumptions, since amazingly the hosting company still has no
comments at all. Who exactly rebooted the server? Not the FBI, for damn sure.

------
ichilton
I hope he securely erased the data rather than just deleting it, otherwise it
might not be just the FBI who have potentially got a copy...

~~~
nettdata
Would there not also be backups at DigitalOne?

~~~
SeoxyS
With dedicated servers, the responsibility is on the client to back up what
matters to him. Ditto with VPSs.

------
Supermighty
I think a neat idea would be to create 1U - 4U lock doors that could be bolted
in front of a server in a rack.

I wonder if this would be enough to give each server it's own legal "separate
space" insomuch as to prevent a whole rack from being seized due to warrants
like this.

~~~
jsatok
The server that was taken from Marco was a blade server, and he just rented a
single node in the blade. Locking the single blade he was on wouldn't have
prevented the situation that occurred, as the FBI seized a number of
enclosures, rather than servers. Most likely, the agents who took the servers
didn't understand a blade server, and took the entire enclosure, rather than
pulling a couple nodes out of the blade.

------
gojomo
Should services consider storing all user data on encrypted volumes?

Pro: after a powerdown, seizure/theft/cloning of the volume won' reveal user
data

Con: need a secure way to supply decryption passphrase on each reboot

------
richardblair
Man, you have an awesome service. It served me greatly through my post
secondary education. I am glad you got your server back, and you can keep on
keepin on.

Again, thank you for the services you offer.

------
suking
This is the exact opposite approach dropbox takes - well done with good
communication. Props.

