
We moved our servers to Iceland - soheilpro
https://blog.simpleanalytics.io/why-we-moved-our-servers-to-iceland
======
teddyh
> _That’s right, we need to enter the key when the server boots. Wait, but
> what happens with a power failure? Are all requests with page views to your
> server failing after a reboot?_

> _Then we found Dropbear, a very small SSH program, that you can run via the
> initial ramdisk (initramfs). This means we are able to allow external
> connections via SSH. We don’t have to fly to Iceland to boot our server,
> yeah!_

Or you can use Mandos to be able to sleep through a reboot:
[https://www.recompile.se/mandos](https://www.recompile.se/mandos)

Disclosure: I am the co-author of Mandos.

~~~
cf498
I am not sure that the offset in security is that small. The assumption, that
your servers will only be evaluated month later doesnt hold true in most
cases. Attacking running machines quickly does happen. If your threat
scenarios are little green men, they are unlikely to just shut down your
server and haul it to some dark room to interrogate later. They will of course
try to get the key from a running system, but thats also something you should
most definitely have in your threat assessment.

edit: blue men*. Forgot that green police uniforms are an outdated local
thing.

~~~
bradleyjg
I’ll give examples for the US but you can translate as appropriate.

I’d say there’s no municipal police force in the country that has a
sufficiently technically sophisticated regular law enforcement to try to
attack a running machine. So if you are suspected of having information
relevant to a murder or drug deal—the seize and examine one month later
scenario is extremely likely.

A handful of states and maybe one or two municipalities _may_ have a computer
crimes division that would be sophisticated enough to know to worry about
encrypted disks, but they’d have to hire expensive outside consultants to
carry out the actual attack and would do so in exceptional circumstances
rather than routinely.

Finally, if your threat model includes the national security apparatus then,
yes, this is probably too much of a compromise.

~~~
cf498
>I’d say there’s no municipal police force in the country that has a
sufficiently technically sophisticated regular law enforcement to try to
attack a running machine. So if you are suspected of having information
relevant to a murder or drug deal—the seize and examine one month later
scenario is extremely likely.

I wouldnt assume that this is still the case, at least in Germany. I think the
days of police shutting down the devices they have a search warrant for are
pretty much over until they are confident enough that there isnt a disc
encryption in place. If there is, they keep the device from going to sleep and
call for an expert. If they are actually targeting server infrastructure,
there will be people with technical expertise involved.

~~~
Scoundreller
The US can have some small police forces. Like, a handful of people
independently serving a city.

If they were the first to respond to a tech crime, they could probably create
a mess. But in theory, they could do their own search warrants, searches,
recommending prosecutions, etc.

~~~
bradleyjg
Even big police forces, e.g. NYPD, have lots of divisions. I don't think
there's any standard operating procedure in place that would lead a random
detective in a random precinct to call in one of the two or three experts
across the entire NYPD on a case he is working that happens to involve seizure
order for servers. If someone in the special city-wide child pornography unit
was involved in a server seizure, that's when the computer expert might be
involved (because he or she would be attached to the computer crimes division
the child porn unit is a part of).

------
Semiapies
It's a little sad to watch someone jump through hoops to try to please the
people behind EasyList, when I sure don't see anything about country-of-
hosting in EasyList's policies. Especially when almost nobody actually cares
about the other members of the Five Eyes or the Enemies of the Internet in
these discussions, but just knee-jerks about US hosting.

~~~
dylz
Yes. Easylist/EasyPrivacy here is intended to block ANY third party tracker.
This is a tracker. I use this adblock list and I don't care what country
you're hosted in, simpleanalytics is a third party tracker and should be
blocked.

> " EasyPrivacy [...] including web bugs, tracking scripts and information
> collectors"

Simpleanalytics is all of the above. There is no arguing this.

~~~
skrebbel
> There is no arguing this.

Yes there is. SimpleAnalytics claim that they "don’t track visitors of our
customers’ websites".

Are they lying? If not, how is it a tracking script?

~~~
satori99
I guess they mean that they aggregate data? rather than storing it for
individuals users?

Otherwise, why do they need your browser to make a request in the first place?

FWIW I agree with dylz. It is crazy to suggest that any company doing
analytics has any sort of right to make my software connect to its servers.

If Easylist did not block all unnecessary requests, I would find a list that
did.

~~~
dagenix
> It is crazy to suggest that any company doing analytics has any sort of
> right to make my software connect to its servers.

No analytics company has a right to make you connect to its servers. But, that
isn't the case at all: You visit some website. The operator of that website
has contracted with the analytics company. You ask the website for some
information and it replies with that information as well as a request to ping
the analytics company that the website contracted with. It's apparently
"crazy" to comply with that request, all while still consuming the information
that you asked for and received.

Ad-blockers wouldn't be controversial if they worked by navigating away from
any site that displayed ads or included a tracker. Of course, that would be
inconvenient for the user, so, they don't do that. The whole point of an ad-
blocker is to allow a user to consume a service in a manner that the entity
running and paying for the service didn't intend. We could at least
acknowledge that. Your position, however, seems to be that doing what the
service you are using is asking you to do in exchange for its information is
"crazy" \- and that is straight up ridiculous.

~~~
satori99
I understand your viewpoint, but I consider all script tags that a server
provides to be merely suggestions.

A browser is, after all, a _user_ -agent. The web was designed this way for a
reason.

Would I be breaking some sort of inferred agreement if I browsed with no
Javascript enabled at all?

In any case, a site server logs should provide ample information for
analytics, unless they are measuring mouse movements or scrolling etc.

~~~
crankylinuxuser
Just wait until that official DRM crap is used for adtech and preventing you
from modifying the HTML the server sends.

This is a war; a never-ending war of users VS adtech. I realize how much
defense I have to run to protect mtself even moderately, and I'm still losing.

------
dagenix
> If you draw a straight line from San Francisco to Amsterdam you will cross
> Iceland. Simple Analytics has most customers from the US and Europe, so it
> makes sense to pick this geographical location.

1\. I would be interested to find out if being geographically between San
Francisco and Amsterdam is actually good for latency.

2\. I think the usual solution to having customers in the US and Europe and
wanting to keep latency down, is to setup servers in the US and Europe. So,
this strikes me as an odd justification of the decision.

~~~
cbg0
Pings to 1984.is:

From Palo Alto : round-trip min/avg/max/stddev = 183.514/184.035/184.898/0.466
ms

From Amsterdam : round-trip min/avg/max = 36/37/39 ms

From London : round-trip min/avg/max = 48/48/50 ms

Obviously this can vary quite a bit depending on what peering you have at your
disposal.

For reference, from Palo Alto to London: round-trip min/avg/max/stddev =
156.236/156.921/158.271/0.834 ms

For latency, I would choose NY/NJ or similar if you really want one location
to serve both EU and US; Though this isn't ideal, it does lower the latency to
West Coast US quite a bit.

From San Jose to Newark : min/avg/max/stddev = 73.462/73.547/73.660/0.078 ms

From London to Newark : min/avg/max/stddev = 72.836/74.323/75.600/1.200 ms

~~~
toast0
From your pings, it seems pretty likely that traffic from your Palo Alto
location is going through europe to get to iceland, most likely through
London.

The submarine cable map shows one cable going west to greenland and then
Canada, one that goes to northern England, and two to Denmark. The Canadian
landing isn't very close to any other transatlantic cables, so it may not be
very well connected (land connectivity is much harder to map though).

Anyway, making location decisions based on assuming internet distance is
similar to physical distance is kind of silly. There are many physically
adjacent countries that don't have direct interconnects, it's pretty common
for traffic to exchange through somewhere much farther away than the ultimate
destination: many south american countries exchange traffic in Miami.

~~~
mattrp
You’re right that it’s silly - from an IP topology layer Iceland might as well
be mars. There was a bunch of news several years ago that Iceland was awesome
because geothermal but a) there are other grids that are just as good if not
better (Norway for ex) and b) doesn’t erase the otherwise terrible trade offs
in network and dc options that are available.

------
guaka
Matomo is analytics for folks who really care about privacy (or folks who want
to simply have more control over their users analytics data). Install
[https://matomo.org/](https://matomo.org/) and run it in the same location as
the project you're already running. Your own instance of Matomo is not going
to be on EasyList any time soon.

~~~
edp
Matomo is blocked by many ad blockers by default, not based on the domain name
or URL but based on the name of the js files downloaded by the browser.

~~~
r3bl
...and those could be easily avoided by referencing "js/" instead of
"piwik.js"[0], or by running Matomo's log analytics tool[1].

Which leads back to the following quote from the author:

> But what happens if we block alternatives even if those alternatives are
> taking the privacy of the user very serious. I care about the privacy of the
> individual. I don't collect any personal information (I don't even store
> IP's). Even if you have your Do Not Track-setting turned on in the browser I
> do not collect any information (see our script).

[0] [https://ericmathison.com/blog/bypass-ad-blockers-and-
track-y...](https://ericmathison.com/blog/bypass-ad-blockers-and-track-your-
visitors-with-piwik/)

[1] [https://matomo.org/docs/log-analytics-tool-how-
to/](https://matomo.org/docs/log-analytics-tool-how-to/)

------
meowface
I appreciate the intent, but I think this is folly. The FBI has imaged
Icelandic servers, like Silk Road's, with the cooperation of Iceland's law
enforcement. Iceland's ISPs are also not neutral and have been pressured by
other countries to block access to websites like Pirate Bay, which would be
beyond the pale for US ISPs.

There is no country on Earth that will meet your requirements for hosting. For
example, if you host in Russia, you probably can evade the US government's
prying eyes, but then you have to deal with the Russian government's prying
eyes.

I strongly doubt Digital Ocean would give customer data to anyone without a
warrant from a judge. And there could be some scenarios I'm not thinking
about, but I also doubt a judge would ever grant a warrant to collect bulk
analytics data. And I think it would be unlikely that law enforcement would
want to request a warrant just for some narrow analytics collected on a few
specific individuals.

Also, as others have pointed out, you actually make yourself totally fair game
for US intelligence agencies by being in a non-FVEY country, and even more so
because Iceland's ISPs peer with ones in FVEY countries.

But way more importantly than all of that, the threat model is wrong. You're
likely at far greater risk from cybercriminals and regular blackhats than you
are from any government. Digital Ocean (very much unlike Linode and some other
big providers) has never had a (known) breach, and probably invests way more
into security than the Icelandic provider you switched to does. DO likely has
many world class security engineers employed; maybe your Icelandic provider
does, too, but it's less likely.

And this isn't even going into the added management and latency issues.

I feel like you're kind of handicapping yourself without any significant
privacy gain or increase in customer acquisition. You're getting feedback from
very suspicious people who want to block all use of your, and others',
services. You should take feedback from such a group with a heavy grain of
salt.

This also does nothing to actually get the domain removed from the block list
- there is probably nothing you can do there, other than gray hat stuff like
constantly rotating domains and IPs, or pivoting and changing your entire
business model and company.

~~~
shoes_for_thee
> I strongly doubt Digital Ocean would give customer data to anyone without a
> warrant from a judge

A subpeona from a grand jury or an NSL would be more than sufficient.

------
BillinghamJ
> We also delete the credit card and email from Stripe (our payment provider).

The email isn't actually deleted - Stripe's logs will permanently hold that
data. If you don't want to retain the email address, you'll need to send
receipts etc yourself.

~~~
harianus
Ah, good point. We already do so it's just a matter of not sending the email
address to Stripe. Thank you for the suggestion. Added this to our roadmap:
[https://simpleanalytics.io/roadmap#109207](https://simpleanalytics.io/roadmap#109207)

------
whoisjuan
I applaud the intention but I doubt this is going to make a difference when it
comes to signing or keeping users.

If the service keeps growing they will quickly realize that latency and
availability matters. Uptime would be now a major concern. If something blows
up in whatever facility it's used for these servers they will be fucked. No
way to shift traffic to another server.

And let's not even talk about computing needs. At scale, that matters. Having
managed services also matters and helps reduce operational cost (a lot).
Having a bare server in someone's garage in Reykjavik is the opposite of
scaling. It's literally a recipe to deprive yourself from the technologies
that can make you successful and definitely a way to slow down your progress.

If SimpleAnalytics' goal is to stay small, then maybe this could make sense,
but that for me makes this business pointless and it doesn't sound like the
vision the founder has either.

Or maybe this is just an MVP until they can have their own on-prem
infrastructure with fully encrypted hardware. Idk, maybe the founder's vision
go as far as that, and I can't see beyond the downsides of a such a random
move.

~~~
mattrp
No you’re right - very little if anything was accomplished here. The other
side of the cables connecting Iceland land in five eyes territory and cost
like n times more for capacity (and are old etc).

------
idlewords
This reasoning is backwards. Host a service in the US and it is at least
protected from arbitrary monitoring by the NSA by US law. Host it abroad and
you have no such protection.

~~~
AdriaanvRossum
I think protection by the NSA is a though sell nowadays.

~~~
tptacek
I don't think that's really the sell. The sell is that hosting within the US
is the only protection you can have _from_ NSA, since they have a total,
unencumbered free hand when it comes to services hosted outside the US,
without even a legal formalism to stop them. Like all national SIGINT
agencies, breaking into everything outside their own borders is literally
their chartered job.

------
Siira
Isn’t the key on the VPS’s memory? What’s the use of this full-disk
encryption?

~~~
sneak
In practice, it is harder to dump the memory of a bare metal server than it is
to simply yank out a hard drive. (It is, of course, possible - the technique
is beyond many hosting companies and local police, and the effectiveness
depends on the temperature.)

In theory, the hosting provider, having physical access, has both the key (in
RAM) and the cyphertext on disk, so logically there is little point.

It’s worth noting that my hosted bare metal boxes have encrypted data
partitions with keys I provide over ssh each boot.

Of course, if it is a VM, anyone with root on the hypervisor (i.e. the hosting
company) can trivially dump the memory and encryption keys.

------
jkaljundi
How did you choose between Iceland and Estonia, what were the pros and cons?

~~~
harianus
Iceland has 100% green energy and their location is slightly better. Maybe the
biggest plus is their laws.

~~~
dmitripopov
And also it's a very cool place to visit :)

------
perilunar
> We are thinking of setting up a very simple CDN with encrypted servers,
> which only serve our JavaScript and store the page views temporarily before
> sending it to our main server in Iceland.

For the JavaScript, just encourage your clients to copy the script and serve
it from their own server with the rest of their files.

------
derpherpsson
I really liked this :)

Thank you for existing.

~~~
harianus
<3

------
caprese
and all of this had zero effect on revenue and customer retention,
congratulations you played yourself

also the FBI has merely made suggestions to Reykjavik Police to infiltrate
servers and they did

> After the initial revelation of the server's location in a data center in
> Reykjavik, Iceland, the filing explains that Reykjavik police accessed and
> secretly copied the server's data. As agents of a foreign government, the
> prosecution argues, they weren't required to seek a warrant from any US
> authority.

The owner of those servers is in US prison right now serving a double life
sentence

Iceland's domestic laws didn't help here, and the lack of a formal arrangement
with cross-sharing intelligence communities didn't help here

So you are either implementing a zero-knowledge service to begin with, or
wasting your time

~~~
mirimir
Where are you getting that quote from? I don't see it in TFA, and you don't
provide any reference.

If the server uses full-disk encryption, and if it's well locked down, it
would be nontrivial to secretly access and copy the server's data.

I mean, adversaries could attach a keyboard and monitor, but they couldn't log
in. And you can even delete the root password, and allow only key-based login
via SSH.

~~~
caprese
source of quote: [https://www.wired.com/2014/09/the-fbi-finally-says-how-it-
le...](https://www.wired.com/2014/09/the-fbi-finally-says-how-it-legally-
pinpointed-silk-roads-server/)

additional: [https://nakedsecurity.sophos.com/2014/10/10/fbis-
warrantless...](https://nakedsecurity.sophos.com/2014/10/10/fbis-warrantless-
hack-of-silk-road-was-legal-prosecutors-claim/)

> If the server uses full-disk encryption, and if it's well locked down, it
> would be nontrivial to secretly access and copy the server's data.

OP's article mentions this, part of the reason they move out of the US is
because RAM can be trivially read even if full disk encryption is used.
Reading RAM still works in Iceland.

~~~
mirimir
Thanks.

> RAM can be trivially read even if full disk encryption is used

I wouldn't say "trivially", but yes, it can be. But if you're _that_ paranoid,
you can embed key parts of the motherboard in alumina-filled epoxy. Its
thermal conductivity is good, and you can add fins and fans as needed. You can
even embed trip wires in the epoxy, to trigger system shutdown if tampered
with.

~~~
dylz
They're running on VMs, not bare metal, presumably, because their (new)
server's reverse DNS is vps-*, and previously on Digitalocean. You can just
dump the VM's memory space while unlocked, can't you?

A lot of this seems like security theater, especially while still hosted
behind Cloudflare.

~~~
mirimir
Oops.

Yeah, here it does seem security theater.

But still, it was a good writeup. I mean, dropbear and all.

I have no clue why they're using VPS, after all that. I mean, if they're a
real business, they ought to just setup a server, and ship it to Iceland. If
the want the ease of VPS, it's easy to do secure KVM in a FDE server. Even
with Docker containers within KVM, if you like.

~~~
philliphaydon
“If they’re a real business” - this is the sort of dribble HN is reduced to? A
real business can’t run on a VPS?

~~~
hal009
I believe the point here is that they claim that they care about security,
while their Icelandic VPS hosting provider can just dump the host server
memory, which would include the encryption keys.

~~~
philliphaydon
Then can’t we say that? “If they truly cared about security they wouldn’t use
a VPS”. It just rubs me the wrong way the way it’s worded.

~~~
mirimir
Yes, I should have been clearer. Sorry.

------
auslander
> ..embed scripts .. hosted via the CDN of CloudFlare

I'm browsing to med.com/somedisease and now CF links my browser to the URL
from Referer header.

~~~
gruez
This can be mitigated by [https://developer.mozilla.org/en-
US/docs/Web/HTTP/Headers/Re...](https://developer.mozilla.org/en-
US/docs/Web/HTTP/Headers/Referrer-Policy)

~~~
harianus
This seems to work for links, but does it work for when you embed a script?

> [...] by using the referrerpolicy attribute on <a>, <area>, <img>, <iframe>,
> or <link> elements

Hm, seems not to work for scripts.

------
LifeLiverTransp
Why not have them on a boat, so you can ship them wherever its cheapest to run
them or they are needed? Also ocena provides free cooling water.

------
dmitripopov
Just for the notice: you keep using "we" through the article, yet your home
page tells that you run it solo.

~~~
derpherpsson
The royal we.

Also makes it easier to hire people if you don't have to update any texts ;)

~~~
dmitripopov
Yep. But it's important to be consistent even for a royal person.

------
chvid
I appreciate what simpleanalytics.io and 1984 is trying to do but I really
think they risk going on a hyperbole and thus lose the broad appeal of their
cause.

The important thing here is rightful skepticism of companies which gives away
valuable complex software (a web analytics tool) for free in return for access
to your customers behavioural data which then is sold or used in other parts
of their business (ie. Google Analytics/Google Adwords).

Making this a thing about the mass surveillance of the five eyes and thus
having to avoid hosting companies such as Digital Ocean and AWS just looses a
lot of us and frankly is a bit too paranoid/silly.

~~~
harianus
A little note on hosting companies. In Europe at least there are corporations
that only host data in Europe and don't use any hosting companies from the US.
This is a question which we repeatedly get at Simple Analytics. So avoiding
those companies has advantages business wise.

~~~
chvid
What exactly is the problem in using digital ocean and select a server placed
in Europe?

~~~
detaro
The concern is that the US government will ask/force US companies to hand over
data or give access even it's stored outside the US. E.g. see the "CLOUD act".

