
Https hurts users far away from the server - antoinefink
https://antoine.finkelstein.fr/https-is-hurting-users-far-away-from-your-servers-and-what-to-do-about-it-5b143080488#.sxjb81794
======
jacquesm
The biggest impact you can have on your users experience is to trim down the
number of connections and the size of your pages. _Long_ after that you can
start worrying about round-trip-times to the server.

This blog post is a nice example: 30 requests (ublock origin blocked another
12, with those enabled the time to load increases to a whopping 28 seconds),
2.5M transferred, 7 seconds load time. And all that for 4K payload + some
images.

~~~
lclarkmichalek
Number of connections isn't that relevant with HTTP2

~~~
jacquesm
Unfortunately back here in the real world only about 10% or so of all websites
support HTTP/2 so it is _very_ relevant.

~~~
dalore
Adding http2 is easier then reducing the number of requests.

And infact reducing the number of requests using things like spritemaps,
bundling js and css is actually an antipattern with http2.

~~~
jacquesm
> And infact reducing the number of requests using things like spritemaps,
> bundling js and css is actually an antipattern with http2.

Those are just ways to lose some of the impact of bloat without addressing the
bloat itself.

If you address bloat directly it will benefit _all_ users.

~~~
tuxracer
If you migrate to HTTP2 or move to a host that already supports it then
cutting down on "round trips" is an obsolete concern altogether.

~~~
jacquesm
HTTP2 does not magically bundle all connections to one. It still very much
depends on how you build up your page.

~~~
floatboth
It does not magically split one connections into many. One domain == one
connection. Sure it won't undo your "domain sharding" hacks and merge your
CDNs, yeah :)

------
alvil
There is also another problem on how much and how often is Googlebot indexing
your site because your site speed is one of the factors of so called Google
index budget. My users are in Germany so my VPS is also in Germany to be fast
for local user (~130ms for http reply), but for US Googlebot is my site slow
(~420ms for http reply). So you are penalized also for this.

~~~
KabuseCha
Hi - I know some tools report a slow site in this case, but these reports are
not accurate - don't believe them! Google is not that stupid ! :D

I am currently working as a dev in an SEO-Agency (in Austria), and we never
believed this hypothesis - so we tested this once with a bunch of our sites:

When moving sites with a German speaking audience to a VPS in America, your
rankings at google.de/google.at will decrease (slightly - the effect is not
that big) - the other way around your rankings will improve (slightly).

However - even if your rankings would improve when moving to America I would
recommend keeping your sites hosted in Europe: The increase in rankings will
not offset the decrease in user satisfaction and therefore the decline in your
conversion rates.

~~~
chatmasta
That sounds like a really interesting experiment. Kudos for taking a
scientific approach to SEO.

A bit off-topic, but out of curiosity, have you run any other interesting
experiments like this? I would love to read a blog post about them.

~~~
KabuseCha
Hi, thanks!

We regularly test different things, but few are as extensive as this "server
location test".

This one was quite easy to do - and to revert - even when doing it for a lot
of sites: just duplicate your sites on another continent and change your dns-
settings.

Sadly we do not blog about this stuff. As our customers are not particularly
fond of sharing their data - and blog posts without precise data are not
useful at all...

Additionally, most of our assumptions and hypotheses were wrong. So most of
our blogsposts would sound like:

"We thought google would work like this, but sadly we were wrong"

SEOs might like these posts - but potential customers probably not so much :D

------
mfontani
Yup, and that's why for thereg we started using Cloudflare's Railgun… with it,
the connection to the servers (hosted in the UK) is "bearable"… without, it's
abysmal:

From a VPS in Sydney, with a Good Enough bandwidth:

    
    
        root@sydney:~# speedtest-cli 2>&1 | grep -e Download: -e Upload:
        Download: 721.20 Mbits/s
        Upload: 117.89 Mbits/s
    

… doing the request through Railgun is "quite bearable":

    
    
        root@sydney:~# ./rg-diag -json https://www.theregister.co.uk/ | grep -e elapsed_time -e cloudflare_time -e origin_response_time
        "elapsed_time": "0.539365s",
        "origin_response_time": "0.045138s",
        "cloudflare_time": "0.494227s",
    

Despite our "origin" server being quick enough, the main chunk of time is
really "bytes having to travel half the world".

Why does railgun help? Because this is what a user would get otherwise; the
"whitepapers" site is hosted in the UK, and doesn't use Cloudflare or Railgun
– it only uses Cloudflare for DNS:

    
    
        ./rg-diag -json http://whitepapers.theregister.co.uk/ | grep elapsed_time
        "elapsed_time": "0.706277s",
    

… so that's ~200ms more, and _on http_.

How much would https add, if it were done without Cloudflare's https and
Railgun? That's easy to check, as our the whitepapers site has TLS (although
admittedly not http/2):

    
    
        root@sydney:~# ./rg-diag -json https://whitepapers.theregister.co.uk/ | grep elapsed_time
        "elapsed_time": "1.559860s",
    

that's quite a huge chunk of time that Cloudfalre HTTPS + Railgun just
saves/shaves for us. Recommend it highly!

~~~
pbarnes_1
Did you try CloudFlare without Railgun?

That would be interesting.

~~~
mfontani
Sure, let's do just that… from the same location in Sydney; origin server
hosting the content is in UK. This domain is on their "free" plan, as it gets
hardly any traffic.

    
    
      root@sydney:~# ./rg-diag -json https://thereglabs.com/ | grep -e elapsed_time
        "elapsed_time": "0.863677s",
    

So that's from Sydney to the UK, with https served by Cloudflare. The webapp
serving that isn't the sharpest knife in the drawer, but when tested on
localhost it replies in 0.015s – the rest is time taken moving bytes across
the world.

    
    
        root@sydney:~# time curl -sH 'Host: thereglabs.com' -H 'Cf-Visitor: {"scheme":"https"}' http://THE_ORIGIN_SERVER/ -o/dev/null
        real	0m0.821s
    

… and this is plain HTTP to the origin server: the free plan is great for
offloading HTTPS at basically no cost in time added.

We've got another domain on the business plan… so let's try that one.

This is an _image_ request, which is _cached by cloudflare at the edge_:

    
    
        root@sydney:~# ./rg-diag -json https://regmedia.co.uk/2016/11/09/hypnotist_magician_smaller.jpg  | grep elapsed_time
        "elapsed_time": "0.239641s",
    

Lovely, the "local caching" of their CDN helps a ton!

… compared to if we were to request the same file from the ORIGIN_SERVER over
HTTP:

    
    
        root@sydney:~# ./rg-diag -json http://ORIGIN_SERVER/2016/11/09/hypnotist_magician_smaller.jpg  | grep elapsed_time
        "elapsed_time": "0.704458s",
    

… but our "origin server" _also_ is likely to have the image in the "memory
cache"…

… and that image was likely in their cache; so… let's add a parameter so they
_will_ have to ask the origin server:

    
    
        $ pwgen 30 2
        Eehacoh2phoo1Ooyengu6ohReWic2I Zeeyoe8ohpeeghie3doyeegoowiCei
    

There you go… two new randomly generated values…

    
    
        root@sydney:~# ./rg-diag -json 'https://regmedia.co.uk/2016/11/09/hypnotist_magician_smaller.jpg?Eehacoh2phoo1Ooyengu6ohReWic2I=Zeeyoe8ohpeeghie3doyeegoowiCei'  | grep elapsed_time
        "elapsed_time": "1.198940s",
    

Yup, took quite a bit longer than the 200ms it took when the image URL was
fully in their cache.

All in all, from the point of view of being able to _easily_ serve people on
the other side of the world with a "good enough" (not great, mind you!)
response time, both "standard" Cloudflare, the "pro" offering _and
specifically_ the "business" offering are just effin AWESOME.

------
hannob
There are a couple of other things you can do with existing TLS technology
that can improve your latency, e.g. using OCSP stapling, use modern crypto so
browsers may use TLS false start, avoid too many ciphers or unnecessary certs
in the chain to make the handshake smaller.

It's a bit older, but here's some info, much of it is still valid:
[https://istlsfastyet.com/](https://istlsfastyet.com/)

~~~
citrin_ru
It is questionable if OCSP stapling reduces TLS handshake time.

Without OCSP browser makes slow request to CA, but caches results for a long
time so slow request happens not often.

With OCSP stapling enabled more data is transferred between client and server
on _each_ TLS handshake.

Main proponents of OCSP stapling are CA, because it saves them
bandwidth/hardware.

~~~
pfg
Thinking about this a bit, it seems to be that clients talking to a server
with OCSP stapling support could still make use of cached OCSP responses by
simply omitting the "status_request" extension in the client hello, which
would cause the server not to send the stapled OCSP response. I don't think
any clients behave that way today, though.

I'm not certain how session resumption plays into this either. If OCSP is
skipped for resumed session as well (which would be my guess), you'd probably
not take that small bandwidth hit all that often.

As an aside, OCSP stapling improves your user's privacy quite a bit as well,
by not giving your CA a list of all IP addresses connecting to a domain.

------
hedora
Presumably, cloudflare is up to its ears in NSL's, illegal wiretaps, etc. If
you care at all about mass surveillance, censorship, oppressive governments
(in the US, or the location of the cloudflare proxy) you probably should look
elsewhere.

It's probably controversial, but I'd love to see a yellow security icon in
browsers when sites are using well known https relays that can see plaintext
(or are doing other obviously bad things, like running software with known
zero day exploits, etc)

~~~
apeace
> Presumably, cloudflare is up to its ears in NSL's, illegal wiretaps, etc. If
> you care at all about mass surveillance, censorship, oppressive governments
> ... you probably should look elsewhere.

This analysis seems flawed. If you care about mass surveillance, you want
their top-tier security and legal teams working for you.

~~~
dublinben
Their security and legal teams don't work for you, they work for the company.
The company can be drafted to work for the government. Just because you pay
them, does not mean they actually work for you.

~~~
dickbasedregex
Or even that their interests and yours are even remotely aligned.

They sell you a widget/service. That's where your relationship ends.

------
sp332
If you're worried about a proprietary solution, you could host your own cache
server in Australia or wherever your customers are having trouble.

~~~
problems
Yeah, at a $200/mo cost, you could spin up a few VMs on DigitalOcean, Vultr or
LightSail which have decent bandwidth and cache from there.

Nice part about cloudflare though is that they can use anycast to determine
location and then send the closest server IPs. For sub-$200/mo, you're not
able to do that, you'd have to find a provider that could do it for you, I'm
not sure anyone offers country-based anycast DNS alone.

EDIT: Looks like easyDNS enterprise may be able to do it,
[https://fusion.easydns.com/Knowledgebase/Article/View/214/7/...](https://fusion.easydns.com/Knowledgebase/Article/View/214/7/geodns)
for about $12.75/mo too. Might be a decent way to brew your own mini caching
CDN for fairly cheap.

~~~
manigandham
> anycast to determine location and then send the closest server IPs

Anycast doesn't determine location or send the closet IPs, it's all the same
IP address announced using BGP (border gateway protocol) to automatically
route to the closest (in network travel) server.

~~~
problems
Of course. Let me clarify - they use anycast DNS to send the closest CF
caching proxy's IP.

~~~
manigandham
That's not how it works. Both their DNS and reverse proxy servers use anycast
IPs without any DNS-based routing.

They did recently release some features called traffic manager that lets you
control the origin server based on geo. If you just need geo-balanced DNS
though, AWS Route 53, Azure Traffic Manager, and NSOne offer DNS based
routing.

~~~
problems
Really? I didn't think they'd do anycast on their reverse proxy servers, that
seems risky to me (ie: a TCP connection changes from one server to another due
to a BGP change), but I suppose the odds are fairly low.

I seem to remember getting different IPs from different locations, but it
could just be random or I could be mistaken.

EDIT: Tried now and it seems I'm getting the same IPs from Canada and
Australia, so you are indeed correct.

~~~
manigandham
Yes, they have a big address space and announce all ips from every location:
[https://blog.cloudflare.com/cloudflares-architecture-
elimina...](https://blog.cloudflare.com/cloudflares-architecture-eliminating-
single-p/)

Pretty much all major CDNs use anycast today for load balancing, rolling
downtime and security/ddos protection. Http/tcp connections are usually short-
lived, relatively cheap to setup (since it's an edge network anyway) and BGP
route updates don't happen that often.

LinkedIn switched to anycast too after testing:
[https://engineering.linkedin.com/network-performance/tcp-
ove...](https://engineering.linkedin.com/network-performance/tcp-over-ip-
anycast-pipe-dream-or-reality)

------
Kiro
I don't understand why I need to use https on a static marketing webpage. No
login stuff, no JavaScript, nothing. Just straight up HTML and CSS. Right now
I need to pay about $150 every year for something that's only used to satisfy
Google PageRank (I can't use LetsEncrypt with my hosting provider). Why?

~~~
eganist
Keeping it extremely high level:

Among other reasons, not encrypting traffic gives an opportunity for bad
actors to replace content in transit to your end users when your end users are
on compromised connections, such as rogue "free" wifi networks in airports or
coffee shops, or even legitimate networks which have in some way been
compromised, e.g. the ISPs of the world who decide to inject other content
e.g. their own ads into unencrypted traffic.

The next question is usually "what could they possibly do, change a few
pictures?"

They could inject malicious payloads, and for all your users would know, it
would appear to them that it came from your site.

> I can't use LetsEncrypt with my hosting provider

Consider switching. For a static site, consider Gitlab; they do a good job of
permitting LetsEncrypt.

\---

I sincerely appreciate the question, though. I have marketing people ask me
this question all the time in private who hesitate to do so in public because
quite a few security types berate them for not doing something "obviously"
more secure. It's not at all obvious to most of the world's web designers and
content creators that a static site should be TLS'd until it's framed (heh) in
this manner. The fact that you asked brings about a massive educational
moment.

Anyway, consider switching hosts. :)

~~~
cmdrfred
May I add an example. Let's say you are a drug company and you offer a number
of different drugs. With TLS I only know that you are interested in a drug
that company produces or the company itself, without it I know you or someone
you care about has erectile dysfunction.

~~~
spand
No that is not all an attacker could know. TLS does not provide
confidentiality of the number of bytes transmitted. So in your example an
attacker would only have to crawl the public website and find the pages
matching in size to the ones you have been browsing.

~~~
mi100hael
There are web server modules that will append random-length comments to the
end of a page's HTML in order to foil this kind of attack

[https://github.com/nulab/nginx-length-hiding-filter-
module](https://github.com/nulab/nginx-length-hiding-filter-module)

------
c0nfused
It seems to me that it is worth considering that HTTPS is not always a panacea
of goodness. We should think about two things.

First that almost every firewall out there right now supports https snooping
via MITM. Example:
[https://www.paloaltonetworks.com/features/decryption](https://www.paloaltonetworks.com/features/decryption)

Second, I just got back from rural China where most unblocked american
webpages take between 5-15 seconds to load on my mobile phone many of them
take upwards of a minute to load fully. This seems to be a fun combo of
network latency, smaller than expected bandwidth, and pages using javascript
with a series of different load events to display content. That
dompageloaded->xmlhttprequest -> onreadystatechanged chain can ad some serious
time on a 500ms round trip, and that's without talking about the css, the
images, and the javascript.

I forgot to pay me electric bill before I flew out and it took me nearly an
hour to login, push pay my bill, accept the terms, and confirm payment. I was
not a happy camper.

It seems to me that while https is a very good thing, in some cases http and
low bandwidth solutions might be worth implementing. It seems to me that one
might actually want to tailor this to your audience, no one in their right
mind is going to waste 5 minutes loading your web page. If they are so
desperate they need to wait, they are going to hate you every minute they do
it.

~~~
rocqua
> First that almost every firewall out there right now supports https snooping
> via MITM. Example:
> [https://www.paloaltonetworks.com/features/decryption](https://www.paloaltonetworks.com/features/decryption)

Seems prudent to mention that this requires cooperation of the client bein
MitMed. Specifically, the client needs to install a root certificate.

------
SEMW
Funny coincidence, I was running into this exact issue earlier today. Had a
customer complain about high response times from even our /time endpoint
(which doesn't do anything except return server time) as measured by curl, and
turns out it was just the TLS handshake:

    
    
        $ curl -o /dev/null -s -w "@time-format.txt" http://rest.ably.io/time
        time_namelookup:  0.012
           time_connect:  0.031
        time_appconnect:  0.000
        time_pretransfer: 0.031
             time_total:  0.053
    
        $ curl -o /dev/null -s -w "@time-format.txt" https://rest.ably.io/time
        time_namelookup:  0.012
           time_connect:  0.031
        time_appconnect:  0.216
        time_pretransfer: 0.216
             time_total:  0.237
    

(as measured from my home computer, in the UK, so connecting to the aws eu-
west region)

Luckily not that much of an issue for us as when using an actual client
library (unlike with curl) you get HTTP keep-alive, so at least the TCP
connection doesn't need to be renewed for every request. And most customers
who care about low latency are using a realtime library anyway, which just
keeps a websocket, so sidesteps the whole issue. Certainly not enough to make
us reconsider using TLS by default.

Still, a bit annoying when you get someone who thinks they've discovered with
curl that latency from them to us is 4x slower than to Pubnub, just because
the Pubnub docs show the http versions of their endpoints, wheras ours show
https, even though we're basically both using the same set of AWS regions...

------
andreareina
One round trip over the course of the time that the user is using the same
OS/browser installation isn't much.

The Cloudflare Railgun is an interesting solution, and one that could be
implemented in the context of an SPA over a websockets connection. Or
conceivably some other consumer of an API.

------
filleokus
A related interesting topic is the possibility of secure cache servers that
don't break the secure channel with "blind caches". Currently just a RFC draft
and probably a long time from mass adoption, but nevertheless interesting.

[https://tools.ietf.org/html/draft-thomson-http-
bc-00](https://tools.ietf.org/html/draft-thomson-http-bc-00), and Ericsson's
article on it
[https://www.ericsson.com/thecompany/our_publications/ericsso...](https://www.ericsson.com/thecompany/our_publications/ericsson_technology_review/archive/blind-
cache)

------
nprescott
I really enjoyed the coverage of the same topic in _High Performance Browser
Networking_ [0]. It effectively explains the key performance influencers
across various networks without being boring.

[0]: [https://hpbn.co/](https://hpbn.co/)

------
aanm1988
> In our case at Hunter, users were waiting on average 270ms for the handshake
> to be finished. Considering requests are handled in about 60ms on average,
> this was clearly too much.

Why? Did it hurt user engagement? Were people complaining the site was slow?

~~~
kuschku
The question is "is this the best we can do?".

If it’s no, then clearly we should improve.

~~~
aanm1988
By spending time and effort on things that may not actually matter?

To their credit this post talks about improving the performance, instead of
just using it to complain about they can't use https because of a difference
in a metric that may or may not actually cause end users any pain.

~~~
kuschku
You do realize that every latency you see on residential connections is
magnified on mobile or even satellite connections?

If you have a RTT of seconds, 10 additional roundtrips can cost an entire
minute.

These things become very noticeable very quickly if the system is in non-
perfect environments.

Additionally, even with modern browsers it takes far too long to open a
website — it should be 100% instant, less than one or two frames (16 or 33ms).
That's not possible, as RTT is usually around 18ms between users and CDN
edges, but at least it should be below perceivable delays (100ms).

EDIT:

My best websites hover around 281ms to start of transfer, and 400ms to the
site being finished.

That's improvable, but most sites out there take literally half a minute to
load.

Now go on a 64kbps connection, and try again. Handshake takes seconds, start
of transfer is almost after 30sec, and by the time your website arrives your
coffee has gone cold (a few minutes for a google search).

Years ago, Google was usable on dial-up. Now even the handshake takes as long
as an entire search used to take. Notice anything?

------
chatmasta
What's wrong with the cloudflare free plan? You can host a static site on
github pages with a custom domain and use the free cloudflare SSL cert.

~~~
amiraliakbari
The "Railgun" feature mentioned in the article is only available in some paid
plans. Using the free plan wouldn't keep an open connection between your
servers and Cloudflare's. It does improve the situation by terminating users'
handshakes early, using better links, warm DNS cache, etc. among servers. But
the latency hard limit is still present between your server and CF. Skipping
https between your server and CL is not an option either for any site
transferring user data.

~~~
chatmasta
Ah, I see. I did not realize that. Accordingly, I edited my comment to be less
inflammatory. :)

I understand that by using the generic CF free cert, https terminates at CF
and the connection CF->Origin is over unencrypted HTTP. Is this why there is
latency overhead? Because CF cannot connect to origin via https so it cannot
open a persistent tunnel? Or is it because the overhead of keeping an open
https tunnel per origin server is prohibitively expensive to maintain for
every free customer?

I assume that even though there is no persistent tunnel, CF still must still
use persistent TCP connections?

~~~
floatboth
Maybe the cost is not completely prohibitively expensive but they do consider
it a premium feature. Have to earn money :D

