
HTTPS on Stack Overflow: The End of a Long Road - Nick-Craver
https://nickcraver.com/blog/2017/05/22/https-on-stack-overflow/
======
mrunkel
At $previous_job we once turned on HTTPS for our entire customer website and
online store, only to have our customer support team be bombarded by phone
calls claiming that our "website was down."

After much teeth gnashing and research, we determined that a large segment of
our user base was still using WinXP and the encryption protocols we offered
weren't available to them.

We didn't think this would be a problem because the current version of the
software wasn't compatible with WinXP any longer.

There was some debate internally whether the better fix was to including the
legacy encryption protocols or just leave the HTTP version of the site running
and use Strict-Transport-Security to move capable browsers to HTTPS.

In the end we had to include the legacy protocols so those customers could use
our online store.

~~~
mattmanser
I know it's hindsight and all that, but why didn't you check your website
analytics first? Seems a fairly massive assumption that should have taken 10
seconds to check.

~~~
y4mi
Some people don't spy on their customers and don't have these kinds of
information available for analyses

They're admittedly few though and their moral high ground is debatable
considering that there are self hosted FOSS alternatives around nowadays

~~~
HalfwayToDice
Calling aggregate anonymous analytics "spying on your customers" is absurd
nonsense.

~~~
jasonkostempski
Declaring "absurd nonsense" isn't an argument.

~~~
obstacle1
In this case, the absurdity and nonsensical character of the 'spying' claim is
fairly self-evident.

When a client voluntarily makes a request to a server, it presents a bunch of
information for the server to see and consume. This information is not meant
to be kept secret from the server. Among such pieces of information can be
some about the characteristics of the user agent, including OS. It is
disingenuous at best to call collecting such voluntarily-presented and
clearly-transmitted data as "spying" on a user.

A basic requirement for spying is for a collecting party to be obtaining
information that can be reasonably considered confidential or restricted.
Details about the system from which you send a request are by definition of
the protocol not confidential or restricted to the recipient of your request.
It is not reasonable to expect a server to not look at or use information you
present to it. Therefore, it isn't "spying" for the recipient to consume the
information. The information might be used in ways some people(e.g., OP) don't
like, but that does not make obtaining the information "spying".

~~~
jasonkostempski
I only posted this because it was the second time that day I saw "absurd
nonsense" used as a comment with no additional content. It annoyed me enough
the first time that it stuck out like a sore thumb the second time, then I
noticed it was the same user and it was their last 2 comments.

------
lmm
> The password to our data center is pickles. I didn’t think anyone would read
> this far and it seemed like a good place to store it.

You ought to have more confidence in your writing. BRB stealing all your
servers.

~~~
DamonHD
Only if you get there first. And it may in fact be pickles2.

~~~
SimbaOnSteroids
It's actually hunter2

------
phsource
This is incredibly detailed; in short, CDNs, cookies/authentication , tons of
subdomains, and 3rd-party/user-generated content make it a pain to move onto
HTTPS.

I was chatting with a non-engineer friend about why it's hard to estimate how
long tasks often take, and this seems like a prime illustration: the
dependencies are endless.

I also love the Easter egg:

"The password to our data center is pickles. I didn’t think anyone would read
this far and it seemed like a good place to store it."

~~~
niklasrde
Just know the username and you can log onto
[https://stackoverflow.com/admin.php](https://stackoverflow.com/admin.php)

~~~
oefrha
Warning: link's NSFW.

~~~
nhylated
It opens a random youtube video everytime (I got several 10 hour vids
including Jeff Goldblum laugh and relaxing hairdryer sound).

~~~
bpicolo
Seems there's a limited set. It's a pool of 10 hour vids

------
dzdt
Stack Exchange is no longer available from my workplace due to this change. We
have a strict no-posting-code-fragments policy, and SE was viewed as too risky
to allow without some restriction in place to make it read only. Before HTTPS,
the IT department had worked out such a read-only restriction by blocking the
SE login with firewall rules. But with HTTPS that kludge is no longer
possible, so the site is blocked.

~~~
teraflop
Leaving aside all the reasons why this policy is super dumb (which I'm sure
others will cover quite adequately), I guess your IT department can't figure
out how to create their own CA certificate and do SSL interception?

~~~
indexerror
Certainly doable but this should not be done.

~~~
BinaryIdiot
There are many enterprise "solutions" that basically do this "out of the box".
Yeah it shouldn't be done and a lot of employees are likely unaware that IT
can see all of their SSL traffic but it's a big business.

~~~
erikbye
HTTPS or not, certainly nothing is private here, but that's expected for this
sort of place.

------
gbrayut
If you like working on these kinds of projects, the SRE team at Stack Overflow
is hiring and we allow remote work full time!
[https://stackoverflow.com/jobs/143725/site-reliability-
engin...](https://stackoverflow.com/jobs/143725/site-reliability-engineer-
generalist-stack-overflow)

------
tomschlick
Just a reminder, HTTPS isn't enough. Be sure to turn the other security knobs
with headers...

[https://securityheaders.io/?q=https%3A%2F%2Fstackoverflow.co...](https://securityheaders.io/?q=https%3A%2F%2Fstackoverflow.com&followRedirects=on)

~~~
Nick-Craver
Yep - we're aware. I thought about putting in our Content-Security-Policy-
Report-Only findings about what all would break, but the post was already a
tad long. It's quite a long list of crazy things people do.

As the headers go, here's my current thoughts on each:

\- Content-Security-Policy: we're considering it, Report-Only is live on
superuser.com today.

\- Public-Key-Pins: we are very unlikely to deploy this. Whenever we have to
change our certificates it makes life extremely dangerous for little benefit.

\- X-XSS-Protection: considering it, but a lot of cross-network many-domain
considerations here that most other people don't have or have as many of.

\- X-Content-Type-Options: we'll likely deploy this later, there was a quirk
with SVG which has passed now.

\- Referrer-Policy: probably will not deploy this. We're an open book.

~~~
tomschlick
Great! Thanks for the detailed response!

Expect-CT is one to look at as well.

Basically just tells the browser that Certificate Transparency should be
available through the provider (DigiCert in this case).

------
kalleboo
Note to self: Use subdirectories, not subdomains in the future

~~~
baby
TLS kills this kind of "cool" features which is kind of sad :( Unless you can
afford wildcard certs.

What's the argument behind LetsEncrypt not doing that? Extended Validation
stuff?

~~~
cookiecaper
Subdomains were killed by SEO a long time ago (afaik, Google does not transfer
domain PageRank credit to subdomains), so this is not limited solely by the
cost of wildcard certs.

~~~
baby
But this is orthogonal to the issue of LetsEncrypt not delivering wildcard
certs.

------
tomschlick
Side question: any plans for IPv6?

~~~
alienth
Not any immediate plans. Decent amount of development is necessary there.
There are so many places in our various systems that work with IP addresses,
and many of them don't support v6 addresses.

~~~
ec109685
One thing that places do is support IPv6 on the outside and translate to IPv4
calls in the next hop beyond the servers that sit beyond the edge.

------
fareesh
Despite the "Google gives a boost to https" reasoning, which comes from Google
itself, in practice I've read several first-hand accounts of how traffic (non
XP) dropped significantly right after the switch.

------
gub09
It would be better if scripts like jquery were not encrypted. This forces
users to use e.g. a google service instead of caching/hosting the scripts
themselves or getting them from another CDN. I do not understand why so many
people do not consider the privacy implications of every single webpage
requiring calls to google services. There are ways to avoid this, but it gets
a lot more complicated when that requires MITM methods for SSL. Please: use a
non-tracking CDN, host it yourself, or at least leave it HTTP.

------
janwillemb
Wow, I didn't expect this ("switching" to HTTPS) to be so hard.

~~~
irrational
Yeah, we've been working on this for about a year (not continually, but as we
have time to try to work through the problems). We do use subdomains though,
so that is part of the problem. We keep feeling like we are getting close, but
then we run into another issue. It's like a rabbit hole that has no bottom.

~~~
skyisblue
> We keep feeling like we are getting close, but then we run into another
> issue. It's like a rabbit hole that has no bottom.

That's exactly what we experienced migrating a bunch of sites to https. There
were so many things that we didn't anticipate.

------
jontro
Regarding the section "Mistakes: APIs and .internal"

Why wouldn't they use split horizon DNS for this? Seems like the perfect use
case

~~~
Nick-Craver
Split horizon would point you at the same data center, rather than the
writeable one. So that's more of a .local than a .internal. We discussed this,
but ultimately the AD version we're on (pre-2016 Geo-DNS) it's not actually
supported the way you'd need, and it's a nightmare to debug.

We'd consider it for a .local, when the support it properly there in 2016.
Even subnet prioritization is busted internally, so that's a bit of an issue.
Evidently no one tried to use a wildcard with dual records on 2 subnets before
(we prioritize the /16, which is a data center) and it's totally busted.
Microsoft has simply said this isn't supported and won't be fixed. A records
work, _unless_ they're a wildcard. So specifically, the
<star>.stackexchange.com record which we mirror internally at
<star>.stackexchange.com.internal for that IP set is particularly problematic.

TL;DR: Microsoft AD DNS is busted and they have no intention of fixing it.
It's not worth it to try and work around it.

~~~
jontro
Interesting, thanks!

------
quintin
Has anyone tried running Fastly behind Cloudflare? Are the tradeoffs worth it?

~~~
tyingq
Is there some reason other than cost to do that? Curious.

~~~
quintin
Mitigate attacks much better than Fastly, flatten CNAME etc.

~~~
bpicolo
> Mitigate attacks much better than Fastly

if that's the concern, probably just better to configure switching than put
both in front all the time.

------
user5994461
Funny how the main reason for lack of SSL is said to be the lack of support
from 3rd party services... and the first service quoted is ads.

[https://nickcraver.com/blog/2013/04/23/stackoverflow-com-
the...](https://nickcraver.com/blog/2013/04/23/stackoverflow-com-the-road-to-
ssl/)

~~~
therealdrag0
Funny how?

------
jcadam
I work at a government facility. Stack Overflow and github are now both
blocked (in addition to all social media and webmail). But Hacker News is
apparently ok.

------
Zekio
Your blog posts are always an interesting read

------
souenzzo
How many questions on stackoverflow to these migration?

------
merb
sadly that haproxy (which stack overflow uses) does not support http/2
directly, you need to terminate it via nginx or anything else.

------
bullen
I said it before and I'll say it again: HTTPS is a waste of electricity.

