
Google Domains blocking all Gitbook URLS: post-mortem - martypitt
https://blog.gitbook.com/tech/post-mortems/06-20-gitbook-domains-blocked-by-registrar
======
SamyPesse
GitBook CTO here:

Our production domains (gitbook.com and gitbook.io) have been blocked and
locked by our registrar (Google Domains).

None of our infrastructure is impacted, all user content and databases are
safe; our domains simply blocked by a heavy handed policy.

As mentioned on Twitter, we are all hands working with Google to fix this
issues ASAP. We'll then share an in-depth post-mortem

[https://twitter.com/GitBookStatus/status/1268554857411227648](https://twitter.com/GitBookStatus/status/1268554857411227648)

~~~
SamyPesse
We've just published a postmortem: [https://blog.gitbook.com/tech/post-
mortems/06-20-gitbook-dom...](https://blog.gitbook.com/tech/post-
mortems/06-20-gitbook-domains-blocked-by-registrar)

let us know if you have any questions!

~~~
cirno
I'm curious how you feel about CloudFlare as a registrar not allowing GitBook
to use an external root nameserver.

Being forcibly stuck on CloudFlare's own nameservers only sounds very
nefarious, and isn't a limitation I've ever heard of with any other registrar.
For instance, it would break my tooling that uses my host's APIs to control
DNS records through their nameserver.

I'd be very appreciative if eastdakota or jgrahamc could elaborate on what
possible reasoning there is for this restriction as well.

~~~
DenseComet
Cloudflare sells the domain at cost. I think the idea is that its an extra
service meant for their customers, not a service for the general public. As
they are a DNS provider, their customers will use cloudflare nameservers. If
they didn't, they would no longer be customers.

~~~
cirno
That does make sense. If I were using Cloudflare I suppose it would be a no-
brainer, and if I were Cloudflare and didn't want people not routing their
traffic through me on my registrar, that would be an excellent way to
discourage it. If they're forced to offer to everyone as part of being a
registrar, then the combination of all of the above is my answer. Thanks!

------
mbreese
Aside from the Google domain issue (which is obviously a problem) -- why on
earth would GitBook be hosting user-generated content under their corporate
gitbook.com domain?

The registrar issue is one problem here -- but I don't see this addressed in
their post-mortem. I thought this was a well-known issue. You don't host user-
generated content on the same domain that handles your corporate email.
Because things like this can and do happen, so you want to make sure that your
company isn't also down while you're busy fixing a problem with your
customers.

I know the exposure risks are different, but isn't this is one of the reasons
why Github moved hosting Github Pages content to github.io from their primary
github.com site? Or why raw data is hosted from githubusercontent.com (in
addition to mitigating cookie security issues).

The registrar was an issue that was largely out of their hands. But this was
something that they could control. And I think it's something that is missing
from their post-mortem. If they had split their domains, then while _serving_
user-content was disrupted, _accepting_ users to gitbook.com and their email
(!?!?) would have still been working. Also, depending on if they had a 3rd
domain for hosting their CDN, users that had custom domains would also have
been protected.This goes along with the idea that you don't use email from
your primary domain (cto@gitbook.com) to register your domain (gitbook.com).
Or host your status page on the same infrastructure as your site.

If I were them, after migrating registrars, this would be the next engineering
change I'd make.

 _(Maybe they do this, I don 't know enough about GitBook to know. But based
on the thread here, I don't think they do, or at least it isn't mentioned in
their post-mortem that I saw.)_

~~~
jtl999
Where do you draw the line? I mean github.com is the corporate email domain of
the GitHub company, it also hosts repositories (user generated content)

~~~
mbreese
But it isn't... directly.

Github.com is their corporate site, hosts their application, and is how we all
interact with repositories (via https or git://). But, the only data you get
from that site has been processed through their application and sanitized.

The only way to get access to the _raw_ user-generated data is through
raw.githubusercontent.com _or_ Github Pages which hosted on github.io. And the
data from raw.githubusercontent.com has the MIME types set so that you don't
get HTML rendered -- only the raw plaintext (I think).

So, in this specific example, if there was someone hosting a phishing site in
a github account, it would have been active only through github.io, _not_ the
main github.com site. (You could have likely seen the source code from the
main site, but it would not have actually generated an HTML form).

For historical views, here is the Github blog post that details the change
(2013): [https://github.blog/2013-04-05-new-github-pages-domain-
githu...](https://github.blog/2013-04-05-new-github-pages-domain-github-io/)

~~~
pmontra
Nevertheless some countries blocked github.com, not (only?)
githubusercontent.com or github.io

[https://en.wikipedia.org/wiki/Censorship_of_GitHub](https://en.wikipedia.org/wiki/Censorship_of_GitHub)

Probably they think that GitHub didn't sanitize the content on github.com well
enough.

------
mikedilger
Could you imagine if "youtube.com" were blocked and locked by IANA/ICANN/?
because of reports that somebody posted a video with misinformation and
phishing content?

I don't expect google appreciates how devastating this, but that would change
quite immediately if it happened to them.

~~~
walrus01
It's almost as if, while the laws and policies and regulations are supposed to
be the same for all, giant entities with huge amounts of money and power
behind them are held to a different standard than everyone else. I can't say
I'm surprised.

~~~
lmm
There's a parade of apologists who will quickly pop up to say that a private
company platform should not be held to any free speech standard since they're
not a government, never mind how much power they wield in practice.

~~~
techntoke
What about when it is the government taking down a domain for hosting illegal
or copyrighted content? What happens when you give the government enough power
to take down a domain for political dissent? The problem you mention isn't
with Google, it is with how TLDs and domain registrars work in the first
place.

------
tzs
> sent to an e-mail address on a domain they had just suspended...

I've always assumed that:

1\. When creating your account at the registrar you should use an email
address that is not at any domain you will be registering through or
transferring to them,

2\. The contact information you give for your WHOIS records (or if using a
WHOIS privacy guard service, the contact information you give them to forward
to) should use an email address at a different domain,

3\. If you use a DNS provider other than your registrar the contact email
address they have for you should not be a domain using them for DNS, and

4\. Same for whatever hosts your email.

The general principle is that the contact method that a service provider will
use to contact you if there is a problem with your account or service with
them should not depend on the account being in good standing and the service
working.

------
Animats
The biggest hoster of phishing sites is Google.[1] Here's a list of major
sites which have live entries in PhishTank.[2] Hosting phishing sites on
Google Drive is very popular.

Many of those are long gone, but PhishTank hasn't cleaned them out, so they're
still listed.

[1]
[http://www.sitetruth.com/reports/phishes.html](http://www.sitetruth.com/reports/phishes.html)
[2]
[http://www.sitetruth.com/fcgi/ratingdetails.fcgi?details=tru...](http://www.sitetruth.com/fcgi/ratingdetails.fcgi?details=true&url=google.com)

~~~
heipei
To be fair, every platform which allows user-generated HTML pages suffers
massively from phishing and most of them don't deal with it very well: Google,
Microsoft, smaller players like Codebox and countless others. Then there's
phishing on Dropbox, phishing on Google Forms, OneDrive, etc. Then you have
phishing at all the various hosters like DigitalOcean, CloudFlare, etc. Even
there you'll sometimes have IPs which have hosting phishing pages for various
brands for a long time. It's not an isolated problem. Some deal with it more
aggressively, true, but the pace and ease with which phishing can be stood up
and modified makes it a whac-a-mole. Plus, the expectation is that most
phishing pages will only be active for a few hours before being taken down
and/or detected, so phishers pump out new ones on a constant basis.

I run the service at [https://urlscan.io](https://urlscan.io) which tracks
phishing and frequently run into these cases which render any kind of
black/whitelisting impossible. Imagine Microsoft phishing hosted on Microsoft
domains and infrastructure. Here's a fun search which will return lots of
phishing on windows[.]net and googleapis[.]com:
[https://urlscan.io/search/#page.domain%3A(googleapis.com%20O...](https://urlscan.io/search/#page.domain%3A\(googleapis.com%20OR%20windows.net\)%20AND%20filename%3A%2F\(jpg%7Cpng%7Cgif%7Csvg\)%2F)

~~~
bigiain
> To be fair, every platform which allows user-generated HTML pages suffers
> massively from phishing

The big difference here is that all the other platforms struggling (and
failing too often) to prevent themselves being used as phishing hosting -
aren't then turning around and hypocritically taking other platform's entire
internet presence offline for doing so.

Google here are acting in the roles of judge and executioner, while being the
biggest offender of the same crime.

------
koluna
Google Domains locked my domains at renewal time and refused to renew them
until I provided proof of identity in the form of a scanned government ID
-AND- a scanned copy of the credit card.

Coupled with the horror stories of non-existent support, the first thing I did
was move my domains out this month.

~~~
CameronNemo
To where?

~~~
bigiain
I've kinda given up, and use Route53 for everything, on the grounds that if I
managed to piss AWS off enough for them too take my domain names down, all my
infrastructure will probably vanish to, so having someone else manage to keep
my zone files up will not be of any use anyway...

It's got all my eggs in one basket - but solving that problem is a much bigger
task that just picking a different domain registrar... (And I sometimes wonder
if my blue-sky cloud-agnostic dreams would all come crumbling down if I even
managed to implement then anyway - if whatever went wrong that pissed AWS off
enough for them to shut me down got shared with and/or triggered the same
reaction at Azure/Google/DO/whoever...)

------
dundercoder
This mirrors my experience with google support, even as a paying gsuite
customer. I lost a YouTube channel and wasn’t given any option to restore it.
Their advice was to just re upload all the content and forget about view
counts and old links that would no longer work.

~~~
boromi
I'm going through issues with G Suite as an admin. They randomly blocked my
account falsely accusing me of sending spam. I can't even access the help
support team sicne I can't login to the system.

~~~
fouric
This is completely insane - there's absolutely no legitimate reason for Google
to _lock you out of your account_ , even if you _were_ sending spam emails.
Disable your ability to send new emails, maybe. Lock you out of your email
inbox, maybe. But completely prevent you from accessing your account, and
therefore even appealing? Inexcusable.

~~~
skrebbel
Come on bear with them. They're not a very big or technically sophisticated
company, so they only have one big on/off button per account.

------
Karupan
I have a simple rule when using Google services (including paid ones) - be
ready to lose access without a moment's notice. Unless you are an enterprise
customer or can reach out to Google employees when things go horribly wrong,
you are at the mercy of their automated systems.

As other posters have pointed out, hope they have a plan to migrate their
domains elsewhere.

------
joepie91_
For the record: if you don't want this to happen to your site, you should
avoid Dynadot also.

Several years ago I ran into almost _exactly_ the same issue with them after
someone sent them a frivolous abuse report - they'd locked down my domain
unannounced to the point that I couldn't even transfer it out, and it took a
call from a journalist(!) inquiring about the suspension before they were
willing to unlock it for transfer.

I've been using internet.bs for most of my domains since and haven't had any
issues with them - they let me know when an abuse report comes in, and give me
the time to handle it. There are probably other registrars that are fine too,
but I don't have personal experience with them.

------
thinkmassive
"Everything is back to normal! The domains have been unblocked by the
registrar. We are monitoring everything."

[https://twitter.com/GitBookStatus/status/1268565887256330241](https://twitter.com/GitBookStatus/status/1268565887256330241)

------
spacephysics
After hearing this I’ll be transferring away from Google domains. Had to use
them for the initial .dev sales

~~~
baryphonic
Same. I bought a few .app names when they came out, but I'm transferring the
rest away. It's not even the threat of someone accidentally locking something
that concerns me, but the fact that Google is now so big that, like a black
hole, all information is pulled in and none can escape, including (effectively
nonexistent or incompetent) support.

------
wolco
Never use google as a registrar. Support someone else. The internet is bigger
than google.

~~~
techntoke
Support who? Government has power to take down any domain, especially TLDs and
registrars in the US. People should be supporting decentralized DNS and name
servers if they want to be safe from takedowns.

~~~
cube00
If the government is taking down a domain at least there's a human behind it.

------
CompuIves
The exact same has happened with CodeSandbox, Google Domains had blocked our
csb.dev (internal domain) and csb.app (domain for projects) without any
warnings. It took us two days (of downtime!) to get them to lift the block.
But then after 3 weeks, they did the exact same thing! They blocked us again
for reports without warning. Luckily we had a fallback domain so that there
was not much service disruption, but it was extremely frustrating working with
them on this.

After 3 sudden blocks from them, they now give us a warning before they're
planning to block us, but it took many calls before we got to that stage.

------
Keverw
Makes me wonder if they don't understand user-generated content, if Google
Domains is more meant for small personal sites maybe?

I used to use a hosting company that had in their terms, that profanity wasn't
allowed... The company seemed to have decent hardware, user interfaces and US
based support too but ran by Mormons based in the beautiful state of Utah. I
don't have much of a opinion of Mormons but I think pushing religious views on
your customers is bad business.

So after noticing that in their terms I was curious since I run a personal
WordPress blog, if someone wrote a comment even if it wasn't approved just by
being in the database if it broke their terms and was told yes... Now that I'm
a bit older, I wonder if they really are scanning database table text fields
for swearwords or that support rep misunderstood what I meant... But after
that discussion, I switched my domain and hosting elsewhere, two separate
companies actually instead of just one account with both domain and hosting.
If some spam bot or troll writes the F word, they are going to take my entire
site down even if no fault of my own? Yeah, no thanks.

They got acquired though by a lot larger company and doesn't seem to have
those same terms now though, and heard they now outsource their support. There
was someone I used to talk to who was a big fan of them though and referred
me. So slightly over a decade later, they are probably not even remotely the
same company anymore though since merged with a much larger company. Wouldn't
surprise me if they ended up getting rid of a lot of support staff and just
kept a handful of people to manage the datacenter, since marketing,
accounting, support, etc would probably be centralized between all the hosting
companies they own I'd imagine, which is probably a blow to the local areas
when they merged unfortunately. I guess that's one of the sad things about
getting bigger, they lose that small town friendly startup feel probably as
seemed like a great company other than forcing religious views on people. Just
did some more research about their company, sounds like they were also anti-
lgbt too sadly. Makes me wonder how many other people working their share
those views or just something management was pushing down since looks like the
church changed their policy last year... I've been wanting to go somewhere
with more opportunities, and actually Salt Lake City is one of the areas I've
been considering since seems like a bit bigger city and on the list as a place
for upcoming startups.

------
wegs
... Which is again why I would never use Google for any business application.
These stories happen all the time.

"I built my business on Google Cloud / Android / Google Apps / Youtube / etc.
I was flagged by an automated system. There was no human to speak to (at least
one competent or empowered to do anything), and my business is now gone."

Some of these go viral on social media, and Google then fixes them. Some
don't.

Support channels built on only having an impact if things go viral aren't what
I'll build my business on.

------
ggm
Having just moved 3 personal domains -> into google domains you can imagine I
find this quite concerning.

I wonder how the domain registry community at large feels about this? ICANN
exists, domains are subject to a legal agreement with ICANN, and it has
customer-protection concerns surely?

~~~
dannyw
I can tell you about my experiences with Namecheap when I used fake WHOIS info
for my privacy and someone reported me.

Their support staff said that due to ICANN policy, they must have accurate
WHOIS info, so they have changed my WHOIS information but also added
WhoisGuard to the domain so the details are not visible unless someone gets a
Panama court order.

My trust in them increased substantially.

------
kerng
Using Google products really becomes a big liability for businesses. Articles
like this have become quite frequent over the last year.

------
antoncohen
Are they conflating domain registrars and DNS hosting? Did Google Domains
block their domain at the registry level, like removing the NS records from
roots? Or did they block is at the domain DNS level, like not resolving
queries to Google nameservers for the domain?

I don't expect many startups or big companies use Google Domains for _DNS
hosting_. AWS Route 53, Google Cloud DNS, or dedicated DNS hosting would be
used for that. But I expect it is fairly common for startups to use Google
Domains for domain registration.

------
coronadisaster
A bit off topic but Google blocks me constantly from accessing my gmail
account... is there anyway to disable the "suspicious activity" "protection"?

------
Ayesh
I don't think it should be registrar's duty to shut domains down.

Gitbook appears to use CloudFlare to host DNS, and likely to use their
CDN/proxy, which makes them the actual "host" of the content. If there is any
phishing, they should be the ones responsible.

The post says even their email was down. I'm deciding to not use Google
Domains anymore.

------
franga2000
A month or so ago, I was setting up a new system for writing internal-ish
documentation and made the rather unpopular decision to not go with Gitbook
because it was too reliant on their servers for my taste. Today, I got to send
a very satisfying "told you so" the the team chat.

To be clear, I think the Gitbook team actually handled this very well and it
wasn't their fault, but it does illustrate very clearly why we shouldn't be
relying on cloud services when we really don't need to.

~~~
Spivak
I’m surprised how few people in this space aren’t packaging their apps up for
on-prem customers.

* Fewer security and compliance hurdles when its on our own infra.

* Perfect uptime because it’s our responsibility. (Not saying we’re better but outages are our fault and don’t make you look bad)

* Fewer hurdles when integrating LDAP and friends. There’s no need for the weird “connectors” that companies come up with.

* We’ll pay you more for the privilege of hosting it ourselves.

* Giving us access to the code in a shared source model buys you an extreme amount of lock-in since once we start tweaking our fork for our needs we’re not leaving.

* The support burden is higher, sure but if it’s your most expensive enterprise tier you’ll almost always be working with a professional IT team.

* We’re the easiest crowd to up sell and we’ll even pay you to develop features you can sell to other people.

I don’t want to make it seem like selling in enterprise isn’t a slog but it’s
a good business to be in.

------
_jal
You know, this is interesting. Google is taking something close to an
editorial stance here - it is clear that they take action against their
customers based on content on their sites.

Some might consider that sites with controversial content who use Google as
their registrar and not taken down are considered reviewed-and-OK by Google.

Others might start pressuring Google to expand the nature of content they find
unacceptable.

And yet others might try pressuring them to expand their policing - who knows
how many bad actors could be shut down with Gmail monitoring?

~~~
notriddle
Phishing is not "controversial." It's illegal, and it undermines the principle
of informed consent that meaningful freedom requires. Even if Google isn't
legally obligated to take down phishing domains, I refuse to be so anal as to
complain about it. It is everybody's job to make the internet safe and usable.

Too bad Google does a terrible job of it. No way am I signing up for a Google
Domain in the future.

~~~
_jal
The point is not whether it is reasonable to do so or not; I agree that
phishing sites should go.

The point is that there is a large lobby out there for knocking different
types of content offline, and they don't show much concern for whether their
demands also seem reasonable.

Most registrars feel they should not be in the content policing or law
enforcement business. Google volunteering to do it in one area that makes
sense opens the door for pressure to do it in other areas that make less
sense.

------
ezoe
When google said "Jump", you must say "How high?". But there is more. Google
won't inform you the required height or doesn't define what jump is.

------
nitwit005
> sent to an e-mail address on a domain they had just suspended...

I remember logging into an Ad Exchange account and discovering Google was
sending emails to it. No one at the company was aware of it.

------
machbio
The one surprising aspect about google domains - it is not integrated into
Google cloud.. they are two separate products while AWS domain registration is
part of aws cloud

~~~
bduerst
Unlike Amazon, Google has more domain-related services than just Cloud - i.e.
Analytics, Sites, Places, etc - which is probably why it's still independent.

------
aledalgrande
Wait until Google controls all of your traffic, with AMP, ads, Google Cloud
and all their services. Flip the switch off for a business and kill.

------
johnghanks
gitbook... who?

------
rydre
Try epik.com

------
strooper
Just out of curiosity- when Google is infamous for hard-to-reach human
support, what in the Internet would make anyone interested to register their
domain with them? Do they provide some sort of security or insurance that I am
unaware of?

All the popular dedicated domain registrars I have used so far have excellent
human support. Godaddy, namecheap, namesilo to name a few. I don't know if big
companies or corporate use something more to secure their domain names and
DNS, do they?

~~~
drusepth
I moved/consolidated from GoDaddy and 101domain to Google Domains because of
the support I've gotten from Google in the past (on Nexus/Pixel devices, Apps,
Fiber, Fi, Stadia, etc).

I always assume the people complaining about nonexistent support from Google
are trying to get support for something they aren't paying for. You pay for
Domains, and the support reflects that. You probably can't get support for
getting locked out of a consumer Gmail account or help uploading a YouTube
video.

~~~
dannyw
As a paying G Suite customer, all of my support experiences have been
horrendous, including trying to unlock an employee’s account that was locked
for “spam”.

The support agent couldn’t do a single thing but tell me to wait for the
possibly robotic appeals process.

------
downerending
Since no one else has gone there, does this qualify as "book burning"?

------
hahadeservedit
I told you guys, don't use Google Cloud or any of their services, but GPC
users thought they are smarter... keep shooting in your own foot.

------
robk
If you were hosting phishing sites then I'm glad they did this. You should
have better controls.

~~~
MattGaiser
There is spam on Facebook, there is spam on Reddit, there is spam on Twitter,
there is spam on Hacker News.

Proactive spam control is not a solved problem anywhere. Banning a website
over it is absurd.

~~~
ben509
Nuking a domain with no warning for any reason is nuts. They started to shut
it down _before_ the email notification.

And any time support or PR is droning on about "muh policies" you know there
are problems.

