
Google Doorman: Global Distributed Client Side Rate Limiting - ryancox
https://github.com/youtube/doorman
======
mixonic
Hopefully there is a Googler reading who can answer this: I'm curious why the
README.md says

> Note: This is not an official Google product.

However the copyright is Google, and you must sign their CLA. That seems
pretty official to me? Or is there some other implication to "official" beyond
ownership?

~~~
wereHamster
That is because Google claims copyright on everything you create while you
work there, even if you create it in your garage in your free time. You baked
some cookies for your daughters birthday party? Sorry, can't distribute them
for free, consumers of those must sign a CLA (Consumer License Agreement) and
Google retains copyright on your recipe.

~~~
kuschku
Question: Does this also apply to Google employees at Google Hamburg, and, if
yes, how far? Because, as far as I know, German law directly prohibits such
contracts.

~~~
schwa571
I can't answer your question, because I don't know the answer.

However, I would recommend against relying on wereHamster's apparent mis-
information... unless Google's policies are different in the jurisdiction that
wereHamster is familiar with, in which case I apologize.

~~~
kuschku
I think wereHamster’s text was satire, but the idea was repeated by many
others: Anything related to your job you do is owned by Google.

You can’t contribute to ejabberd when you work on Hangouts.

~~~
wereHamster
What does it tell us if people don't recognise that as satire? Is it so close
to reality to think that Google could claim rights on our cookie recipes?

And if you really have to ask if such a clause applies to you, you haven't
properly read your contract and/or you don't know your rights. People, please
educate yourselves, don't let large corporations, even if they claim not to be
evil, violate your rights.

~~~
kuschku
> And if you really have to ask if such a clause applies to you, you haven't
> properly read your contract and/or you don't know your rights.

I’m not working at Google, but I’m a compsci student in Germany, and obviously
interested in the situation in the job market.

> What does it tell us if people don't recognise that as satire? Is it so
> close to reality to think that Google could claim rights on our cookie
> recipes?

It’s certainly possible that some might expect that – from the standpoint of
someone who only heard of German law, for example, a large part of US
employment laws would sound just as crazy as being able to have the copyright
of your employee’s cookie recipes.

------
rixed
Rate limiting looks outdated. Suppose you can spawn new servers (and turn down
unused ones) fast enough to react to demand, and have a good LB that sends the
traffic where you have capacity, then you do not need to limit anything until
you reach full capacity, in which case you probably want to degrade (by query
class and/or client class) rather than limit anyway.

Rate limiting is an inefficient way to distribute a service, that makes sense
only if you preallocate your resources and have queries with predictable cost.
Let's use this technique only as a bug-prevention tool not for resource
economy, as organised scarcity is likely as inefficient for the data center as
it is for the distribution of goods :)

~~~
greenleafjacob
If you degrade by client or query class then in the case of just one class of
query or one class of client you would have an unfair distribution, because
one client could consume many resources and others would be limited to
whatever's available on the margins.

I would like to see an adaptive system that, when resources are scarce, pushes
towards a more equitable distribution.

------
nickpsecurity
I know it expects cooperative behavior. However, I wonder if the protocol
design could be used in a setup where an embedded firewall did the rate
limiting and mediated traffic from the possibly-malicious host. Boeing already
has rate-limiting in their embedded firewall but cheap or OSS project could
use a OSS rate limiter prebuilt to save time.

As in, is this protocol inherently cooperative or could an implementation have
checks/controls added?

------
wslh
Offtopic but related: I want to like GRPC but I think Google is not working
hard to make it quickly usable. Just try to perform a pip install in Visual
Studio and obviously... it will not work. This is not the first time it
happens, try to compile V8 (NodeJS is easy) or Chrome in Windows and it will
be difficult even following the step by step instructions.

~~~
secure
I have the opposite experience: The Go version of gRPC works like a charm,
following the official instructions.

------
harryf
So do I understand this right: Doorman could be used to rate-limit access to a
website by integrating it with the sites proxy server, the result being
individual clients would only get a certain number of requests per minute
based on available server side resources?

~~~
tonfa
I think it's more meant for internal RPC within a cluster, to e.g. avoid
overloading a service.

~~~
stingraycharles
This is the correct answer, since this exact problem is a major issue when
dealing with distributed system failovers. Here are a few examples of AWS
service disruptions due to exactly this problem, where they were unable to
recover the system because nodes kept failing over under load:

[https://aws.amazon.com/message/5467D2/](https://aws.amazon.com/message/5467D2/)
[https://aws.amazon.com/message/2329B7/](https://aws.amazon.com/message/2329B7/)
[http://aws.amazon.com/message/65648/](http://aws.amazon.com/message/65648/)

An example quote:

"When this network connectivity issue occurred, a large number of EBS nodes in
a single EBS cluster lost connection to their replicas. When the incorrect
traffic shift was rolled back and network connectivity was restored, these
nodes rapidly began searching the EBS cluster for available server space where
they could re-mirror data. Once again, in a normally functioning cluster, this
occurs in milliseconds. In this case, because the issue affected such a large
number of volumes concurrently, the free capacity of the EBS cluster was
quickly exhausted, leaving many of the nodes “stuck” in a loop, continuously
searching the cluster for free space. This quickly led to a “re-mirroring
storm,” where a large number of volumes were effectively “stuck” while the
nodes searched the cluster for the storage space it needed for its new
replica. At this point, about 13% of the volumes in the affected Availability
Zone were in this “stuck” state."

So these things are very hard, can occur in totally unexpected situations, and
I'm not at all surprised that a company like Google comes out with something
like this.

------
dedalus
It should have been called "Velvet Rope" :-)

~~~
cm3
Care to you explain the joke for the culturally unaware?

~~~
CaveTech
When waiting to get into a club or party, you wait behind such a rope until
the bouncer allows you to enter (or doesn't).

