
It’s Time for Some Queueing Theory - sogen
https://kottke.org/19/01/its-time-for-some-queueing-theory
======
jinpan
Another fun surprising result from queuing theory is that if you have a buffer
of pending RPC's that have some deadline, it is better to service them in a
"unfair" LIFO order, as opposed to a fairer FIFO order. [0] goes into more
gory details.

In the bank example from the article, LIFO would dramatically shorten the
median wait time (~5 minutes), at the cost of really upsetting customers [1].
In the case of a support queue, where the waiters are blind to each other,
this cost could be reconsidered ...

[0] [https://arxiv.org/pdf/1008.4895.pdf](https://arxiv.org/pdf/1008.4895.pdf)

[1] from the article, "We really, really hate it when someone shows up after
us but gets served before us.

~~~
ulucs
After a while people would grow wise of this tactic and start redialing after
waiting for some time. We could build a whole lot of efficient and beautiful
systems, if only not for the people...

~~~
SilasX
Exactly. I've never had the time to dig into it, but these LIFO-optimality
results always strike me as being non-physical or introducing some implicit
hard-to-model effects.[1]

Like you say, people can just re-enter the queue, or (in meatspace) form a
meta-queue that vies for entering the moment it has an open slot, reproducing
FIFO all over again.

Furthermore, you get a _lot_ of pro-cooperation effects (necessary for queues
to work at all) by giving people "skin in the game" in the form of valuing
their place in line. Once people lose nothing by inventing a new name and re-
entering the line, that's all gone, and they no longer have an incentive not
to be disruptive, which throws off disproportionate negative utility onto the
rest of the system.

One day, I promise, I will unpack this result.

[1] My comment from 2015 expressing similar reservations:
[https://news.ycombinator.com/item?id=10182781](https://news.ycombinator.com/item?id=10182781)

~~~
papln
What's wrong with degrading from LIFO to a random queue due to client retry,
when you're _starting_ from a FIFO that is _already_ failing client
expectations?

~~~
SilasX
Nothing, if it's just used for that (narrow, least-bad, degraded) situation. I
was addressing the more general point that LIFO queues have some actionable
superiority over FIFO in the general case because of the theoretical results
cited in the literature.

------
davidw
I'd definitely appreciate some 'next steps for the working programmer' that
aren't a deep dive into math. What we need to be able to do is realize what
sort of problem we're facing, how to describe it, and maybe what solutions
might look like or where to learn more about them.

At a certain company, we were dealing with a limited number of phone numbers
that could be used in on-line ads to get people to call for more information.
Reuse the numbers too quickly and you wouldn't be able to track what people
were calling about. Having too many idle numbers cost money.

Seemed like an interesting problem to me and it looked like "queuing theory"
even if I didn't know the details. I suggested we could probably figure out a
way to optimize our usage of the numbers. Blank looks and "let's just use 10
numbers per client and see how that goes".

I'm definitely someone who's wary of coding up a big, complex solution as the
first iteration, so I could live with "let's start with a number", but I think
that needed to feed into a further iteration using the knowledge we gained and
a bit more thinking about the problem.

~~~
kragen
Maybe the next steps for the working programmer are, in fact, a deep dive into
math.

~~~
davidw
Adam Smith's words about specialization apply to programming too.

One of the best places I ever worked, I sat right next to a guy doing computer
vision algorithms. He had a PhD, and was extremely smart and knowledgeable and
fun to talk with.

He also wasn't the most... practical programmer, and I was able to give him a
variety of suggestions about how to improve his code, and of course I had
ideas about data modelling and web programming that he didn't know much about.

Neither one of us knew much about the details of the firmware, or the optics
(mirrors and lenses and such) that another colleague was working on.

I guess the point is that for some of us, the point is that we're probably not
good candidates to become domain experts, but knowing enough to find help is
probably a good move.

~~~
kragen
Not learning math because it's not what you want to specialize in is self-
defeating. It's like not learning to read and write, or save money, or
communicate with other people, because it's not what you want to specialize
in.

Math isn't a domain. It's the technique of abstraction we apply to _any_
domain to make it tractable for computers, and it's the technique of reasoning
we apply to any domain to figure out where our thinking has gone wrong. At
least, any objective domain — except in rare cases, math isn't that useful for
figuring out why your wife feels you don't love her anymore. But for you to
program something on a computer, someone needs to mathematize it first. This
can be done well or badly.

------
jph
Great article! I'm adding it to the queueing theory intro repo:
[https://github.com/joelparkerhenderson/queueing_theory](https://github.com/joelparkerhenderson/queueing_theory)

------
z3t4
You should not stare blindly on the average. It's in the 1% slowest requests
the dragons are hidden.

~~~
vitus
Not when you're looking at high utilization, as presented in the example in
the article.

For that particular case (a M/M/1 queue with arrival rate of 5.8 customers per
hour and a service rate of 6 per hour), even your median response time is 3.5
hours.

Unbounded FIFOs are _really_ bad once you get to high utilization.

------
takk309
If any one is interested in a diving into queuing theory as it pertains to
vehicular traffic, check out chapters 11 and 12 of Traffic Flow Fundamentals
by Adolf May. [1] Chapter 11 discusses shockwaves and moving queues. Chapter
12 discusses both deterministic and stochastic queuing.

[1] [https://www.scribd.com/document/253416450/Traffic-Flow-
Funda...](https://www.scribd.com/document/253416450/Traffic-Flow-Fundamentals)

------
numlock86
From the article:

> We assume customer arrivals and customer service times are random (details
> later).

Where are those details? I expected some math about the random distribution
and how that adds up, but I can't even find the corresponding text passage to
explain anything related to that.

~~~
moandcompany
They probably intend to discuss the Poisson distribution which is commonly
used to characterize random customer inter-arrival times, and similar or other
distributions for how long it takes to process the customer's request.

[https://www.wikiwand.com/en/Poisson_distribution](https://www.wikiwand.com/en/Poisson_distribution)

Queueing theory is a pretty common topic in Electrical Engineering and
Computer Science curriculum related to scheduling or computer networks. Also
part of Operations Research curriculum.

Probably worth reading more about Leonard Kleinrock's work if you are
interested in this topic and the context where it was applied:

[https://www.wikiwand.com/en/Leonard_Kleinrock](https://www.wikiwand.com/en/Leonard_Kleinrock)

~~~
Animats
Reading that, years ago, was what led me to figure out how to approach
congestion control. Everybody was reading Kleinrock and assuming random
arrival rates. Klienrock had done a study of Western Union Plan 55-A, which
was Sendmail for telegrams, built with paper tape for buffering and telephone
switches for routing. That led to the original analyses of how the Arpanet
should work.

The trouble with that approach is that it assumes packets just arrive randomly
no matter what the network is doing. That was true for Plan 55-A; delays in
the network had little influence on the submission rate for new telegrams. But
it's not true once you put a protocol with backpressure, like TCP, on top of
the raw packets. Now congestion becomes a feedback control system. I figured
that out in the early 1980s and wrote RFC 970.[1]

The idea of having one big line was introduced by banks some time in the
1970s. It helped some. But the big breakthrough came from Walter Wriston [2],
CEO of what is now Citibank, who pushed his people to develop in-bank
terminals and then ATMs, to get rid of the lines entirely.

A classic piece of advice from retail consultants is "never place an obstacle
in front of a customer who is ready to buy". Amazon's "one-click ordering" is
the classic on-line example. Gap stores used to be noted for getting this.
They had big, empty counters and more than enough people ready to check
customers out. All that crap retailers put at checkouts? Few customers buy it,
and it limits how much merchandise the customer can put on the counter.

The big queuing theory insight for line management is that if you don't have
some idle time, in an open loop situation the line length will go to infinity.
In the real world, you start to lose customers. That's how closed loop
feedback reacts to retailer incompetence.

[1] [https://tools.ietf.org/html/rfc970](https://tools.ietf.org/html/rfc970)

[2]
[https://en.wikipedia.org/wiki/Walter_Wriston](https://en.wikipedia.org/wiki/Walter_Wriston)

~~~
iamnotacrook
"All that crap retailers put at checkouts? Few customers buy it, and it limits
how much merchandise the customer can put on the counter."

People select less stuff when they're shopping because they're concerned it
won't fit on the counter, or they take a bunch of stuff there but end up not
paying for it because it won't fit, or what?

~~~
setr
I think the issue is that it slows down processing time, reducing overall
throughput, for little to no gain

And the bigger the line, the more likely customers drop out (at least, I do,
especially if the purchase is small/insignificant)

------
tomsthumb
There’s a great Kubecon talk on Queueing Theory if you want a bit more:

[https://youtube.com/watch?v=yf6wSsOFqdI](https://youtube.com/watch?v=yf6wSsOFqdI)

------
tombert
About a year ago, I was waiting in line at Dollar Tree (guilty pleasure), and
I had to wait almost a half-hour in line. Even though there were two cashiers,
they were constantly communicating back and forth, introducing a constant
blocking factor and effectively reducing it one queue, and I was trying to
think about how to model this with pi-calculus while I was waiting.

Obviously there's a math for basically anything, but reading this makes me
glad I'm not the only one who has thought about this. This article has given
me some reading fodder for this weekend...thanks!

------
wodenokoto
Not really any theory, but some interesting anecdotes and research results.

I’d wished the author at least explained the 5 hour average wait. I mean, if
the queue started at 30 people, then I can see it, but not if we start at 0.

~~~
isaacg
In that setting, the line queue is 30 people long, on average. Sometimes more,
sometimes less. Because 5.8 people arrive an hour out of the maximum of 6 that
can be served an hour, and 6/(6-5.8)=30, the average queue length is 30.

------
oftenwrong
Relevant to queueing theory:

"Stop Rate Limiting! Capacity Management Done Right" by Jon Moore

[https://youtube.com/watch?v=m64SWl9bfvk](https://youtube.com/watch?v=m64SWl9bfvk)

------
hinkley
My favorite bit from queuing theory is the observation that if you execute the
shortest tasks first, the average wait time for any one task is minimized.

~~~
lapinot
Are you sure? Aren't you talking about the average wait time for all tasks
combined (sum of wait times) instead of the wait time for _any one_ task.
Obviously the optimal wait time any one task is 0, if you start right away.
Intuitively i'd say if you sort the queue by size and always process the
shortest task, you may indefinitely stall some long tasks.

~~~
hinkley
I think that depends on whether your queue is a fixed data set or a stream of
new entries.

I learned queuing theory in a class about multimedia, and I came away with a
good theoretical understanding but my practical understanding was rubbish.
When synchronizing two streams of tasks, the queue is not finite, but the
pattern is fixed. For A/V it might be 22 frames per second, best effort, and
_exactly_ one second of audio per second.

But any queue with even a vague notion of deadlines will only let so many
tasks 'cut in line'. At some point the undesirable task gets scheduled, even
in the face of a constant stream of other work. Or it gets dropped. From what
I understand, in a realtime system a low priority, long task might just be
aborted. Because it's never a good time to run it.

------
c3534l
I'm always annoyed that whenever someone purports to have an article or book
on queue theory, no one actually has any math. So I'm just left with a bunch
of anectdotes and no understanding.

------
pcvarmint
Myron Hlynka

[https://web2.uwindsor.ca/math/hlynka/queue.html](https://web2.uwindsor.ca/math/hlynka/queue.html)

------
lcuff
Does it get more light-weight than this? Grrr. It's time for fluff.

~~~
kyllo
I had never heard of queueing theory until I read about it in a supply chain
management textbook, and would have appreciated being at least aware of its
existence much earlier than that. This is light, but it's a great way to
introduce the topic to people who have never been exposed to it.

By the way, I also worked in a supply chain management department of about two
thousand people for three years and I never heard anyone else mention queueing
theory while I was there, so now I really wish at least this blog post were
required reading for everyone in that department.

------
winrid
Thought I'd hear about some AMPQ... :)

~~~
nurettin
What is there to hear about AMQP? Make exchanges, bind queues and send to
routing keys. Add cluster and load balance for bonus points.

Now fast-food queues, that is an interesting problem.

