
Linux QoS for humans - ktsaou
https://github.com/firehol/netdata/wiki/You-should-install-QoS-on-all-your-servers
======
Animats
When I used to work on network congestion, I took the position that each
stream is entitled to one packet in flight. After that, we can talk about
throttling. That's why I invented fair queuing.

TCP will throttle down nicely to one packet in flight, and the connection
won't stall, although it may be slow. Routers which discard all the packets
from a single stream are doing a very bad job, unless they're dealing with
hostile action.

Much of the trouble comes from network devices with big, dumb, FIFO buffers.
DSL routers are notorious for this on their uplink side. So are Ethernet
interfaces, which have a big FIFO buffer. It's important to avoid too much
dumb buffering in the chain between TCP and the Ethernet cable/WiFi air link.

A typical problem is ACKs from downloads being lost during heavy uplink usage.
If you're a choke point (much more bandwidth in than out), big dumb FIFO
buffers will suck. You at least need fair queuing. The "bufferbloat" crowd is
into this.

Linux QOS is more about fixing this as an endpoint. You're trying to avoid
overloading the next choke point upstream. This works only if everybody using
the next choke point upstream plays nice. One bad guy will ruin it for
everybody.

~~~
sargun
Nagle, much of the issues you're talking about with bad middleboxes is about
signaling or optics into their status. We've had a standard for some years to
deal with this (ECN). Why don't you think ECN is deployed at all? It seems to
me it would be beneficial for all parties.

~~~
Animats
An overloaded middle box can do two basic things - drop packets, or reorder
them. Many middle boxes do too much dropping and not enough reordering. Random
Early Drop was probably a mistake. It's easy to implement, and beats dropping
from either the tail or the head of a dumb queue. But that's its only virtue.

Any packet you drop eventually has to be resent. Packet dropping should be
rare. Again, each flow should be allowed at least one packet in flight. This
keeps the connection moving forward, and it means the endpoints can measure
round-trip time and bandwidth, which should be reasonably stable because the
algorithm is deterministic.

If you have so many flows that you can't handle one packet in flight per flow,
you're so badly overloaded that no congestion management strategy will work.

Fair queuing is about dropping the newest packet on the flow with the longest
queue. Thus, each flow competes with itself, and over-sending penalizes
itself.

We really need to get home router/modems, cable headends, and outgoing routers
just before the air link for mobile fixed. Those are the big choke points. You
can't fix this from the end points alone.

------
bcook
QoS _can_ help with DDoS if the DDoS is L7, but most DDoS attacks are
bandwidth exhaustion attacks, which requires upstream mitigation.

I completely agree that "tc" is one of the most confusing CLI programs I have
encountered. tc & iptables pushed me away from Linux (for router purposes) to
*BSD and pf has been a refreshing experience.

~~~
txutxu
Come on, it's not so hard. You mark the packages in the mangle table and you
distribute the defined bw with tc.

OK, I confess... all my life I've reused a tc script I inherited from a gentoo
sysadmin.

It works, I edit the available bandwitch and magic happens. The iptables cmd
is more easy to understand and I even did an output parser and input
generators, but tc is hard.

~~~
ktsaou
share the magic script. We might learn something.

~~~
txutxu
It's under NDA, but what if I tell you everything networking related are
numbers?

It just uses a packet mark convention. There is no magic.

------
mindslight
How does FireQOS/FireHOL handle dynamic behavior like interfaces coming/going?
My problem isn't so much the tc/ip/iptables syntax or semantics, but stateful
mismatch like 'ip route' disappearing when an interface goes down, while 'ip
rule' sticks around. Once a config gets that complex it's nice to keep in one
place rather than scattered through various files. The easiest hacks are to
let rules build up, or to reload everything on any ifup, but ugh. Also,
manually allocating numeric table/mark/prio/class numbers is annoying.

~~~
ktsaou
iptables (FireHOL) is generally autonomous and does not depend on interfaces
being available at activation. tc (FireQOS) requires the interfaces to be up
and running. To overcome these issues a) the configuration of both tools is a
BASH script itself - do whatever you like in them: variables, conditions,
loops, etc and b) in the distribution there is another tool called 'link-
balancer' that applies inheritance to routing tables (!) and dynamically
alters ip rule filters - it can also manage the balance of gateways.

~~~
ktsaou
And all these tools (FireHOL, FireQOS, link-balancer) are just BASH scripts
themselves (big ones of course).

FireHOL:
[https://github.com/firehol/firehol/blob/master/sbin/firehol....](https://github.com/firehol/firehol/blob/master/sbin/firehol.in)

FireQOS:
[https://github.com/firehol/firehol/blob/master/sbin/fireqos....](https://github.com/firehol/firehol/blob/master/sbin/fireqos.in)

Link-Balancer: [https://github.com/firehol/firehol/blob/master/sbin/link-
bal...](https://github.com/firehol/firehol/blob/master/sbin/link-balancer.in)

------
vesinisa
I am personally using tcng, which is a configuration language that already
makes tc much more usable.

tcng on $ourceForge:
[http://tcng.sourceforge.net/](http://tcng.sourceforge.net/)

Source code mirror by Debian (if wary of downloading from $ourceForge):
[https://packages.debian.org/source/wheezy/tcng](https://packages.debian.org/source/wheezy/tcng)

~~~
ktsaou
Interesting. It seems however that its latest release is 2004 !

~~~
vesinisa
No problem, it still works. It just internally calls tc. It's a language that
makes it easier to write tc classes and matching rules. The documentation is
here: [http://linux-ip.net/gl/tcng/](http://linux-ip.net/gl/tcng/)

The QoS userspace interface (tc) has not changed much at all, and at least my
configuration has no issues.

------
tra3
This looks intriguing, thank you! Over the last 10 years I've tried a number
of things to get QoS working on Linux, with not much success. My most
successful (and recent) attempt was to purchase a Mikrotik, but to be honest
I'm not totally satisfied. This seems like it might be a better solution!

~~~
pcunite
What trouble are you having with MikroTik? I enjoy them.

~~~
tra3
Mikrotik has been rock solid for me. It's my lack of knowledge really. I
adapted someone's QoS configuration and it's been working really well. Much
better than anything I've tried before.

My only complaint is the fact is the proprietary nature of Mikrotik. When I
was running a small linux box as a router, or when I was running DD-WRT I
could always extend the functionality in the ways that's not possible with
Mikrotik. The configuration, QoS, everything is Mikrotik specific. I
understand that it's reminiscent of enterprise grade hardware though.

------
mixmastamyk
How does the QoS interact with containers? Would probably need to configure at
the host level, correct?

Wow, that "netdata" daemon/dashboard this is associated with is awesome. I'm
wondering if it can be extended for more app metrics, like a custom solution
we had at a former employer using grafana.

~~~
ktsaou
Regarding QoS and containers, it can be applied at either side (veth is good
too).

Regarding netdata, of course it can be extended. Check this:
[https://github.com/firehol/netdata/wiki/External-
Plugins](https://github.com/firehol/netdata/wiki/External-Plugins) A lot of
other users are already writing their plugins - check the github issues

------
brbsix
That first paragraph is so right. `tc` is as close to deep magic as anything
I've come across in a Linux box.

------
MattSteelblade
Project looks extremely interesting. Will have to try it out.

~~~
ktsaou
thanks!

------
homarp
previously
[https://news.ycombinator.com/item?id=11477915](https://news.ycombinator.com/item?id=11477915)

~~~
ktsaou
You are right! The title (or the timing) did make a difference though.

~~~
dfc
I hope your take away is not that it is a good idea to embellish in the
titles. There is a reason editorializing in titles is frowned upon; it's a
hyperbolic arms race and everyone is a loser.

~~~
ktsaou
I am just trying to understand how this works. Probably is the timing. How do
you explain the difference in involvement between the 2 posts? I would love to
understand what made the difference. Can you explain it?

~~~
dfc
I cant explain it. What I am concerned about is that you are focusing on it.
Would you give your links linkbaity titles if it meant more traffic?

~~~
ktsaou
I am an open source developer. I don't sell anything. I am just trying to let
people know there is something they can use to make their lives easier.

I don't care about traffic (actually I don't want unrelated traffic - a lot of
noise - the key resource wasted is my time), but I do care a lot about
involvement.

Since a title can actually trigger involvement, I understand it is good,
especially for the readers. Isn't it?

To my understanding, it is good since people that should have been informed
about the context in question, have actually been found and informed.

Check this thread for example. A lot of people found something new, learned
something. Compare it to the previous one. No one learned anything.

So, what do you think? Shall I pay some attention on titles and their
effectiveness?

~~~
dfc
_In Submissions

Please don't do things to make titles stand out, like using uppercase or
exclamation points, or adding a parenthetical remark saying how great an
article is. It's implicit in submitting something that you think it's
important.

...

Otherwise please use the original title, unless it is misleading or linkbait._

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

~~~
ktsaou
I didn't do any of these.

Take also into account, I am the author of the original article too. So, would
it be better to copy the old article to a new one with the new title?

You are avoiding the question though: Since a title triggers involvement, is
experimentation on the titles something good or bad?

------
throw7
Is it overly pedantic for me to say QoS has nothing to do with inbound traffic
or is that an older, say, traditional definition?

My understanding is one can only "shape" outbound traffic. You are under no
control at all of the incoming packets.

~~~
brbsix
You can install an ingress queue and apply a filter to inbound traffic. There
are few other more arcane techniques but you're right that control is quite
limited.

Here's an example for something I was working on yesterday (traffic policing
to simulate slow download):

    
    
        sudo tc qdisc add dev eth0 handle ffff: ingress
        sudo tc filter add dev etch0 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 30kbit burst 3k drop flowid :1
    

Here's an example of using IFB to shape incoming traffic (what the OP was
referencing):
[http://serverfault.com/a/386791/301799](http://serverfault.com/a/386791/301799)

------
baghali
This is awesome. Would be great if it gets OpenWRT integration as well.

~~~
baghali
Oops, never mind:
[http://firehol.org/installing/openwrt/](http://firehol.org/installing/openwrt/)

------
geggam
Guess this is too old school ?

[http://lartc.org/wondershaper/](http://lartc.org/wondershaper/)

~~~
tra3
I'm curious -- Have you ever tried it? I could never get it to work.

~~~
eosrei
It works well for me. I run it on my local machine to make the network more
responsive during large downloads.

sudo wondershaper eth0 7500 850

~~~
wtallis
You'll get better results from

sysctl -w net.core.default_qdisc=fq_codel

on as many machines as possible, and use something like OpenWRT's SQM instead
of wondershaper on your router.

~~~
tra3
Slightly off topic -- is there a solid openwrt router you can recommend that
supports 802.11AC?

------
cbsmith
Congratulations on breaking the trend of "for humans" projects being written
in Python. ;-)

------
candl
I am not that familiar with iptables/tc, but could the tc definitions created
by fireqos be consumed by iptables + ipset (say I have an ipaet storing ip
addresses and I want to provide a different qos profile for each ip) ?

~~~
ktsaou
QoS and Firewall are different things in Linux. I have developed FireHOL
(iptables, ipsets) and FireQOS (qos) to handle everything. They cooperate to
share for example MARKs, but can also be used independently.

------
Scramblejams
Also featured recently on HN is this nice tool:
[https://github.com/thombashi/tcconfig](https://github.com/thombashi/tcconfig)

------
Klingner
I’d just like to add some other cool QoS-projects to the discussion so that
maybe more people can find a project that suits their own needs. I’ve always
been fond of QoS, mainly because it was a necessity with five family members
and just one measly DSL-connection, second because it’s just neat to use QoS
to prevent people from breaking your connection and making everything much
more effective.

1\. It seems like the first QoS-solution I ever used is still being actively
developed. It’s called Mastershaper and you can find it here:
[https://oss.netshadow.net/redmine/projects/mastershaper](https://oss.netshadow.net/redmine/projects/mastershaper)
Though it seems like the maintainer is doing some heavy work right now so
there is no usable version at the moment. In my old days Mastershaper was
programmed in PHP and it either worked with iptables as filter or with the
faster tc-filters. It had a nice GUI and also some flow analysis through a
graph. It was made to run under Debian and the like.

2\. The other project I would like to mention is a router firmware called
Tomato. The original website might be still available but the original author
has ceased development for several years now. It’s only being developed by its
community now. You can find the forum here:
[http://linksysinfo.org/index.php?forums/tomato-
firmware.33/](http://linksysinfo.org/index.php?forums/tomato-firmware.33/)
This firmware can be run on several open commercial home-routers (MIPSR1,
MIPSR2, ARM), one of them being the old and very popular WRT54GL. But of
course it also runs on more recent models. Tomato has several flavours.
Toastman’s Tomato is known for its well implemented QoS and stability. You can
find it here: [http://www.linksysinfo.org/index.php?threads/toastman-
mips-a...](http://www.linksysinfo.org/index.php?threads/toastman-mips-and-arm-
releases.36106/) Tomato probably has one of the most easily usable QoS-systems
there is. Once installed, all you need to do is measure your line speed
throughout the day, deduce a safety margin of about 15-30% to start with,
enter your upstream and downstream values into the GUI, hit “Save” and you are
running a pretty decently pre-configured QoS-system. It even supports overhead
calculations for ADSL. Unfortunately it doesn’t work with IPv6, yet, and also
uses the old netfilter L7-filters, which are pretty outdated. A switch to nDPI
hasn’t been undertaken, yet. If you think about using Tomato with QoS, keep in
mind that QoS needs a lot of CPU power, so it’s important to know your line
speed and buy accordingly.

3\. Another firmware that probably has well-made QoS is Gargoyle:
[https://www.gargoyle-router.com/](https://www.gargoyle-router.com/) I haven’t
been using it but maybe it’s worth a look.

