
SACK Panic – Multiple TCP-based remote denial-of-service issues - cdingo
https://access.redhat.com/security/vulnerabilities/tcpsack
======
LinuxBender
This is the way I block such things on my own VM's (not at work) using
iptables:

    
    
        iptables -t raw -I PREROUTING -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m tcpmss ! --mss 640:65535 -j DROP
    

Here it is in action:

    
    
        iptables -L -n -v -t raw | grep mss  
        84719 3392K DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 tcpmss match !640:65535
    
    

My settings may be a little aggressive and may block some old pptp/ppoe users.
Perhaps 520 would be a safer low end. As a funny side note, this also blocks
hping3's default settings (ping floods) as it doesn't set mss. This also
blocks a slew of really poorly coded scanners.

For everything else at work, we are behind a layer 7 load balancer that is not
vulnerable.

You may also find it useful to block fragmented packets. I've done this for
years and never had an issue:

    
    
        iptables -t raw -I PREROUTING -i eth0 -f -j DROP
    

If you have the PoC, then feel free to first verify you can browse to
[https://tinyvpn.org/](https://tinyvpn.org/) then send the small MSS packets
to that domain, then see if you can still browse to it. I don't care if the
server reboots or crashes. Just don't send DDoS please, as the provider will
complain to me.

To see the counters increase, here is a quick and silly cron job that will
show you the MSS DROPs in the last minute, that I will disable after a couple
days: [1]

[1] - [https://tinyvpn.org/up/mss/](https://tinyvpn.org/up/mss/)

~~~
chupasaurus
FYI Debian Security Team recommends setting new sysctl value
_net.ipv4.tcp_min_snd_mss_ to 536, even though they (Debian) are preserving
the default kernel value of 48 for compatibility.

~~~
londons_explore
It's worth noting that lwIP, the network stack used by most microcontroller
based IoT devices, has a very low MSS. It's configurable, but typically
defaults to 512.

~~~
CloudNetworking
That's very interesting. I have only seen those defaults in some... "special"
satellite links.

------
talawahtech
AWS Bulletin: [https://aws.amazon.com/security/security-
bulletins/AWS-2019-...](https://aws.amazon.com/security/security-
bulletins/AWS-2019-005/)

FYI if your instances are behind an Application Load Balancer or Classic Load
Balancer then they are protected, but _NOT_ if they are behind a Network Load
Balancer.

A patched kernel is available for Amazon Linux 1 and 2, so you won't have to
disable SACK. You can run "sudo yum update kernel" to get it, but of course
you have to reboot. Updated AMIs are also available.

Amazon Linux 1:
[https://alas.aws.amazon.com/ALAS-2019-1222.html](https://alas.aws.amazon.com/ALAS-2019-1222.html)
Amazon Linux 2:
[https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html](https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html)

For Amazon Linux 2 the fixed kernel is kernel-4.14.123-111.109.amzn2. Looking
at my instances, it look like I have been on that version since Friday.

~~~
ones_and_zeros
Even if your instances are behind ALBs or ELBs they may not be protected if
they make outbound connections to the internet.

~~~
pferde
Is this so? Can this kernel panic also be triggered in TCP connections
initiated by the victim? I can't find a conclusive mention of this anywhere.

As each direction of a TCP connection has its own MSS, it would make sense
that an attacker's server could exploit this.

~~~
dsp
This is so. Both passive and active connections are at risk.

------
trollied
Looks like most of the internet should be getting patched ASAP! Or not, as is
usually the case :-(

I remember the original ping of death back in the 90s
[https://web.archive.org/web/19981206105844/http://www.sophis...](https://web.archive.org/web/19981206105844/http://www.sophist.demon.co.uk/ping/)

------
usrname
Some solution: echo 0 > /proc/sys/net/ipv4/tcp_sack

[https://github.com/Netflix/security-
bulletins/blob/master/ad...](https://github.com/Netflix/security-
bulletins/blob/master/advisories/third-party/2019-001.md)

~~~
href
Thanks for pointing this out. Applying this buys us time before we can
properly patch all our systems. In our case this was easy to roll out in a
jiffy.

I do wonder though, can anyone guess what kind of impact one might see with
TCP SACK disabled? We don't have large amounts of traffic and serve mostly
websites. Maybe mobile phone traffic might be a bit worse off if the
connection is bad?

~~~
wademealing
Disclaimer: I worked on initial Red Hat article linked above.

In my personal AWS instance from the last few days less than half a percent of
the traffic had hit the firewall rule to log the error.

Most of that traffic seemed to come from the China, this was possibly port
probing / portscans or really old hardware accessing my the server.

I would say that the iptables rule is a 'better' solution than dropping sack
as you may find you use significantly more CPU/bandwidth when dealing with
retransmits when not using selective acknowledgements.

~~~
Twirrim
I was involved with this one for another cloud provider.

I have a personal Digital Ocean (not my employer) instance that is frequently
being probed for stuff (primarily Russian and Chinese IPs). Same old, same
old.

I've been running with the rule for around a week just logging & dropping
small MSS packets out of curiosity, but hardly seen anything worth writing
home about. I was somewhat surprised. I'm curious to see how long it takes for
that rule to go nuts (my shellshock rule still triggers from time to time,
that had a definite curve of action)

~~~
londons_explore
Small MSS is often IoT devices which only have a kilobyte or so of RAM, so
often have an MSS of below 256 bytes. They won't be rendering a webpage, but
are totally capable of doing REST API requests.

More and more are moving away from $0.25 microcontrollers, and up to $5 SoC's
running Linux, so the problem is going away gradually...

------
hoffie
Red Hat's article on these issues also provides further explanations:
[https://access.redhat.com/security/vulnerabilities/tcpsack](https://access.redhat.com/security/vulnerabilities/tcpsack)

~~~
dang
That seems to have a bit more information, so we switched to it from
[https://www.openwall.com/lists/oss-
security/2019/06/17/5](https://www.openwall.com/lists/oss-
security/2019/06/17/5). Thanks!

~~~
loeg
[https://github.com/Netflix/security-
bulletins/blob/master/ad...](https://github.com/Netflix/security-
bulletins/blob/master/advisories/third-party/2019-001.md) is the advisory by
the party that discovered the issue. (Disclosure: I have met Jonathan Looney
and know some of the Netflix engineering staff, but I don't work for Netflix.)

~~~
caf
This mentions FreeBSD impacts as well which the RedHat link doesn't.

~~~
trasz
For FreeBSD only the RACK stack seems to be affected - that's an alternative
TCP/IP stack, not the default one.

------
geggam
Is this the same thing ?

[https://www.cvedetails.com/cve/CVE-2005-0960/](https://www.cvedetails.com/cve/CVE-2005-0960/)

Multiple vulnerabilities in the SACK functionality in (1) tcp_input.c and (2)
tcp_usrreq.c OpenBSD 3.5 and 3.6 allow remote attackers to cause a denial of
service (memory exhaustion or system crash).

~~~
wademealing
Similar concept, different operating system.

------
jkern
So you can remotely cause a kernel panic on basically any system with kernel
newer than 2.6.28? The doesn't sound great

~~~
syn0byte
No, not "any system". Besides needing SACK enabled (which is by default) you
also need segment offloading and non-shite networking hardware that will
respect and preserve stupid MSS fields in packets.

pending a patch simply disable SACK: ~$ echo 0 > /proc/sys/net/ipv4/tcp_sack

and/or disable segmentation offloading: ~$ ethtool -K eth? tso off

TCP and Checksum offloading still aren't super standard on customer grade NICs
or virtual machines. I'd assume less than half of the internet's linux hosts
are actually at risk.

~~~
oasisbob
Isn't TSO enabled on EC2? Their bulletin implies it at least, I seem to
remember the same.

~~~
loeg
Enabled on Azure Ubuntu, FWIW:

    
    
      $ ethtool -k eth0 | grep tcp-seg
      tcp-segmentation-offload: on

------
bhauer
Sorry, this is probably a bit simplistic, but I am curious: How likely is this
to affect embedded devices? E.g., hardware firewalls, routers, IoT devices
that all use a Linux kernel?

~~~
LinuxBender
If you are not exposing any tcp ports or reaching out directly from those
devices to a malicious host, then very unlikely. Either way, it's best to
check the vendors site or open a ticket with them, if that is an option.

An alternate option would be to put the device behind another firewall or load
balancer or proxy that you know is not vulnerable.

~~~
xyzzy_plugh
I guess a question I have is then: can I hose you during a TLS handshake? If I
can forge DNS, then I can DoS, right? Which makes BGP a prime target right
now?

~~~
LinuxBender
Well, anything that gets a vulnerable device to talk to a malicious device
would be an issue. Probably best to check with each vendor and see what their
story is if you can't filter the traffic between your devices and potentially
malicious devices.

~~~
xyzzy_plugh
I'm just thinking how an attacker could disrupt e.g. IoT devices. Pretty
terrific.

------
jandrese
That BUG_ON is a full up kernel panic? That's a pretty severe issue if so.

~~~
Twirrim
Yes. In all seriousness, this is a "drop everything and patch" situation, as
soon as the patches are available.

It's a little bit more involved than a ping of death, but still, relatively
easy to exploit.

~~~
jandrese
Seems like anybody with an open TCP port and SACK enabled (the default) is
vulnerable.

~~~
wademealing
and Generic Segment offloading.

------
nitinics
Here's the series of patches - looks like applied few hours ago.

[https://lore.kernel.org/netdev/20190617.104121.1475407136257...](https://lore.kernel.org/netdev/20190617.104121.1475407136257245934.davem@davemloft.net/T/#t)

------
cmurf
These are patched in linux 5.1.11, 4.19.52, 4.14.127, 4.9.182, 4.4.182.

[https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.1.1...](https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.1.11)

5.0 is EOL as of 5.0.21.

------
rphlx
When was the last time Linux had a similar, reliably-remotely-exploitable
kernel panic in the TCP/IPv4 stack? Pre-2000?

~~~
foofc7c8
Never, from what I can recall.

~~~
rphlx
Though not as bad as Win9x it definitely had some frag-of-death/ping-of-death
vulns around 1997/98\. teardrop et al.

~~~
acdha
Here's a golden oldie from 1996:

[https://packetstormsecurity.com/files/15507/CA-96.26.ping.ht...](https://packetstormsecurity.com/files/15507/CA-96.26.ping.html)

My favorite of that era was simply the working-as-designed simplicity of
sneaking the Hayes modem hangup sequence into various protocols: actual Hayes
modems used +++ with a time-delay to send commands such as ATH0 (hangup) but
everyone else skipped that time-delay in an attempt to avoid the patent so you
could disconnect any modem-connected system if you could figure out how to get
it to echo "+++ATH0". Some IP stacks (e.g. Windows 95) would simply send the
received ICMP payload as the response so a simple `ping -p …` would do it but
people found ways to cause similar problems with sendmail, FTP, etc.

[https://dl.packetstormsecurity.net/new-exploits/modem-
DoS.tx...](https://dl.packetstormsecurity.net/new-exploits/modem-DoS.txt)

~~~
jlgaddis
IRC was a fun venue for that one ~25 years ago.

Pop into some random channel, send "/ctcp #channel ping +++ATH0", and wait
patiently... a moment or two later you would be rewarded with a flood of
"signoff" messages as the users' TCP sessions to the IRC server timed out (by
responding to the CTCP, they had, in effect, told their modems to hang up).

The goal, of course, was to get the highest "body count" possible from a
single CTCP message.

Smurf attacks, the "ping of death", AOHell, the latest sendmail and wu-ftpd
holes of the week, open proxies... the Internet was a very entertaining place
for a bored teenager from the midwest back then.

Thanks for the flashback!

~~~
acdha
I was having a similar moment of nostalgia. Hard as it may be to believe some
weeks, we have gotten a lot better at securing things.

------
mkj
Will any Android phones be affected, or they don't support segment offloading?

~~~
Droobfest
Android seems to be affected:
[https://android.googlesource.com/kernel/common/+/5d625e9b4a6...](https://android.googlesource.com/kernel/common/+/5d625e9b4a6c7a43cc071ae0528f3b29299e0b81)

This might be fun...

Edit: Don't know if segmentation offloading is on by default in Android, but
on my default Arch kernel it is, so I wouldn't know why not.

~~~
strcat
It is impacted, but patches being present in kernel/common doesn't imply that
since all of the upstream changes from the LTS branches are merged.

------
fortran77
Will there be a big performance hit if I just turn sack off (for now) until I
finish running all our tests on the new kernel?

~~~
CloudNetworking
Depends on your traffic, but in general terms you might want to instead drop
packets with low MSS.

------
ahoka
Maybe this helps someone:

    
    
      [Unit]
      Description=Disable TCP SACK
    
      [Service]
      Type=simple
      ExecStart=/sbin/iptables -A INPUT -p tcp -m tcpmss --mss 1:500 -j DROP
    
      [Install]
      WantedBy=sysinit.target

------
keyle
What does OpenBSD/NetBSD do differently to not be affected by this?

~~~
rphlx
They use a different TCP/IP stack which implemented SACK without introducing
this bug.

It's a Linux-specific implementation defect, not an intrinsic problem with the
TCP SACK wire protocol or spec.

~~~
jontro
They found a bug in FreeBSD too: [https://github.com/Netflix/security-
bulletins/blob/master/ad...](https://github.com/Netflix/security-
bulletins/blob/master/advisories/third-
party/2019-001.md#2-cve-2019-11478-sack-slowness-linux--415-or-excess-
resource-usage-all-linux-versions)

~~~
loeg
FWIW the FreeBSD bug only affects the RACK tcp algorithm, which isn’t the
default (I think).

~~~
bifrost
Correct, its not enabled by default.

------
sl-1
Is there any way to quickly test if any of my machines are vulnerable?

~~~
loeg

      [ "$(uname -s)" = Linux ] && echo "Vulnerable"
    

("CVE-2019-11479: Excess Resource Consumption Due to Low MSS Values (all Linux
versions)", "CVE-2019-11478: SACK Slowness (Linux < 4.15) or Excess Resource
Usage (all Linux versions).")

[https://github.com/Netflix/security-
bulletins/blob/master/ad...](https://github.com/Netflix/security-
bulletins/blob/master/advisories/third-party/2019-001.md)

~~~
superkuh
I know you're being glib, but are 2.6.x kernels vulnerable? The big corps tend
define all linux as all linux that they support and isn't end of life. As far
as I read early 3.x kernels on the Ubuntu side _are not_ effected. Like
version 12 and before.

So there's plenty of linux being used out there that's probably not effected.

~~~
tim--
Yes, 2.6.x kernels are vulnerable, starting with 2.6.18-8.el5 Edit: on Red Hat
EL5

------
davoti
Folks,

imo, TSO is intel NIC card function, does this affect others like from Cavium
CPU?

thanks

------
shereadsthenews
Looks like the issue was fixed upstream a month ago. Might have been nice to
know earlier? Is this how long it takes for the distros to lurch into action?

~~~
megous
The fix was pushed just now to stable kernels.

~~~
shereadsthenews
That's just a detail of how linux is developed. The fixing patch was mailed on
May 17th and it mentions the CVE.

~~~
jauer
That's more or less the time it takes to test the patch and for cloud
providers to apply the patch so there isn't large-scale mayhem.

------
bloopernova
Ouch.

Good luck out there, folks.

------
xeno56
how does one apply this patch?

------
gravitas
I'm collecting vendor links internally for work:

 _Red Hat / CentOS_

[https://access.redhat.com/security/vulnerabilities/tcpsack](https://access.redhat.com/security/vulnerabilities/tcpsack)

[https://access.redhat.com/security/cve/cve-2019-11477](https://access.redhat.com/security/cve/cve-2019-11477)

[https://access.redhat.com/security/cve/cve-2019-11478](https://access.redhat.com/security/cve/cve-2019-11478)

[https://access.redhat.com/security/cve/cve-2019-11479](https://access.redhat.com/security/cve/cve-2019-11479)

 _Ubuntu_

[https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SACKPanic](https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SACKPanic)

[https://people.canonical.com/~ubuntu-
security/cve/2019/CVE-2...](https://people.canonical.com/~ubuntu-
security/cve/2019/CVE-2019-11477.html)

[https://people.canonical.com/~ubuntu-
security/cve/2019/CVE-2...](https://people.canonical.com/~ubuntu-
security/cve/2019/CVE-2019-11478.html)

[https://people.canonical.com/~ubuntu-
security/cve/2019/CVE-2...](https://people.canonical.com/~ubuntu-
security/cve/2019/CVE-2019-11479.html)

 _Oracle Linux_

[https://linux.oracle.com/errata/ELSA-2019-4686.html](https://linux.oracle.com/errata/ELSA-2019-4686.html)
(RHCK kernel)

[https://linux.oracle.com/errata/ELSA-2019-4685.html](https://linux.oracle.com/errata/ELSA-2019-4685.html)
(UEK5 kernel)

[https://linux.oracle.com/errata/ELSA-2019-4684.html](https://linux.oracle.com/errata/ELSA-2019-4684.html)
(UEK4 kernel)

 _Amazon AWS_

[https://aws.amazon.com/security/security-
bulletins/AWS-2019-...](https://aws.amazon.com/security/security-
bulletins/AWS-2019-005/)

[https://alas.aws.amazon.com/ALAS-2019-1222.html](https://alas.aws.amazon.com/ALAS-2019-1222.html)
(Linux 1)

[https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html](https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html)
(Linux 2)

 _Debian_

[https://security-tracker.debian.org/tracker/CVE-2019-11477](https://security-
tracker.debian.org/tracker/CVE-2019-11477)

[https://security-tracker.debian.org/tracker/CVE-2019-11478](https://security-
tracker.debian.org/tracker/CVE-2019-11478)

[https://security-tracker.debian.org/tracker/CVE-2019-11479](https://security-
tracker.debian.org/tracker/CVE-2019-11479)

 _SUSE / SLES_

[https://www.suse.com/de-
de/support/kb/doc/?id=7023928](https://www.suse.com/de-
de/support/kb/doc/?id=7023928)

[https://www.suse.com/security/cve/CVE-2019-11477/](https://www.suse.com/security/cve/CVE-2019-11477/)

[https://www.suse.com/security/cve/CVE-2019-11478/](https://www.suse.com/security/cve/CVE-2019-11478/)

[https://www.suse.com/security/cve/CVE-2019-11479/](https://www.suse.com/security/cve/CVE-2019-11479/)

 _CoreOS_

[https://coreos.com/releases/#2079.6.0](https://coreos.com/releases/#2079.6.0)

 _Arch_

[https://security.archlinux.org/AVG-983](https://security.archlinux.org/AVG-983)

[https://security.archlinux.org/CVE-2019-11477](https://security.archlinux.org/CVE-2019-11477)

[https://security.archlinux.org/CVE-2019-11478](https://security.archlinux.org/CVE-2019-11478)

[https://security.archlinux.org/CVE-2019-11479](https://security.archlinux.org/CVE-2019-11479)

(please reply with additional vendor links if you have them)

~~~
whyleyc
Thanks for this. Any chance you can you edit the comment to make the links
clicky please?

~~~
gravitas
I am finding that HN has limited formatting options - if I don't indent by two
spaces (aka code mode) it's a run-on mess, which then requires a large amount
of ugly line breaks making the post 5x as large and unreadable. HN formatting
instructions are all of 3 sentences long.
[https://news.ycombinator.com/formatdoc](https://news.ycombinator.com/formatdoc)

~~~
whyleyc
I agree that formatting could definitely do with some work!

If you put each link on a line and then leave a newline between each one you
should get a nice list though.

~~~
floatingatoll
Nope, a single newline would just be translated to a space character without a
paragraph break. There's no way to <br> in HN.

------
SEJeff
I guess this post beat mine?
[https://news.ycombinator.com/item?id=20205859](https://news.ycombinator.com/item?id=20205859)

~~~
floatingatoll
Nope, it's just the random dupe that ended up getting the upvotes for whatever
reasons. This happens constantly and the one that gets the upvotes has more to
do with chance than any other factor.

~~~
cookiecaper
s/chance/timing/. Social media scoring can be fickle indeed, but there are
_entire industries_ devoted to optimizing and reverse-engineering the hotness
algos of various traffic-drivers. This case is probably coincidental, but it's
naive to ascribe high scoring merely to luck. It looks SEJeff posted around
12pm PDT, maybe on his way out to lunch. This post was two hours later, as
everyone got back from lunch. :)

