
IP Spoofing - majke
https://idea.popcount.org/2016-09-20-strange-loop---ip-spoofing/
======
mrb
I will never understand why some people disregard IP spoofing as a real risk.
For example when I reported a vulnerability to the nginx developers
([http://blog.zorinaq.com/nginx-resolver-
vulns/](http://blog.zorinaq.com/nginx-resolver-vulns/)) about their DNS stub
resolver using predictable transaction IDs, they refused to consider it a
vulnerability, effectively saying no one could exploit it because spoofing the
IP of the DNS server can't be done on the Internet. And yet
[https://spoofer.caida.org/summary.php](https://spoofer.caida.org/summary.php)
shows ~25% of network prefixes can be spoofed... _sigh_

~~~
hueving
It's a form of victim blaming. We have hoards of people chasing down any
developers that have written a fast response UDP protocol and any operators
running them. Yet the cause is incompetent ISPs that fail to implement BCP 38.

Even if the crusade to end useful UDP protocols is successful, all of that
short-sighted behavior is for nothing because untraceable DDoS attacks are
still a problem. Someone with a spoofing capable, 100,000 node botnet (or the
bitcoin required to rent on) with or without reflection capabilities is going
to be able to saturate 40gb links.

Name and shame ISPs that allow spoofing, stop wasting developers' time with
this crap.

~~~
majke
This is precisely the point of this article. The only real long term solution
to stop large DDoS is to stop spoofing.

The only way to stop IP spoofing is to shame misconfigured networks. For that
you need attribution.

~~~
silverwind
Recent HTTP DDoS attacks have shown that there is no dependency on spoofing.
The problem will be there as long as there's insecure hardware out there.

~~~
hueving
Once spoofing is solved, you have attribution. Once you have attribution, you
can use blacklists, notify ISPs, etc. Cleaning up or isolating infected
machines becomes tractable at that point.

~~~
dogma1138
Blacklists and attribution do not hold that much value.

Blacklists are only useful to stop only a few DDoS attacks mainly amplified
ones like the ones that abuse open DNS resolvers which should be cleaned up on
their own, in which case you want to be blacklisting the resolvers not the
machines that use IP spoofing since they do not attack you directly.

Many (heck most) DDoS attacks do not use IP spoofing, the recent and largest
recorded in history just used 145,000 IP enabled cameras, and this isn't even
remotely the largest botnet out there, blacklisting an entire botnet is
effectively DDoSing yourself since many of those machines can be on a network
with 100's or 1000's of other computers behind the same IP which many of them
would be legitimate users of your service.

Notifying ISP's also doesn't help, some ISP's can do some preventative
measures but what often they do is simply kill routes to your service, again
it would work if you are dealing with some ISP in Slovakia if you aren't
serving any customers there (granted they actually respond to you in due time)
but if you are dealing with large ISP's there is little they could do and
cutting them off isn't an option.

And as for as cleaning and isolating these machines this is laughable, you
aren't going to be tracking down customers, even the ISP won't do that for
you, and even if it did what would it do send and email that would likely be
ignored? Even if you identify customer the IP belongs to it doesn't tell you
anything about the device, not to mention that the customer would likely have
no capability of cleaning up their machines to begin with.

IP spoofing is a problem because there are currently ways to abuse IP spoofing
to amplify the attacks, if you aren't amplifying your attack using UDP
protocols you will usually not going to be using IP spoofing in the first
place, and automatically banning IP addresses might sound like a good tactic
until you are hit with DDoS from a big botnet within your market region at
which point you can't use that any more, you might ban a few IP's that seem to
have access to excessive bandwidth but even that might be problematic since it
can affect legitimate users.

~~~
hueving
>Blacklists and attribution do not hold that much value

Only because spoofing prevents them from being reliable in any way.

>in which case you want to be blacklisting the resolvers

This approach is idiotic and its the reason these attacks are still a problem.
EVERY DNS SERVER can be used if it resolves anything, not just if it's an open
resolver. So your blacklist would have to include the root name servers, the
.com names servers, etc.

>if you aren't amplifying your attack using UDP protocols you will usually not
going to be using IP spoofing in the first place

Completely false. SYN floods and non amplified UDP floods very frequently
spoof because it makes pruning attacks upstream by source impossible.

I'm not sure where you got this information, but stop spreading it. It's
protecting incompetent network operators, harassing service operators, and
doing little to improve the security of the Internet.

------
AndyMcConachie
A couple things:

1) There is no evidence that the recent giant DDOS attacks on Brian Krebs used
IP Spoofing. In fact, there is every reason to believe that they did not since
the generators of the packets were low powered IoT devices. There is
increasingly little reason for attackers to even bother with IP spoofing given
how easy it is becoming to capture giant herds of low power IoT devices. The
attackers don't care if some of their herd gets taken offline due to effective
attribution.

Amplification/reflection attacks which will still require IP spoofing. What
I'm curious about, and only time will tell, is how much IP spoofing will
continue to play a part in lsrge DDOS attacks? Why bother spoofing IPs if your
botnet herd is already large enough to bring someone offline?

2) Go and download CAIDA's Spoofer application. Test it and give them bug
reports. I gave them one a few weeks ago.
[https://www.caida.org/projects/spoofer/](https://www.caida.org/projects/spoofer/)

~~~
marcosdumay
You answered your question there. Without spoofing, only the largest botnets
can launch a successful DDoS attack. That's a big barrier to entry, and if the
police were effective on this area, would be a huge boom to fighting those
attacks.

~~~
mschuster91
> Without spoofing, only the largest botnets can launch a successful DDoS
> attack. That's a big barrier to entry

In the days of Shodan, NO. Absolute no. 100/10 or 50/5 MBit/s household
networking is becoming the norm in Germany, and other countries are way ahead
of us Germans. Add in the fact that people with lots of (crappy) IoT devices
are people who also have the money for high-speed internet connection...

Then throw in a couple of stro's at DCs with good interconnections, and you
got yourself a niiiiiiiice huge botnet. You can do LOTS of damage if you get
maybe ten or twenty servers with 10 GBit/s links under your control! (10
servers = 100 GBit/s, 10% of the Krebs attack IIRC)

~~~
user5994461
800/500 Mbits/s on speedtest here. (less if we pick a server on another
continent).

Fiber is coming to the household. It's gonna bring the potential for DDoS to
another order of magnitude. :D

Well. The IoT device will be limited by the bad Wifi or 100 Mbps Ethernet.
Hopefully.

------
IgorPartola
I have used IP spoofing for good in the past: I had a large number of sensors
reporting real time data to our servers. As we wanted to migrate to a
completely new infrastructure we wanted to have replication from the old
servers to the new. Instead of setting up some kind of higher level system, I
wrote a tiny service in C which received the datagrams and then re-sent them
to the new servers but spoofed the source IP so it matched the sensor. This
worked incredibly well, and the tool was later used for various other
purposes.

Of course toward the end of it I learned that I could have done this all with
iptables, but I like my way better because I got to learn a lot.

~~~
hueving
Ugh, if you're identifying sensors by IP instead of some other ID in the
payload, you're doing it wrong. NAT between the sensors and you can throw a
wrench into that whole scheme in a hurry.

This is just a smell emanating from another poor design, not a justification
for IP spoofing. If you want to spoof addresses, do it in the privacy of your
own VPN tunnel between the servers. Don't expose that sickness to the
Internet.

~~~
IgorPartola
You sir are quick to jump to very incorrect conclusions. It's a bad habit and
you should abandon it at once.

First, I never said we identified the sensors by IP, but it was important that
we record the sensor's IP for diagnostic information.

Second, IP spoofing was not done over the Internet, but rather only between
our own servers inside our data center. Obviously doing this over the internet
would not only be bad, but also highly unreliable.

In conclusion... well I'm trying to be polite.

~~~
hueving
If you have _anything_ , including diagnostic information, that depends on the
source IP being correct, you're doing it wrong.

~~~
colejohnson66
Port forwarding on some consumer grade routers work using IP addresses (not
MAC addresses). If I assign a computer a static 192.168/16 IP and forward a
port to it, I'm doing it wrong?

------
lossolo
There is no excuse for not securing your network to allow spoofing from it.
Most of the big players like leaseweb or ovh do not allow that. But there are
some providers that still allow you to spoof source ip address. There should
be consensus about droping routes on BGP level to networks that send packets
with source ips that they do not announce. It's really simple to drop packets
on switches/routers that do not originate from your network. It would make
ddosers life harder.

~~~
jacquesm
That would penalize a whole pile of parties that probably have nothing
whatsoever to do with the spoofers. It's akin to blackholing mail from
yahoo.com because there are spammers on yahoo.com.

~~~
x1798DE
Isn't it more akin to blackholing mail servers that don't set up DKIM, which
almost all major mail providers do these days?

~~~
jacquesm
No, if you drop the BGP route the whole range disappears, not just the host
that spoofs the IPs. So it kills off a whole pile of innocents as collateral
damage.

~~~
x1798DE
I don't know if I understand. At first I thought you meant that we shouldn't
drop entire providers because one of their customers spoofed an IP, but of
course the burden should be on those providers to make sure spoofed packets
can't leave their network (since further up the chain you're getting packets
that are originating from many networks). Are you talking about something
else?

~~~
niij
BGP route is on the receiving end.

------
zmanian
Netflow is a great example of the dual use aspects of tech between
surveillance and defense. Making Netflow data more widely available looks like
it is going to be essential for defending that Internet but at the same time
Netflow data can threaten the anonymity of Tor users.[0][1]

[0] [https://blog.torproject.org/blog/traffic-correlation-
using-n...](https://blog.torproject.org/blog/traffic-correlation-using-
netflows)

[1]
[https://gitweb.torproject.org/torspec.git/tree/proposals/251...](https://gitweb.torproject.org/torspec.git/tree/proposals/251-netflow-
padding.txt)

~~~
mordocai
Of course, I'm not sure if cloudflare could care any less about tor users.
They see a lot of attack traffic from tor so this probably isn't something
they are concerned about. I can't really blame them from a business
perspective, but I avoid cloudflare due to it.

~~~
dom0
> They see a lot of attack traffic from tor

This was debunked.

~~~
mordocai
I don't believe this is actually arguable. People use tor to attempt to
anonymize their (generally non-ddos) attacks.

I don't believe tor can support the type of ddos the OP is talking about, of
course.

If you could provide the source of this debunk it'd be appreciated.

~~~
ryanlol
This is ridiculous. Anyone can grep their access logs for signs of obvious
attacks and very quickly verify that very few, if any, of them originated from
Tor exits.

~~~
jerefa1
It amuses me greatly that probably no one in this thread realizes who you are.
(Except maybe Cloudflare.)

~~~
jgrahamc
I have no clue who ryanlol is. Would it make a difference if I did?

~~~
z92js71g
He's a subject matter expert: Julius Kivimaki.

~~~
jgrahamc
OK

------
nodesocket
I consider myself pretty technically savvy... Developer and DevOps.

However, when I see tricks like in this talk using iptables with BPF
bytecode[1] to block SYN packet floods, I get completely humbled. I know that
I know nothing.

[1] [https://idea.popcount.org/2016-09-20-strange-loop---ip-
spoof...](https://idea.popcount.org/2016-09-20-strange-loop---ip-
spoofing/strange-loop3-iwi-09-09.045.jpg)

------
biot
"Operating a content neutral service in today's internet is a tough job. Some
people dislike some websites, and they want to stop them from being available
on the internet. The easiest way to do so is to launch a DDoS attack."

... which you can conveniently buy on one of the CloudFlare-protected DDoS
service websites! I know this point has been hammered to death before, but I
find it curious that despite their strong advocacy of being content neutral
and not removing a site unless they receive a court order, their Terms of
Service explicitly allows them to shut off service based solely on their
opinion:

    
    
      "SECTION 11: INVESTIGATION
       CloudFlare reserves the right to investigate you, your
       business, and/or your owners, officers, directors,
       managers, and other principals, your sites, and the
       materials comprising the sites at any time. These
       investigations will be conducted solely for CloudFlare’s
       benefit, and not for your benefit or that of any third
       party. If the investigation reveals any information,
       act, or omission, which in CloudFlare’s sole opinion,
       constitutes a violation of any local, state, federal, or
       foreign law or regulation, this Agreement, or is
       otherwise deemed harm the Service, CloudFlare may
       immediately shut down your access to the Service."
    

Does CloudFlare ever make use of this provision?

~~~
astrodust
That's a standard clause you include in a contract to give you an out if
things get so bad you have no choice but to cut the client loose.

They're not offering a safe harbor, they're offering a non-discriminatory
service. Those are two different things.

------
superkuh
Am I the only one that believes that the kind of non-preferential routing and
anonymity that the current exchange setups provide is a benefit? In terms of
society as a whole this far outweighs the downsides of DoS attacks using IP
spoofing.

"Solving" the "problem" of ip spoofing is only a benefit for centralized
authorities and services. The loss of privacy is also serious. People
advancing this idea are advancing it to better their commercial interests
rather than the interests of individuals using the 'net.

~~~
xorcist
Is that an actual use case? What could you possibly do without return traffic?

Reverse Path Forwarding is best practice, and has been since more than a
decade. The normal reason packets appearing on the wrong interface is a
routing loop somewhere and this alone makes it worth it.

~~~
superkuh
I'm not saying IP spoofing is useful for many real tasks. I'm saying the
solution proposed in the article will end up being abused and worse than the
problems it solves. It's one more step to an internet where commercial traffic
is prioritized over actual users at all steps along the path.

------
MOARDONGZPLZ
Can someone explain the consequences of DDoS attacks to me? My understanding
is that the worst case is that the target server goes offline for the duration
of the attack.

If that's indeed the endgame, it seems like a lot of work on the attacker's
part to disable a company's servers for a bit, but maybe I"m missing
something? The article did mention servers boiling, but that was likely
hyperbole unless there's a way to physically damage servers with DDoS.

~~~
Z1nfandel
Imagine you are a news outlet. You make money by creating articles that entice
people to view your website. You make money to pay your expenses by having
advertisements on your website.

DDoS = No Website. No Website = no users viewing ads. No users viewing ads =
No income.

Yes, that typically resolves quickly. Depending on your hosting agreement, you
may have to pay for all the excess bandwidth that the DDoS bots used.

Now imagine if you are a bank, users want to be able to access their
accounts/funds/etc. How long would you remain with your bank if you frequently
couldn't access your information because they were being DDoS'd.

~~~
devicenull
And, depending on your transit agreements you may have to pay for all that
incoming bandwidth.

------
Thaxll
UDP spoofing is one thing but the latest and largest attacks are TCP based.

~~~
packetized
Flatly incorrect. The latest and largest attacks are generally UDP reflection
attacks, with a smattering of TCP SYN flood & pure L7 attacks thrown in.

~~~
dsp1234
_latest and largest attacks_

Except for the latest and largest attacks: OVH @ near 1Tb/s in combined
tcp_ack traffic on 2016/09/20 [0]

When attackers have 150k+ comprimised hosts, with 1-30Mb/s each, there is no
need for reflection/stealth/source spoofing/etc. At those levels, the
attackers can just point them to an IP with 'normal' traffic and it's enough
to be devastating.

[0] -
[https://twitter.com/olesovhcom/status/778830571677978624](https://twitter.com/olesovhcom/status/778830571677978624)

~~~
packetized
Apologies, I stand corrected.

------
tuxidomasx
Since IP spoofing could be done by a few malicious hosts, is it really
accurate to consider a DoS via ip spoofing 'distributed'? Sure the source IPs
may appear to come from all over, but that's just an illusion.

When I think DDos, I envision thousands of compromised hosts all over the
internet making requests to a target to consume resources.

~~~
majke
It's impossible to say if the attacker owns one beefy server or a thousand of
bots.

------
bikamonki
Am I correctly understanding that big/most attacks are against unwanted
content? For instance, in my country the two most popular political blogs that
make strong opposition to the government are regularly DDOSed. If that is the
case, maybe an easier solution would be to make this a _software problem_ and
not a network one. For that, I think we already have the tech to make P2P
websites feasible. My phone and data plan cannot serve hundreds of requests
per second but I'm sure I can get ALL the content I read on a given day from
peers as well as pass it along to others. Clearly not a solution for all
Internet use cases but a viable alternative to blogs.

------
lukasm
[https://www.youtube.com/watch?v=79u7bURE6Ss](https://www.youtube.com/watch?v=79u7bURE6Ss)
youtube video

------
nodesocket
This is a great talk. Marek does an awesome job of explaining something that
is very technical in a clear and casual manner. I definitely learned some new
things.

Video:
[https://www.youtube.com/watch?v=79u7bURE6Ss](https://www.youtube.com/watch?v=79u7bURE6Ss)

------
LyalinDotCom
I've been a software engineer for 18 years, but this amazing post reminded me
how little I know about how the internet works lol. Going to read this a few
times for sure!

------
scurvy
Large bandwidth attacks might look sexy, but they're trivially easy to block.
Network operators care about pushing packets, not bits. The OVH attacks look
huge to the average AWS user, (ZOMG a terabit!) but to even an average tier 2
transit provider it's a trivial attack to block. Especially when the attackers
are hitting a single endpoint.

To be honest, attackers are not very smart. They almost always use the same
old attacks, attacks that are easily stoppable, and attacks against a single
IP.

100gig coherent is easily turning volumetric attacks into old hat. Ending
spoofed attacks today is a tactic that's 10 years too late.

~~~
hueving
It's not trivial by any means other than preventing it from clogging up
internal stuff. If they are targeting a specific IP and the traffic is
indistinguishable from normal traffic in the l3/l4 headers, there is no way
you are going to block the attack without taking down the service.

This is why solving spoofing is such an important problem. Once source IPs
become sticky, we can actually block based on them to truly stop the attack
without taking down the service.

~~~
scurvy
You block volumetric attacks by filtering the payload, not by blocking IP's.
If you block by IP, you're going to have a lot of upset customers. Also, what
are you going to do on an IPv6 attack? Block the entire /64? Unlikely. Also
remember that a single IP doesn't always mean a single user for IPv4. An IP
might represent several thousand customers in the case of NAT and especially
CGNAT.

If the traffic is indistinguishable from normal traffic via L3/L4, then it's a
layer 7 attack, which cannot be spoofed. You're not dealing with spoofing in a
L7 attack.

The first D in DDoS stands for distributed. Even terabit sized attacks are
easy to stop if they're sufficiently distributed (the attack is geographically
dispersed enough to be diffused). It's when you get a terabit from a handful
of AS's (like in S. Korean attacks) that you can feel pain. Even then, it
usually only knocks out access in that part of the world, but not everyone.

IP's will not be sticky. IPv6 is showing why it's important to not be sticky.
Ending spoofing isn't going to stop DDoS attacks. I've got 10+ years of
experience on the subject, and I'll gladly answer any questions you might
have.

~~~
hueving
Yes, you block by IP and piss off customers or the whole /64 if necessary. IPs
will almost always be sticky to the advertised prefix.

A TCP SYN, or any UDP initial request indistinguishable from legit traffic of
the same protocol at L3/L4, and both can be spoofed so I don't know where you
got the idea that they cant be spoofed and are L7 only.

If you're "stopping the attack" through dispersion, you're not actually
stopping it, you're just paying the price of absorbing it. I know this is
common practice, but it's pathetic than an arms race is the state of the art
in defense. It concentrates power into orgs like cloudflare, akamai, and
Google (project shield) because the only way to participate is to use them.

------
garaetjjte
Ironically, Mobile IPv6 uses spoofing for route optimization.

------
indolering
If only we had a method for authenticating packets : /

------
equalunique
The slide showing the Internet Exchange's switch says those are Ethernet
cables, but they're actually fiber.

I'm really impressed with this presentation, but I wish that one flaw could be
fixed.

------
grogenaut
Warning: Sales pitch masquerading as a technical talk.

~~~
astrodust
Warning: Someone's cranky.

It's a technical talk. It's much better to hear from people who are being
attacked regularly and have a track record of dealing with it rather than
those who have no clue.

~~~
grogenaut
sure except their paper just describes the issue and their solution is "pay us
money". Not really up to the same standard as even a bad academic article,
more like the paid talk track at a conference, you know, the kind lots of
people avoid.

~~~
astrodust
They laid out several solutions, and none of them are specific to their
platform. They also highlighted how the attacks are changing and if you're to
effectively defend yourself you'll need to have more resources on hand.

You can pay them, you can do it yourself, you can pay someone else, but at
least you have options.

------
davedx
Fantastic article.

------
happytrails
Time to spoof!

------
complaint
That was an excellent presentation, very informative.

Except, why put that ridiculous meme in the middle of it? It's cringeworthy
seeing an excellent technical presentation littered with such childish
imagery.

(Not that I agree with this bastardization of "meme" to mean "silly image with
text overlaid in capital letters", but unfortunately that is what everyone is
calling these things.)

~~~
majke
To wake up the audience and emphasize the message :) Also, to divide the
presentation sharply between: problem statement and the solution.

