

Something is amiss with the Interwebs: BGP is a flapping - btilly
https://isc.sans.edu/forums/diary/Something+is+amiss+with+the+Interwebs+BGP+is+a+flapping+/18523

======
rwg
The default partitioning of CAM space on Cisco gear is the obvious issue, but
the root cause is the massive deaggregation of announced IPv4 routes on the
Internet. You can see various statistics about this problem at
[http://www.cidr-report.org/](http://www.cidr-report.org/), but the short of
it is that if the top 30 networks (based on announced route savings)
completely aggregated their announcements as much as possible, ~41,000 routes
(~8% of the routing table) would be eliminated.

And that's just the top 30 networks — if every network cleaned up their
announcements, it would eliminate ~232,000 routes (~45% of the table).

Adding to the deaggregation problem is the inability to easily filter out
route announcements based on RIR minimum allocations without having to add
tons of exceptions for CDNs that operate as islands of connectivity and carve
out IP space for each island from a single address space allocation. (There's
no covering route for the islands of connectivity since these CDNs have no
"backbone" connecting the islands, so if you filter out those smaller
announcements, you lose connectivity to those islands.)

There are many people who think this problem will just magically go away as
IPv6 adoption increases, but all increased IPv6 adoption will do is make
limited CAM space even more limited as network engineers have to balance
dividing precious CAM space between a ballooning-quickly IPv4 route table and
a ballooning-slightly-less-quickly IPv6 route table.

(To be clear: I think ubiquitous, functioning, end-to-end native IPv6
connectivity needs to happen sooner than later, but it's not a magic bullet
for the Internet's technical problems.)

~~~
sp332
This is one of the reasons IPv6 addresses are being given out in such massive
allotments. ICANN doesn't want anyone to have to get multiple "chunks" and
blow up the routing table later.

~~~
MichaelGG
So people multi-homing (which is the only reason I end up getting and
announcing /24s) is not a significant amount of the route table? Cause IPv6
can't really fix that part (multihoming) can it?

~~~
cnvogel
But it solves the problem of you, after some growth, having received several
distinct v4/24s, instead of one huge v6/64...

~~~
waps
You're assuming that just getting a massive opportunity for more de-
aggregation won't cause de-aggregation out of shear laziness and convenience.
It's much easier to de-aggregate than it is to aggregate.

The maximum IPv4 de-aggregation possible is 2^24 - 2 ^ 21. The maximum IPv6
de-aggregation with what's currently being handed out (mostly between /32 and
/48's) is way, way, way more.

So IPv6 has more addresses, yes. Just so long as you don't actually use them.
The problem of course is that the memory requirements for IPv4 assignments
were going up linearly. If you bought gear with a good amount of memory, you
could therefore expect it to last a few years. Clearly the network vendors
that designed IPv6 saw this as a problem ... how can we make it explode ? Well
IPv6 was the answer. Problem is that they went completely batshit insane
overboard.

"If" IPv6 deaggregates we'll need routers with about 2^(48-24) TIMES more
memory. Storing a deaggregated IPv6 routing table (which has to be in memory)
requires 524288 terabytes of memory.

The rest of IPv6 isn't much better. It's some academics wish-list, with total
disregard for real-world concerns. It is not possible to use 1/10th of the
IPv6 features. Multicast ? Won't work (on any public or even just somewhat
large network). Encryption ? Won't work (too many devices don't support it).
Anycast ? Won't work. Site-local anycast ? Won't work. Larger address space ?
Won't work (in half the world). NAT-avoidance ? Won't work (didn't get past
security engineers). Autoconfiguration ? Actually kinda handy in some
scenarios, but again, won't work on most networks, where you fall back to the
IPv4 mechanism. Faster routing due to "smarter" headers ? Doesn't work
according to my load stats. Better Qos ? Doesn't work in either of the major
routing gear vendors. Mobility ? Let's not go there. Instead of implementing
sane "live" re-addressing they went with ... Aargh. Let's not go there.
Automatic network renumbering ... riiiight. Heh. I wonder if this was put in
as a joke.

There are known solutions to all these problems (well, except the multicast
and anycast ones), but of course, the IPv6 designers knew better.

Welcome to the "solution". On the other hand, solving the solution pays rather
well.

------
namecast
I dropped this into another HN thread, so I'll just put it here:

[http://www.cisco.com/c/en/us/support/docs/switches/catalyst-...](http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-
switches/116132-problem-catalyst6500-00.html)

Takeaways: a) 512K routes isn't necessarily a hardware limitation, it's the
default TCAM allocation for IPv4 and B) most people most of the time don't
need their routers to take a full BGP feeds worth of routes - and I hope those
that do aren't running 6500's in Q3 2014 ;)

~~~
oasisbob
The backside of the reload-required TCAM reallocation is really brutal, as
there are apparently a huge number of faulty cards out there in the wild which
won't survive the reload:

[http://www.cisco.com/web/about/doing_business/memory.html](http://www.cisco.com/web/about/doing_business/memory.html)

Reminds me of when I worked at a large public university... the networking
group planned on losing/replacing dozens of switches and power supplies during
the summer when the power plant underwent maintenance. Losing power and
reloading surfaces all sorts of defective components. And, when you have
thousands of switches...

~~~
namecast
Ouch. In that case, woe be to anyone who tries to get Cisco to replace their
grey-market way-beyond-deprecated almost-definitely-not-covered-by-SmartNet
Sup720 linecards in 2014. Even if TAC went for that, I imagine the turnaround
time would take forever...

Btw, thanks for the link! I suspect more than a few 6500 linecards that I've
seen die in years past were due to something similar, as opposed to what we'd
previously assumed to be the root cause - either damage in transit or "cosmic
rays" ;)

~~~
s_q_b
It's generally a two-week turnaround for uncovered equipment, if they have
remaining stock, which they always do. Given that SMARTnet-covered equipment
turnaround can be 2-hour, 4-hour, or NBD and Cisco's massive global network of
warehouses, it appears this constraint is artificially imposed to keep
SMARTnet attach rates high.

It's very odd that "grey market" equipment is considered the standard way to
refer to genuine Cisco equipment when sold by a company other than a Cisco
partner. I understand keeping out counterfeits, but given the first-sale
doctrine, how is a resold piece of equipment anything other than completely
legitimate?

If you think about it, and get past the "it's just industry standard"
mentality, it's generally insane the way that Cisco uses these pseudo-monopoly
tactics. In the old days, say with maintenance on IBM Selectric typewriters,
such schemes were called "bundling" and "tying," and the DOJ would pursue the
companies for anti-trust violations.

Now, the DOJ arrests people based upon nebulous complaints from Cisco's
general counsel. See e.g. [http://abovethelaw.com/2011/07/sue-a-giant-
corporation-get-r...](http://abovethelaw.com/2011/07/sue-a-giant-corporation-
get-rewarded-with-audacious-criminal-charges/#more-84911) wherein a British
citizen was arrested in Canada for starting a company that competed with Cisco
maintenance.

The Canadian court quashed the request for extradition after the DOJ's request
trapped him in a foreign country for years. He remains under indictment here,
despite the Canadian judge stating that the DOJ's case was a fairly
transparent copy of Cisco's civil suit. The ruling was incendiary, stating
that _The extradition process to bring the applicant before United States
Courts… involved innuendo, half truths and complete falsehoods._

The judge concluded:

 _The only reasonable inference I can draw from the facts is that the criminal
process was used to pressure (unsuccessfully) the applicant into abandoning
his antitrust suit against Cisco…. Any well-informed person acquainted with
the truth would conclude that the collective result of the mistreatment of Mr.
Adekeye offended fundamental notions of justice._

~~~
MichaelGG
Doesn't this just make them want to go sidestep the issue and say you're not
licensed to run the software, even if you can resell the hardware? Didn't
Autodesk win on that, with respect to licensing software?

~~~
s_q_b
The "first-sale doctrine" is a legal concept that applies to combination
hardware and software products. Essentially it holds that the copyright of any
combination of a piece of intellectual property (e.g. Cisco IOS, or the
information in a textbook) and a physical product (e.g. a router or the actual
copy of a textbook book) is exhausted after the first sale. [0]

Now, there are court-recognized exceptions to this rule in various circuit
courts (most significantly by the 9th Circuit in _Vernor v. Autodesk_ , SCOTUS
cert denied) for digital goods under the so-called "shrink-warp licensing"
exception. [1]

So given that the highest court has allowed the ruling to stand in a large
circuit, yet has consistently expanded the first-sale doctrine for four
decades [2], the precedent is unclear.

Cisco does rely on _Vernor_ and its progeny as it's justification for
blacklisting resold items. But that argument is unlikely to hold water outside
of the software-laden 9th Circuit.

The question, stated in layman's terms, is, "To which is a Cisco router or
switch more similar, a copy of Windows or an iPod?" This is an open question
from a legal standpoint, and thus we must unfortunately resort to that most
unreliable of legal tools: reason.

Both options are almost equally unpalatable to the Supreme Court for policy
reasons. Expand the shrink-wrap exception to cover Cisco and you risk
encompassing all products that contain firmware, from TVs to microwaves, and
thus strangling the half-trillion dollar secondary sale market for electronic
goods. Keep the exception narrow and you choke off the single largest source
of revenue (all told, including gains from new purchases, maintenance
contracts, and paid software updates close to $20 Billion, or 40% of annual
revenue) to the government's largest IT hardware provider (over 85% of DoD in
particular), leaving the government stranded with vast networks of legacy
Cisco hardware with little to no new development from the company.

The existing legal precedent creates a tough needle for the Court to thread.
As a result, we have the current state of limbo, and thus discussion of "grey
markets," which are neither clearly legal nor clearly illegal.

[0] [https://en.wikipedia.org/wiki/First-
sale_doctrine](https://en.wikipedia.org/wiki/First-sale_doctrine) [1]
[https://en.wikipedia.org/wiki/Vernor_v._Autodesk,_Inc](https://en.wikipedia.org/wiki/Vernor_v._Autodesk,_Inc).
[2] Most recently in Kirtsaeng v. John Wiley & Sons, Inc., No. 11-697 (U.S.
Mar. 19, 2013)

~~~
MichaelGG
Thank you for the explanation.

>leaving the government stranded with vast networks of legacy Cisco hardware
with little to no new development from the company.

Do we have reason to believe this is true? If resale was allowed, do we
believe network hardware companies would just shrivel up? How do we square
this with the facts of many platforms being intentionally crippled for
marketing reasons? Or does that not happen on top-end platforms?

~~~
s_q_b
>If resale was allowed, do we believe network hardware companies would just
shrivel up?

Network hardware companies in general wouldn't suffer, but Cisco's increasing
reliance upon support, service, and firmware for revenue means that
unrestricted resale would bite into Cisco's revenue _hard_ , perhaps 25-50%.

Right now Cisco hardware powers the most critical government information
systems, including military and financial communications (over 85% of the gear
is Cisco.) Since many of these networks use Cisco proprietary functions, have
scores of personnel trained on Cisco gear, and relationships with networks of
Cisco suppliers, transition to another hardware manufacturer would be both
expensive and difficult, at a time when the government has little funding for
new initiatives.

Since the government needs Cisco to be a smoothly functioning company to meet
critical information needs, it makes sense that they maintain tight relations.

------
oasisbob
As discussed on NANOG from a few months ago:

[http://markmail.org/message/n32fmeb2dmtnbsff](http://markmail.org/message/n32fmeb2dmtnbsff)

I find the economics of the routing table to be fascinating. When someone
announces a route, it makes use of a constrained (and often expensive, TCAM-
based) resource on routers all over the world. More discussion:

[http://markmail.org/message/6sunzqtffav5jmfb](http://markmail.org/message/6sunzqtffav5jmfb)

------
ztnewman
512k is surely enough...
[http://www.nux.ro/archive/2014/08/512k_routes_ought_to_be_en...](http://www.nux.ro/archive/2014/08/512k_routes_ought_to_be_enough_for_everyone.html)

------
disbelief
Slate article on using BGP hijacking to redirect mined bitcoins from an hour
ago. Relevant?
[http://www.slate.com/articles/technology/future_tense/2014/0...](http://www.slate.com/articles/technology/future_tense/2014/08/bgp_hijacking_cybercriminals_used_internet_architecture_to_mine_bitcoins.html?wpisrc=burger_bar)

edit: I'll take it by the downvotes without responses that's a "no"?

~~~
ephemeralgomi
The downvotes are probably because you didn't explain why you think it's
relevant, and therefore didn't actually contribute to the discussion.

~~~
tempodox
How DOES downvoting work? All I see are upward triangles, if that has anything
to do with it at all. How much more hidden functionality is there for
“privileged members”? And how do you join the club?

~~~
bodski
Downvotes require a karma threshold (of 500 IIRC).

BTW, whatever you do don't overuse the 'flag' feature on stories as it seems
when this is revoked it is permanent! I lost mine when trying to encourage
news other than the passing of Steve Jobs.

------
phkahler
Wouldn't it be nice to allocate a small part of the IPv6 space to
geographically encoded IP addresses? In other words, the address itself can be
used to physically locate the destination and a route could be chosen
partially according to geography. It seems like this should be less arbitrary
than the routes you need to chose now. Or is it?

~~~
agwa
IPv6 addressing does sort of work that way: each RIR (regional Internet
registry) has just a single prefix allocated out of the global unicast address
space, instead of a whole bunch of prefixes all over the place as with IPv4.
And the IPv6 address space is large enough that ISPs can be given a single
huge prefix that they can grow into. Consequentially, the global IPv6 routing
table should be a lot smaller than the IPv4 routing table is today.

~~~
akira2501
Not quite. The current IPv6 global unicast table is a bit of a poorly planned
mess.

[http://www.iana.org/assignments/ipv6-unicast-address-
assignm...](http://www.iana.org/assignments/ipv6-unicast-address-
assignments/ipv6-unicast-address-assignments.xhtml)

~~~
p1mrx
There were some smaller chunks in the experimental days, but note that in
2006, each RIR was given a /12, and since then there have been zero
allocations at the top level.

8 years of stability is hardly a "poorly planned mess".

~~~
akira2501
In 2006 they also gave ARIN a /23 and APNIC a /20\. Then, one month later they
expanded some previous RIR allocations into /12's; separate from the previous
two allocations I mentioned. They are also allocated in a seemingly random
order throughout 2000::/3, which they've decided to constrain their
allocations to for some reason.

To the point that the linux documentation project still recommends the
following as an ipv6 default route example: /sbin/ip -6 route add 2000::/3 via
2001:0db8:0:f101::1

Yea.. it's /workable/, but a lot of decisions with respect to IPv6 feel like
wasted opportunities that have only slowed adoption and promoted general
confusion.

------
fivre
Neat, I was up late last night and noticed several sites stop working at
almost exactly the time of the spike on Cyrmu's graph that went away when I
proxied through my Linode in the Dallas datacenter (normally I just go
straight through Comcast), though I didn't really think much of it at the
time.

------
NKCSS
Dutch business ISP/Colo/Hoster Bedrijvenweb also experienced outages:
[http://noc.bedrijvenweb.nl/dashboard/244/op-dit-moment-
ervar...](http://noc.bedrijvenweb.nl/dashboard/244/op-dit-moment-ervaren-wij-
packetloss)

------
tuna
[https://secure.ciscodude.net/blog/2014/08/12/major-
internet-...](https://secure.ciscodude.net/blog/2014/08/12/major-internet-
issues-today/)

------
serverascode
I'm surprised this isn't getting more press. Shaw in Alberta has been having
problems all day, not sure if it is because of this but I would hazard a guess
that it is.

------
antihero
Could this be anything to do with why my ping has been spiking to 20 seconds
today? Or is it just Sky being useless?

------
DinooD
Fun times today at work.

~~~
graup
Yup. It was 5pm in South Korea when this hit. Suddenly much less quiet than
usual on the Engineering floor.

------
induscreep
Will SDN fix such issues?

------
wnevets
ruined my morning at work, ugh

------
nextweek2
I love this. There is too much conservative behaviour at the expense of
innovation. Things break, let's be prepared, let's upgrade, let's not worry
about 100% up time and start actually building something bigger.

~~~
davis_m
By building something bigger, are you referring to larger BGP routing tables?

------
exabrial
Standard response: Quick, deploy IPV6 and fix the problem!

OH WAIT, WE DIDNT FKIN THINK OF THIS IN OUR RUSH TO PUSH A BROKEN INCOMPLETE
SOLUTION.

