

The Internet hit 512K BGP routes today, causing widespread issues - mprovost
http://www.cidr-report.org/as2.0/#General_Status

======
mprovost
As I understand it we came close a few days ago but finally rolled over 512K
today at which point some routers hit their hardware limits and either stop
learning new routes or start forwarding in software.

There is a lot of chatter about it on Twitter and some network mailing lists:

[https://puck.nether.net/pipermail/outages/2014-August/007091...](https://puck.nether.net/pipermail/outages/2014-August/007091.html)

It looks like some providers like Level3 have been having a bad day.

------
yebyen
The linked page "CIDR Report" shows a rrdtool graph that seems to show we had
just as many routes 3 days ago. The page doesn't say anything about widespread
issues, and I don't find any source for that when googling.

So, where is your source about these widespread issues?

(Link title at the time of this post: The Internet hit 512K BGP routes today,
causing widespread issues)

~~~
mburns
NANOG[0] warned about it in May, and there is a blog post [1] that expands on
the risk. Cisco 6500/7600 (if they are set to hold the complete BGP table) are
known to be affected.

[0]
[http://mailman.nanog.org/pipermail/nanog/2014-May/066880.htm...](http://mailman.nanog.org/pipermail/nanog/2014-May/066880.html)

[1] [http://packetlife.net/blog/2014/may/6/global-ipv4-routing-
ta...](http://packetlife.net/blog/2014/may/6/global-ipv4-routing-table-
hits-500k-routes/)

~~~
cpncrunch
Neither of those sources confirms the 'widespread issues' today.

------
Zikes
Can someone provide a lay explanation?

~~~
namecast
Sure. Lots of old/older routers can't hold more than 512K IPv4 routes in their
"really fast routing table" memory (TCAM). This means that they'll have to
either drop some routes - meaning some places on the internet will not be
accessible from those routers - or that they'll have to use slower memory to
store those routes - which means routing in general will be slow across the
board.

Many, many ISPs, hosting and content providers use routers with this
limitation. So their customers are going to be impacted, in some
indeterminable way.

(Network geeks - I've glossed over a point or two, I know. Caveat lector.)

~~~
Zikes
Is this limitation specific to IPv4? Would those same routers be able to
handle a larger quantity of IPv6 addresses (if any), or does this highlight
the need to upgrade those routers?

~~~
yebyen
As a person of similar layness but some parts bit-wise...

I think the answer is no, plain and simple because IPv6 addresses are
literally four times as large.

If we were using IPv6 for years leading up to this, would it still be an
issue? Obviously not the same question, but IPv6 having much larger potential
address space and more opportunities for abbreviation in expressions of "what
is a netblock range" means for one thing that you don't always need as many
network blocks, you can have larger network blocks where a smaller one would
be used in IPv4 because of limits and efficiency with the overall available
and provisioned blocks...

Disclaimer: I have never been in charge of a large router, let alone a large
network.

~~~
tjgq
IPv6 addresses are larger, but routers only store prefixes, not full
addresses. And yes, one of the design goals of IPv6 is to reduce address
fragmentation, allowing the global routing table to contain fewer and shorter
prefixes.

That's only achievable in the long term, though - the currently assigned IPv4
prefixes will have to coexist with the IPv6 ones for a long time.

~~~
yebyen
I was trying to say something like this; I don't think it's true that you
don't ever store full addresses in a router (You don't always store the
address, sometimes just a prefix), but when you are describing a destination
for a block, you would need a full address to finger that gateway, wouldn't
you?

Unless the gateway is always at the front of the address block, which I'm sure
"they never taught us" in small network design school, at least I believe I've
seen routes where the gateway was inside the netblock range or at the end
rather than beginning, but again... big networks maybe.

What I was trying to say was that when you have the freedom to write bigger
ranges by an exponent factor of four like you have if you're the governing
bodies of the internet with IPv6 vs IPv4, the expressive part of your network
number can be made shorter, at least until your new network begins to approach
the size of your old network.

You don't avoid this problem at all, even if you are not storing 128 bits of
address there are probably never scenarios where two internets of the same
size last longer on the same hardware because IPv6. You're also right that you
don't really ever have a scenario in the real world where we are on IPv6 and
there is no need for IPv4, at least not for a very long time.

~~~
tjgq
> I was trying to say something like this; I don't think it's true that you
> don't ever store full addresses in a router (You don't always store the
> address, sometimes just a prefix), but when you are describing a destination
> for a block, you would need a full address to finger that gateway, wouldn't
> you?

Oh, I see your point now. You mean the IPv6 addresses assigned to the nodes
that the router forwards the packets through (i.e., the egress nodes), not the
destination addresses for those packets.

> Unless the gateway is always at the front of the address block, which I'm
> sure "they never taught us" in small network design school, at least I
> believe I've seen routes where the gateway was inside the netblock range or
> at the end rather than beginning, but again... big networks maybe.

Sure, the gateway address almost _never_ belongs to the address block you are
forwarding to. You could be forwarding to a very distant network which isn't
directly connected to, nor belongs to the same address block, as your gateway.

I can't tell for sure how routers implement this, but it's safe to say that
the total number of egress nodes for a router is much, much smaller than the
total number of routes (prefixes). So the obvious optimization is to store the
egress node addresses in a lookup table, and only store an index into that
table for each route. That way, the fact that you are now forwarding through
IPv6- instead of IPv4-addressed gateways does not increase the routing table
size (though it does increase the size of the much smaller lookup table).

> What I was trying to say was that when you have the freedom to write bigger
> ranges by an exponent factor of four like you have if you're the governing
> bodies of the internet with IPv6 vs IPv4, the expressive part of your
> network number can be made shorter, at least until your new network begins
> to approach the size of your old network.

I think you probably meant "at least until the used fraction of your new
address space begins to approach the used fraction of your old address space".

> You don't avoid this problem at all, even if you are not storing 128 bits of
> address there are probably never scenarios where two internets of the same
> size last longer on the same hardware because IPv6.

I'm not sure I understand the last sentence. The sense in which the IPv6
network will "last longer" in the same hardware is that, for the foreseeable
future, you will be able to satisfy most new allocations by choosing an
address block that is adjacent to an already existing allocation for the
requesting entity. Because the two blocks can then be aggregated into a single
prefix, the routing table stays the same size.

