A few years ago a team of researchers tried to use a feature attribute flag in BGP. Their experiment caused an important BGP software to crash.
The BGP community reacted by thanking the researchers for uncovering that flaw and worked on improving stability. No, just kidding. They shouted at the researchers and asked them to stop, which they eventually did.
I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers.
> I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers.
I suspect a Network Operators mailing list is going to be full of people trying to keep production up, with the tools they've got, rather than the tools they'd like. Thinking the software quality on the routers they're using are dogshit is probably just called "Monday" and thinking their peers' routers are even worse is called "Tuesday".
I can understand they don't appreciate "research" done on production systems that might cause issues when they already know their equipment sucks.
I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks.
Bitching at researchers (note: not "researchers") who are finding and calling out those flaws before someone really bad comes along and exploits them is somewhere between unhelpful and negligent. Internet security is always improved by more sunlight.
I practised picking locks on your house and broke the lock because of a poor lock design. Yeah okay the lock sucks. But I probably shouldn't have been doing it on your house in the first place and now you've got a broken lock. Who are you mad at?
A researcher could have bought their own copy of the lock and picked it there. Or even borrowed yours with some notice, or let you know so you could have them use an interior door instead, or any number of other things that didn't break your actual house.
That is not really a matching comparison. With BGP, there is kind of public space (global BGP tables) and participants are free to attach valid attributes to their routes. From a cursory look, these researchers used announcement that was compliant with BGP standards (in contrast to Ben Cartwright-Cox from this article, who used fuzzed invalid attributes), so they did something that was generally acceptable behavior for BGP participants.
"I can get into this lock" will not have the same effect or call to action as "I can get into your doors" but it admittedly has to be aimed at the landlords, not the doormen.
> I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks
We call out vendors when their quality suffers but you only have two or three practical vendor options for high end routers depending on your use case/feature requirements. All three have varying cycles of hardware and software quality, with hardware and software rarely being in phase for the same vendor.
The fact that all three vendors make most of their money on non-network focused enterprises with less sophisticated engineers doesn't help get development attention where it needs to be for global infrastructure.
>I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks.
I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. When "fixing" a router means you break half the internet, you just don't fix it. This isn't some walled garden where you can just say "tough shit if you broke, fix it".
Most of the "issues" these researchers find are well known, but backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.
> I think you fail to realize that the internet is a bunch of disparate networks that no one person controls.
The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.
> backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.
Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. This is a security lesson that has been learnt in recent years by many sectors of the IT industry and, judging by the woeful response that Ben Cartwright-Cox describes in his post, it's one that the network backbone is going to be learning soon.
The internet does not owe a negligent operator peering. No one person controls the internet but you sure as hell can find yourself kicked out if your goal is to be right on principals instead of a good peer. The only hubris which could be at play is someone thinking the rest of the internet should follow their desires instead of the other way around.
Research is great but if you're poking around breaking things via DoS in the internet you're going to be judged as a bad peer regardless what the route daemon does. The security of the internet constantly moves forward but that doesn't mean the expectation of improving has to hold more precedence than the expectation of running. It's still very much a human to human system for all but the actual route exchange.
>The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed.
So you've never worked in a corporate environment. Here's how that conversation would go:
Hey guys, some researchers found out if you use a test flag meant for the lab on the public internet, it breaks all our BGP sessions.
OK, so drop their feed and block them?
Great, done.
>Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.
And yet the "vulnerability" in question was known, and was not being immediately exploited if you read through the mailing list or were participating in NANOG at that point in time. So your statement is provably false.
>Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP.
And yet we're having this conversation in 2023, they have operated the same way for 40+ years, and somehow the internet is still working. Bad actors get blackholed, it worked in the past, it'll continue working in the future. The reality is that backbone routing is expensive, and expecting everyone to update their kit on YOUR timeline isn't reasonable.
I am aware that network operators have behaved this way for very many years. The point I am making is that all of the IT industry used to attempt to "work around" security vulnerabilities in the same way, until log4shell and all the others gradually beat that propensity out of them.
I am prophesying that a similar reckoning is likely to come upon the network backbone. You're arguing that the cost of entry to the game (BGP peering) is such that the old ways will continue to work. Let's hope you're right.
Software ecosystems like libraries and frameworks have completely different propagation and remediation mechanics compared to federated systems like core Internet backbone routers and switches is the thing. Try as we might conceptualize otherwise the modern Internet from a packet’s purview is more like a loose confederation of ultimately privatized or state-run fiefdoms than a cellular automata digraph explosion. So actors that try to act maliciously against the network will be basically shut out given the rule of an iron fist being the default.
Conceptually, there is no special difference between a remotely exploitable vulnerability in an HTTP server and a remotely exploitable vulnerability in a BGP router. It would have been theoretically possible to have dealt with log4shell by blocking every IP address that was sending malicious payloads, for instance.
Practically, I accept that there is a real difference. ASNs are much harder to acquire than IP addresses, and there are far fewer of them. That difference might mean that blocking continues to be an effective mechanism. However, in a world where malicious actors are increasingly just nation states wearing a flimsy mask, I'm not sure that the difficulty of getting access to sufficient numbers of ASNs is going to be a forever mitigation.
The main thing I have taken away from this thread is that when it comes to vulnerability management, network operators seem to be pretty far behind the curve in mindset terms. The arguments being made in here ("you can't break other people's stuff", "we have a lot of boxes and SLAs mean we can't patch quickly") aren't even slightly new. The well-proven reality is that threat actors don't give one single shit about any of that.
Backbone operator who was effected by this. We had a large number of routers in production with this bug, we were aware and upgrading as fast as we could, but with 99.999% uptime SLA's we only have so many minutes per router we could afford for downtime/outages. We had schedules in place (approx 3 months of out-of-hours upgrades) 1 week warning was a bullshit move. Dropping the BGP sessions on 1000's of routers globally was stupidity.
Bearing in mind we also couldn't just "apt-get upgrade" in place, most boxes required hard reboots to apply the patches.
The answer to your question is no, as per a few others have said, Don't violate Rule #1. We see bad actors very often, our job is to keep the bits flowing and the internet online.
Keeping the internet online is painful enough as it is without "researchers" dropping thousands of routers to "prove a point."
I suppose you have never seen, ever, in your professional career, the "C" suits making business deals and thus decisions on what's to be acquired, with big network/server/storage/database/software vendors, over the head and against the advice of those then called to support crap in production.
They've only had over a decade to get their act together. I'm being generous, seeing IPv6 is older than the original IPv4 internet was when it went mainstream.
Except there's no "staging Internet" to test such equipment, and even if there was it would still sometimes not translate to the real open network. I'm sorry, were they inconvinienced with the world at large?
If you read the links: they discovered that using this flag broke EVERY installation of FRR on the internet. They let the developers know. And then said they were going to test again in a week.
Does any sane person think that 1 week is enough time to both notify every user of FRR in the WORLD, as well as ensure they have upgraded their installations for a "bug" that only is triggered by someone using experimental flags meant for testing purposes only?
Mind you, these researchers didn't test ANYTHING in a lab environment first "because it's a lot of work to test all the open source BGP implementations in the wild". AKA: I'm lazy.
> Mind you, these researchers didn't test ANYTHING in a lab environment first "because it's a lot of work to test all the open source BGP implementations in the wild". AKA: I'm lazy.
That's a gross exaggeration: they said that they first tried it "in a controlled environment with different versions of Cisco, BIRD, and Quagga routers" [0] without issue. Presumably, the problem we know about would have been avoided if they had happened to include FRR in their test suite beforehand. But that raises the next question, exactly how many implementations does one need to test before one is no longer "lazy"?
2) You are usually the architect of your own destruction.
3) It’s easier to see other people’s problems than your own.
Stir into a death spiral and let it rest for six to twelve months. Bake at 375° for two years, then serve cold with a sprinkling of schadenfreude to offset the bitterness.
Is there a reason why the researchers did this on live production BGP network rather than in a test environment, running their own BGP and FRR routers? Seems a bit haphazardly, as doing so first seems to have uncovered the issue.
Overall I agree with your message though, and of course things shouldn't stop working just because you use attributes reserved for development.
The researchers claimed that they had first tested it with a few different vendors' routers without any issues, but they had not included FRR in those tests [0]. There was a bit of discussion on the mailing list regarding just how many BGP implementations (and configurations) a responsible researcher is obligated to test with before exposing something new on the public Internet.
The correct answer is "All of them on your own network first".
Thou shalt not blow up another's network knowingly.
Blowing up another's network because you didn't test your theory in your own isolated lab space first counts as blowing up another's network knowingly. Because what the hell else did you think would happen?
I don't think that is an accurate representation of the "comunity", it was a "singular complaint from a company advertising unallocated ASN and IPv4 resources", and got told off by literally most of the mailing list. I'm counting at least 20 other individual responses and all of them are in support of the experiment https://mailman.nanog.org/pipermail/nanog/2019-January/threa...
- I think it's OK not to want experiments like these to be conducted connected to the global internet and rather in a lab such as the one benjojo used.
- NANOG is known to be a bit of a joke within the networking space, not too long ago someone complained about NTP issues with traffic passing through TATA and they were told to make their devices stratum 0 or something to that effect, so don't always take them too seriously.
> someone complained about NTP issues with traffic passing through TATA and they were told to make their devices stratum 0 or something
Not saying your summary of that thread is wrong, I just wanted to expand a bit to maybe clarify:
- someone posted to NANOG about issues with their (European) servers reaching 0.freebsd.pool.ntp.org
- they noticed they were being routed through Tata (AS6453)
- they mentioned that their other servers (e.g. in Africa) had no issues
- they posted to NANOG to ask about this
- they thought it was a routing or FreeBSD issue (they later posted about the issue to a FreeBSD list as well)
So AFAICT, the correct response was posted about a day and a half (~42 hours) later:
- the NTP pool uses GeoDNS to map clients to servers
- the OP's server IP address was incorrectly geolocated to Mauritius
- the Mauritius zone in the pool has only one server
- so the OP would only ever get that IP as an answer
The short-term fix was to use a different pool address (e.g. europe.pool.ntp.org), and the responder also mentioned that the NTP pool folks were working to fix the problem.
Now the interesting part of that thread was that shortly after (~4 hours) the OP posted about their problem, someone replied and recommended that they setup their own GPS-based NTP network.
From there, that sub-thread was mostly about NTP attacks and GPS security and mounting receivers on top of datacenter buildings and such.
Counting up the messages (if my mail client's threading capabilities are to be trusted), of the 94 or so total messages in the thread, 79 (84%) of the messages were in that GPS-related subthread, and only 15 messages responded to the OPs issue directly.
I didn't really see this as a failing of NANOG per-se, it's just a public mailing list and some folks went on a nerd sniping tangent (cf. XKCD #356).
The sane thing to do is to drop that particular announcement with that invalid attribute or discard the invalid information. But some people tend to be more catholic than the pope.
Dropping the whole session, especially in a DFZ scenario is really bad. The churn that you generate is immense for all your neighbors.
And indeed GREASE the TLS feature where you just propose nonsense because every other party should go "Er, no I don't speak nonsense?" to prevent ossification also comes from Google engineers.
GREASE means that when you build your "advanced" security device which "protects" customers by treating everything different as hostile, it won't work, so you'll need to tweak it to at least just ignore such differences as irrelevant, which is enough that we can come back later and intentionally improve things.
Previously, without GREASE, we'd have to guess what oversights we could exploit to avoid this "protection" to deliver protocol improvements, if we guessed wrong nothing works, or everybody's security is broken, sometimes both. e.g. for TLS 1.2 the oversight we found is, if you're resuming an existing session the "security" products just wave that through because otherwise they break people's real workflows. So in TLS 1.3 protocol as actually spoken essentially an initial connection goes like this:
Client: Hi some.dns.name.example I'm a TLS 1.2 Client, I'd like to resume our previous conversation #randomNonsense. Also, completely unrelated, I happen to speak FlyCasualThisIsReallyTLSv1.3 and so I have these TLSv1.3KeyAgreementParameters.
then either:
TLS 1.3 Server: Hi Client, of course, let's continue from there. [ whereupon everything further is encrypted because this is actually TLS 1.3, but to a dumb middlebox it makes sense that they're just resuming a prior encrypted conversation #randomNonsense which it doesn't remember ]
OR
TLS 1.2 Server: Er, no I don't remember any such conversation and I don't know FlyCasualThisIsReallyTLSv1.3 so let's start a fresh conversation as normal.
More like the secret to getting ignored/blackholed/sued into the ground the moment your fuzzer hits on a combo that actually does damage to someone else's system, since the implication here is you're not doing it on your own stuff.
If you are, good on ya. Continue doing $deity's/Bob's work.
More like the secret is to buy up some routing equipment and run these tests on your own network, sell the exploits you find to nefarious actors for big bucks, and have a bunker to survive the crash of the internet in.
If your system is on the internet it is now a potential piece in a potential global war.
Always has been, even since it's inception. The only thing that'd kept it safe and unutilized in that regard had been a grassroots efforts to not facilitate that transformation. That only lasted so long as the main population of Ops/devs/netizens were in agreement.
We've blasted past that age at this point, do everybody gets to don their robe and wizard hat, pick up the staff, and yeet patches, routing rules, and packets betwixt one another.
Yeah, but I don't think the people who created that RFC thought about all the bad side effects their advice causes.
BGP is one of the most robust protocols out there: there's no state (as in protocol state), you can treat each message independently, you can extend it super easy and so on.
And because you can treat every message independently, you can have a really secure and stable mechanism to deal with any badness that might happen.
To be honest, if you are a responsible researcher you are NOT running “experiments” in live environments that have potential to impact multiple regions.
The outrage here is understandable. I would be pissed if I was paged because halfway across the world, some asshole caused this intentionally. Also this mailing list is aimed at the operators, not the vendors. Different communities.
The vendor response here, I do agree is irresponsible. Besides OpenBSD, every one here just sucked.
Ya the researcher could just sell their findings to whatever blackhat groups are out there instead. You generally get less pushback and more cash from them.
I think the difference is that network gear is supposed to handle the whole spec and gracefully deal with (drop) things that aren't to spec. If there was a law that all cars had to be able to get hit by another car and keep going without issue (and this was physically plausible), then it would be similar.
That only makes sense if all the people using self-driving cars would have driven drunk (or drugged, or sleepy) instead. Otherwise, you're replacing a decent driver with a sub-mediocre automatic system.
Unfortunately at the time when drunken people are returning home from pubs/bars (e.g. 2 in the morning) most public transport do not drive or at least is significantly limited.
This is a problem with North Americas suboptimal implementations of public transit. Most public transportation systems in the rest of the world do actually take care of this.
It’s 2023 and driving kills like 40,000 people a year.
We’ve been building trillions of dollars of public transit since about the same time as we started building the National highway system. So when exactly does public transit start saving all these lives?
A new train from SF to LA has been in the works since 1996. They are expecting partial completion for $30 billion by 2030 which is projected to carry ~15 million passengers * ~400 miles = 6 billion passenger miles per year.
The fatality rate for passenger cars is ~0.57 per 100 million miles, so eliminating 6 billion passenger miles could be reasonably expected to save about 60 * 0.57 = 34 lives a year out of the ~40,000 that are dying on the roads annually.
This is just rough napkin math showing the scale of the problem would require a roughly $30 trillion investment in public transit to appreciably shift passenger-miles away from passenger cars.
Public transit can be very useful. But it’s clearly not the solution to eliminating driving deaths.
It seems unfair to say that public transport doesn't have a role in reducing deaths caused by drunk drivers just because the hugely flawed and half-assed public transport systems in our cities (which were designed to make public transport ineffective) haven't yet delivered on that promise. In places where cities and public transportation systems are well designed and useful people do use them and those places see far fewer deaths from drunk drivers than the US does as a result.
You argue that it would be very expensive to fix the problems we've made for ourselves in order to get useful public transportation and that's certainly true. It's also very expensive to invent self-driving cars and part that cost has already been paid with lives lost and human suffering, but that hasn't stopped us from developing those systems. We could decide to invest instead in improving our cities and infrastructure. We just choose to let obscenely rich people kill innocent Americans while they play with their profit generating spying car tech instead.
Haven't many cities actually dismantled public transport that they had already built (or rather, sold it to car companies "to operate more efficiently", who then proceeded to dismantle it so it didn't compete with their cars)?
The robustness principle: "be conservative in what you do, be liberal in what you accept from others", hardly applies here. Whether or not it was followed, a crash shouldn't happen.
The robustness principle is also long considered to be not that great of an idea for protocols. There's a good document here going into this:
It isn’t a crash, BGP terminates the session and retries because it considers the connection to be malformed. So, “be liberal in what you accept” certainly does apply.
That is the exact opposite of what RFC 7606 says. As it explains in its introduction, it deprecates session reset in favour of treat-as-withdraw in most cases, unless there is an explicit exception to the general rule.
> With that being said, I would like to thank the OpenBSD security team, who very rapidly acknowledged my report, and prepared a patch. My only regret with dealing with the OpenBSD team was reporting the issue to them too quickly, as they are uninterested in waiting for any kind of coordination in disclosure.
Lesson to network engineers: switch your routers to OpenBSD/OpenBGPd and keep up on your syspatches to escape this particular hell. ;)
> My only regret with dealing with the OpenBSD team was reporting the issue to them too quickly, as they are uninterested in waiting for any kind of coordination in disclosure.
This BS with disclosure coordination and all the other crap vendors have been trying to train researchers to do for years needs to stop. You find a issue, you publish it, vendors need to deal with bad code and products in realtime, not when it suits them.
While I agree in theory, in practice this will lead to bad actors ranging from trolls to cyberwar troops exploiting it immediately.
There is IMHO no ethical alternative to responsible disclosure, and using actual people who can get harmed by having their information disclosed as pawns isn't the way to go forward.
> vendors need to deal with bad code and products in realtime
One of the main reasons for coordinated disclosure is that, over time, it has become evident that vendors will not do that. We can say it until we're blue in the face, but vendor behavior will remain largely unchanged regardless (there are a few bright spots, but not nearly enough).
So you go to war with the army you have. In this case, that means frequently deferring/coordinating security event communications. As much as I wish the market would appropriately penalize vendors that encourage researchers to sit on 0-days, that rarely happens, and the cost of insecurity is passed onto consumers instead.
Tl;dr delaying disclosure does suck, but you should still do it.
Alas, it is the consumer that apparently drives pain reception for vendors, as they have no interest in minimizing consumer pain, but their own pain. Their own pain is defined as lost business.
'Tis the world we live in, so might as well get used to it.
I would very much doubt any of those carrier-grade routers run BGP in hardware. Chances are the control plane is handled by a bog-standard Linux/VxWorks/BSD that does all the logic of talking BGP/etc and then populates the right registers in the switching/routing hardware which handles the actual data plane.
You could technically replace all the control and decision-making with whatever off-the-shelf computer platform you want, and have it drive the carrier-grade router and "manually" populate its routing table via some API/Telnet/SNMP. In fact, as long as the hardware packet-routing core is still good, this might be a good way to upgrade out-of-support equipment that is no longer getting updates to its control-plane computer.
I could have sworn I saw someone on here a few weeks ago saying how they switched their router to openBSD and ended up in some horrible broken state after a power outage or something. I can’t recall the details, maybe someone else would recall.
It was only broken in the sense that it requires you to get out a keyboard and manually continue to a root prompt and type `fsck` (and then Y a bunch of times) and presumably some files may have been corrupted but i haven’t seen any.
For a router I’d much prefer that it comes up on its own after I lose power, but the system isn’t totally broken or anything. It’s happened probably 3 times in the last year or so.
To another commenter saying it’s crazy to not run a UPS with your router: this is residential. 99% of people don’t have a UPS for their router. They probably don’t even have a separate router and all and just use an all in one modem/router/wifi combo. None of these people are crazy, they’re just not made of money and a UPS is expensive. IMO it’s more crazy to not have a journaled file system in this day and age, no other systems have issues with sudden power loss at my house.
Huh, I've been running an openbsd router as my prod home network router for ~ a year now without issue, and it has handled power outages and more without issues.
I wonder if the outage happened during patching or something maybe?
why not switch to bird that was not affected at all? not just acknowledged and fixed. isn’t it “safe be default” better than “fixed after they pointed the flaw”
This is not helped by the RFCs that specify the behaviour of these core protocols. It is rare to find a network protocol rfc that is clear, specific and not self contradictory (IPSec rfcs come to mind as a positive example).
Say, compare RFC3174 (sha1) and e.g. rfc4664 (l2vpn)
the first is almost a copy-paste implementation, with tests and all.
the second reads like a "write a presentation, ten pages minimum"
or look at the base bgp4 rfc - that overspecified imaginary state machine is a good illustration of how bad things are there.
The software on big routers that run the Internet is far more reliable than it was 20 years ago. It's not perfect and each vendor has cycles of good and bad code releases but it's so much better than it used to be.
[bgp.tools Alert] for AS200676
A hijack on 2a06:a005:2730::/44 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured)
[bgp.tools Alert] for AS200676
A hijack on 2a0a:6040:b1f4::/48 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured)
[bgp.tools Alert] for AS200676
A hijack on 2a0a:6040:b1f1::/48 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured)
[bgp.tools Alert] for AS200676
A hijack on 2a0a:6040:b1f0::/48 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured)
Bird is unaffected, so I assume their response is satisfaction for a job well done. FRR, on the other hand, appears to be less responsive to concerns -- which does actually concern me since I use FRR, though not presently for eBGP. Disclosure seems to have done the job thankfully, as there appears to have been a little scramble over at FRRouting/frr about an hour ago.
This seems like a good time to mention Yuan et al., “Simple Testing Can Prevent Most Critical Failures.”
“We found the majority of catastrophic failures could
easily have been prevented by performing simple testing
on error handling code – the last line of defense – even
without an understanding of the software design.”
How do you figure? The IP address to route to in DNS is still an IP which from within that network segment will believe that prefix is provided by that AS if they aren't honoring RPKI.
Therefore, within that routing zone, that traffic would be blackholed I'd think, unless I'm seriously missing some nuance that I'd love it if you would share.
The impacted systems get stuck on a retry loop... they can neither forward the message blindly nor consume it. They break the BGP connection, reset, come back alive and get the same message and break the connection again.
Effectively the impacted instances are now no longer able to do their job, they cannot propagate more rules.
That's the flaw.
When they receive something that they don't understand they need to propagate without breaking connection, and they do not.
Yes. It’s also important to emphasize that when a BGP session between two ISPs is lost, all connectivity between those ISPs on that path is also lost. This is why the author emphasized when you can reconfigure the routers for error tolerance without affecting active BGP sessions.
> When they receive something that they don't understand they need to propagate without breaking connection, and they do not.
And this is how things are done, per the article:
> On 2 June 2023, a small Brazilian network (re)announced one of their internet routes with a small bit of information called an attribute that was corrupted. The information on this route was for a feature that had not finished standardisation, but was set up in such a way that if an intermediate router did not understand it, then the intermediate router would pass it on unchanged.
> As many routers did not understand this attribute, this was no problem for them. They just took the information and propagated it along.
However, the information was corrupted at the source, and those devices that did understand the attributed detected the corruption and rejected it:
> However it turned out that Juniper routers running even slightly modern software did understand this attribute, and since the attribute was corrupted the software in its default configuration would respond by raising an error that would shut down the whole BGP session.
The question is whether the session shutdown was appropriate (at least by default).
I think Juniper's default is reasonable, as attributes can communicate important information about the intentions of the network operators, and rejecting a particular attribute and basically ignoring the intentions of the operators may also cause problems.
I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).
> However, the information was corrupted at the source, and those devices that did understand the attributed detected the corruption and rejected it.
No they did not, they shutdown a whole session down for a malformed announcement. The right decision would have been to ignore the corrupt attribute and maybe not install that particular route.
A logged message saying "Attribute 129 with value 0 is invalid" would have helped operators more than just blindly shutting down sessions through which this was announced.
The right decision is specified in RFC 7606, which is usually to treat the malformed announcement as a withdrawal. (BGP attributes cannot in general be discarded safely.)
The current behavior is akin to an HTTP server shutting down whenever it receives an invalid HTTP request. Or, for an even better analogy, it is like an HTTP server that, upon receiving a malformed HTTP request, drops all current and future traffic from that IP. A malicious user can then issue a malformed HTTP request through an open proxy, and deny all other connections to that server throgh that proxy for all other users who were sending proper requests.
> The current behavior is akin to an HTTP server shutting down whenever it receives an invalid HTTP request.
It is nothing like that.
HTTP requests do not propagate across an organization's entire network, or possibly the entire Internet.
> Or, for an even better analogy, it is like an HTTP server that, upon receiving a malformed HTTP request, drops all current and future traffic from that IP.
So just like (e.g.) fail2ban blocking IPs that continuously send bad login requests? That's awesome! I would want my web server to block bad clients.
Heck, it would be nice if web browsers would do the same thing with content they downloaded. In the early days of the web there were all sorts of garbage files out there and instead of forcing people to correct their HTML they tried to be clever in mind reading:
It's fine to have a flag to have stringency be a policy that can be toggled, but I think Postel's law ("be liberal in what you accept") can cause issues over the long-term:
> So just like (e.g.) fail2ban blocking IPs that continuously send bad login requests? That's awesome! I would want my web server to block bad clients.
Well, somewhat, except that it's not awesome. It's like fail2ban, but implemented on a server that sits behind a load balancer. So, when it receives a bad login request from an IP and it bans that IP, it actually banned the IP of the load balancer, so now it won't receive any other requests. So that a malicious client can just send 1 bad request, and take down everyone else using the whole server.
Basically, this whole discussion is not so much about Postel's law, it's about limiting failure domains. A bad route advertisement should make that 1 route inaccessible. It is not good design for a bad route advertisement to take down all the other good routes. It is particularly bad design for a protocol where routes are automatically propagated by devices that don't think there's anything wrong with them.
If you want to compare it to the HTML situation, the bug can also be seen as a browser that, when it receives invalid HTML from www.google.com/search would not only throw an error, but also refuse any other connection to *.google.com/*. The proposed fix is not to interpret anything as valid HTML. It is to show an error only when accessing www.google.com/search.
It’s like your origin web server unceremoniously dropping the http connection when its front end reverse proxy (fastly, cloudfront, akamai, etc) forwards an invalid request, and when your web server restarts the proxy retries the bad request.
Bad input -> shutdown provides a method by which an external attacker can shut down your service by providing bad input, otherwise known as a denial of service attack.
The input in question was a particular attribute: these attributes can have important information about the intentions of how the operators wish things to work. Simply ignoring the bad input and carrying on the BGP session without the attribute(s) in question may lead to working against the desired intentions of the network operators.
I think Juniper's default is reasonable: rejecting a particular attribute and basically ignoring the intentions of the operators could also cause problems, and instead of trying to read the minds of those running the networks, the software doesn't try to be clever.
I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).
The problem is that one bad route in a BGP session caused the whole session to shut down. So if ISP1 advertises a new route with a corrupted attribute to ISP2 who doesn't understand it, and ISP2 advertises it to ISP3 who does understand the corrupted attribute, all traffic between ISP2 and ISP3 will shut down. So ISP1 has caused communication failure between ISP2 and ISP3 for all clients, not just for its own routes.
My point still stands: attributes can have important information about the intentions of how a network operator wishes their network to be seen to the rest of the world. Having a corrupt attribute, and thinking it's okay to simply assume that without that attribute the rest of the advertisement is still valid is not fine IMHO: you're basically trying to guess/assume the intentions of the source of the advertisement.
For all we know ignoring that attribute and accepting the rest of the advertisement as-is could have caused flood of traffic over the connection causing a just-as-effective-DoS.
As the fine article explains, your concerns are addressed by RFC 7606. It recommends that in many cases, an error should be treated as a route withdrawal, not by dropping the session. (The RFC specifies other ways to handle errors in specific situations.) There’s no guessing or assuming.
In the scenario that started off the author’s investigation, the bad attribute kicked COLT off the Internet even though COLT did not connect directly to the Brazilian ISP that sent the bad route update. When they received the bad update, COLT’s routers dropped their BGP sessions with intermediate ISPs that were unrelated to the Brazilian ISP, causing complete loss of connectivity.
If COLT’s routers had treated the error as a route withdrawal, they would have lost connectivity to the Brazilian ISP but not the rest of the Internet.
That is not what is being done. If you advertise a route with a corrupt attribute, your route will be considered inaccessible by any device that understands the attribute.
But! All your other routes that you advertised earlier with perfectly fine attributes will continue working. This is what prevents this from becoming a DoS vulnerability.
Let's take a more complete example.
ISP1 advertises 2 routes to ISP2:
35.67.0.0/16 attr1=val1
37.61.0.0/16 attr1=corr
ISP2 doesn't know what attr1 is, and it advertisea the following routes to ISP3:
The bug is that ISP3 will then stop routing any traffic to 35.67.0.0/16, 37.61.0.0/16, or to 78.0.0.0/8.
After the fix, ISP3 will keep routing traffic to 35.67.0.0/16 (respecting the value val1 for attribute attr1) and to 78.0.0.0/8 through ISP2. It will not route traffic to 37.61.0.0/16.
It sounds like a system designed to stop doing things on unexpected input. The designers chose correctness over robustness. It's not an accidental vulnerability.
Maybe switch off on request is a dubious design choice for infrastructure that you'd rather stay alive, but it sounds like a new version of the spec has already been released to make that an option.
Imagine if Gmail and Outlook refused to talk to each other and transmit any email anymore, because Gmail tried to deliver one email whose content was invalid (bad MIME, invalid attachment, etc).
If you shutdown the whole session because of one bad forwarded message, you are indeed a broken implementation.
Agreed, the article title is a bit sensationalist. Really should be titled “RTFM: Juniper routers shutdown BGP on bad input”. That being said, I found it be a good deep dive and I appreciate the author looking into this. BGP isn’t perfect but we made it this far and I think it does a pretty good job for handling ~75K autonomous systems.
Err no, it was not just Juniper. Many other systems were fuzzed, half a dozen vendors were fine, another 5 weren’t and some remain unpatched and vulnerable.
If it were just a Juniper implementation flaw it’d be less interesting than explaining an aspect of BGP error handling more generally which is indeed what the article was, for those of us aware of BGP but not familiar with specific protocol design decisions and implications.
If you quibbled with the word “Grave” I might agree since it’s not industrywide and impacting major vendors like Cisco and Huawei and those implementing RFC 7606 (although there is no patch for Nokia or Extreme or Bird!) I’d counterpropose that a more neutral TLDR title/summary for an HN audience might be “Grave flaws in non compliant BGP error handling.”
But on balance I think the author is better off with raising the alarm with a stronger headline; he clearly put his passion to good use and I learned a little something as a result.
A few years ago a team of researchers tried to use a feature attribute flag in BGP. Their experiment caused an important BGP software to crash.
The BGP community reacted by thanking the researchers for uncovering that flaw and worked on improving stability. No, just kidding. They shouted at the researchers and asked them to stop, which they eventually did.
I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers.
Sources: https://mailman.nanog.org/pipermail/nanog/2018-December/0984... https://mailman.nanog.org/pipermail/nanog/2019-January/09876... https://mailman.nanog.org/pipermail/nanog/2019-January/09918... https://mailman.nanog.org/pipermail/nanog/2019-January/09914...