
BGP and RPKI - tolien
https://www.aa.net.uk/etc/news/bgp-and-rpki/
======
pbowyer
Andrews & Arnold are the best UK ISP I've used in 20 years of being online.

As well as going above and beyond when it comes to sorting problems with BT, I
appreciate that like with this post they are not afraid to stick their neck
out and say what they think.

~~~
mystcb
I have recently (within the last year) joined AAISP, and I am in agreement
with you on this.

One thing it does show to me is that, even though the site was made available
by Cloudflare only recently, my ISP has already acknowledged and responded to
the post. Given that they have explained the situation, what it means, and
that they are actively looking into it. The trust they built up with me since
I started with them, means that I do trust that they are looking into this.

BGP as a whole does need a solution, and this may be one of a few ways to
achieve this.

I am glad they are not rushing their solution so that my connection stays
online while I have to stay isolated.

On the flip side, I do agree that some ISPs will drag their feet, and try to
ignore everything for the sake of "money" or "can't be bothered" types of
excuses. These are the ones that really should be pushed hard against to make
the changes that are required. - I am just glad AAISP isn't one of them.

------
tialaramex
I have been an A&A customer for years and so this is disappointing because
it's exactly the sort of crappy response I'd criticise other ISPs for.
Hopefully in the not too distant future we can talk some sense into RevK and
co.

The choice to block one route to make isbgpsafeyet apparently pass is bogus,
it's like silencing a fire alarm (which is also something I see far too many
businesses choosing to do, oh Zone A keeps alarming, we'll just disable it
even though now that means more danger, engineer calls are too expensive)

Those familiar with my work will know that I read a lot of accident
investigation reports, and a recurring theme in maritime accidents involving
large vessels is that the BNWAS (a technology which alerts sleeping crew
elsewhere on a vessel that for some reason nobody on the bridge is paying
attention, probably because they're asleep) is "somehow" muted, with its
logging disabled or even entirely switched off during an incident. That might
not seem like a big deal but big ships are _slow_ and so these incidents often
take place over many minutes or hours, thus there is plenty of time for
somebody to wake up, realise there's a problem, wake someone else up to help
fix it, and fix the problem before you smash the ship into an island clearly
marked on your chart.

If Cloudflare has any sense they'll change the bogus route up every week or
two so that outfits like A&A keep getting these "annoying" alarms and concerns
from customers never cease.

> This is marginally better than routing to somewhere else (some attacker) but
> it still means a black hole in the Internet.

No. It's the difference between an inconvenience (routing problem which can be
reported and then corrected by a human) and a grave security problem (data is
being sent to completely the wrong place because you didn't bother with the
mechanism to prevent that).

Also, this glosses over the _very common_ scenario where the bogus routes are
more specific than legitimate ones. If AA doesn't have a route to one specific
/24 containing secure.bankofamerica.com that has "somehow" been advertised by
a Taiwanese ISP for the past hour, that data will be bundled along with the
rest of the /16 or whatever else (BoA's assignment is old and a mess) to the
Akamai boxes (I think) which actually serve it anyway. Rather than a "black
hole" there was no problem except for the bogus advertisement, which RPKI
would have helped block.

------
vbernat
[https://isbgpsafeyet.com/](https://isbgpsafeyet.com/) is a bit misleading.
They mention both IRR filtering and RPKI. Both of them are needed. RPKI
currently only is used to validate the origin. When a peer only relies on
RPKI, it is trivial to announce a route you are not authorized to announce:
just append the legit ASN at the end of the AS path. That's what IRR filtering
prevents.

Both are therefore useful. RPKI protects from mistake and mitigate a bit the
ability to spoof a route (adding the legit ASN makes the AS path longer and
less preferred).

------
samcday
> and also our feelings about Cloudflare attempting to build support in this
> manner, especially now, during the Corona Virus situation.

Weird angle. Unless the RPKI standard is somehow actively encouraging people
to violate social distancing policies, I don't see any connection with
Covid-19..

To me this whole article just reads like a network operator complaining that
someone else is trying to hold them accountable.

~~~
gorgoiler
It’s a small business. Their staff could be infected or furloughed, or worse.

In terms of our day to day lives it might feel like the proverbial _month of
Sundays_ right now, but for operations teams it’s more like an unending stream
of Friday afternoons in terms of sensitivity to making big infrastructure
changes.

~~~
tolien
Yeah, that was how I read it - the impact of getting this wrong is that you
break the internet for your customers (and staff, if they're all or mostly
WFH) at a time when they're potentially depending on it to eat (e.g. if you're
in a vulnerable group and need to order food for delivery) or work.

We've known BGP's been vulnerable in this way for years, so it's a bit of a
weird time to actively encourage people to publicly shame their ISPs for being
"unsafe".

~~~
oasisbob
Cloudflare's BGP activism isn't exactly a new thing, so critiquing their exact
timing here seems misplaced perhaps?

~~~
tolien
isbgpsafeyet.com only appeared at 4 p.m BST yesterday, a Friday [1]. It's the
timing of _that_ which I took the OP to be commenting on. The GGP mentioned
that we're in a month of Friday afternoons, this page dropped literally
towards the end of the working day on a Friday afternoon!

As you say, Cloudflare have been promoting RPKI for a couple of years now and
it's disappointing that more of the big players haven't implemented it yet but
is _now_ the time?

1: [https://blog.cloudflare.com/is-bgp-safe-yet-rpki-routing-
sec...](https://blog.cloudflare.com/is-bgp-safe-yet-rpki-routing-security-
initiative/)

> 17/04/2020, 4:00:00 pm BST

> Today, we are releasing isBGPSafeYet.com, a website to track deployments and
> filtering of invalid routes by the major networks.

~~~
Avamander
For some, it's never the time they should do something. ISPs are notorious for
dragging their feet and they'd just find new excuses if CF had delayed the
publish.

~~~
tolien
I mean, the bigger ISPs will just ignore it like they've ignored IPv6
¯\\_(ツ)_/¯

On the other hand, AAISP started automatically assigning IPv6 addresses ~9
years ago, so you can hardly accuse them of dragging their feet. The OP was
published on a Saturday, after all.

~~~
gorgoiler
...and /48s at that too.

------
hugoromano
Cloudflare approach is pushy, but many companies need a push. No RPKI and also
no DNSSEC support. When will Andrews & Arnold do something about it?

~~~
tptacek
Hopefully never, on the DNSSEC side. Of course, very few other ISPs (or tech
and telecom companies of any sort) use DNSSEC either.

~~~
mlyle
Yay, let's leave DNS pwned because we have some disputes and the alternative
isn't perfectly technically pretty. /s

~~~
tptacek
This is snark but it's snarking up an interesting architectural issue, that
being: "where do we draw the line?" Say we protect DNS (we could, with a
better protocol design). But other redirection protocols aren't protected. Do
we similarly authenticate OSPF? IP options? ARP?

The easy response to your snark is "no, people are just going to use DoH".

~~~
mlyle
We've had decent authentication in OSPF for ... decades.

Most of the egregious problems with IP options have been worked around well,
again for decades.

ARP is problematic, but only within a segment. Your attacker has to be close.
For IPv6, we have SEND, but it is not without problems.

> The easy response to your snark is "no, people are just going to use DoH".

DoH doesn't prevent someone from poisoning well-trafficked DoH resolvers to
redirect individual domains. DNSSEC does. I run it on my zones, and it's not
so bad... About 40% of resolver traffic to my domains is verifying, too, so
we're getting there...

~~~
tptacek
DNSSEC does nothing to protect queries on the wire on their way to resolving
servers (it collapses down to a single bit in the header), unlike DoH, which
does protect those queries. Those queries are the most likely attack vector
against DNS. Meanwhile, however many resolvers are validating queries to your
own domains, virtually none of the most important (ie, well-trafficked)
domains on the Internet are signed, or likely ever to be signed. At the same
time, browsers have _removed_ DNSSEC code. DNSSEC is moribund. DoH has already
protected more people in the brief time that it has existed.

~~~
mlyle
> DNSSEC does nothing to protect queries on the wire on their way to resolving
> servers (it collapses down to a single bit in the header), unlike DoH, which
> does protect those queries.

Yes, and DoH does nothing to protect queries between resolving servers and
authoritative servers, unlike DNSSEC, which does protect those queries.
Together, you end up with end to end protection.

~~~
tptacek
That's true: DOH doesn't protect outbound communication from caches, until
authority servers expose DOT or DOH endpoints for them to talk to --- which
should happen, and is likely to happen long before DNSSEC. But DNS attacks
almost always target endpoints (DNS attacks are rare, and heavily mitigated by
the rest of the networking stack, which is why practically nobody deployed
DNSSEC).

~~~
mlyle
DOH resolvers don't recurse via HTTPS, and there are few/no authoritative DOH
authorities. DOH just exposes much larger, more dangerous targets for
poisoning at this point.

Something like 30% of domains in .COM have DNSSEC, in part because of things
like Cloudflare making it so dang easy. Yes, the top domains / services tend
not to, and this is unfortunate. But we're well on our way.

And the possibility for poisoning worsens other attacks (it's still trivial in
many cases to steal email with DNS cache poisoning... 25 years after I first
played with the idea).

There's no reason not to deploy DNSSEC today. The DOH resolvers you're
enamored with are almost all validating, so it's a way to get end to end
validation right now.

~~~
tptacek
Nothing resembling 30% of the domains in .COM are DNSSEC-signed. The number is
much closer to 1% (like, it may by now be 2%; it is not 4%). This is an easy
stat to look up, but it's often presented misleadingly; you may need to look
at whatever chart you're looking at knowing in advance how many domains are in
.COM.

And, of course, that 1-2% is almost entirely a bunch of random tiny zones
nobody cares about. You can quickly discover that for yourself by feeding any
list of the Internet's most popular zones (the Moz 500 is an easy one to
download) through a bash for-loop against "host -t ds".

There is no evidence of any cache-poisoning attacks on DoH servers that I'm
aware of. DNS attacks are in general far rarer than you'd think given the
attention they've received in the past (again: there isn't much you can do
with a DNS cache-poisoning attack other than trying to get a certificate mis-
issued, and CAs have multiple levels of defense against that attack).

It's interesting that you played with DNS cache poisoning 25 years ago. So did
I! Small world. I wrote a QID-brute-forcing cache poisoner in late 1995 (to
get onto IRC from funny domains with). Towards the end of 1996, I wrote the
DNS cache poisoning checks for Secure Networks Ballista scanner, which did QID
stuff a la Johannes Ulrich, and the DNS authority record caching tricks
Kashpureff used, plus some others.

It's fun stuff! There's not much to do with it anymore, though. In 1995,
reliable domain forgery would get you past rutils filters and NFS exports; it
was a very big deal, so much so that people took the time to do crazy things
like elaborate source-routing attacks. But just a year or two later,
everything was SSH; a few years after that, and most sensitive things on the
web were SSL (it wasn't good SSL, but DNS wasn't the low-hanging fruit).

It's been hard times for people who wanted to spend their careers protecting
the DNS.

There are, of course, lots of good reasons not to want to deploy DNSSEC. A
good starting point would be that it drastically harms the reliability of all
your sites, because when DNSSEC fails, every site in your zone vanishes as if
it was never there. That's what happened to HBO the week they rolled out HBO
NOW.

But there are strong architectural arguments against it, too.

I end up saying a version of a bunch of this stuff any time people complain
about DNSSEC not being available on this or that online service. I appreciate
the opportunity to write a new and hopefully more interesting version of the
comment. Thanks!

~~~
hugoromano
My Logs, for the last 30 days, domains validated with DNSSEC was 5.34%. In
general, it feels that ISPs like to be slow adopting standards, e.g. IPv6 it
is still rolling.

~~~
tptacek
Just look it up on dnssec-tools. As of March of this year, there were ~1.7MM
signed domains in .COM. As of Q3 of last year, there were 144MM domains in
.COM. Wherever you got that "30%" stat from, it was _wildly_ off. Even I was
overcounting.

And ( _I 'm ninja-editing this in, so sorry if I caught you while replying_),
as you noted, much of the growth in .COM signings is attributable to
Cloudflare auto-signing --- which is the same thing that is happening in
places like .NL. But registrars auto-signing domains is security theater.
Ironically: attacks against registrars are far more common than attacks on the
DNS protocol itself; the recent spate of "DNS attacks" from the end of last
year turned out to have been phishing attacks.

~~~
mlyle
Yes, I slipped a zero. Still, your argument is: It's not adopted much, so we
shouldn't adopt it--- plus it doesn't matter if naming isn't trustworthy? :P I
don't find that a very good argument.

Your same argument seems to imply that since we have good TLS on many
protocols, we shouldn't be too heartbroken about people stealing traffic with
BGP, either.

~~~
tptacek
So, that's actually not my argument. But, though you're trying to dunk on it,
it is a solid argument. Here's why: the cryptography in DNSSEC is obviously
outmoded (it's PKCS1v15 RSA signatures, most of them 1024-bit, with an
outlandish hack that combines spoofing and a KDF to prevent zone enumeration).
The protocol "supports" P-curve ECDSA signatures, and, of course, both the
P-curves and the NIST DSA signing algorithm are themselves outmoded --- and
that's the _frontier_ of crypto in DNSSEC; Geoff Huston did studies of this
and found a substantial fraction of deployed DNSSEC verifiers can't even
handle _that_. The point being: it will take _decades_ , literal _decades_ ,
to get the baseline level of cryptographic competence we expect from secure
messengers (to say nothing of modern signing crypto) into DNSSEC.

 _And every new deployment of 1990s-grade DNSSEC makes that problem harder_.

If we started from scratch, the very first thing that would change would be
the notion that cryptography is too expensive to support online signing, which
is one of the two original sins of DNSSEC. The entire protocol would be
drastically cleaner, easier to deploy, and, of course, much more secure.

This is _not_ the core argument I have against DNSSEC (for that, just Google
[against DNSSEC]). But it's a good one. What they say about PHP holds true for
DNSSEC: it's a fractal of bad design.

I don't know what BGP has to do with any of this except to say that DNSSEC
obviously doesn't do anything to prevent BGP hijacking attacks; attackers who
control BGP control IP addresses, the things DNS maps in the first place.

I'm doing my best to keep these responses varied and interesting, because an
HN search will show this is an argument I've made here a lot. Hopefully I'm
doing an OK job of that.

~~~
mlyle
> So, that's actually not my argument. But, though you're trying to dunk on
> it, it is a solid argument. Here's why: the cryptography in DNSSEC is
> obviously outmoded (it's PKCS1v15 RSA signatures, most of them 1024-bit,

Root zone has been signed with 2048 bit keys since 2016. Large numbers of
domains are signed with 2048 bit keys, too.

> If we started from scratch, the very first thing that would change would be
> the notion that cryptography is too expensive to support online signing,
> which is one of the two original sins of DNSSEC.

Online signing still looks really expensive. Asking an authoritative server to
do a 2048 bit signature per answer is costly, given the other conversation
you've followed me into and argued basically the other side of. Especially
since work you do to make DNS authoritative servers verify things is easily
used for DOS, unlike work done for prefix verification in BGP).

> don't know what BGP has to do with any of this

Because we're in a thread talking about preventing BGP hijacking of prefixes
with crypto, and the tangential discussion of preventing DNS hijacking of
names with crypto (DNSSEC) came up. Some arguments you presented against
caring about DNS security work equally well wrt: caring about BGP security---
who cares if someone intercepts our traffic, we're probably using TLS!

~~~
tptacek
No. All modern DNSSEC implementations _are online signers anyways_ , because
the BCP mitigation for zone enumeration, "whitelies", depends on that. So what
we in fact have is the worst of both worlds: a protocol that makes painful
compromises to work in an offline-signer model, and all the operational
concerns of online signers. A DNSSEC that was designed for online signers from
the jump would be cleaner and more secure.

That the root zone was signed with 1024-bit RSA in 2016 is its own dunk, and I
think you for making it for me, but of course the root zone isn't the only
zone; the 1024-bit root just meant every TLD was capped at that level of
security. Now they're not, but the rest of the DNS is still littered with
them. See: every DNSSEC survey.

And, of course, you've addressed only a small part of that previous argument
--- just the key sizes.

I don't know what point you're trying to make about BGP. I think you think I'm
opposed to RPKI. I'm not.

~~~
mlyle
We don't exactly need signatures that will stand forever to greatly mitigate
DNS interception.

I agree DNSSEC isn't wonderful, but perfect is the enemy of good. I'd take 768
bit RSA over "doing nothing"\-- it raises the difficulty level of a successful
DNS attack enormously.

Some of us don't care about enumeration. If we do, there's whitelies, as you
say, and eventually NSEC5 will be around.

> I don't know what point you're trying to make about BGP. I think you think
> I'm opposed to RPKI. I'm not.

You argued earlier that worrying so much about redirection isn't justified:
"Say we protect DNS (we could, with a better protocol design). But other
redirection protocols aren't protected. Do we similarly authenticate OSPF? IP
options? ARP?"

In which case, maybe we should just ignore RPKI, too. :P And unfix source
routing, not bother with SEND, etc.

~~~
tptacek
If DNSSEC were free, rather than a many-tens-of-millions-of-dollars
proposition, and if it were safe, rather than a demonstrated reliability
nightmare, and if it was a well-contained solution to a specific problem,
rather than a transparent attempt to bootstrap a government-controlled PKI,
and if complicated new protocol deployments weren't a giant source of new bugs
to exploit, sure. Sure, it would make sense to deploy a little bit of 768-bit
RSA to fix some marginal problem the rest of the stack already heavily
mitigates.

~~~
mlyle
I don't think it's a many tens of millions of dollars proposition. It was
about 45 minutes of my time to get it deployed on my zones, tested, automated,
and then installed by my registrar.. and it's been reliable for 3 years since.
I understand the tooling is better now, too, so some of the crap I fiddled
with is probably not necessary anymore. And it's even mechanized entirely for
you if you use Cloudflare DNS, etc.

~~~
tptacek
No. TLS is a many-tens-of-millions proposition, demonstrably so, and TLS is
easier than DNSSEC. You're intuiting global cost from a single extremely
simple test case; I think you're mistaken both about the multiplicand and the
multiplier.

Further evidence on this is easily obtained both from survey papers that
collect DNSSEC misconfiguration statistics (they're hilarious) and from the
IANIX collection; many of the organizations IANIX notes are extraordinarily
clueful operators.

I'm right about this. It's feasible to get DNSSEC deployed globally (it would
be a catastrophe, but we indeed have the tools to make that catastrophe
happen). But it would be very, very expensive.

This has been a fun thread, but we are way into the right margin now and are
probably the only two people reading it, so I suggest we tie it off at
whatever your next response is.

~~~
mlyle
If you're stating a global cost, make it explicit. Because most organizations
can deploy DNSSEC cheaply, and who knows the global cost for total deployment
of anything.

------
anfilt
I am not a fan of RPKI since it's centralized. I guess better than nothing,
but what I find annoying is that IP protocol has the ability for source
routing. So instead blinding trusting routes in BGP, clients/nodes who knows
the route could avoid BGP entirely. It also allows the internet to be more
resistant to censorship. Yet network operators seem to hate the idea of source
routing because someone can spoof an ip address. So they just disable it so
they don't have deal with it. Also smaller network operators seem to disable
it as well to make firewall rules simpler

IPv6 implements source routing too, but again is hardly used. However it does
include an other standard that also works like source routing for mobile
networks, but is no where as powerful.

------
judge2020
I'd like to see what the potential issues with RPKI are (if there are any),
this seems like a blog post that would address those but anything about the
protocol isn't mentioned, outside of "it takes time to implement and we'd
rather our upstream do it".

------
znpy
Possibly related: [http://rpki.me/](http://rpki.me/)

------
ngcc_hk
Do not trust any centralised authority. But if PKI Involved, what are those
and what is meant by court in the article? We have to be aware certain
countries (China, Russia) may not be of best interest of humanity in mind.

