
Hijack of Amazon’s domain service used to reroute web traffic for two hours - coloneltcb
https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f
======
bklyn11201
Getting a basic certificate issued is incredibly easy these days. If you own
the DNS resolution, you can get a cert.

Here a few ways I can think of to make this harder for an attacker:

* Add a Strict-Transport-Security to all HTTPS requests. This means the attacker will need to get a valid cert (still easy if you can hijack BGP).

* Pick a preferred SSL issuer and stick with them. Add a CAA DNS record only allowing that one issuer.

* Go through the EV SSL process with your preferred issuer. Now modify your CAA DNS record to add an EV policy. This should make it very difficult for an attacker to get a cert issued during a BGP hijack.

* DNSSEC? Make sure to choose an SSL issuer that will correctly test DNSSEC?

~~~
EE84M3i
Forgive my ignorance but I have a few questions:

* Why would HSTS help in this case? While HSTS is active, does it prevent clicking through the warning (which was done here)?

* How would a CAA record help against cert issuance in this case? Is it only helping against compromise of the authoritative during the remaining TTL of the record in recursives AND if the CAA record points to something that doesn't have on-demand DV like Lets Encrypt?

~~~
bklyn11201
HSTS wouldn't help users clicking through warnings, but it's a good thing to
have (myetherwallet doesn't use HSTS).

CAA record would only help in remaining TTL. Once expired, then it doesn't
matter.

So yeah, these seem like decent steps to help protect but certainly not going
to 100% prevent an attack like this one.

~~~
f2n
>HSTS wouldn't help users clicking through warning

Actually it would have! Chrome and possible other browsers do not allow
clicking throw certificate validation issues on sites with HSTS. For example,
try to get to [https://badssl.finn.io](https://badssl.finn.io) in Chrome.

~~~
cft
Sorry how does that help if the attackers purchase a new "valid" SSL
certificate since they control the DNS and thus email?

~~~
rocqua
Once you control DNS, it doesn't require email, you can just use lets-encrypt.
The lets-encrypt verification does not check HSTS.

Makes sense because it keeps HSTS from the lockout scenario that makes HPKP so
scary.

~~~
cft
Is it possible to get a TLD registrar to set a very short TTL like 5min on
your NS record? Then you can switch to a backup DNS hosted on another network
fast.

~~~
rocqua
Yes, you can set a short TTL on your NS record, but that would not keep this
attack from hijacking your site. This attack intercepts the clients DNS
lookup, so the DNS a browser is talking to is ill-behaved. No amount of
correct setup here will work because the ill-behaved DNS server will just
replace your setup with whatever that server wants.

There is no strong defense against this as a website. With an app the solution
would be certificate pinning. You could try HPKP but that comes with a host of
issues and I think it is being deprecated.

~~~
cft
Wait: in example.com say .com registrar sets a 5min TTL on NS record for
example.com that resolves to 1.2.3.4. That means that your DNS server is at
1.2.3.4, serving your A and MX records. An attacker BGP hijacks 1.2.3.4. You
change NS record in your .com registrar settings to 5.6.7.8 that is not
compromised. Notice I am not talking about your A or MX records that you
controlled on a compromised IP, but of NS record that a .com registrar
controls. So after 5 min the browsers contact a non-compromised nameserver at
5.6.7.8 to get your A records.

I think unless a TLD registrar gets hijacked that mitigates the attack on your
own DNS after the NS TTL

~~~
rocqua
You are right, I was thinking of a different attack. Something like a BGP-
hijack of 8.8.8.8 , 1.1.1.1 or similar often used DNS resolvers.

Here though, people using area53 for DNS probably can't move away from it as
they are stuck on amazon.

------
mlosapio
Wonder if services like Let’s Encrypt were affected. I imagine a scenario
where a small hijack of DNS could allow for properly signed certificates for
domains that are not owned. If I operated a CA service, I would carefully
examine the requests received during this time frame. Maybe someone can audit
the Transparency Logs during this period for anomalous activity.

~~~
erulabs
I would assume they go far out of their way to resolve the address from a
large number of distributed DNS resolvers - the question is if they're just
round robin instead of multiple simultaneous queries, but I would hope that's
part of why the DNS test usualy takes >60 seconds. If they're not treating DNS
as a consensus protocol instead of as an immutable database, they'll always be
open to poisoning attacks.

If any certs were issued for hijacked domains (which as far as I've ready was
only one, __not __using LetsEncrypt), it 's a pretty glaring failure on the
issuer, assuming they used "DNS Validation"

~~~
jaas
Let's Encrypt does not currently validate from multiple perspectives but we're
working on it. We're cooperating with a research team at Princeton to make
sure our strategy is an effective mitigation against BGP attacks.

Any given production validation right now may come from one of two of our
datacenter locations, but not both.

You may see multiple validation requests if you use our staging servers, but
that's just an early version of multi perspective validation which doesn't
work well enough to promote to production. There are some performance issues,
and the external validation points are all on AWS, which uses the same
autonomous system (AS) across datacenters. In order to effectively defend
against BGP attacks we're probably going to have to deploy validation servers
on 3+ different ASes so that our network perspectives are sufficiently
diverse.

If multi perspective validation is effective and nothing is done to
significantly improve BGP security (we're not holding our breath) then I
expect multi perspective validation to be a requirement for all CAs down the
line.

~~~
peterwwillis
I'm trying really hard here not to be snide, but it is amazing to me that an
organization that is responsible for securing the world wide web is basing
that security on the hope that nobody can spoof 3 AS's at once.

Just give up on BGP. Strongly suggest people use DNSSEC. For TLDs that don't
support DNSSEC, require a public key issued to the registrar. You (the
Certificate Authority) can get the public key from [R]WHOIS, IRIS, RDAP,
whatever method you like, directly from the registrar. This is standard
practice using the thin WHOIS data model, where .com and .net require domain
registrars to maintain their own customers' data. Other TLDs simply run thick
WHOIS servers that store all the data for their domains.

Registrars are already required to pass up Delegation Signer records to a TLD
to support DNSSEC. Passing on a similar record to a Certificate Authority
would be practically the exact same thing. So we know the registrars can
support it, and we know it would ensure that people would only be able to
generate certificates for those domains that they actually own (and not
"whomever can control the IP space currently").

~~~
pawal
We should not "give up on BGP". What we should do is to improve security in
all layers. This includes BGP, and as you mention DNS. DNSSEC should be
mandatory, just as TLS, for any business that take themselves seriously.

~~~
rphlx
Not to start another DNSSEC melee (as I start another DNSSEC melee..) but for
it to be effective we really need browsers to a) be able to tell whether a DNS
response was properly DNSSEC signed or not, b) produce some large, scary, red
warning for ones that aren't, mark them "Not Secure", etc. [1]

There is a major chicken-and-egg/game-theoretical problem though: any browser
that does that today will piss off/irritate its users, forcing them to use
other browsers or older versions of the same browser, as DNSSEC is not widely
deployed on the corporate side. And until most/all browsers do something major
to make the current DNS security crisis obvious to large numbers of users,
most companies won't care enough to deploy DNSSEC.

At least it will be interesting to see whether, and how, this eventually gets
solved.

[1] Certain abysmal DNSSEC cryptographic choices, such as 512-1024-bit RSA,
should also be addressed.

~~~
zjs
> There is a major chicken-and-egg/game-theoretical problem though: any
> browser that does that today will piss off/irritate its users, forcing them
> to use other browsers or older versions of the same browser, as DNSSEC is
> not widely deployed on the corporate side. And until most/all browsers do
> something major to make the current DNS security crisis obvious to large
> numbers of users, most companies won't care enough to deploy DNSSEC.

This seems analogous to the problem browsers faced with Flash. Perhaps they
can leverage the same incremental approach here.

For example: 1) Validate DNSSEC. If present and valid, the HTTPS "green lock
icon" gets a bonus glow. 2) 6 months later, not having a DNSSEC response gives
a little red X badge on the "green lock icon". 3) 6 months later, not having a
DNSSEC response graduates to a little ignorable info/warning box near the
location bar. 4) 12 months later, you have to click through a big scary
warning to access the site.

Essentially, start by providing a carrot and then introduce a progressively
larger stick. Communicate the whole plan up front so that large organizations
can get the ball rolling.

------
dice
It was certainly noticed in under 2 hours. The Outages list thread about the
issue started at 11:54 UTC:
[https://puck.nether.net/pipermail/outages/2018-April/011257....](https://puck.nether.net/pipermail/outages/2018-April/011257.html)

I'm sure that there were ops teams working on it before then. The people
involved in actually fixing the problem wouldn't have been posting to public
mailing lists.

~~~
kolpa
> The people involved in actually fixing the problem wouldn't have been
> posting to public mailing lists.

Surely, but their coworkers should have been.

------
reaperducer
How do we know it was unnoticed? It could very well have been noticed and
taken two hours for Amazon to mitigate it. I don't know the technical process
behind making such a correction, but in any large organization there are steps
that have to be followed. It's not like some kid typing out a shell command on
his basement Linux box.

Inflammatory/clickbait headlines do not make the internet a better place.

~~~
kolpa
If no one mentioned this important outage, it's likely that many/most/all
people didn't notice.

~~~
ohiovr
How big was the attack? You can't fool all the people all the time. But
fooling all their targets for even 30 minutes is a significant achievement
considering what info they could have obtained. What do we know about the
victims? What reason do we have for believing it was a random victim attack?
Besides the insane difficulty part..

------
omarforgotpwd
Wow. Just wow. These BGP vulnerabilities are ridculous. Imagine, someone
taking over DNS for even a small subset of people and being able to basically
just rewrite the internet as they see fit, completely taking control of
anything. Even without being able to get a valid SSL certificate you could do
a lot of damage. For example, let's say I rewrote requests for
SomeNationalBank.com to my proxy server. I make a request to the real
SomeNationalBank.com with the weakest set of ciphers allowed, and then pass
those bytes along to the client. Then I can try and decrypt it and log into
your bank account as soon as I do.

OR better yet I wait for you to log into an unsecured site like neopets.com or
something and then pair that unencrypted password wait the fact that I saw you
request the domain for a certain bank and if your password is the same (which
it is for most people) then that can also get me in.

We need a new more secure system to replace DNS.

~~~
cowkingdeluxe
How similar was this to the BGP attack by Russia in late 2017?

[https://arstechnica.com/information-
technology/2017/12/suspi...](https://arstechnica.com/information-
technology/2017/12/suspicious-event-routes-traffic-for-big-name-sites-through-
russia/)

~~~
ryanlol
>by Russia

Sounds like xenophobic bullshit.

~~~
balls187
More correct to say "Russian Government"

~~~
ryanlol
The correct thing to say is “ISP in Russia”, could be Norwegian hackers using
their routers for all you know.

------
925dk
Why do AWS [http://status.aws.amazon.com](http://status.aws.amazon.com) say
that only Google resolvers were affected?

Between 4:05 AM PDT and 5:56 AM PDT, some customers may have experienced
elevated errors resolving DNS records hosted on Route 53 using DNS resolvers
8.8.8.8 / 8.8.4.4. This issue was caused by a problem with a third-party
Internet provider. The issue has been resolved and the service is operating
normally.

~~~
fooblitzky
It seems like either the title is misleading, or the article does a poor job
of explaining the situation. From my reading of the article text, it seems DNS
traffic was rerouted to Route53, which the attackers then used to serve false
DNS records. That does not sound like Route53 was hijacked at all, just that
the attackers happened to use the service it provides.

~~~
925dk
Traffic to Route 53 was rerouted to an alternative DNS server. But that has
nothing to do with my question re. AWS calling out Google in particular in
their status update.

~~~
dgemm
_Some_ Route53 traffic was redirected somewhere else. Traffic can only be
hijacked from neighbors that actually accept the routes. You would think
Google of all networks would have effective ingress filtering to prevent this,
but it seems like they did accept it in this case.

~~~
dward
I don't think ingress filtering on Google's edge would have helped if the
rerouting happened in any of the transit ASs between AWS and Google.

------
nodesocket
If BGP is open and anybody can modify routes, why don't these attacks happen
constantly? What prevents these sort of attacks and how are they fixed?

~~~
noselasd
There's an informal trust chain. You only set up BGP peering with companies
you trust, and you implement filtering of it's traffic and advertised routes
that you know can be trusted.

You know that if you do something nefarious, or if someone at your downstream
does something nefarious, your upstream will identify it and block your
traffic, or the part of it that they identify to be the issue.

Thus as long as you need to appear on the internet, you do your best to play
nice, otherwise your traffic gets blocked, and you also do your part to
identify and block traffic when something goes bad.

~~~
nodesocket
So if BGP is trust based and hard to compromise then how were these Russians
able to?

> They re-routed DNS traffic using a man in the middle attack using a server
> at Equinix in Chicago.

~~~
notyourday
Because large networks do not register all their prefixes or enforce policies
on what they accept from other large networks. So a large network can do this.
Allowing a small network to do this demonstrates incompetence. Frankly, I am
having a hard time believing that HE.NET failed that badly.

AS7007 incident demonstrated this being a bad idea in the nineties.

~~~
pyvpx
> Frankly, I am having a hard time believing that HE.NET failed that badly.

Believe it. HE has many, many netiron-based routers in their network and many,
many ASNs as transit customers. the hardware doesn't have the capacity to
implement complete filtering for everyone.

between hardware-constrained equipment being replaced and the tireless work
Job Snijders (& others) are doing with route security, these kinds of episodes
will become far more difficult and occur much, much less.

~~~
notyourday
Half if not all of HE business model is to sell dirt cheap flat rate IPv4 with
BGP to other companies in major carrier facilities. I find it extremely hard
to believe that not filtered access to HE.NET is possible.

------
7ewis
How would you see this in ThousandEyes? I see he has a screenshot of it in the
post, and doesn't appear to be logged in.

Was troubleshooting this for the two hours, until I started reading reports of
the BGP leak. Not in networking, so don't fully understand how it works.

I'd just like to know how to troubleshoot this in the future, and check for
leaks, if any. Are there any other online tools I could use? Obviously don't
have an edge router or anything like that.

~~~
adamb_
> How would you see this in ThousandEyes?

Checkout this video analysis from the team on the hijack:
[https://youtu.be/YXm4GJMUlP0](https://youtu.be/YXm4GJMUlP0)

~~~
7ewis
Thanks! Do you know how I can access that page he is on? I'd like to have a
click around myself.

~~~
ruddell
Here's the link he starts on in the video:

[https://webysvi.share.thousandeyes.com/view/tests/?roundId=1...](https://webysvi.share.thousandeyes.com/view/tests/?roundId=1524576600&metric=availability&scenarioId=dnsServer&testId=619444&serverId=239927)

------
rgbrenner
A website I own was affected by this. I got an alert from our monitoring
saying the website was down for 1hr 2min 59sec. Luckily, they didn't redirect
it to anything.

I have other domains using Route53 (and hosted at AWS, just like this one)..
that weren't affected AFAICT.

~~~
isostatic
> I got an alert from our monitoring saying the website was down for 1hr 2min
> 59sec.

That's a very accurate time. What system do you use to allow sampling at under
1 second intervals? My nagios boxes poll every minute, so an outage could be 2
seconds, or nearly 2 minutes, and nagios would report the downtime as 1
minute.

~~~
outworlder
They could be alerting based on metrics, instead of "pinging" the site.

~~~
isostatic
Yes, if the site serves up lots of traffic (multiple hits per second), then
the hole in the logs would certainly give an indication.

I get very suspicious when people quote things accurate to 1 part in 1000

------
conistonwater
> _This traffic was redirected to a server hosted in Russia, which served the
> website using a fake certificate — they also stole the cryptocoins of
> customers._

What does this sentence mean? Sorry if I'm being slow. I thought the whole
point of certificates is that you can't generate a legitimate one for a domain
you don't control, so messing with name resolution wouldn't affect it.

~~~
polpo
It means just that. Since they had control of DNS, they could have easily
gotten a DV certificate, but in this case, they didn't and people clicked
through certificate warnings.

~~~
polpo
Here's a source on Twitter showing some documentation of the self-signed
certificate that myetherwallet.com users ignored warnings about:
[https://twitter.com/GossiTheDog/status/988785871188045825](https://twitter.com/GossiTheDog/status/988785871188045825)

------
peterwwillis
This same attack can be used to generate valid TLS certs for any website, for
example using Let's Encrypt. The best part? Your target doesn't need to use
Let's Encrypt at all. _Anyone_ can use them to forge certs _for any domain_.
Of course, this is possible with most other cert providers, but Let's Encrypt
automates it.

So.... TLS certs mean jack squat if you can pull off a BGP+DNS attack. You
might want to start pinning certs for your bank's website. (but not using
HPKP, Google is removing it from Chrome)

~~~
24gttghh
Excuse my ignorance but how does this allow someone to forge TLS certs for a
given website/domain name that already has active certs? Wouldn't they have to
revoke existing certs and then get new ones _during_ the attack to pull this
off?

~~~
jerf
Step one: Hijack the DNS entries so they point at your site.

Step two: Obtain a certificate from Let's Encrypt using website validation.

Step three: Proxy traffic through a proxy you provide that SSL cert for and
capture anything you like. In the event that a website legitimately has a
higher-class Extended Validation cert or other such thing, hope users don't
notice the downgrade from Extended Validation correct SSL cert to a merely
correct SSL cert, which is a pretty solid hope.

Certs don't have any sort of overriding or superseding property. A given cert
and its chain of signatures is either accepted because you trust the signers,
or it isn't; at the time you are making that decision, you don't have access
to know whether any other certs have been issued since then or anything.

One thing that comes to mind is that browsers really ought to scream if they
see a domain that was previously Extendedly Verified drop down to a Let's
Encrypt-level cert. Though how much good this would do in practice is hard to
tell... "Hmmm, what's this about the SSL cert being less good? It still has
one, right? Let's just click through this... hmm... looks like the web site I
expected... must just be the computer being weird _enters password_ "

~~~
24gttghh
For some reason I thought CA's at least talked to each other to see if someone
else had issued a valid cert for a given domain along the lines of owning a
domain name itself through a registrar. Clearly that only works when DNS isn't
being abused like this...

~~~
xnyanta
Why should websites be limited to having only one certificate?

For example, if you have a service running on multiple backed servers, you
might want to have once certificate per server for all of your domains.

~~~
notriddle
That use case still wouldn't explain a domain having certs from two different
CAs.

------
tribune
How did they manage to steal coins if they didn't get a valid cert? Did people
logging into MyEtherWallet just ignore the invalid cert warnings and log in
anyway?

~~~
Already__Taken
In addition they could be behind rubbish proxy's that aren't passing the
warning forward. I know ours in work will happily resign that expired cert to
mitm you for compliance and the user will see that green padlock.

~~~
rocqua
I get that companies need to control access to their internal networks. But
surely when you do it this badly you have to realize you are only making your
network less secure.

~~~
tialaramex
So far as we've been able to tell all the middleboxes on the market for this
sort of purpose are worse than useless. If you remember that NCSC blog post
that annoyed Adam Langley, its author Ian Levy insists that "there are some
good products out there" I actually replied to that comment, requesting a list
of these "good products" which presumably are warranted by the NCSC not to
actually be worse than useless. Unsurprisingly Ian has elected not to in fact
list any such products. There aren't any.

------
sajal83
[https://pulse.turbobytes.com/results/5adf2844ecbe40692e003ad...](https://pulse.turbobytes.com/results/5adf2844ecbe40692e003ad2/)

Some traceroutes captured during the incident. The results that show "Target
unreachable" are the ones seeing the hijacked paths

------
jwbensley
A lot of people in here are talking about securing their DNS setups but not
their BGP setups.

For any BGP operators out there, you could automate your inbound/outbound BGP
prefix filters using IRR/LIR databases, peeringdb.com, RADB etc. Tools like
[https://github.com/snar/bgpq3](https://github.com/snar/bgpq3) can help with
auto-generation. Even manual prefix lists will do, anything is better than
nothing!

It would be good if operators could also use AS-SETs so that other operators
can see what downstream ASNs should be seen via their peers (and which ones
should not).

Also BGP RPKI (Resource Public Key Infrastructure) based ROA (Route Origin
Authorisations) can also help to ensure that prefixes are announced by their
true originator.

Most networks follow the Pareto rule - 20% of prefixes learnt via BGP (in some
cases even less!) are the source/destination for 80% of their traffic flows.
Some operators use an approach of writing explicit filters for those 20% of
prefixes explicitly blocking them from all their BGP peers except the origin
peer sessions.

None of these techniques are fool proof and we all need to play along to
improve their impact. They cost nothing, only man hours, and we can all
benefit. In metaphor - no encryption is unbreakable but we can try and make it
unappealingly difficult.

------
notyourday
This is another one of the articles that says absolutely nothing.

The only reason why such attack was possible is because some of the providers
did not filter announcements from hobos claiming to be able to advertise AWS
space. That could have only happened if:

a) AWS did not register all their policy prefixes

b) providers did not apply filters on the transit announcements from those
that announced AWS prefixes based on the policies registered by AWS.

This is a modern day "let residential IPs to send data directly to tcp/25 and
not check".

~~~
lathiat
If you think the majority of BGP speakers world-wide are applying these
announcement filters you are sadly mistaken.

~~~
notyourday
I dont remember when was the last time that I was able to just announce
anything I wanted either as a customer or peer to any large network. So as
long as the large networks filter their customers and peers it seems to
minimize the exposure. [1]

[1] That statement excludes Google whose idea of filtering is to let me
announce anything I want and build filters based on what I announce.

------
djsumdog
So they obviously couldn't or didn't hijack certs, so they were depending on
people who clicked through or APIs/systems that didn't do cert checks?

~~~
Diederich
Update: see other comment chains invalidating the security I'm claiming HSTS
provides.

APIs will probably go directly to [https://](https://), and hopefully cert
failures will cause the API to blow up with minimal harm.

If you want to visit my web site, and you need to type it in, you'll probably
just put realms.org into the URL bar of your browser. Most secure websites,
such as paypal.com, have something called HSTS
[https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security)
to force the browser to always hit the site with https, even if requested to
it it http:

dana@araman:~$ curl -D - [https://paypal.com](https://paypal.com) 2> /dev/null
|grep -i strict-transport Strict-Transport-Security: max-age=63072000
dana@araman:~$

So even if going to paypal.com lands you on a blackhat server, the cert will
still blow up, protecting you.

Slightly related: much to my annoyance, it seems that neither AWS ELB nor ALB
support sending HSTS, which is a real bummer.

~~~
philjohn
Except, it would have been trivial for them to get Lets Encrypt to issue them
a valid cert.

~~~
Manozco
it depends, let's encrypt might be performing request from different sources
(and then it would need a bigger BGP hijack to fake them)

~~~
tialaramex
Yes, Let's Encrypt acknowledges that they perform their validations from
multiple observation points on the Internet. They (intentionally) don't
provide a list.

This occasionally annoys people who have an idea like "I will firewall
everything but whitelist Let's Encrypt" and then find out that while you're
welcome to try to puzzle out some way to make this work, Let's Encrypt won't
help you and when it breaks you get to keep both halves.

------
rodionos
If you use Route53, I encourage to you to check crt.sh if any SSL certificates
were issued for your subdomain over the last few days.

Example:
[https://crt.sh/?Identity=%.MyEtherWallet.com](https://crt.sh/?Identity=%.MyEtherWallet.com)

Keep in mind that crt.sh has a certain time lag between the time a certificate
is issued and a certificate entry is inserted into the crt.sh database.

~~~
YouKnowBetter
Your advice should get much much more attention. You got mine, thanks.

------
quentusrex
Another write up of the event: [https://blog.cloudflare.com/bgp-leaks-and-
crypto-currencies/](https://blog.cloudflare.com/bgp-leaks-and-crypto-
currencies/)

------
zegl
We experienced the same thing. DNS lookups via Googles public DNS servers
where failing for one of our hosted zones.

Interestingly this only seemed to be a problem with Google DNS, other large
providers as well as smaller ones where not affected.

~~~
fernandotakai
we saw the same thing. dns stopped resolving while using google's dns for
about 2h.

there's a route53 status update on their page that says basically that -- only
affected google dns.

~~~
eastdakota
That’s simply wrong. Best case, this is ignorance on the Route53 marketing
team. Worst, it’s actively misleading. Disappointing either way.

------
textmode
"Between 11am until 1pm UTC today, DNS traffic-the phone book of the internet,
routing you to your favourite websites-was hijacked by an unknown actor."

Is it worth keeping a record of the IP addresses for "your favorite websites"?

For example, with IP addresses saved, would this enable reaching the websites
ven if DNS is not working?

How often do these "favorite websites" change IP addresses?

Are they all the same in that regard?

(The frequency with which they chaange IP addresses.)

Is it worth paying attention when one of them changes its IP address?

Is it worth noting where these addresses are thought to be located (e.g. the
countries)?

What if several of "your favorite websites" each change their IP address on
the same day, the same week or even the same month?

Is this worth noting?

~~~
viraptor
Depends on how the website is hosted. If it's a tiny website on a single
server/VPS, then it's going to be fairly stable with the IP. For anything
else, you're out of luck. Shared hosting or PaaS will change ips when needed
internally, or to rebalance the traffic, or when new hosts come up. Anyone
with large enough traffic will use some edge network like CloudFlare which
would also get a different address depending on your location.

Some IPs may stay stable for a long time, but virtually every change to them
is going to be intended. And you can't tell easily which one want.

~~~
textmode
"Anyone with large enough traffic will use some edge network like Cloudflare
which would also get a different address depending on your location."

What if the request is always sent from the same location?

What if when the user moves location, e.g., from one country to another, she
updates her stored IP addresses?

"And you can't tell easily which one want."

Personal experience as a user is that when it is an "intended" change, the
previous IP address no longer hosts the content.

Personal experience is that this is surprisingly rare.

In the case of the more common load balancing, the user who may store several
IP addresses for the website, all of which continue to serve the content. She
can choose which of those she prefers.

Personal experience is that this works very well.

Further questions:

If this was a BGP hijack, why did the hijackers target the IP addresses of DNS
servers, rather than the target IP addresses of the websites?

If they had targeted the IP addresses of the websites, then what would be the
possible mitigations, if any?

~~~
viraptor
> What if the request is always sent from the same location?

Then you _may_ get the same IP.

> What if when the user moves location, e.g., from one country to another, she
> updates her stored IP addresses?

It doesn't have to be a different country. Different ISPs with different
peering may get different results. Moving between your home wifi and you phone
tethering may get different results. So depending on your usage style, it's
between "change multiple times a day" and "no change".

> Personal experience as a user is that when it is an "intended" change, the
> previous IP address no longer hosts the content.

In some cases I'm sure it's true. In others, it isn't. Basically it's not a
_reliable_ indicator.

> If this was a BGP hijack, why did the hijackers target the IP addresses of
> DNS servers, rather than the target IP addresses of the websites?

Haven't seen the site's announced block, but it's possible that it's announced
as /24 block. This attack was possible because Amazon announced /23 and the
attacker could announce something more specific.

~~~
textmode
"So depending on your usage style, it's between "changes multiple times a day"
and "no change"."

For me, for each ISP, it's been "no change". Can only relate personal
experience.

------
maltalex
Something similar happened once in 2014 [0]

> That BGP hijack allowed the hacker to redirect the miners’ computers to a
> malicious server controlled by the hijacker.

Also, the entire blockchain ecosystem is very susceptible to BGP hijacks [1]

[0]:[https://www.wired.com/2014/08/isp-bitcoin-
theft/](https://www.wired.com/2014/08/isp-bitcoin-theft/) [1]:[https://btc-
hijack.ethz.ch/](https://btc-hijack.ethz.ch/)

------
vasilipupkin
can someone ELI5 how this works? like, how do they physically do this ?

~~~
chatmasta
I’m not an expert, but I think it goes something like this —

You register an AS and get a dedicated server [0] at at an internet exchange
point (IXP) and install a BGP router on it. Your router advertises a route to
the original route53 IP, with a shorter AS path to the target IP than the
route advertised by the “real” Amazon router. You peer your router with at
least one provider who peers with the real route53 servers (they accept your
route advertisements).

Then you redirect incoming traffic on port 53 to a mitm dns proxy, where you
either do nothing or return an IP address you control, which could be
anywhere.

At the IP address you control, you run a transparent HTTP proxy that signs
HTTPS responses with the valid DNS cert you obtained by spoofing MX replies in
the dns proxy. You then either do nothing, or modify the page to return
arbitrary content (like rewriting bitcoin addresses).

[0] or you get a VPS at a provider willing to advertise BGP routes on your
behalf without validating them

~~~
rocqua
Not an expert either, but I think it was slightly different.

Firstly, I believe BGP hijacking works by announcing a more specific prefix
rather than path-length. That is to get to 3.3.3.3 A 3.3.0.0/16 route takes
precedence over a 3.0.0.0/8 route.

Secondly, this attack did not obtain a valid cert. They probably could have
using spoofing (either of MX or just of A records and using lets-encrypt), but
they did not. This might be because the BGP hijack did not affect any CA, or
it might just have been laziness.

~~~
chatmasta
Yeah you're correct. There was a great defcon presentation on this in 2015:
[https://www.blackhat.com/docs/us-15/materials/us-15-Gavriche...](https://www.blackhat.com/docs/us-15/materials/us-15-Gavrichenkov-
Breaking-HTTPS-With-BGP-Hijacking.pdf)

Really the hardest part of the attack, and what makes it so infrequent, is the
social engineering aspect of registering the AS and getting an influential
peer to accept your advertisements. To really cover your tracks you would need
fake identities, fake businesses, fake bank accounts...

~~~
vasilipupkin
what's an AS?

~~~
notriddle
Autonomous System.

Basically, a top level Internet shard. You need to become a Real ISP to pull
off this attack.

~~~
chatmasta
That may be technically true, but not by any layman’s definition of an ISP. An
AS can be a tiny company (I’ve been one) as long as it has some IP space
SWIPed (registered) to it with ARIN/RIPE. All you need is to fill out a form.

Then you need to find other ASes to peer with you of course.

Also note that technically all you need is the ability to advertise routes.
That could be mutually exclusive from AS status if your server provider is
willing to advertise routes from their AS on your behalf (like vultr does, I
think). In that case the provider is acting irresponsibly and should validate
the routes.

------
mercora
I wonder why they did not reroute traffic for their targets directly. I assume
the amount of traffic the Route53 servers get is quite a lot to handle and
failure to do so would make a lot of people aware of some issue. Also, how is
it possible to reach the original servers in order to forward non targeted
lookups to them? Is it possible to have your own routes not affected by your
own BGP announcements? Maybe they were able to do this by using EC2 internal
DNS resolvers?

Maybe Route53 does use individual IP addresses for the nameservers of
registered domains? Or at least are using them hihgly segmented? I guess this
would help to handle the amount of traffic quite a bit if so. Otherwise it
seems inplausible that the mentioned site is the only target or that there was
just a few targets because that should be easier to pull off.

On the other hand i am not sure if these targets might share IP addresses with
a large pool of websites and handling "only" authoritative DNS requests for
them (and any other that happens to have the same IP addresses setup as
nameserver) would be alot easier. How does Route53 handle delegation exactly?
Would i have to handle all NS requests that happens to resolve to Route53 or
just some specific set of domains? Only handling these delegation related NS
requests might reduce the amount of traffic alot in any case. But maybe they
setup glue records for registered domains. In this case the amount of traffic
they would have to handle would be even larger and i guess disrupt quite alot
of DNS clients because they would not be able to sign the zone properly.

I also wonder why they were not able to issue a new valid certificate. This
appears to be very easy even without any interruption at this stage.
Especially this feels like it was only done in order to direct the attention
away.

Hopefully a more detailed writeup appears because this seems rather
interesting.

------
lifeisstillgood
Can I check my understanding

(Somehow) A BGP route was advertised and accepted for the IP range for amazons
route53 - so that some / many DNS requests hit a fake server instead of the
real amazon one. The only dns requests that changed seem to be for a bitcoin
site, and they were redirected to a site that seems to have then grabbed their
passwords and then emptied their wallets on the real site.

?

Mitigation strategies that come to mind

\- certificate pinning (where I have my browser only accept certificates that
I have approved, not just are signed by a CA (this has not really taken off)

\- Certificate warning - surely browsers should shout of a certificate that
was valid for domain X changes today? (suffers similar problems to pinning)

\- BGP route acceptance. We are out of my comfort area now, but this is surely
where the real solution lies - why did people accept a BGP chnage to redirect
route53 to russia? Can routes be advertised with signing ?

~~~
jimktrains2
> surely browsers should shout of a certificate that was valid for domain X
> changes today

Why? If the new cert is valid, why sound any alarms?

~~~
lifeisstillgood
I visit my bank site and my browser gets a certificate for barclaysbank Ltd,
to expire in 12 months. Tomorrow i visit the same site and get a different
certificate, or from a different CA.

Should my browser warn me?

I don't know. the chances are high that 99% of people will click OK and 99% of
the 1% left will look at it and think "how do i verify this?"

That is probably the reason browsers don't bother with pinning and the rest -
there is simply no chain of trust i as an individual can possibly use to
verify a fraction of the sites i visit.

but still ... if I pay my landlord every week in cash and one day someone else
turns up and says "hi i am your landlord, pay me" I have visual signals to
warn me of a possible problem.

The signals are there - my browser could fingerprint the servers, the headers,
the response times, a lot could be done. But then most days my browser would
act like Donald Sutherland and point at people and say "they have been
replaced" and i would have no means to tell.

we really have become reliant on one piece of technology haven't we.

~~~
jimktrains2
> The signals are there - my browser could fingerprint the servers, the
> headers, the response times, a lot could be done.

You realize this stuff can change by the minute, nay, second?

> but still ... if I pay my landlord every week in cash and one day someone
> else turns up and says "hi i am your landlord, pay me" I have visual signals
> to warn me of a possible problem.

No, it's more like you walk into your landlord's office and there is a new
secretary. Sure, usually there won't be, but it's not unheard of for companies
to get new secretaries.

> I visit my bank site and my browser gets a certificate for barclaysbank Ltd,
> to expire in 12 months. Tomorrow i visit the same site and get a different
> certificate, or from a different CA.

> Should my browser warn me?

If the new cert is valid, and there are no CAA records (which, admittedly a
dns hijack would make worthless), then why should the browser warn you?
Everything is valid, what's wrong?

What you should be asking yourself is why we don't use mutual authentication
for banks. The answer is banks think we're all idiots and couldn't handle it.

------
chatmasta
What separates BGP hijacking from BGP anycast? Amazon is already advertising a
bunch of routes to its route53 IP's for BGP anycast. How can an upstream peer
tell if a new advertisement is another anycast route vs. a hijacked prefix? Is
the only way to tell by validating the advertisement came from Amazon?

~~~
xraystyle
The issue here is that the attackers were advertising more specific prefixes,
and when doing route selection, routers pick routes to these over less
specific prefixes that cover the same ranges.

e.g. if Amazon was announcing 205.51.192.0/21, that would cover all the /24s
that were announced in the hijack. Not sure what Amazon actually announces,
this is all just examples.

Say I'm trying to get to 205.51.192.12, for example. If I have two routes, one
advertising 205.51.192.0/21 and another advertising 205.51.192.0/24, the
router will forward the traffic to the next-hop advertising the /24, all else
being equal.

So if you have the ability to advertise routes over BGP to the internet, you
advertise a more specific route than what someone else is already advertising,
and your peer has no filtering in place with regard to what they're accepting
from you, you can potentially hijack IP space like this.

ETA: Anycast DNS generally advertises the exact same prefix to multiple peers.
That way, the best path to that IP will be dependent on where your traffic is
coming from, and will hopefully come in over the closest peering to you.

~~~
lathiat
As much as it's bad for routing table inflation, given how much content they
host Amazon should probably consider de-aggregating their DNS resolver
prefixes to /24 announcements which would limit the effect to be much more
local for the majority of the Internet.

------
dharma1
Also experienced this today - was doing user testing on an app, and suddenly
our AWS hosted app that uses Route 53 stopped working, didn't resolve. When I
checked [http://status.aws.amazon.com/](http://status.aws.amazon.com/) it
showed all green for Route 53 though.

Luckily I noticed the app kept on working on my own devices where I had
previously switched the DNS to 1.1.1.1 (Thanks HN for the heads up on that a
few weeks ago!) - otherwise would have been a waste of half a day in an
expensive user testing lab.

Assumed it was Putin's blocking of Amazon and Google IP's that had been on the
news, but guess it was just hackers after some ETH. Or was it?

------
pierd
The attackers are getting richer:
[https://etherscan.io/address/0xb3aaaae47070264f3595c5032ee94...](https://etherscan.io/address/0xb3aaaae47070264f3595c5032ee94b620a583a39)

------
r1ch
It's sad that HPKP is being deprecated. It's one of the best ways to defend
against an attacker with bgp hijack capabilities assuming you pin your own
public key. Difficult to scale though and prone to disastrous
misconfiguration.

~~~
tomc1985
Too many people have foot-gunned themselves with that one. As someone who has
to readily fight technical fires for other people (when they should have
googled their problems) I am glad to see HPKP go.

~~~
dcbadacd
Honestly there are a lot of ways for people to foot-gun themselves when
dealing with providing a service. HPKP isn't perfect, but deprecating it just
for the sake of keeping some incompetent people safe is just silly. Might as
well remove HSTS and not roll out Expect-CT because everything allows someone
to foot-gun themselves.

~~~
pfg
The difference between HSTS, Expect-CT and HPKP is that the former two offer a
way out (support HTTPS, provide qualified SCTs) whereas HPKP can effectively
brick your domain for a couple of months, and it's not even hard to pull off.

~~~
dcbadacd
HPKP can brick your domain if you use HSTS alongside with it, if you don't
then one can fall back to http. Not an ideal solution, but it's a way out.
HSTS is an extreme and possibly dangerous measure that should be carefully
considered (maybe it shouldn't be TOFU but TOTU - trust on third use), but I
still think it's not reasonable to totally deprecate it. There are cases where
such measure would provide clear and strong security benefits.

------
adamb_
Route 53 DNS queries getting black-holed at a small ISP in Ohio:
[https://pbs.twimg.com/media/Dbka0oAVMAAFEXx.jpg](https://pbs.twimg.com/media/Dbka0oAVMAAFEXx.jpg)

------
donttrack
Isn’t dns hijacking a major problem? I could go to a Starbucks and setup my
own dhcp server on my laptop. A certain fraction of the WiFi devices would
probably pick up my dhcp given dns which would reroute requests to my own dns
server. Nobody would ever know... or am I misunderstanding something about
https?

Or if I was a malware writer I would discretely target hosts files or the
local dns configuration - exploits in certain routers to try to reconfigure
the complete local network and such.

~~~
jmanderley
Let's hope your local Starbucks has device isolation enabled so that isn't
possible.

~~~
donttrack
I will find out and report back.

------
rodionos
> which served the website using a fake certificate

This would cause an untrusted certificate error in browsers.

> failed to obtain an SSL certificate while man-in-the-middle attacking

The sentence doesn't make sense. MITM is by definition an attack where an
actor was able to impersonate the target. In this case, they presented an
certificate that was not signed by a CA in the root CA file on the clients. So
this can be qualified as a failed MITM attempt.

------
profalseidol
Time to use the Ethereum Name Service (ENS). As long as you keep your private
key safe, your domain name is safe.

------
mcqueenjordan
What's with the clickbait title? It is simply a falsehood.

Here's the update from the horse's mouth:

[5:19 AM PDT] We are investigating reports of problems resolving some DNS
records hosted on Route53 using the third party DNS resolvers 8.8.8.8 and
8.8.4.4 . DNS resolution using other third-party DNS resolvers or DNS
resolution from within EC2 instances using the default EC2 resolvers are not
affected at this time.

[5:49 AM PDT] We have identified the cause for an elevation in DNS resolution
errors using third party DNS resolvers 8.8.8.8 / 8.8.4.4 and are working
towards resolution. DNS resolution using other third-party DNS resolvers or
DNS resolution from within EC2 instances using the default EC2 resolvers
continues to work normally.

[6:10 AM PDT] Between 4:05 AM PDT and 5:56 AM PDT, some customers may have
experienced elevated errors resolving DNS records hosted on Route 53 using DNS
resolvers 8.8.8.8 / 8.8.4.4. This issue was caused by a problem with a third-
party Internet provider. The issue has been resolved and the service is
operating normally.

~~~
jsmthrowaway
The horse's mouth, in this case, speaks nonsense. Multiple /24s were announced
more specifically _to the DFZ_ , and that has _absolutely nothing_ to do with
Google's DNS resolvers. "Simply a falsehood," indeed.

[https://twitter.com/InternetIntel/status/988792927068610561](https://twitter.com/InternetIntel/status/988792927068610561)

[https://arstechnica.com/information-
technology/2018/04/suspi...](https://arstechnica.com/information-
technology/2018/04/suspicious-event-hijacks-amazon-traffic-for-2-hours-steals-
cryptocurrency/)

~~~
lathiat
The horses mouth actually does say it, you'll note they said

"This issue was caused by a problem with a third-party Internet provider."

As soon as I read this and saw the myetherwallet announcement I called it last
night that's exactly what they did, and today I find out that was the case.

------
kentt
Wow myetherwallet is trusted for transaction very non-trivial amounts. What
would a good mitigation strategy be for these types of attacks.

~~~
artursapek
Making sure the cert is valid.

~~~
quickthrower2
Also do a dry run before each usage, generating a throwaway wallet, and
inspect the network tab / use fiddler to check if there is any suspicious
sending of data.

Even this isn't 100% as they could seed keys if bad person gets control of the
website. This has been done many times in the IOTA community for example,
albeit those sites were dodgy from the get-go.

So I'd probably get a snapshot of their client code when you trust it, and
serve it to yourself locally thereafter.

It's a sad state of affairs that people use services like this. I have done so
myself :-(

The reason is that the Ethereum desktop clients suck. More than suck, they are
untenable. The official client never synced even after running for 2 weeks. I
tried this twice. It is awful.

~~~
g09980
Also, you can save the MEW webpage locally and use it.

------
pierd
Is there any official post mortem available?

------
ed_balls
How does one prevent an attack like this? APIs and Browsers.

------
unabridged
With all the talk of blockchain, dns should be one of the best uses.

------
technologyvault
Does anyone know if this happened to affect Amazon's Seller Central?

Amazon sellers (including me) couldn't access their control panels for several
hours last night.

------
solotronics
was this related to the myetherwallet.com hack? they hijacked BGP and DNS to
redirect traffic to their own page

[https://altcoinreport.co/my-ether-wallet-
hacked/](https://altcoinreport.co/my-ether-wallet-hacked/)

~~~
AlphaWeaver
Yes, the article says the only compromised site they are aware of was
MyEtherWallet.

------
OJFord
Surely related (and warning!):

During this period I received an email from:

    
    
        ship-confirm@amazon.com
    

purporting to notify me of the despatch of a non-existent order; with an
attached .zip.

Nothing about the message, except the .zip and suggestion of an order's
existence, was at all suspicious - all links point to amazon.com.

I checked on amazon.com (without following a link) and on amazon.co.uk (where
I would usually shop) and an order with the given number does not exist for my
account.

~~~
ars
That's just regular spam. I bet if you checked you would see the email was not
actually sent by Amazon, and did not have a DKIM signature.

~~~
OJFord
Maybe. I've deleted it now, but the headers looked OK to me. If I'd actually
ordered something (e.g. it had come a day earlier) I'm sure I'd have taken it
as being legit.

