
Multi-Region Latency Based Routing now Available for AWS - jeffbarr
http://aws.typepad.com/aws/2012/03/latency-based-multi-region-routing-now-available-for-aws.html
======
ryanbales
I am utterly floored by the frequency at which new features are being released
for AWS... Are you Amazon people getting any sleep?

~~~
jeffbarr
Five hours does it for me. And I have to write all of those posts!

As one of my colleagues mentioned, we are hiring (currently 516 open jobs on
the AWS team). Here's a list that's easier to read:

<http://awsmedia.s3.amazonaws.com/jobs/all_aws_jobs_list.html>

~~~
jedberg
And if you want to consume the fruits of their labor, Netflix is hiring too!
<http://jobs.netflix.com/>

(Sorry Jeff, I couldn't resist)

~~~
werner
And if you are looking for really cool jobs in which you can apply all your
distributed systems and cloud skills on amazingly interesting products, beyond
just video streaming, check out Amazon.com (75+ pages of 20 jobs per page in
software development alone)...

[http://www.amazon.com/gp/jobs/ref=j_sq_btn?keywords=&cat...](http://www.amazon.com/gp/jobs/ref=j_sq_btn?keywords=&category=Software+Development&location=*&x=43&y=14)

(sorry Jeremy, coudln't resist) :-)

~~~
glenngillen
Well given we're doing this here now for anyone that takes advantage of AWS...
;)

I'm looking for engineers and UX people to come help me make the Heroku Add-
ons platform even more amazing. If putting these cloud services into the hands
of developers and changing the way people think about provisioning these
services sounds like something you'd like to be part of: glenn at heroku dot
com

~~~
lukevenediger
What's it worth to do great work if you can't live in a great city? Amazon's
Development Center in Cape Town, South Africa is situated in the heart of the
Mother City, and is surrounded by the Atlantic Ocean and Table Mountain - you
can't get better views. Combine this with never-ending beaches and sunny
weather and you have the perfect work and play environment.

We build software for AWS and are looking for all sorts of engineers: from
kernel development to building great web front-ends for our customers. Check
out <http://www.amazon.co.za/> for more info.

------
gibybo
So I'm a little fuzzy on how DNS works and this seems like a good place to
ask:

I was under the impression that end users typically talked to DNS cache
servers rather than directly to the authoritative servers in the domain's
registration. If that's true, how can AWS provide different records based on
the requesting user?

If end users are talking directly to one of the ~4 authoritative name servers
listed in the registration, how does that scale to billions of queries?

~~~
mcpherrinm
The most important thing going on here is Anycast. See
<http://en.wikipedia.org/wiki/Anycast> for a complete description, but
basically, multiple machines have the same IP address, and different routes
are advertised to different networks. This results in packets being routed to
the nearest DNS server.

Then, that DNS server can decide where the other end is and provide them with
the correct IP.

You're right that most users use a caching DNS server, so it is actually the
location of the DNS caching server. Of course, since most users choose a DNS
server close to them (usually, their ISP), this should still result in a
correct approximation. If you're using, say, 8.8.8.8 (google public dns),
that's (probably) more than 1 server -- you're using the closest one to you.
So then AWS will provide you with the closest AWS region to the closest Google
DNS server, which is hopefully the closest one to you.

~~~
isb
As mcpherrinm described, anycast plays a big part in how queries are routed to
a DNS server in Route 53's global fleet.

You are right that a DNS server sees the query coming from the resolver
instead of the user. So how does it pick the region closest to the user? As
the blog post describes, we measure latencies from client networks to AWS
regions and we also have a mapping of which resolvers are used by which client
networks. If you put both of them together, you can compute which region is
closest to the users of the resolver.

~~~
nl
_we also have a mapping of which resolvers are used by which client networks_

How did you build this map?

I can think of a few complicated ways to go about it, but I'm wondering if
there is something easy I'm missing.

~~~
colmmacc
Take a look at <http://whatsmyresolver.stdlib.net/> for one simple way.

Full-disclosure: I work on Amazon Route 53, and although we don't quite use
that same method - it will give you an idea of what's possible. PS; we're
hiring.

~~~
nl
How does it work?

1) I visit the site, it gets my IP address

2) Magic happens

3) It displays my nameserver address.

What is going on in step 2?

Edit: worked it out.

For those interested, it uses a Javascript include from a unique subdomain
name. Because the subdomain is unique the app can work out the relationship
between client IP and resolver.

~~~
colmmacc
The html page at <http://whatsmyresolver.stdlib.net/> loads a javascript
script from the url <http://whatsmyresolver.stdlib.net/resolver/>.

Fetching <http://whatsmyresolver.stdlib.net/resolver/> triggers a 302 redirect
to a url of the form;

    
    
      http://$guid.nonce.stdlib.net/resolver/
    

The DNS server authoritative for nonce.stdlib.net has a simple wildcard
configured, so *.nonce.stdlib.net all resolve to the same web-server.
Obviously the DNS request for the globally unique id domain name has to come
before any HTTP request to the guid url, so when the DNS request comes in the
authoritative server can record it in a simple lookup store (guid -> resolver
source ip).

Then, when the HTTP request makes it to the web-server, it can inspect the
Host: header to determine what the guid was. It then uses this guid to
correlate the HTTP request it is handling and the resolver source ip, and
generates some javascript with the data we need;

    
    
      var resolver="192.0.2.53";var edns=true;
    

It's just a hack I wrote up for my own reasons years ago. But if you'd like to
avail of it for any reason (ie helping end-users debug things), feel free to
embed;

    
    
      <script language="javascript" src="http://whatsmyresolver.stdlib.net/resolver/"/>
    

and use the variables it populates. No warrantees or guarantee implied :-)

------
kogir
Amazon is such a tease. Just give us Anycast Elastic IPs that work in multiple
regions already!

There's still no good high-availability story for region-wide outages, and
there won't be until they do this.

~~~
jscott11
Seems unlikely. Not every AWS user is in every region. They would need to have
a separate range of anycast IPs for every possible combination of regions (2^N
possibilities) to prevent packets from going into a black hole. If they wanted
to be able to serve a conservative number of hosts in every region (say 32768)
that would require 15-bits of IP space per combination. If AWS expands to 10
regions, that means 25-bits of the IPv4 space. A /8 only gives you 24.

I guess an alternative might be to do some kind of NAT for unbound anycast
addresses which forwards packets to an available region, but that is hugely
complicated.

~~~
kogir
I'd be delighted if they offered it for even just two regions. Right now if
you want your service to be highly available you either run your own routers
in two collocation facilities and forward to AWS, or you don't use AWS.

------
donavanm
I created a quick demo at <http://region.strewth.org/> and a introspective TXT
record at region.strewth.org. More details <http://www.strewth.org/words/>.

------
sargun
This has been one feature that I've been waiting for a while for a while.
Otherwise, multi-region load balancing had to be done in the application, via
geo-IP, or a third-party provider.

The only thing I'm curious as to is what kind of measurements is Amazon
gathering, and how is it gathering them? Is it using ELB, and looking at TCP
latency (delta time between SYN/ACK? Curious minds want to know...

------
ez77
Sorry for bringing this off-topic issue on AWS: is it possible to set hard
caps on monthly spending in AWS? I just asked this [1] but didn't receive a
single upvote =S. Thanks.

[1] <http://news.ycombinator.com/item?id=3737595>

~~~
ryanbales
No. Just keep an eye on it.. it's cheap :p

~~~
esrauch
Except if you go on vacation to unplug for a week and had some bug or
misconfiguration couldn't you easily run up a bill in the hundreds or
thousands when you expect it to be in the tens?

If this is so implausible then why can't they provide a cap? It feels to me
that they are purposefully trying to capitalize on people's mistakes. Call me
paranoid but it is honestly the primary reason why I've been wary to play with
AWS.

~~~
ryanbales
Most sites with the kind of traffic that could potentially run up the bill
thousands of dollars will have checks in place to ensure a mistake like that
wouldn't happen. Anything else would be irresponsible.. I'd challenge your
assumption that Amazon is intentionally trying capitalize on mistakes... A
company a big as Amazon doesn't have the time or need to even consider
something like that.. And yes, I think you're being paranoid.

~~~
ez77
I don't think Amazon is being dishonest, but you can see that, regardless the
individual degree of paranoia, there are quite a few small developers who
don't try AWS just to be safe.

My point is that it makes business sense for Amazon to lure this old-fashioned
crowd, and it could do so by implementing a simple feauture such as spending
control.

