
The Amazon Builders' Library - mooreds
https://aws.amazon.com/builders-library/
======
azinman2
Wow. I haven’t gone through most of the content but shuffle sharding not only
is a great idea but their write up is quite clear:
[https://aws.amazon.com/builders-library/workload-
isolation-u...](https://aws.amazon.com/builders-library/workload-isolation-
using-shuffle-sharding/)

Impressive stuff!

~~~
t0mas88
Yes, that's a great one. But now I have a brain teaser stuck in my head:
Imagine you have a multi-tenant system like they describe, but for each
customer A, B, C they have users A1, A2, A3, B1 etc.

You need ordered delivery of messages, but only at user level (if all messages
needed to be ordered you could not do this sharding type setup). Is there a
way to apply shuffle sharding while maintaining user-level ordering
guarantees?

~~~
colmmacc
Yes! we do recursive, nested, shuffle sharding in these cases. Our Open Source
infima library doesn't quite include direct support for this, but you can get
a sense of how we put it all together:

[https://github.com/awslabs/route53-infima](https://github.com/awslabs/route53-infima)

~~~
t0mas88
Cool, thanks!

------
appwiz
Colm and Marc frequently post great threads about distributed system design on
Twitter. Definitely worth a follow:
[https://twitter.com/mndoci/status/1202641319643869184](https://twitter.com/mndoci/status/1202641319643869184)

And on a personal note, I’m astounded and so lucky that I get to work with
these amazing engineers!

------
planckscnst
I've worked in AWS for five years. I get paid, but the kind of knowledge in
these pieces are by far the most valuable thing I've gotten from working
there. I've been in it. I've used these principals. I've seen what happens to
drive these recommendations first-hand. I've worked to design and build
systems that follow and support these guidelines and principals.

------
gokulj
Amazingly little to no use of images in the articles. Turning the 'an image is
worth a thousand words' on its head. Seems to work well though. Examples:
[https://aws.amazon.com/builders-library/avoiding-
insurmounta...](https://aws.amazon.com/builders-library/avoiding-
insurmountable-queue-backlogs), [https://aws.amazon.com/builders-
library/caching-challenges-a...](https://aws.amazon.com/builders-
library/caching-challenges-and-strategies/)

~~~
dyanacek
Good point. We should use more images. I’ll think about where we can add some
for clarity! Now that we’ve done a bunch of talks at reinvent in these topics,
we should have some that we can incorporate easily.

------
ignoramous
Disclaimer: ex-AWS

Wowzer. Amazon, and in particular AWS, sits on a treasure trove of hard-earned
lessons abt designing, building, operating, maintaining systems at scale. I'm
glad they have began to share _some_ of that know-how with the external world.

I wrote abt longing for AWS to do this some 6 months back [0] as I dearly
missed access to those docs and videos that helped me understand so much abt
the underpinnings of some of the most staunchest challenges dealt with by
behemoth-esque services like S3, EC2, Redshift, DynamoDB, Kinesis, LB, Lambda,
Edge, Security to name but a few.

I must point out though, re:invent does have a _ARC_ track [1] which gives
glimpses into AWS' design principles.

There's so much they could still share (for instance, abt how they track and
run their operations, how they deal with service outages, how they do change
management, how they continuously monitor and improve the weaker and slower
parts of the systems, how they gear up for launches, how they deal with issues
facing larger customers, how they go abt their design process, how they refine
their internal tools and processes, the internal tech talks, the review
meetings esp the one run by Charlie Bell...), and I'm glad they've committed
to be doing just that.

It'd be fun comparing literature coming out of Amazon to what's published by
Google et al.

[0]
[https://news.ycombinator.com/item?id=20251589](https://news.ycombinator.com/item?id=20251589)

[1]
[https://www.youtube.com/playlist?list=PLhr1KZpdzukfzSjpgH4E5...](https://www.youtube.com/playlist?list=PLhr1KZpdzukfzSjpgH4E5PiDwBM2o4UHD)

~~~
packetslave
why do you keep typing 'abt' instead of 'about'? what are you planning to do
with all the time saved?

~~~
ignoramous
Isn't _abt_ [0] a generally accepted abbrv for _about_ [1]?

[0]
[https://www.dictionary.com/browse/abt-?s=t](https://www.dictionary.com/browse/abt-?s=t)

[1] [https://www.abbreviations.com/ABT](https://www.abbreviations.com/ABT)

~~~
SAI_Peregrinus
It's understandable, but very rare in most text. It's the sort of thing you'd
expect in a telegram, not a modern forum comment.

~~~
shpongled
Especially when they take the time to write "management", instead of "mgmt".

------
dint
Does anyone know why Amazon engineers always seem to write so well? Any time I
read technical content from Amazon, I am struck by how clear and concise it
is.

~~~
kevan
Amazon has a very strong written culture (search for 1 pagers and 6 pagers).
To be effective at higher levels in any role you need to be good at it.

~~~
social_quotient
Any tips on how to get better at this?

~~~
hartzell
Write more _and_ find ways to get feedback on your writing. It's important to
get outside input to help you see the things that don't work and experienced
readers/editors/friends/coworkers can often provide suggestions on how you can
fix it.

~~~
sokoloff
This. and: It really helps if you work in a culture that values critical
feedback on thinking and writing. Too many readers will pass on an opportunity
to offer critical feedback, either because they don’t want to offend or
because they didn’t read it carefully and critically enough. Having someone
read a doc and tell you it’s good feels good, but in general doesn’t help you
improve (and is actively harmful if the document is mediocre).

One way to encourage this helpful criticism is for authors to accept feedback
gracefully and have a policy to ask questions _only_ for clarification of the
feedback (to understand the feedback better) and never to engage in the debate
response of explaining why what they wrote is good enough.

(The minute you debate them on the first of seven points they have, they’re
going to deprive you of much of the value of their feedback on the next six
because you’re seeming defensive at worst or exhausting at the least.)

------
pacohope
Reading this snippet in the article about leader election in distributed
systems: "Each item of data still belongs to a single leader, but the whole
system contains many leaders.". It could not be a clearer statement of
Conway's law[0]. Orgs ship their org chart. The technical architecture at AWS
absolutely reflects the org chart and business organisation of Amazon
generally.

[0] [https://www.techopedia.com/definition/31927/conways-
law](https://www.techopedia.com/definition/31927/conways-law)

------
TheSoftwareGuy
Lot of amazon content in the past week or two. What's going on? did they have
a keynote or something?

~~~
doctoring
AWS re:Invent
([https://reinvent.awsevents.com](https://reinvent.awsevents.com))

------
Pfhreak
It's great to see some of the stuff that I used while at Amazon becoming more
publicly available. Good documents, these.

------
diehunde
Is that something new? Or they just compiled all their tutorials in one place?

~~~
bbgm
This just got released. It’s a living source authored by senior engineers at
Amazon and covers some of the best practices that we use at Amazon.

------
kevan
I'm really glad this is public now, really good experience-driven (often
straight from major post-mortems) recommendations here.

------
rajul
Some real gems there.

------
whoisjohnkid
yeah this is great stuff!

------
dang
Lists don't make such great HN posts, because the discussion inevitably
becomes about the lowest common denominator of the items on the list. The
interesting material is in the specific items on the list, but these never
escape the gravitational pull of the generic topics (in this case Amazon and
AWS). It's usually better to pick the most interesting thing on the list and
submit that instead.

[https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20denominator%20list&sort=byDate&type=comment)

~~~
jedberg
Normally I would agree with you, but the existence of the library was the most
significant announcement of today's re:invent keynote. So the existence of the
list, and the fact that the list will keep growing, is the news here.

One option might be to change the URL on this submission to the blog post
about the list instead:

[https://aws.amazon.com/blogs/aws/check-out-the-amazon-
builde...](https://aws.amazon.com/blogs/aws/check-out-the-amazon-builders-
library-this-is-how-we-do-it/)

~~~
dang
That's a fair point. Thanks for the correction!

