
Uncomfortable AWS Truths - mooreds
https://twitter.com/QuinnyPig/status/1173367909369802752
======
mabbo
> AWS purportedly puts design documents forward in the form of six-pagers.
> They start meetings with a 20 minute silent reading session. It's like the
> book club from hell.

Not just AWS- that's an Amazon-wide technique. And it's freaking amazing. You
should try it.

Look, there's two other options. 1- let someone bullshit you with PowerPoint
for an hour and skim over critical details. 2- send out the doc ahead of time
for everyone to read before the meeting and have every single attendee say "ah
yeah I only had time to skim it, but looks good to me". Wasted time.

The Amazon design review meeting involves no bullshitting, no homework before
the meeting that gets skipped. We all arrive. We read the entire doc, on
paper, red pens in hand. Then we dive into questions on the important bits and
anything circled by those red pens.

Generally takes a lot less time as far as meetings go.

[Bias note: I've been at Amazon, but not AWS, for 7 years. These are my own
opinions and not company statements]

~~~
falcolas
Perhaps I’m just a particularly slow thinker, but 20 minutes is not enough
time for me to read 6 pages of highly technical documentation and consider all
of the implications and consequences of the proposed technical design. Even a
full one hour meeting is insufficient. I need at least a day. I don’t want to
come out of the meeting and two hours later have a “L'esprit de l'escalier”
about a flaw in the design I just approved come up.

If I’m being abusively frank, if you can’t be trusted to read a 6 page
document the day before a meeting and put some serious thought into it, you’re
not doing your job.

~~~
grogenaut
Typically a 6 Pager is:

1 Page of Press Release and FAQ

2 pages of arch overview

2 pages of business justification

1-2 pages of appendix

Or some mix thereof. 20 minutes is a bit short but doable. Generally at 20
minutes the person running asks "need more time" and it can go longer.

For a design review I ask for at least 2 hours for the meeting. Or more.

Doing everything in the meeting greatly respects everyone's time. It's a thing
I'm trying to get evenly applied at the subsidiary I work at as often we don't
as people have trouble scheduling the meeting.

~~~
falcolas
See, that’s what I don’t get. A 2 hour synchronous meeting, to me, is a waste
of resources. (side note: If you’re going to destroy an engineer’s morning (or
afternoon) with a 2 hour meeting, you may as well make it a 4 hour meeting and
jam out a POC.)

A day (or week) to review on their own availability, plus the asynchronous
feedback during that day (or week) means that the synchronous meeting can
conclude in 20-30 minutes (if its necessary at all), with more thought put
into it than any two hour review.

In the end, the asynchronous method of syncing state between people may take
more overall time, but that time is taken from when an person is naturally not
engaged with their current project. A meeting, on the flip side, occurs
whether they were in the zone or not.

~~~
pas
Yes and no. In theory the asynchronous mode is more efficient, but then it
might mean you never get the ACKs from all the people.

If the async mode works, no need for the meeting, after all do it over email
or in GDocs.

The problem with the sync mode is that inevitably you're wasting someone's
time, at best only yours as you wait for others to read.

------
Scoundreller
> Despite no fewer than 6 attempts to patch the Open S3 Bucket problem, it
> remains. You can't patch people--legally, anyway.

Heh, I use S3 for hosting static sites only.

2 weeks ago they sent an email saying "[...] your AWS account xxxxxxxx has one
or more S3 buckets that allow read or write access from any user on the
Internet."

And I go "oh shit, what has write access set up?".

But nothing did. When they said "read or write", they meant 1, the other, or
both. They just sent the same ambiguous email to everyone.

~~~
jjoonathan
That's so AWS it hurts.

"Surely this AWS service can't be as poorly integrated with that other AWS
service as it seems, because if it were that poorly integrated, it would be
almost completely useless."

"Oh. It is. FML."

------
vijucat
Most businesses can probably run on a single server from OVH (see Paul Tyma's
Mailinator architecture for inspiration: 1.2 billion emails on a single
server, [http://highscalability.com/mailinator-
architecture](http://highscalability.com/mailinator-architecture)), but I
suspect a lot of folks in tech want to pad their resume with cloud buzzwords,
so they recommend overcomplicated architectures to business that they
absolutely wouldn't spend their _own_ money on. So much of tech design, nay,
tech thinking, is based on following fashion trends, it's shocking.

~~~
erik_seaberg
It's really good machine sympathy for keeping the unfunded service cheap to
run, but it's one bad fan bearing away from losing a _lot_ of messages, so I
wouldn't say its design assures four nines. And the optimization work to get
all the way down to n=1 wouldn't have paid off if Tyma needed to hire someone
at market rate to build that and maintain it, because an engineer-month costs
hundreds of instance-months.

~~~
parliament32
Honestly physical servers have better uptime than some of amazon's regions.

~~~
erik_seaberg
YMMV. I've seen an adtech company who purchased hardware to install in colos,
and a couple of years later their failure rate was almost 5%/year. I would
never bet on four nines from a single commodity box.

------
giancarlostoro
> Amazon's managed ElasticSearch offering is awful because it's ElasticSearch.

Had I never used ES or ELK I wouldnt even bat an eyelash. But man... This one
hurt me in the tech feels. I already dont like ES when its on premise, I cant
imagine on the cloud where you have even less control of it.

~~~
idnefju
Is ES one of these necessary evils? I have been looking for a product similar
to it, but it seems ES is always recommended.

~~~
falcolas
The problem with ElasticSearch is that it does multiple things at once. It
indexes documents. It stores documents. It searches indexes. It serves
documents. The indexing and searching should be separated from the storage and
serving at any kind of reasonable scale.

But, ES makes it “just work” at the proof of concept and low volume stages,
and you can’t easily back out of it when you reach its limitations.

~~~
castillar76
For all that people (rightfully) give them grief for being expensive, this is
something Splunk got right. You _can_ put your entire Splunk infrastructure on
one host: like stem cells, a single Splunk box can perform any of the required
functions (ingest, index, search). But you can also separate out each layer
into separate hosts and then cluster them, so you have clusters of ingesters
feeding clusters of indexers, accessed by clusters of search heads. That
should be the architecture ES aims for, where people are given the flexibility
to take whichever function is currently a pain point and break it out to a
separate set of nodes.

------
Scoundreller
> If you've got an old account charging you 22¢ a month, don't get mad. Start
> a snarky Twitter account and make sure you cost them orders of magnitude
> more than that in doing damage control each month instead.

I use S3 for hosting static sites only, and only in North American zones.

Some time ago, I see some billing lines stating:

"US West (Oregon) data transfer to Asia Pacific (Tokyo) 0.001 GB $0.01"

I had no idea why I'd be paying for transfer to a zone outside my own.
Obviously I don't care about the 1cent, but my small problem may be someone
else's big problem.

Instead of looking into it, they refunded me a month of service (a few
dollars).

I guess that's the opposite of @QuinnPig's thought, but seriously, what was
the charge for? Someone running their own crawler on EC2 so I paid for
internal DC-DC charges?

~~~
wj
I saw that too and figured it was due to Cloudfront.

~~~
Scoundreller
Not using Cloudfront though.

------
ISL
' "AWS stole our open source project and turned it into a service!" is the
rallying cry of people who suck at business models. '

Thanks, @QuinnyPig. I needed that laugh today.

------
Scoundreller
> If you're a GM of a service at @awscloud, and you price it at a simple fixed
> fee of only $X per month, you can expect to be walked out that day.

As a micro-customer, I like the "Only pay for what you use" model.

But AWS charges fixed fees for Route53. You have to pay 50 cents/month/domain
when hosting static sites. My volume is small enough that I pay 4x$0.50/month
for Route53 DNS, and like 29 cents total for the actual storage/transfer.

The profit margin on that 50cents/month must be 99%.

~~~
revicon
Why use route 53 vs your registrar’s dns? Or use Cloudflare, it’s free

~~~
Scoundreller
I thought I couldn't use my registrar's DNS unless I wanted it to redirect to
the weird URL for the s3 bucket. But maybe I've got it all wrong and just took
someone's S3 static site build instructions too literally 5 yrs ago.

~~~
revicon
no, you can set your cname to the s3 bucket's url in your registrar's dns
settings...

[https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosti...](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingCustomURLs)

You can also use cloudfront to do SSL if you need it.

Note: If you're really only need to host a static website and want SSL for
free, Github static hosting will give you all that for free.
[https://pages.github.com/](https://pages.github.com/)

------
etaioinshrdlu
Corey Quinn is my spirit animal. He calls himself a stand-up cloud comedian.

------
rgoldste
For the uninformed - myself included - Who is this guy?

Is or was he ever affiliated with AWS?

~~~
jdonald
> Is or was he ever affiliated with AWS?

Does not appear to be affiliated, but launched "Last week in AWS" more than 2
years ago so may know a thing or two.

[https://www.linkedin.com/in/coquinn/](https://www.linkedin.com/in/coquinn/)

[https://www.linkedin.com/pulse/last-week-aws-corey-
quinn/](https://www.linkedin.com/pulse/last-week-aws-corey-quinn/)

[https://www.lastweekinaws.com/](https://www.lastweekinaws.com/)

~~~
QuinnyPig
I have never worked at AWS, largely due to my personality.

~~~
nixpulvis
I think I'd currently have trouble working for AWS as well...

~~~
QuinnyPig
[https://www.lastweekinaws.com/blog/why-i-turned-down-an-
aws-...](https://www.lastweekinaws.com/blog/why-i-turned-down-an-aws-job-
offer/) Same. Same.

------
peterwwillis
Best takes (so far):

    
    
      - Nobody has figured out how to make money from AI/ML other than by selling you a pile of compute and storage for your AI/ML misadventures.
      - "AWS stole our open source project and turned it into a service!" is the rallying cry of people who suck at business models.
      - Amazon's managed ElasticSearch offering is awful because it's ElasticSearch.
      - A major reason to go public cloud that @awscloud can't say outright is "you people freaking suck at running datacenters."
      - Route 53 isn't really a database, but then again, neither is Redis.
      - MultiCloud is a good idea if you're tetched in the head; it treats cloud solely as "a place to run a bunch of VMs." If that's all you're doing, go you I guess. Bring money!
      - Reserved Instances are the best way to take the on-demand promise of the cloud, and eviscerate it completely by forcing customers to think of it like it's an ancient datacenter. "Enjoy your three year planning cycles, schmucks!"
      - Baby seals get more hits than the [AWS] forums do.
      - "You should deploy everything to be HA across multiple regions" is the rallying cry of armchair architects who don't pay their own AWS bills by a long shot.
      - "What does AWS have that GCP doesn't?" "A meaningful customer base"
      - There's only one place to see every resource in your AWS organization, in every region: the AWS bill.
      - DocumentDB isn't a perfect MongoDB clone yet, and can't be until it's just as good at trashing your production data.
      - Netflix has assembled many of the most brilliant engineers on the planet so they can... use @awscloud to stream movies. Draw your own conclusions.

~~~
cwyers
My favorite, although one I have to take with a grain of salt:

"Despite all of the attention Serverless, AI/ML, etc. get on stage, the
_majority_ of AWS's income comes from EC2."

------
djsumdog
> The purest form of "static site" is the @awscloud status page

I'm convinced every companies status page is just static content hosted from
an S3 bucket.

~~~
StavrosK
I mean, that's what it's supposed to be, except the joke is that AWS uploaded
their page once in 2004 and never changed it since.

------
erichurkman
How many retweets will it take to get this for GCP?

~~~
akhilcacharya
I just retweeted it just for that

------
SomaticPirate
Lets see how far he gets

~~~
mooreds
I wonder if the snark well will run dry. Hope not.

~~~
mjulian
Corey's business partner here. I can attest: it does not. I sometimes wish it
would...

~~~
QuinnyPig
Half of it's trying to get your humorless stoneface to crack a smile. We're
approaching dangerous levels of snark, with limited success to date...

~~~
ethbro
I optimized costs by replacing my face with a static site.

------
eropple
I'm really here for the dunking on IBM Cloud. Five months there stole literal
years from my life through the power of condensed frustration.

And it still ain't a real cloud.

------
yodon
> [https://totes-not-amazon.com](https://totes-not-amazon.com)

It's like the Git Man Page Generator [0] but trained on the AWS docs. Each
time you click the "ASW" home page logo, it regenerates the docs.

[0] [https://git-man-page-generator.lokaltog.net/](https://git-man-page-
generator.lokaltog.net/)

------
uikoeixueo
As this thread will draw people that know 'the cloud' I'm wondering if I could
get experience learning 'the cloud'?

It it as simple as taking each individual service listed in the AWS console
individually and learning exactly how they work, or is there something more
in-depth that matters?

~~~
itgoon
Integration between the various services. Any individual service is pretty
straightforward, but bringing all the pieces together can be a challenge
(security, networking, etc.) Billing is also a big deal - your solution can
end up being very expensive.

A better way to learn it, IMO, is to come up with some project, and implement
it. For example, a typical webpage with a backend db, some storage, DNS, maybe
load balancing. Avoid EC2 options, only because that's too easy (it's just a
VM).

It won't be incredibly difficult, but it isn't as easy as spinning up a VM on
your local machine.

Hope this helps.

------
sdan
> Nobody has figured out how to make money from AI/ML other than by selling
> you a pile of compute and storage for your AI/ML misadventures.

Thinking more about it, have companies made much with ML other than in
analytics/self driving companies/Google?

------
overgard
This is really funny, but I'm curious for people in the know on this: if you
were starting from scratch (so no legacy), would GCP or Azure be better? As
far as I can tell AWS's main advantage is cost -- is that fair?

~~~
Jedd
Depends on your budget, your risks, your technology sophistication, your
location, your variation / cylces of in compute & storage requirements.

Recently from the same person this story[1] on the minefield that is AWS costs
for data transit.

So, whether the 'Main advantage is cost' is accurate for you is very much an
'It depends' proposition.

[1]
[https://news.ycombinator.com/item?id=20972687](https://news.ycombinator.com/item?id=20972687)

------
dexterdog
My most uncomfortable truth about AWS is that bandwidth is crazy expensive and
it does not get cheaper over time like just about everything else does.

------
eternalny1
Wow ... this is like the AWS portion of my subconscious mind spilled out all
over Twitter.

------
timeimp
>Everyone likes to make fun of outages in us-east-1 that break the internet,
but Azure takes outages and everyone's websites all stay up. One wonders why.

huehuehue, doesn't the have i been pwned service use Azure? I can't recall
that ever being down due to Azure outages...

------
joneholland
We use autoscaling groups and spot instances.

¯\\_(ツ)_/¯

------
buboard
Great marketing

