
AWS X1 instances – 1.9 TB of memory - spullara
https://aws.amazon.com/blogs/aws/x1-instances-for-ec2-ready-for-your-memory-intensive-workloads/
======
jedbrown
Does anyone have numbers on memory bandwidth and latency?

The x1 cost per GB is about 2/3 that of r3 instances, but you get 4x as many
memory channels if spec the same amount of memory via r3 instances so the cost
per memory channel is more than twice as high for x1 as r3. DRAM is valuable
precisely because of its speed, but the speed itself is not cost-effective
with the x1. As such, the x1 is really for the applications that can't scale
with distributed memory. (Nothing new here, but this point is often
overlooked.)

Similarly, you get a lot more SSDs with several r3 instances, so the aggregate
disk bandwidth is also more cost-effective with r3.

~~~
sun_n_surf
Not sure I quite understand your math here. The largest R3 instance is the
r3.8xlarge with 244 GB of memory. 4 times of that would only get you to 1 TB.
Also, this: "DRAM is valuable precisely because of its speed", is wrong
([https://en.wikipedia.org/wiki/Dynamic_random-
access_memory](https://en.wikipedia.org/wiki/Dynamic_random-access_memory)).

~~~
jedbrown
1\. 4 of those R3 instances cost less than the X1 but offer nearly double the
bandwidth. The X1 is cheaper per GB, but much more expensive per GB/s.

2\. If DRAM was not faster than NVRAM/SSD, nobody would use it. "Speed"
involves both bandwidth and latency. Latency is probably similar or higher for
the X1 instances, but I haven't seen numbers. We can make better estimates
about realizable bandwidth based on the system stats.

------
lovelearning
This is probably a dumb question, but what does the hardware of such a massive
machine look like? Is it just a single server box with a single motherboard?
Are there server motherboards out there that support 2 TB of RAM, or is this
some kind of distributed RAM?

~~~
schlarpc
Don't know what hardware AWS is using, but Ark has server boards supporting
1.5TB, which is close enough to make 2TB believable:
[http://ark.intel.com/products/94187/Intel-Server-
Board-S2600...](http://ark.intel.com/products/94187/Intel-Server-
Board-S2600WTTS1R)

Edit: Supermicro has several 2TB boards, and even some 3TB ones:
[http://www.supermicro.com/products/motherboard/Xeon1333/#201...](http://www.supermicro.com/products/motherboard/Xeon1333/#2011)

(Disclaimer: AWS employee, no relation to EC2)

~~~
yuhong
This would require expensive 64GB DDR4 LR-DIMMs though.

------
MasterScrat
As a reference the archive of _all_ Reddit comments from October 2007 to May
2015 is around 1 terabyte uncompressed.

You could do exhaustive analysis on that dataset fully in memory.

~~~
jedberg
Your point is accurate, but I'd like to point out that the dataset isn't
actually all the comments on Reddit -- it's inly what they could scrape, which
is limited to 1000 comments per account. So basically it's missing a lot of
the historical comments of the oldest accounts.

I only point this out to try and correct a common error I see. You're
absolutely right that it is awesome that the entire data set can be analyzed
in RAM!

~~~
MasterScrat
Are you sure? The dataset is from here:
[https://archive.org/details/2015_reddit_comments_corpus](https://archive.org/details/2015_reddit_comments_corpus)

Looking at the thread from the release, I see no explanation of how he got the
data, but I see several people commenting that they finally have a way to get
comments beyond the 1000 per account:
[https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...](https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/)

~~~
jedberg
It looks like they got clever, so what they are doing is getting 1000 links
from each subreddit, indexing the comments and linking it back to the user. So
yeah, they did figure out how to got back more than 1000 comments for some
users, but they'll still be limited by 1000 links per reddit per sort. So for
a very active reddit, you can't get really old comments.

~~~
_wmd
IIRC you can use the search API to enumerate 1000 at a time, using a
"created_at>=..." restriction to move the cursor starting position. I had to
poke in the code to find out how to talk Amazon's search syntax directly

~~~
jedberg
Yeah, that might work too. That would be pretty clever.

------
ChuckMcM
That is pretty remarkable. One of the limitations of doing one's own version
of mass analytics is the cost of acquiring, installing, configuring, and then
maintaining the hardware. Generally I've found AWS to be more expensive but
you get to "turn it on, turn it off" which is not something you can do when
you have to pay monthly for data center space.

It makes for an interesting exercise to load in your data, do your analytics,
and then store out the meta data. I wonder if the oil and gas people are
looking at this for pre-processing their seismic data dumps.

~~~
ddorian43
Why does everyone(really!) compare aws to colocation ? I've never head an aws-
believer ever mention dedicated servers.

Why don't you compare aws to building your own cpu?

~~~
ChuckMcM
I suspect it is because "everyone" (which is to abscond with your definition)
believes that colocation is an alternative to AWS (well the EC2 part anyway).
I would be interested to hear how you seem them as not being comparable.

On your definition of "aws-believer" is that someone who feels that AWS is a
superior solution to deploying a web facing application in all cases? Does
your definition include economics? (like $/month vs request/month vs latency?)

Can I assume that you consider comparing AWS to building your own CPU as an
apples to oranges comparison? I certainly do, because I define a CPU to be a
small component part of a distributed system hosting a web facing application.

~~~
ddorian43
by "everyone" I mean >90% of comments on this forum comparing aws to an
alternative (90% of cases it's straight to colocation)

Building your own datacenters is also an alternative to aws-ec2, but you go to
that step by step (I think?) (dedicated > colocation > datacenter). In some
cases when you have crazy growth you can skip a/some steps (ex: dropbox going
from aws to their own datacenter)

They don't even compare them in whatever $/request/latency/$metric since they
don't even mention them. And dedicated has also many options from low-
price+no-support-shitty-network to high price/good-network/support etc.

------
1024core
Spot instances are about $13 - $19/hr, depending on zone. Not available in
NorCal, Seoul, Sydney and a couple of other places.

~~~
snewman
Do you mean on-demand instances? The announcement says "Spot bidding is on the
near-term roadmap." And $13 / hour is the on-demand price in US East.

~~~
dharma1
Indeed, doesn't look like it's there yet. Based on that I guess the spot
prices will be around $1-3/h - not bad, if you have a workload that can be
interrupted.

------
dman
Going to comment out the deallocation bits in all my code now.

~~~
sedachv
JonL White actually wrote a serious paper about just this idea in 1980:
[http://dl.acm.org/citation.cfm?id=802797](http://dl.acm.org/citation.cfm?id=802797)

~~~
gaul
Full paper:

[http://3e8.org/pub/scheme/doc/lisp-
pointers/v1i3/p17-white.p...](http://3e8.org/pub/scheme/doc/lisp-
pointers/v1i3/p17-white.pdf)

------
pritambarhate
Question for those who have used monster servers before:

Can PostgreSQL/MySQL use such type of hardware efficiently and scale up
vertically? Also can MemCached/Redis use all this RAM effectively?

I am genuinely interested in knowing this. Most of the times I work on small
apps and don't have access to anything more than 16GB RAM on regular basis.

~~~
chucky_z
Postgres scales great up to 256gb, at least with 9.4. After that it'll use it,
but there's no real benefit. I don't know about MySQL. SQL Server scales
linearly with memory even up to and past the 1TB point. I did encounter some
NUMA node spanning speed issues, but numactl tuning fixed that.

I setup a handful of pgsql and Windows servers around this size. SQL Server at
the time scaled better with memory. Pgsql never really got faster after a
certain point, but with a lot of cores it handled _tons_ of connections
gracefully.

~~~
anarazel
I've very successfully used shared buffers of 2TB, without a lot of problems.
You better enable huge pages, but that's a common optimization.

------
vegancap
Finally, an instance made for Java!

~~~
granos
I dislike developing in Java. I am not a fanboy by any stretch of the
imagination. That being said, someone who takes the time to understand how the
JVM works and how to configure their processes with a proper operator's
mindset can do amazing things in terms of resource usage.

It's easy to poke at Java for being a hog when in reality its just poor coding
and operating practices that lead to bloated runtime behavior.

~~~
placeybordeaux
For a long time I wondered if it was a failing of the language or the culture.

After spending 4 days trying to diagnose a problem with hbase given the two
errors "No region found" and "No table provided" and finally figuring out it
was due to a version mismatch I now believe it is the culture.

At the very least you should be printing a WARN when you connect to an
incompatible version.

~~~
breischl
Definitely the culture. It seems to be a Java axiom that you should never use
one object when eight will do.

I've never seen another community that would actually have a builder for the
configuration for the factory for the settings for a class.

~~~
mpd
I assume you've never had the pleasure of configuring a Sendmail installation?

~~~
ddalex
Actually configuring sendmail is a piececake these days thanks to Google and
countless engineers documenting in excruciating detail the problems they
faced.

------
krschultz
A bit under $35,000 for the year.

~~~
helper
If you paid hourly (instead of using a reserved instance) it costs $116840.88
per year (13.338 per hour * 24 * 365). A 1 year reservation + hourly charges
for a year is $67197.96 (all up front).

~~~
kabdib
You can buy a server from Dell with 1.5 TB of RAM and four beefy processors
(80 cores) for about $60K. You can power and cool it for a year for maybe $4K.

Every time I look at AWS, it just doesn't make sense from a financial
standpoint (even after you add another machine for redundancy, and remote
hands -- you're ahead after 12 months).

~~~
helper
I'm sure there are companies that will run this instance type 24/7 for years.
But you can also spin one up for a short period of time, use it to do things
you would normally need to do across a bunch of machines and then tear it
down.

Having this type of instance available via an API call is within seconds is
really cool.

------
realworldview
Recompiling tetris with BIGMEM option now...

------
Erwin
I'm curious about this AWS feature mentioned:
[https://aws.amazon.com/blogs/aws/new-auto-recovery-for-
amazo...](https://aws.amazon.com/blogs/aws/new-auto-recovery-for-amazon-ec2/)

We've experiemnted with something similar on Google Cloud, where an instance
that is considered dead has its IP address and persistent disks taken away,
then attached to another (live or just created instance). It's hard to say
whether this can recover from all failures however without having experienced
them or even work better than what Google claims it already does (moving
around failing servers from hardware to hardware). Anyone with practical
experience in this type of recovery where you don't duplicate your resource
requirements?

------
zbjornson
How does this thing still only have 10 GigE (plus 10 dedicated to EBS)? It
should have multiple 10 Gig NICs that could get it to way more than that.

------
jayhuang
Funny how the title made me instantly think: SAP HANA. After not seeing it for
the first 5 paragraphs or so, Ctrl+F, ah yes.

Not too surprising given how close SAP and Amazon AWS have been ever since SAP
started offering cloud solutions. Going back a couple years when SAP HANA was
still in its infancy; trying it on servers with 20~100+ TB of memory, this
seems like an obvious progression.

Of course there's always the barrier of AWS pricing.

------
amazon_not
The pricing is surprisingly enough not terrible. Given that dedicated servers
cost $1-1.5 per GB of RAM per month the three year price is actually almost
reasonable.

That being said, a three year commitment is still hard to swallow compared to
dedicated servers that are month-to-month.

~~~
bdcravens
It's not just the commitment, but the fact you have to cough up
$52,000-$98,000 up front.

------
0xmohit
Wow!
[http://codegolf.stackexchange.com/a/22939](http://codegolf.stackexchange.com/a/22939)
would now be available in production.

------
manav
Hmm around $4/hr after a partial upfront. I'm guessing that upfront is going
to be just about the cost of a server which is around $50k.

------
micro-ram
What happened to the other 16 threads?

18(core) * 4(cpus) * 2(+ht) = 144

~~~
zbjornson
Some fraction has to go to the physical host for its resources.

------
ben_jones
I'd be guilty if I ever used something like this and under utilized the ram.

"Ben we're not utilizing all the ram."

"Add another for loop."

------
mrmondo
I'm taking it this is so people can run NodeJS or MSSQL on AWS now? Heh, sorry
for the jab - what could this be used for considering that AWS' top tier
provisioned storage IOP/s are still so low (and expensive)?

Something volatile running una RAM disk maybe?

------
samstave
~$4/hour???

Thats amazing.

~~~
tobz
Only if you reserve them for three years, which carries an upfront cost of
~$98K.

~~~
bdcravens
Partial upfront is _only_ $52,166, and over the life of the reservation, only
costs an extra 2%.

------
samstave
16GB of ram should be enough for anyone.

Edit, y'all don't get the reference: famous computer urban legend...

[http://www.computerworld.com/article/2534312/operating-
syste...](http://www.computerworld.com/article/2534312/operating-systems/the--
640k--quote-won-t-go-away----but-did-gates-really-say-it-.html)

~~~
dang
We detached this subthread from
[https://news.ycombinator.com/item?id=11723858](https://news.ycombinator.com/item?id=11723858)
and marked it off-topic.

~~~
samstave
A joke about an ancient myth in our tech community is "off topic" \- you guys
are too stringent with this. I think it's lame.

~~~
minimaxir
The joke doesn't make any sense because the Bill Gates quote refers to
personal computing, not development.

------
0xmohit
Encouraging folks to write more inefficient code?

I'd be interested in hearing what Gates [1] has to say about it, though.

[1] "640 kB ought to be enough for anybody"

~~~
bdcravens
Too bad no one has a real source for that quote, probably because he never
actually said it (though like "Beam me up Scotty" which was never quoted in
the original Star Trek, he may have come close, but it's not nearly as
memetastic)

