
Announcing UltraWarm (Preview) for Amazon Elasticsearch Service - mihirsoni
https://aws.amazon.com/blogs/aws/announcing-ultrawarm-preview-for-amazon-elasticsearch-service/
======
whalesalad
I’d love an ES service that exposes the same external API but can operate on a
single node. So many “legacy” tools or systems rely on ES yet no one runs it
correctly because it’s so costly. You can run single node but you’re
definitely gonna lose data at some point.

Some orgs just need a simple ES-compatible system that won’t lose data. No
frills. I inherited an e-commerce contract that uses ES and maintaining a
cluster has been more costly than it should be, even with t2 nodes. The demand
on this isn’t high but it needs a few nodes to protect the data.

~~~
blorenz
Could you elaborate on what of ES becomes so costly? I see it alludes to it in
the article but I'm in the dark honestly. Is it the storage of the data since
it mentions S3? Is it memory bound or CPU bound?

~~~
outworlder
Not the parent but I'll pitch in.

For any non-trivial cluster, you'll want 30GB of RAM for Elasticsearch(no
more), plus whatever you can spare as OS cache. 64GB is common. Plus a lot of
storage, preferably on fast disks.

CPU consumption really depends on what you are doing but both
ingestion/indexing and searches are costly. Ingestion is mostly I/O bound, but
still requires a surprising amount of CPU.

Since we are talking about hot-warm, retention is the problem. You'll need
enough nodes with capacity to hold all the required data. Every open index
uses some amount of memory for each shard. A shard should ideally not exceed
50GB, so if you have big indices you'll have lots of shards (or you have lots
of small indices but still lots of shards).

You could have really enormous disks, but you shouldn't. If there is a problem
with any given node (and at scale, there will be), now your cluster will want
to move ridiculous amounts of data if it detects a node failure. You can
mitigate this, but you don't really want to turn of shard relocation.

Now multiply the storage requirements by two, as I assume you'll be using at
least one replica per shard.

So now you have a bunch of nodes doing ingestion, and a bunch of other nodes
storing data and answering search queries (in many clusters, they are the
same, but you can specialize). It's not uncommon to end up with hundreds of
nodes. At this scale, network communication becomes a bottleneck too, as the
cluster is trying to send enormous state updates. And now masters will have
trouble keeping up, so you'll end up scaling them as well.

It's a snowball. Small ES clusters are trivial to manage, large ones are a
full time job, even with automation. If this solution can help offload data
from the primary ingest or hot cluster, it will be a godsend.

~~~
__blockcipher__
Lots of small indices is an anti-pattern due to performance overhead. I’ve
seen such a setup wreck cluster stability.

By sticking to shards that are 10-100GB, and only using i3 instances, you can
get great performance. My company ingests about 2 TB per day and we use less
than 10 i3 instances.

------
ignoramous
Elasticsearch has had support for hot/warm, and now with 6.7+, for
hot/warm/cold via Index Life Management [0], where you'd typically run _hot
nodes_ with NVMe or SSDs, _warm nodes_ with HDDs, and _cold nodes_ with
network attached storage.

AWS' key innovation seems to be that _hot nodes_ are the ones you provision to
keep both primary and secondary indices for _hot data_ (writable and recent),
and _ultra-warm nodes_ , of different ec2-instance-type than the _hot nodes_ ,
prefetch speculatively / predictively caches of index-data downloaded on-
demand from S3 required to execute read-only requests transparently to the
rest of the cluster.

Impressive: It is _almost_ like running ultrawarm nodes with Amazon FSx for
Lustre instead of EBS.

I wonder if the Lucene / Elasticsearch changes required to do this will make
it to OpenDistro. If not, that'd be tremendously disappointing [1].

\---

[0] [https://www.elastic.co/blog/implementing-hot-warm-cold-in-
el...](https://www.elastic.co/blog/implementing-hot-warm-cold-in-
elasticsearch-with-index-lifecycle-management)

[1] _In addition, the innovation focus has shifted from furthering the open
source distribution to making the proprietary distribution popular. This means
that the majority of new Elasticsearch users are now, in fact, running
proprietary software._
[https://news.ycombinator.com/item?id=19359602](https://news.ycombinator.com/item?id=19359602)

------
reilly3000
Damn, I just learned about ChaosSearch yesterday. This is their whole business
model.

I've faced that pain on AWS ES service and will definitely give it a try.

------
manigandham
This is a great feature to add. Something that Elastic should've created a
long time ago.

Interesting how well AWS is adding features on top of open-source in
competition with the original vendor now.

