
Amazon Elasticsearch Service - johns
https://aws.amazon.com/blogs/aws/new-amazon-elasticsearch-service/
======
necubi
Seems very expensive compared to just running ES on instances yourself. Using
i2.xlarges (as we do for our ES cluster) it's $1.194/hour for managed ES and
just $0.853/hour for spot i2 instances. If you're will to reserve for a year,
it's only $0.3555/hour.

Given how easy ES is to operate, I don't know why anybody would pay for a less
flexible, more expensive hosted version.

~~~
balls187
> Given how easy ES is to operate, I don't know why anybody would pay for a
> less flexible, more expensive hosted version.

Easy. Amazon's service is fully managed. Getting ES running on EC2 is easy
enough. Making it fault tolerant, automated backups to S3 and multi-az
capable, takes more work. Even more to make sure it works properly.

Add the time and complexity required to test your
recovery/tolerance/redundancy plans, and you pretty much lose out on the
savings.

Given the choice of spending 2-3 days getting everything configured right, vs
letting Amazon deal with it, I'd pick Amazon.

Using platitudes: the time I save not dorking with infrastructure is time that
I can spend solving my customers problems.

~~~
dangrossman
+1. This is the reason I pay for Amazon RDS/Aurora, where the price difference
versus the same EC2 instances is even greater. Sure I could set up a
database... and replication to another database... and script scheduled
snapshots... and build monitoring and fault tolerance against all the ways
things can break... but by the time I've set all that up, I've wasted a week
or more building servers instead of building a business.

~~~
matt2000
Are you using aurora? How's your experience been so far? It sounds interesting
but I haven't heard much real world experience yet.

~~~
dangrossman
I'm running W3Counter on it now. That's ~100GB of data and a couple hundred
queries per second. It's about equal read/write load with one insert and one
select for every page view I log. Nothing you can't do on a single server, but
it's all I can talk about.

I was renting two ~$300/mo servers at Softlayer to run MySQL (Percona), with
one as a backup + failover + to run my monthly browser/OS market share reports
against without killing the website by overloading the main server. I
regularly spiked high loads that killed latency anyway this spring. I was
going to have to upgrade soon anyway, so I decided to try out Aurora during
the free preview period, and migrated everything over there one weekend.

Turns out a single R3.Large instance (the smallest) was more than sufficient.
It's sitting at under 25% CPU and 50% memory usage right now, which means
there's plenty of room to grow too. I'd say performance is as good or better
than native MySQL based on that. I'm paying less now than I did for the two
servers, and get more for it, with free multi-AZ replication, nightly
snapshots, and the ability to spin up a live replica in just a few minutes to
run backups/large reports against, then shut down and only pay for the hour.

~~~
bdcravens
That's awesome; thanks for sharing. To me, the managed aspect where I don't
have to worry about monitoring the hard drive space is a big win.

How long does it typically take for the live replica to come online?

------
spankalee
Will Elasticsearch ban queries for AppleTV and Chromecast?

~~~
phpnode
Solr and Sphinx

------
andrewvc
Interesting, This would be Amazon's version of Found (
[https://www.elastic.co/found](https://www.elastic.co/found) ), which is
Elastic's (makers of elasticsearch) aws based cloud solution. I wonder what
this means for cloud search?

I should note that I work for Elastic on the Logstash team and used to work on
Found.

------
bitroliest
We will definitely be checking this out. Our experience with another vendor
and ES has not been great. From having the whole cluster come down when we're
doing batch inserts for initial loads to having 10s timeouts on random queries
during low activity windows with the vendor saying "dunno, everything looks
good here" (when we measured that this 10s were spent waiting for ES).

~~~
evboyle
If you're looking for a search backend without having to worry about managing
infrastructure, you should check out Azure Search. I'm a dev on the product.
If you're interested in trying it out let me know. You can ping me at my
username @microsoft.com if you have questions or need pointers.

~~~
nullrouted
Side question: Do you know if Azure will every allow traceroute through their
network?

~~~
evboyle
no idea, sorry

~~~
nullrouted
ipv6 support?

------
SEJeff
For reference:

[https://aphyr.com/posts/317-call-me-maybe-
elasticsearch](https://aphyr.com/posts/317-call-me-maybe-elasticsearch)

And the updated:

[https://aphyr.com/posts/323-call-me-maybe-
elasticsearch-1-5-...](https://aphyr.com/posts/323-call-me-maybe-
elasticsearch-1-5-0)

------
Jgrubb
Well that's a tiny bit annoying, since I just set up our own cluster 3 days
ago to centralize all our logging...

------
bdcravens
I wonder what this says about Amazon CloudSearch?

~~~
room271
It says it is not very good. There are a lot of problems with Cloudsearch. By
contrast, Elasticsearch is great and not too difficult to run yourself in AWS,
or indeed elsewhere.

Aside from product differences, Cloudsearch was phenomenally expensive,
particularly if you had a larger dataset but low read/write volume, as
Cloudsearch did not use EBS (at least not in a way you could control), unlike
RDS for example.

------
aaronkrolik
Will amazon support the head plugin? Can we enable our own plugins?

~~~
sscarduzio
They support just ICU and Kibana. Shield is not there too.

------
ssalat
This new aws service offer is really boring, I don't see a real advantage
migrating to it anytime soon.

What will be the expectation here:

\- Autoscaling functionalities out of the box

\- Better speed than on regular EC2 instances (I/O, Network)

\- More visiblity, better monitoring out of the box (e.g. Marvel)

\- Lower pricing than running on plain EC2

Still a long way to go here...

~~~
dangrossman
RDS seems to be doing just fine with a large markup over EC2. Why would the
managed version of a service be lower priced than the same infrastructure
being managed?

~~~
ssalat
Amazon has more possiblities to run a really huge cluster and balance the
different load peaks of the search nodes. The whole business model of AWS is
based on this... but you're right as well, price isn't everything and comfort
worth money. And yes, I love RDS as well, especially Aurora... it's a great
deal!

~~~
p0rkbelly
Running one big cluster and using application level separation would make lots
of customers apprehensive...

------
fideloper
Opinions on the price being high seem to miss the point of AWS.

AWS is very expensive relative to other clouds, with exceptions so some _very_
nice services (S3, SQS, SES, Route53).

"Core" (Very Important™) services such as EC2/RDS are (appropriately)
expensive due to the tooling and high availability options available. I'd
expect the tradeoff for their hosted ES to be price, since I'm buying
availability and less headache, not just the search service.

I know for sure that I don't know enough to monitor JVM applications. I'd love
to pay for their ES service if/when it made sense for a business.

------
Thaxll
It seems weird to handle the number of instances on top of AWS service, I mean
it's AWS they should provide a solution where you don't have to think about
that...

It looks like ES installed by Puppet on top of EC2.

------
icpmacdo
I thought there was already an AWS Elasticsearch product? Am I remembering
something different or is this coming out of beta or something to that effect?

~~~
ihsw
There is CloudSearch however it doesn't support schemaless stuff.

Looks like this is AWS' answer to that.

There's also the issue of CloudSearch taking in document batches rather than
individual documents, which means you have to do a fair bit of more leg work
to get your stuff in there. ElasticSearch can take in Logstash inputs which is
handy and quite performant.

Plus a lot of data analysts are used to Kibana so there is some appeal there.

------
johnhkg
How would this compare to Algolia ? Does anyone have experience with
elasticsearch and algolia to shed some light on this ?

------
odiroot
Nice, they even support Kibana.

Just few days after I managed to finally set up ELK stack correctly by hand.

------
ShaneOG
sa-east-1 gets the short straw again.

> The service does not support any i2 and r3 instance types in the sa-east-1
> region.

