
Things to know before using AWS’s Elasticsearch Service - good_regex
https://read.acloud.guru/things-you-should-know-before-using-awss-elasticsearch-service-7cd70c9afb4f
======
jonaf
Something else people should know: AWS ES is on the Internet. You can't deploy
it to a vpc yet, and you can only lock it down using IAM, which may or may not
be good enough for your use-case.

For those that prefer the ease provided by AWS ES Service, consider Elastic
Cloud, which affords most of the same capabilities but is run by Elastic
themselves (it was previously known as Found, which Elastic purchased a few
years ago). There's also an Enterprise offering. If you're looking for a
hosted Elasticsearch solution, it's probably better than what AWS is offering.
Side note: they update about as often as elastic releases, whereas AWS ES is
consistently behind.

~~~
luhn
The IAM authentication is really annoying. It's not supported by many client
libraries, nor have I found an easy way to make arbitrary HTTP calls with
signature v4.

The only other options are completely public or IP-based whitelist, the latter
which is untenable in most cloud environments.

~~~
ecnahc515
You can also use a signing proxy.

~~~
luhn
I wasn't aware of that option. I'll look into it.

~~~
Leon
A simple solution in this vein is to white list your the EIP addresses of your
NAT. This would give access to all resources in a private subnet (this is
useful for Lambda's running in subnets).

------
cavisne
Article is a bit naive about what it takes to run a shared service. Any API
that AWS ES exposes has to be there forever, clearly pending_tasks had some
risk of leaking internal implementation details that either couldn't be
exposed, or that they didn't want customers building a dependency on.

Likewise with the doubling of nodes, this is obviously a blue-green style
deployment. In place updates would be quicker but ES can get into all sorts of
weird states that require manual debugging to fix, with blue green for most of
the deployment you can simply flip back.

I've been pretty impressed with AWS ES compared to running it myself (other
than the poor fit of IAM auth)

~~~
dmix
> Any API that AWS ES exposes has to be there forever, clearly pending_tasks
> had some risk of leaking internal implementation details that either
> couldn't be exposed, or that they didn't want customers building a
> dependency on.

If this is a reality of using cloud based ES then clearly it's something to
seriously consider before using it - which is all the author is saying. The
article is titled 'things to consider' not 'things AWS needs to fix'.

ES is big complex beast of a Java app. This is good advice regardless from
someone who has used both approaches (self hosted vs AWS) in production.

I did not get the impress that he's saying that AWS can resolve this easily.

------
Andys
Worth mentioning that elastic.co themselves run a hosted service on AWS that
is of a high quality and has none of these flaws.

------
jknoepfler
I can confirm the author's frustrations with AWS ES. Having set up clusters on
my own (on EC2 hosts) and using the service... The latter is expensive,
inefficient, behind on features, hard to integrate with, and generally just a
really crappy piece of work (like almost every peripheral AWS service, ie
anything but EC2, S3, and DynamoDb).

Elastic search is honestly pretty simple to set up, save yourself money and
trouble and just do it.

~~~
theparanoid
I had the same experience with their code hosting products (CodeCommit). It
was better to just setup an EC2 instance and manage it myself.

------
rpedela
This appears to be somewhat out of date. If you use version 5.x then
pending_tasks is available.

[http://docs.aws.amazon.com/elasticsearch-
service/latest/deve...](http://docs.aws.amazon.com/elasticsearch-
service/latest/developerguide/aes-supported-es-operations.html)

~~~
good_regex
Good to know, I didn't realize 5.x has that API available. Why it's only
available in 5.x makes no since ES has had the API since at least 1.x

------
Roritharr
Having a DevOps Engineer that wants me to go the AWS Dedicated Everything
Route, I need articles like this to explain to him my fear that our problems
will just change, not go away, by going that route. + Adding a fat layer of
dependency.

~~~
meddlepal
Complexity never goes away... it just shifts. I dunno if that is a common
saying or not but a former coworker of mine once said it and it's very true
IMO.

I do infrastructure engineering for a small startup and really I think with
any of these managed systems you need to step back and evaluate them within
the context of TCO, lock-in, security, reliability, performance and
flexibility/customizability. I've heard ES isn't that much of a PITA to manage
on its own, but on the flip-side I'd never sign up a small team to run PgSQL
at scale.

~~~
vacri
> _I 've heard ES isn't that much of a PITA to manage on its own_

I just run ES for my logstash setup, and ES is lovely and rock-solid... except
when it isn't. For example ES deciding to just silently refuse input when its
disk is 90% full - that was a bit hard to find when it happened. ES looked
alive, but hunting down the reason why it stopped wasn't trivial. I've had a
couple of similar but lesser gotchas as well.

I guess you could say of my experience that it's _not that much_ of a PITA (as
you say), but it is still a bit of a PITA.

Disclaimer: if these things weren't a bit of a PITA, there'd be no need for us
sysadmins, so I should be grateful...

------
hendzen
Not all AWS services are created equal... Some are rock solid and others (
_cough_ Data Pipeline _cough_ Kinesis Firehose _cough_ ) must have been
written by interns.

~~~
aianus
What issues did you have with kinesis firehose? I just deployed a couple of
those and would like to know what to watch out for.

~~~
bpicolo
One issue we've hit is that logrotate with copytruncate enabled breaks
firehose, but afaik it's mostly been good.

~~~
otterley
Using copytruncate breaks a lot of software, not just Firehose. I generally
discourage its use in favor of addressing whatever root cause is making you
want to use it.

~~~
vacri
In my experience, any 'improve logging' ticket goes straight to the back of
the dev's backlog. Followed the next week by complaints about the logging
system not being all that good... :)

------
kiernanmcgowan
Another point of frustration is that because these endpoints are locked
down[0] you cannot fully use management tools like curator -
[https://github.com/elastic/curator](https://github.com/elastic/curator)

[0][https://www.elastic.co/guide/en/elasticsearch/client/curator...](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/faq_aws_iam.html)

~~~
jbyers
Curator support got better in the AWS ES 5.3 release a few weeks ago:
[https://github.com/elastic/curator/releases/tag/v5.1.0](https://github.com/elastic/curator/releases/tag/v5.1.0)

"AWS ES 5.3 officially supports Curator now. Documentation has been updated to
reflect this."

~~~
yissachar
You still can't use Curator to take snapshots, because AWS ES doesn't expose
the `/_snapshot/_status` endpoint.

[https://github.com/elastic/curator/issues/639](https://github.com/elastic/curator/issues/639)

------
skywhopper
Like almost everything else you can build on AWS managed services (RDS,
Elasticache, API Gateway/Lambda, Kinesis, etc), if it's truly critical to your
application's uptime, you should be managing it yourself.

But if your need for ES is to support a backend system that would make your
life inconvenient for a while if there are problems, is relatively small and
won't grow too fast, but isn't business-critical, then the AWS managed service
is fine.

------
justonepost
Good warning. Yeah, beta software released to production.

~~~
StreamBright
There is that and also from fragility from ES. I was wondering what
alternatives are out there to ES. I know of Solr only.

~~~
justonepost
Solr has the added complexity of zookeeper. ES isn't bad, but in an MT context
you really have to layer a lot on top for security and configurability.

It's possible it's not MT and they just didn't write the facade APIs. That'd
be pretty crazy.

My biggest complaint would be lack of plugin support.

~~~
bpicolo
To be fair, ZK is great at it's job, and that responsibility is something ES
has had a lot of trouble replicating.

------
trengrj
I looked at using AWS Elasticsearch Service for a project but had to back out
due to the lack of plugin support. Running elasticsearch yourself, even in a
HA setup is actually fairly easy.

------
CD1212
Does anyone have any experience of Compose.io's hosted Elasticsearch offering?

I have been using it on a new project the last couple of weeks and it seems to
be working well.

------
petethepig
Had a very similar experience with Redis on ElastiCache. When things go south,
it's really hard to debug. You don't get access to logs, you don't get to
change a lot of config parameters.

Had to provision our own EC2 instances.

It was 2 years ago though, things might be different now.

------
jakozaur
It even get worse if you use Aws ElastSearch for logs. Logs are usually high
volume and it can quickly beczme nightmare.

~~~
coredog64
It used to be worse. The max EBS volume size was 512GB (with 15% reserved for
Amazon) and a max cluster size of 20.

We hit that limit and had to ruthlessly prune live data.

You can now add 1.5TB per node (with very large and expensive instance types)
as well as scale past 20. Requesting the limit increase was a lot more
difficult than most other limit increases.

------
MightySCollins
I just want to get rid of the stupid proxy I have just so I can make it
work... Amazon just let me put it in a VPC

------
manigandham
Side note: reading anything on medium.com is frustrating, such a slow and
janky site for a glorified blog.

------
ianamartin
Bookmarked to look at the next time my boss wants to lock us into yet another
AWS service. Thank you.

------
jdc0589
there's also the bit about how adding a whitelisted IP for access takes like
20 FREAKING MINUTES to take effect.

------
whatnotests
Wow WTH Amazon? Take the training wheels off already or fix the defaults for
this service.

~~~
moonka
Preferably both.

------
xchaotic
Brutal plug, but MarkLogic is a good alternative if you want a good search
solution that runs and scales on AWS (and you can migrate to another cloud or
on prem)

