
Open Distro for Elasticsearch - jeffbarr
https://opendistro.github.io/for-elasticsearch/
======
dang
We merged the related threads. Here are the relevant blog posts:

[https://aws.amazon.com/blogs/opensource/keeping-open-
source-...](https://aws.amazon.com/blogs/opensource/keeping-open-source-open-
open-distro-for-elasticsearch/)

[https://aws.amazon.com/blogs/aws/new-open-distro-for-
elastic...](https://aws.amazon.com/blogs/aws/new-open-distro-for-
elasticsearch/)

------
sciurus
AWS explains their rationale at
[https://aws.amazon.com/blogs/opensource/keeping-open-
source-...](https://aws.amazon.com/blogs/opensource/keeping-open-source-open-
open-distro-for-elasticsearch/) . Here's a snippet"

" Unfortunately, since June 2018, we have witnessed significant intermingling
of proprietary code into the code base. While an Apache 2.0 licensed download
is still available, there is an extreme lack of clarity as to what customers
who care about open source are getting and what they can depend on. For
example, neither release notes nor documentation make it clear what is open
source and what is proprietary. Enterprise developers may inadvertently apply
a fix or enhancement to the proprietary source code. This is hard to track and
govern, could lead to breach of license, and could lead to immediate
termination of rights (for both proprietary free and paid). Individual code
commits also increasingly contain both open source and proprietary code,
making it very difficult for developers who want to only work on open source
to contribute and participate. In addition, the innovation focus has shifted
from furthering the open source distribution to making the proprietary
distribution popular. This means that the majority of new Elasticsearch users
are now, in fact, running proprietary software. We have discussed our concerns
with Elastic, the maintainers of Elasticsearch, including offering to dedicate
significant resources to help support a community-driven, non-intermingled
version of Elasticsearch. They have made it clear that they intend to continue
on their current path. "

~~~
rushabh
When you publish open source, this is what you sign up for. As a publisher of
open source software, I think "open core" does a huge disservice to the
community as it feels deceptive. While there is no denying that creators needs
to be paid, communities thrive on freedom and openness, and dual licensing
takes that away.

Companies that publish open source must realize that they cannot make money
from the "software". Open source gives companies a brand, that they can then
leverage to do other services like support, consulting, hosting, merchandise,
events, etc.

There was a time when service companies wanted to move into products because
of high margins and artificial scarcity, but enough open source will ensure
that software licensing is not a sustainable model to seek rent, and product
companies must also move to services to be able to grow and sustain.

~~~
anth_anm
I think Amazon does a much bigger disservice.

> but enough open source will ensure that software licensing is not a
> sustainable model to seek rent

So your solution to this is to just make it so no one can make money doing
work.

Except Jeff. Jeff gets all the money.

~~~
skywhopper
Just be honest, and don't pretend to be open source, when you are attempting
to harness goodwill towards open source in order to make a buck. As pointed
out, ES builds on the work of hundreds of other contributors to other open
source projects. Is Elastic compensating those developers?

~~~
weberc2
> don't pretend to be open source

In my mind, a company that makes lots of free contributes to open source is an
open-source-friendly company, but this thread and the other on the Elastic
post made me realize just how many people think that a True Open Source
Company (whatever that means) should prefer to go out of business to dealing
in proprietary software.

I personally don't care for any of these puritanical viewpoints that deny
economic realities (there's only so much a person or a collective can afford
to give away for free, money doesn't grow on trees, etc). Open source is good,
but thriving, sustainable open source depends on a symbiotic relationship with
for-profit institutions.

------
pauldix
This is a very surprising move from AWS. In the past they haven't seemed to be
willing to contribute to the OSS that they pick up and host. Even though they
claim they'll contribute changes upstream, I doubt that Elastic would accept
changes that are competitive with their commercial offerings. So you
effectively get a fork.

I think this is a win for the Elastic community as a whole, but presents a
real problem for Elastic the company. And that begs the question of what
happens to the Elastic community if there's a real fork. And what happens if
Elastic the company is very negatively impacted? Do we see a fracture of the
community?

Looking even farther out, at this point does it make sense for any startup
looking to create infrastructure software to open source it? If their project
becomes successful, they'll get eaten up by the hosting providers. It makes
smaller scale open source more commercially viable because you won't attract
the attention of the providers that would come in a take your business.

Will be interesting to at least watch what early stage VC investment in open
source companies looks like over the next 12-24 months. Will the open source
pitch work for series A investments or will investors shy away?

~~~
zzzcpan
> Looking even farther out, at this point does it make sense for any startup
> looking to create infrastructure software to open source it?

Of course it still does, just not under permissive licenses. Hopefully the
industry will converge on Kyle Mitchell's license for this, instead of
everyone coming up with their own licenses:
[https://github.com/kemitchell/api-copyleft-
license](https://github.com/kemitchell/api-copyleft-license)

~~~
pauldix
I view that as a significant negative impact to OSS. I have a strong
preference for liberal licenses like MIT or Apache 2.0. Infectious licenses
like copy-left put restrictions on it that make it less appealing as open
source. I wrote a bit on it here: [https://www.influxdata.com/blog/copyleft-
and-community-licen...](https://www.influxdata.com/blog/copyleft-and-
community-licenses-are-not-without-merit-but-they-are-a-dead-end/)

~~~
LamaOfRuin
While I understand your argument, I dispute that copy-left makes things
inherently less appealing as open source, or that you can make any objective
claim that liberal licenses ultimately result in broader utility. 1\. I
personally find copy-left inherently _more_ appealing (as something to
potentially contribute to) purely because of my ideological leanings. 2\.
While the first order calculation of use for liberally licensed software seems
clearly in favor of them, it's impossible to ever know what benefits the copy-
left restrictions would bring one or more orders removed from the original
distribution.

------
weberc2
> Unfortunately, we are seeing other examples where open source maintainers
> are muddying the waters between the open source community and the
> proprietary code they create to monetize the open source.

As I understand it, Amazon took ElasticSearch and monetized it, therefore
competing directly with Elastic (the company that develops ElasticSearch).
Elastic felt Amazon was taking advantage of Elastic's commitment to open
source, and started developing new features under a proprietary license, but
releasing them for free. The idea was to prevent their work from benefiting a
competitor.

Basically in this blog post, Amazon is trying to make the case that they're a
Good Tech Company and Elastic are the Bad Guys or something. Generally I'm all
for nuance and "the truth is probably somewhere in the middle", but Elastic is
the rare company that truly is committed to open source (at least at the
current moment in history) and Amazon is, well, Amazon.

~~~
abtinf
> [Elastic] started developing new features under a proprietary license

> Elastic is the rare company that truly is committed to open source

On their face, those two statements seem mutually exclusive. What context am I
missing that would make them not so?

~~~
dragonwriter
Elastic, it seems to me, is the not-at-all rare company who is truly committed
to using “open source” as a marketing gimmick and tool to recruit free labor,
and like most such companies resents being reduced to itself playing the role
it envisions for FOSS community members by a bigger fish.

~~~
speedplane
> Elastic, it seems to me, is the not-at-all rare company who is truly
> committed to using “open source” as a marketing gimmick

Don't think it was that nefarious. More likely that the original developers
were committed to open source, and then took lots of venture capital money to
keep coding and making cool things. Now those VCs want a return on their
investment, and effectively have control over the product.

------
jchw
My personal, probably disagreeable opinion, but... The narrative that Amazon
is somehow screwing over open source is getting old. I don't really think
anything is 'screwing over open source', but if it's _anything_ , it is the
notion that we should severely restrict the licenses to ensure only the right
people can monetize it. To me, the original spirit of open source is "I don't
care what happens to this after I release it," like Linus releasing Linux so
long ago. To that end, Amazon has certainly leveraged hundreds or thousands of
FOSS projects over the years, but only a few have cried foul about it, and
they happen to be startups releasing open source software.

I'm totally supportive of developers and their rights to release software
under whatever license they feel appropriate, but if you release open source
software, benefit from your software being open source, for probably years,
and then someone else monetizes it... I'd argue you really ought to have seen
that coming from the get-go. The license explicitly allowed it, it's not a
loop hole that nobody knew about, it's absolutely intentional.

If people want to move over to shared source licenses like SSPL, or heck, even
just close the source... that's fine too. But please don't try to call it open
source or defending open source. It's defending a profit model that stopped
working. Totally reasonable thing to do, but it's nothing to do with open
source being corrupted.

~~~
throw2016
If we think of open source as an ideology that drove people to plant trees and
tend them so that users get fruits for free at a time when fruits are only
commercially available then the point becomes a bit more complicated.

We now have a cottage industry of people who harvest the fruits and sell them
so end users don't have to pick them themselves and also others who use the
free fruits to manufacture and sell milkshakes at higher margins than if they
would have to grow their own fruits or buy them.

The relationship with end users and ideology is broken as is the pipeline of
new contributors as end users do not anymore see the value and ideas that
drove the initial plantations as they deal with fruit pickers and milkshake
makers.

At this point open source loses its reason to exist and the people still
planting and offering fruits for free will inevitably question what they are
doing if they are only enabling third party businesses, but inertia means they
will continue as that's their life work. But there is unlikely to be a second
generation after them so that's the end of the movement. The story here is the
commercial interests have cut the branch on which they and everyone else sit.
For them it doesn't matter its just money not ideology and they will find ways
to profit but the loss is to end users. So for open source to be meaningful
its has to think carefully about how it will interact with commercial
interests as mix and match will kill it in a generation.

~~~
est31
There _are_ free milkshake plants, it's the distros. The distributors take all
of the open source projects, combine them, root out the bad fruits (DFSG non-
compliant, etc.), and offer it to everyone for free. I don't have to pick my
fruit from the AOSP tree and build my own image. I can just go to Lineage OS
and get prebuilt images for a multitude of phones. These projects of course
only reach a minority of people: Lineage OS has about 2 million users, while
the entire Android market has > 2 billions. But they exist. The free
milkshakes set a minimum bar that every milkshake plant has to meet.

Another thing to consider is the aspect of tree tending. If you are a
professional tree planter who has worked at a tree plantation with a big wall
around it, once you left, you won't be able to point at the tree any more and
say "look, I planted it". E.g. when trying to get hired at another tree
plantation. The only thing left of your own creation would be the diminishing
memories you had of the tree growing, having its first leaves, first real
bark, etc. Maybe your company won't like your tree for some reason and burn
it. If it's outside, it can't be logged. If the company pays you to tend the
tree, if they fire you you will still be able to tend it at a different
company. Red Hat or Sun can be bought but the trees were outside of the walls,
so the damages to the community were comparatively minor.

------
praseodym
Last year Elastic opened up the source code of their commercial X-Pack
offering: [https://www.elastic.co/blog/doubling-down-on-
open](https://www.elastic.co/blog/doubling-down-on-open). This means that
these components (security, alerting, etc.) are now available in the GitHub
repository, but they are covered by the proprietary Elastic license instead of
the Apache 2.0 license like the rest of the software. Some of the proprietary
parts can be used for free, some need a paid commercial subscription. None of
them are true open source (free as in freedom), though.

However, this is also when Elastic started muddying the waters (to quote the
AWS blog post). Most of the new features across the Elastic stack that were
added in the past couple of months (Index Lifecycle Management, APM UI,
Infrastructure and Logs UI, Kibana multi-tenancy, Kibana Canvas) are _not_
added to the Apache 2.0 codebase but are only available in the free-but-not-
open version. These features can be used for free with the Basic subscription
(no registration needed) but only under the terms of the Elastic license.

And this Elastic license is where AWS feels the pain. It clearly prohibits
SaaS offerings:

> You agree not to: [..] use Elastic Software Object Code for providing time-
> sharing services, any software-as-a-service, service bureau services or as
> part of an application services provider or other service offering
> (collectively, "SaaS Offering") [..]

(from
[https://github.com/elastic/elasticsearch/blob/6.6/licenses/E...](https://github.com/elastic/elasticsearch/blob/6.6/licenses/ELASTIC-
LICENSE.txt))

The license is very similar to moves that MongoDB and Redis made recently.
Only Elastic is just now doubling down on adding new (and highly demanded)
features to their proprietary offering.

~~~
SeaCow
If you read the Elastic License, it's actually worse than Mongo & Redis -- you
can't modify it and use it for anything other than testing. You can basically
install & use it only (definition of License, then the restrictions sections);
if your testing uncovers something wrong... what are you supposed to do, under
the terms of their license?

~~~
sam_lowry_
Discover Lucene.

~~~
samcday
Comparing Lucene to Elasticsearch is very apples to oranges, in my opinion.
It's like if MySQL did something people didn't like and the answer was
"Discover InnoDB".

Sure, Lucene might be all you need, if all you need is some basic freetext
search on a small set of data that fits on a single machine.

Though maybe you misspoke and meant to say "Discover Solr"...

~~~
sam_lowry_
Hm... Nope, ElasticSearch is a good entry point into Lucene. As your software
needs grow, you may discover the benefits of going a level below. Start with
ELK, then discover ElasticSearch, then use Lucene with your own way of
distributing data over multiple machines if needed.

Personally, I never had to use more than 1 machine even for huge data sets
(JFYI, Orbis and Wikipedia fit onto one bare metal server).

------
Mizza
This is an interesting new "threat" to the "open core" business model. ES
makes proprietary extensions to support their FOSS core product - then a tech
behemoth clones these features for their own hosted service, but makes them
Free as well. Good for the consumer, but bad for the company that originally
created the core FOSS technology and best for Amazon.

Have there been any examples of this happening before?

~~~
sl1ck731
The whole MongoDB thing recently.

And it doesn't appear that AWS is hosting these right now, just distributing
so they must be paying some kind of licensing for ES. But I'm sure that will
change.

~~~
Mizza
The blog post says that some of these features now appear in ESS (AWS hosted
ES), and that the rest will arrive soon.

Did AWS do the same thing with Mongo?

~~~
sl1ck731
They created a compatible hosted database.

I believe in response to some licensing changes MongoDB made, so the AWS
service targets the last permitted license release by Mongo.

[https://aws.amazon.com/blogs/aws/new-amazon-documentdb-
with-...](https://aws.amazon.com/blogs/aws/new-amazon-documentdb-with-mongodb-
compatibility-fast-scalable-and-highly-available/)

 __*I forgot to mention, they did not release the code for this as you may
have been looking for apple-to-apple comparison with the ES situation.

~~~
_msw_
Hi, I work at AWS on Compute services, but not directly on database services.
I also work on broader open source topics at Amazon.

Amazon DocumentDB was in development for a long time, and development started
well before any of the license changes that MongoDB (the company) made to
MongoDB (the database).

------
spectre256
I run a company that offers consulting and hosted services around a 100% open
core ([https://geocode.earth](https://geocode.earth), shameless plug).

As someone who has a very strong desire to continue to be able to make a
living while making open source software, I've been thinking long and hard
about all the possible paths to take while maintaining open-source software as
a job that pays the bills enough to run at least a small business.

Some paths and possible outcomes are:

\- Open-core: Make most code pure open source, and then save some good stuff
for pure proprietary. The lines are not _too_ blurred here, but this can get
annoying. (Elastic's X-Pack was _super_ annoying to deal with back in the 2.x
days). Risks: you have to balance crippling the core versus building
compelling proprietary features. You have to add additional complexity into
both codebases to deal with a clean delineation. Amazon (or other bigco) can
re-implement open-source replacements for your proprietary parts.

\- Source-available: This in my mind includes any of the recent Redis Labs
style licenses. Pros: more convenient from a setup perspective. Cons:
Incredible legal risk for your users (compliance is harder). Uncertainty as
all these licensees are new and possibly one-offs. Currently, there is a lack
of goodwill around these licenses. Amazon can re-implement your entire
interface (like with Mongo).

\- All open source: Simple, clean, but Amazon can just take all your code and
replace you, right?

It's possible to assume that all scenarios end poorly, but I really don't
think so.

Take any famous chef. They probably have published a cookbook with recipes for
many, if not all of their most popular dishes. People still eat at their
restaurants for at least the following reasons:

\- They want to experience the dish from the actual chef's establishment.
Whether it's to be sure they're getting the real deal or simply for the image
aspects of the experience, it's still worth it.

\- They know that the chef is always experimenting and pushing new things that
aren't _yet_ in a cookbook, and they want to try it.

Bringing this back to software, I am confident Elastic still has a reputation
of being able to build a better Elasticsearch hosted service than AWS (in my
experience, Elastic's is far superior). I also am confident that Elastic will
continue to innovate using their proven experience building search products in
the future, and that's a good reason to use their products or software over
AWS.

I believe that if you want to make money in open source you should do it by
having valuable experience with some particular open source software that
others are willing to pay for, not by building legal barriers to others doing
the same thing. That's what I plan to do for my business and I believe Elastic
can and will do it too.

My hope is that in a few years after all these "clever" attempts to build
moats around open source have proven futile, people will go back to good old-
fashioned experience leading the way. But maybe I shouldn't hold my breath.

~~~
anth_anm
You're incredibly naive and I hope you don't end up feeling screwed over by
one of the big companies in a few years.

~~~
spectre256
Honestly, if the open source project we maintain ever becomes popular enough
that a big company like Amazon puts effort into supplanting us, great.

We'll be known as the team that created and grew the project, and we'll have a
guaranteed job at Amazon and a bunch of other places doing something we have
unique experience with.

My goal isn't to leverage open-source software into enough of a moat to build
a billion dollar company. If it was, then I might be unhappy if Amazon broke
that moat.

I'm totally happy building a sustainable small business. Maybe even one that
won't last forever. Our team is small, our costs are low, and we are
comfortable. We are growing, sustainably, and providing real value to our
customers in the process (or they wouldn't pay us the rates we require to stay
in business).

~~~
loukrazy
It seems like VC funding is the real key to changing the calculus. If you get
as much money as Elastic did, you damn well start digging that moat and get to
monetizing.

~~~
spectre256
Exactly! VC funding has its place, but it also greatly restricts the types of
outcomes that are favorable.

I mean, in our case, we're building a geocoder. We'd have to raise, no
exaggeration, tens of billions to build a moat considering the current market
leader is Google. There's no way to guarantee that outcome.

On the other hand, there is _so_ much room left after Google for other
companies if they don't have to get massive. There's room for us, and at least
a dozen other companies I know of that have their own take on geocoding that
their own customers appreciate.

------
HillRat
Definitely creates a new risk model for hybrid OSS-Enterprise software: build
something attractive enough for the big players to co-opt and they might just,
ah, fork you up. Now Elastic either accepts pull requests that create clean
OSS versions of their pay-locked functionality, or they accept the existence
of a feature-plus “Amazon Elasticsearch” fork in the market. While I give
Cockcroft all the credit in the world (remembering him from his Sun days),
this is still a tough spot for Elastic to be in. (More discussion in the
Twitters:
[https://mobile.twitter.com/_msw_/status/1105260461149151232](https://mobile.twitter.com/_msw_/status/1105260461149151232))

~~~
quickben
From that twitter feed

"adrian cockcroft:

What should we have done instead, that Elastic would have agreed to? We have
customers to support"

And this is where the spot he is in gets tighter. Because now the aws business
model threats anybody using X as Amazon's user that should be monetized.

So I don't see Amazon backing off profits, and I don't see the OSS community
getting any less pissed off from not seeing a piece of the cake.

This story (Mongo, elastic co, etc) seems far from being over.

~~~
jake_morrison
Amazon should pay money to the people who create the open source. Not
necessarily because a license compels them to, but because it's the right
thing to do. If they don't, then the ecosystem collapses.

A good example is RedHat. There is basically no money in simply selling
operating system licenses anymore, but someone has to keep developing it.
Amazon makes lots of money off of EC2 instances. They could easily throw a few
million a year into paying kernel developers. But will they? Are they just
going to be a free rider, while keeping anyone else from being able to make
any money too?

~~~
cthalupa
>Amazon should pay money to the people who create the open source. Not
necessarily because a license compels them to, but because it's the right
thing to do. If they don't, then the ecosystem collapses.

Elasticsearch is built on top of Lucene. When Elastic received triple digit
millions in VC funding, did they pay the Lucene developers? They're not even
Apache Foundation sponsors, as best as I can tell.

>Amazon makes lots of money off of EC2 instances. They could easily throw a
few million a year into paying kernel developers. But will they?

Amazon contributes back to the Linux kernel. How much contribution is needed
to make it "right"? How do we determine it? Does Amazon need to employ them
directly, or sponsor projects? What about their Platinum sponsorship of the
Apache Foundation?

~~~
Yeroc
Actually ElasticSearch is paying the salaries of a number of Lucene
committers. At least they did so already about 5 years or so ago. Haven't
followed things closely more recently.

~~~
tasuki
Yes, this is still the case.

Source: I work at Elastic.

------
sailfast
As an active developer and maintainer of a cluster, I strongly disapprove of
this move by AWS.

By forking on an earlier instance of the codebase for ES and Kibana they're
not only creating an open source version of the code but also attempting a
fork of the community - plugin developers, user groups, etc. It's extremely
frustrating. EDIT: If folks actively building plugins have felt pain here
because of decisions by Elastic, then my view on this point changes
considerably, obviously.

I find it very hard to believe there was no compromise to be had here between
the current Elastic roadmap for their product and their previous work.

How is this good for users of the current tools? How does this help create
better capabilities on a common platform? Will this encourage users to build
businesses on elasticsearch open source elements?

I'm not sure where the community ends up after this, so I'm not sure I can
support by using AWS' elasticsearch tools.

~~~
sidi
As someone who is involved in the ecosystem and maintains some open-source
projects based on ElasticSearch, I am similarly curious about the move to not
build upon existing projects (there could be a good reason, but haven't heard
it yet).

Why wasn't ElastAlert used / built upon for alerting?
[https://github.com/yelp/elastalert](https://github.com/yelp/elastalert)

Or why wasn't ElasticSearch SQL used / built upon for SQL?
[https://github.com/NLPchina/elasticsearch-
sql](https://github.com/NLPchina/elasticsearch-sql)

Or why wasn't SearchGuard used / built upon for Security?
[https://github.com/floragunncom/search-
guard](https://github.com/floragunncom/search-guard)

All of them have Apache 2.0 license.

~~~
ssgao
For SearchGuard, only the basic auth is Apache 2.0. Anything practical you
need to pay to use it.

~~~
sagichmal
Why isn't AWS paying to use it, then? It's not like they don't have the money?

~~~
ssgao
there's no "they", there's only us. AWS paying to use it means we are paying
to use it. Between that and a truly Apache 2.0 project, I think it's a no-
brainer for everyone.

~~~
sagichmal
Why shouldn't you pay to use it, then?

More generally: why should AWS get to leverage its near-monopolistic position
in the SaaS market to perform essentially hostile takeovers of the software
projects it packages and runs? Simply because they have the resources to do
it?

~~~
e1ven
I'm very glad that they've done this.

The code was open, and the company decided to close it for the new versions,
so it forks. Forks are common when people don't like the corporate direction
of an OSS project - It's a strength, not a weakness. Openoffice/Libreoffice,
Hudson/Jenkins etc.

Imagine if Linus said that the next version of Linux would be paid-only.

Within 15 minutes there'd be a new librelinux repo that people could
contribute to instead.

Someone was going to make a new fork anyway - This is Amazon putting their
money where their mouth is to fund and support that new fork.

~~~
dhd415
To be clear, no Elasticsearch code or features that were OSS were closed. As
new Elasticsearch features are developed, some of them are now released as OSS
and others are released with a commercial license.

------
mcescalante
I recently decided to experiment with ES and prefer to evaluate things with
Apache 2.0 licensing if available, even if I may pay for to license it at a
later time. Elasticsearch Still maintains -oss suffixed docker images and
builds that are true Apache 2.0. You can read more about this here:
[https://www.elastic.co/products/x-pack/open](https://www.elastic.co/products/x-pack/open)

I jotted down instructions to run the docker containers based on a few
different solutions from different places:
[https://gist.github.com/mcescalante/6be03751c820677cf0f15c7b...](https://gist.github.com/mcescalante/6be03751c820677cf0f15c7b864fa23a)

I'm sharing this because I got the vibe from the linked post in the comments
and the overall marketing of "Open Distro for ES" that Elasticsearch was 'not
open anymore' and had just discovered the Apache 2.0 builds within the last
week.

------
kristoff_it
"The maintainers of open source projects have the responsibility of keeping
the source destination open to everyone and not changing the rules midstream."

Since when is anybody entitled to any free software. Projects can do what they
want with their code, and while free software (the Stallman kind) is nice, if
projects cite you as the main reason for their change of direction, you are
intellectually dishonest if you just blame them for not being willing to do
volunteer work (for you) anymore.

This, to me, looks like an abusive "look what you made me do".

~~~
unethical_ban
No one has to "not be a dick" but the world is better when you try for the
goal.

What the blog is saying is "If you build an open source tool and people put
their time and effort to using it, closing more and more of the toolset to
paid add-ons isn't cool".

AWS just said "Hey, we can build those ourselves and give back to the
community". They volunteered and made it happen.

~~~
anth_anm
They just said "Fuck you, we have more money so we own you now".

------
jake_morrison
Basically Amazon screwing Elastic. Take their code and monetize it, not giving
anything back. When Elastic changed their license to prevent it, Amazon forks
it.

It's the biggest risk to open source right now. They are following the
strategy of "commoditize the complementary good". Since Amazon makes money
selling machine cycles, they want the software to be free.

~~~
__david__
No, no, no. Amazon is using the software _in the way it was intended_ ,
legally and ethically. It's like companies published software under a license
and never gave a thought to the consequences of the license they chose. If you
license permissively, be prepared for your competitors to use it!

~~~
weberc2
Which is why Elastic changed their licensing, and now Amazon is playing the
hapless victim.

~~~
Twirrim
They're not playing the hapless victim. They're doing _exactly_ what the open
source license was designed to do and support. They're forking it because they
don't like the direction of the current mainters, and are releasing their own
open sourced updates to it.

We've seen this happen repeatedly through many open source projects over the
last couple of decades. Take the Jenkins/Hudson split, for example, or
OpenWrt/LEDE, Node.js/IO.js, or to stretch even further back Mambo/Joomla.

The whole purpose of the open source license is to ensure that you are always
able to patch, support, and maintain what you're running. That you're not
stuck depending on a single company to keep your software running. That you're
always able to add missing features as you see them, etc. (obviously, the
license you choose dictates whether you have to provide those back).

By mixing up proprietary code with open source code _Elastic_ betrayed those
central ideals of open source, as have Nginx and any number of companies that
have adopted this model. The reason why should be pretty obvious. By building
the company model around premium modules, they have disincentivised themselves
to fixing or enhancing the open source version of the software. Worse, they're
not incentivised in any fashion to even accept patches that open source
contributors provide that provide the missing features etc.

By way of example, there's a fix for a common issue in Nginx that has landed
in their upstream commercial version of Nginx, that hasn't made it in to the
open source version (at least last I saw it hadn't, and the fix had been in
upstream for quite a while). Nginx ignores DNS TTLs, and can seriously trip
you up unless you happen to know to use a particular combination of variables
and statically configured DNS resolvers in nginx to make it DNS TTL for
servers it proxies too. The company behind Nginx has had zero incentive to
provide that in the open source version, _or_ accept patches that fix that
behaviour. That leaves people tripping over the same bug for years on end,
until they discover, just like so many before them, those blog posts that
detail the behaviour and how to save yourselves from it.

What each of these companies have done essentially makes actions like Amazon's
actions here completely inevitable.

~~~
judge2020
While I agree with you, companies should really know what a free-as-in-free-
beer license means for an open source product, AWS effectively plays the
helpless victim in the linked blog post. If it was a simple "we knew this was
going to happen, so we forked it" they would have made a short announcement
instead of a 1500 word essay on the issue.

------
devhead
Hosted ES by AWS is a good solution to regret later in life.

They have minimal tooling, poor change management, poor logging, inability to
address broken state and no support what so ever for a service you paid for. I
won't go back to their hosted ES.

Edit: spelling

~~~
softwaredoug
I really recommend services like Bonsai and MeasuredSearch (I know there's
others, but haven't used them). They have a lot of tooling and provide a lot
of great support. In fact, the support and attention to your account is
probably the biggest differentiator they have to AWS tooling.

------
gtsteve
For further reading, this is also mentioned in the AWS blog:
[https://aws.amazon.com/blogs/aws/new-open-distro-for-
elastic...](https://aws.amazon.com/blogs/aws/new-open-distro-for-
elasticsearch/)

~~~
jeffbarr
Thanks for sharing; I wrote the post and also submitted the distro to HN. In
most cases (apparently per HN policy), blog posts that announce new services
are removed in favor of direct links to the offering.

~~~
karussell
Is it really honest to say "Keeping Open Source Open"? Elasticsearch is 100%
open source, even Apache Licensed, and you are implying that this is not the
case "somehow". Sure "security" is missing, but who says that this has to be
part of the open source project "Elasticsearch"?

Maybe it would be more honest to say "Forcing open source backing companies to
avoid open core that is hated by AWS" ;) ?

(btw: I'm not associated with Elastic except loving their open source
projects)

~~~
karussell
Ah, I probably did not correctly catch the news:

> Unfortunately, since June 2018, we have witnessed significant intermingling
> of proprietary code into the code base.

And you mean the x-pack stuff?

[https://github.com/elastic/elasticsearch/blob/master/LICENSE...](https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt)

Interesting, wasn't aware of it.

------
jillesvangurp
I'm a long time Elasticsearch user (from before 1.0). I've also used Solr and
I've built some lucene based search solutions before that as early as 2003.
I'm also a customer of their hosted elastic cloud and I know several people
inside that company; both former colleagues and through meetups. They do
awesome stuff technically and it's a great product to use.

In short, I think Amazon doing this is a good thing. I've had the pleasure of
using their hosted solution and I can't recommend it with a straight face. Key
management features in the API are blocked and upgrades are a pain in the ass
(as in, not supported). Support and documentation is minimal. It's not
terrible but you get a much better bang for buck by using Elastic cloud. Self
hosting is not something I recommend you do until you need to. It's a PITA and
requires a non trivial amount of devops by someone who knows what they are
doing to do it right. Rarely worth the price and hassle unless you need to run
this at petabyte scale on custom hardware.

First, lets look at what this is not. This is not a fork of Elasticsearch or
Kibana. If you go to the github account associated with this website, you'll
see that this is simply Amazon publishing the source of plugins they use in
their hosted solutions. The readme of those plugins basically instructs you to
download the appropriate version of Elasticsearch from Elastic.

Second, I was there when Shay Banon announced the licensing changes at Elastic
ON in early 2018. At the time I already thought this was a somewhat
unfortunate move as it muddles the water in terms what is and isn't open
source. Technically what they announced was shared source for things that were
previously closed source. Which is better than nothing but not ideal. The
whole thing was designed to get more people to try this out than was the case
when you needed to get a license before you could play with it. It's basically
a form of shareware.

We use Elastic Cloud, it seems all the cool stuff they talk about is mostly
off limits to their basic tier users astill nd requires quite a bit of skill
to get started with. I appreciate that they are trying to upsell this stuff
but I seem to be doing fine with just the standard OSS features (and I know
they are OSS because this is what I used before elastic cloud as well). I
could switch back to self hosted and not lose much (other than convenience).

Third, I think the Elasticsearch ecosystem benefits from having outside
contributors; especially from big corporate entities like Amazon. Their
history is literally bootstrapping off Apache Lucene as a small group of
developers and consultants. They employ core committers and one of the co-
founders is a core committer as well (I worked with him before Elastic
happened). They've been working together with the Solr people on improving
Lucene for the past six years and have made some awesome improvements. Lucene
sees regular contributions from academics as well. It's a healthy project and
it is core to what Elasticsearch does. IMHO this is something they need to
nurture and keep and something that is core to their culture.

The plugin ecosystem around Elasticsearch is where the action is commercially.
They've built a lot of cool stuff around Elasticsearch that adds a lot of
value. That too sees a lot of outside innovation. So, Amazon publishing some
plugins is great. Also, given that they use it at scale, they probably have a
thing or two to say about how to improve Elasticsearch and probably should be
regularly creating pull requests, creating issues, etc. That's a good thing.
Anything blocking that would be a bad thing

In short, Elasticsearch created a big mess by mixing open and closed source
and people are working around them. Instead of fighting that, they should
double down of being open source at their core and benefit from this rather
than attempting to fight it. I'd recommend them to 1) re-create and market
their community edition, 2) actively considering offering support for Amazon
customers and other people using their technology but outside of their bubble
(e.g. Graylog). Work with the community instead of against it. Everybody wins.

------
fredsted
I think a fork/distro of ES was bound to happen.

As a heavy ES user, I've looked into getting a license to access the so-called
X-Pack features, most of which are very basic and should belong in the OSS
distribution anyways. Lots of those features are now available as plugins, but
until now there's been no overall solution for getting the missing features.

After an almost two hour long phone call, we kind of got the pricing in a
roundabout way from the salesperson; tens of thousands of dollars per year for
a small cluster. We REALLY wanted to get this, which is why we dealt with the
sales process, but there was just no way to justify that to our management.

------
hardwaresofton
If you're looking for options in the search space:

\- [https://github.com/toshi-search/Toshi](https://github.com/toshi-
search/Toshi)

\- [https://github.com/tantivy-search/tantivy](https://github.com/tantivy-
search/tantivy) (more like lucene)

\-
[https://github.com/blevesearch/bleve](https://github.com/blevesearch/bleve)

------
david_p
For those curious:

\- the website is copyrighted to Amazon Web Services

\- all commits in the github repositories are made by AWS staff.

~~~
scarhill
> all commits in the github repositories are made by AWS staff.

Given that it was just made public this morning, it would be surprising if it
were otherwise. The real question is what the contribution model looks like
going forward. The blog post says "Contributions are welcome, as are bug
reports and feature requests"[1], but of course the devil is in the details.

1 - [https://aws.amazon.com/blogs/aws/new-open-distro-for-
elastic...](https://aws.amazon.com/blogs/aws/new-open-distro-for-
elasticsearch/)

------
all_usernames
Title isn't quite right: "As was the case with Java and OpenJDK, our intention
is not to fork Elasticsearch, and we will be making contributions back to the
Apache 2.0-licensed Elasticsearch upstream project as we develop add-on
enhancements to the base open source software."

~~~
tyingq
Unless they assume every enhancement they put in their, uh, "not a fork" repo
will make it upstream, unaltered, it's a fork.

That seems inevitable to me. There's probably something eventually that either
isn't wanted by Elastic, or wanted...but implemented in some different way.

------
cnj
It looks like Amazon and Elastic have maneuvered themselves into a lose-lose
situation.

Amazon has to invest into an Open-Source fork, probably won't benefit from
some of the feature development Elastic will be doing, and won't get any help
running their ES-as-a-Service.

Elastic won't see any revenue of the AWS ES-as-a-Service, and has lost a lot
of goodwill with (potential) customers.

I think they could have reached a compromise where AWS makes sure a share of
their revenue from their ES-as-a-Service ends up at Elastic, and Elastic
recommends and supports ES-as-a-Service for their customers who are on AWS.

But these are apparently not the times for compromise, so we end up in a
situation that is worse for AWS, Elastic and everyone in the wider community.

------
jerrac
I like seeing a, hopefully, solid OSS implementation of security for ES. I
understand the companies need to make money and pay developers, but I've never
been very comfortable with them keeping the security features out of the OSS
version.

------
jmartrican
As I long time user of ES I applaud Amazon's effort here. ES is not new and it
is missing basic functionality.

------
rooam-dev
IMHO using "open" is just strategy to attract devs and companies that are
looking for FOSS solutions, but in reality it's just another way to vendor
lock-in. I hope i am wrong.

Honest question, besides not paying licenses, is there any technical reason to
do this?

~~~
sl1ck731
I was trying to implement ES previously but kept running into things I thought
we're included "open source" but turned out to be an extra paid feature. With
this I know what I'm getting.

There is no more vendor lock-in with this than with Elastic.

~~~
rooam-dev
Not yet, but it will be. Open Distro ES. At some point features will increase
and migration from AWS will be more difficult.

~~~
otisg
Then, if that happens, we will see the "Open Open Distro for ES" ;)

------
kureikain
Is this some kind of AWS is trying to cleanup their reputation in ES? Their
offer of ES is super weird. THe data node is only support 2 AZs, instead of 3.
They also require using the IAM for authentication(you sign the request with
access+secret keys) in other words, hard to simply `curl` it.

So if they added this, maybe they will eventually bring these into AWS ES
offer, so that they can do the thing that X-Pack offer - because we cannot
install plugins on AWS ES yet.

~~~
jen20
Of all the (many and varied) criticisms that can be levelled at the AWS
Elastic Search service, integration with IAM is a strange one: it’s ‘harder to
curl’, because it’s more secure.

------
anth_anm
Hot take, the future of open source is just massive companies sponsoring
projects to get free dev hours.

You can try to fight it, but if Amazon wants your project they will either
take the code or build a compatible API. There is no defense, IP rights only
favour proprietary software.

------
bratao
If someone is looking for a third opensource alternative to Elastic Search, I
highly recommend [http://vespa.ai](http://vespa.ai) We migrated from ES, and
it is truly next-gen compared to ES

~~~
otisg
I'd love to hear more. Could you describe your experience and elaborate on how
and why it's next-gen compared to ES?

------
otisg
WDYT: Would it have been better for AWS to initiate Open Distro for ES as part
of e.g. Apache Software Foundation, whose mantra is "community over code"?

(or if not ASF for some reason, then Eclipse or CNCF for example)

~~~
fredsted
Yes, and I think they should have done so, just look at the responses in this
thread. It's just been launched and everybody is afraid of Amazon vendor lock-
in. For many techies, just the Amazon name is scary even though this is Apache
2.0 licensed.

------
cik
I couldn't be happier about this. We build on Elastic today - and there are
several features that we've written into our codebase from scratch, that are
available in commercial Elastic only. Quite frankly if there's a community
that once again "open", and that we can even contribute some of our technology
to with everyone benefiting - I'll be pretty thrilled.

In tomorrow's test I'm going to throw ~3 billion messages at Elastic 6.x and
then immediately stand this up over the same database... Time to see what
happens :)

~~~
cik
I'm going to respond to myself - but so far so good. I just ingested ~14B
messages into a local instance, on top of an existing Elastic.

Interestingly enough - and to Amazon's credit, the have SSL running by
default, with a default admin password. Our (soon to be old) solution to this
was to run nginx in front of Elastic - and do another bunch of things with
stunnel for balancing. I'm happy to see default encryption.

Pretty happy to start!

------
sidi
This is a good project to happen, and kudos to AWS for doing this! However,
like some in the thread, I suspect AWS has been building this for a while for
their own product and Elastic's license change may have influenced the
decision to open-source.

Getting a good out-of-the-box granular security has been a long overdue pain
point with ElasticSearch, there have been good alternatives for the other
problems it is replicating.

1.) ElastAlert -
[https://github.com/yelp/elastalert](https://github.com/yelp/elastalert) for
alerting, 2.) ElasticSearch SQL - [https://github.com/NLPchina/elasticsearch-
sql](https://github.com/NLPchina/elasticsearch-sql) for writing SQL queries.

Would very much like to hear from OP why these weren't contributed to instead
of creating another alternative.

We've been working on a similar project in Golang and now well may be as good
a day to open-source and put it out there:
[https://github.com/appbaseio/arc](https://github.com/appbaseio/arc). It's an
extremely light-weight API gateway for ElasticSearch that at the core comes
with an out-of-the-box security system based on users and permissions (set
granular ACLs, Rate Limits, TTL) inspired from a proprietary security system
we built over the past year -
[https://appbase.io/features/security/](https://appbase.io/features/security/).

~~~
edraferi
According to the release blogpost [0], "SQL Support [...]is an improved
version of the elasticsearch-sql plugin." They're not replacing the SQL
extension you linked -- they're integrating it.

[0] [https://aws.amazon.com/blogs/aws/new-open-distro-for-
elastic...](https://aws.amazon.com/blogs/aws/new-open-distro-for-
elasticsearch/)

~~~
sidi
That's definitely good to read. And gathering from other comments, they have
done something similar for Security too. :-)

------
jahewson
Who is acting in the best interest of the open source community, Elastic or
AWS?

~~~
benatkin
Neither. Right now AWS seems to be acting in the best interest of the
ElasticSearch community, but this doesn't extend to open source as a whole.

For the long-term, the open source community should look to projects where the
center of gravity is outside of an open core or FANG company. ElasticSearch
went open core a while ago, and Elastic's form of open core is more like
MongoDB or Redis than it is like nginx or Docker. As for Amazon, I think I'd
rank them below facebook, and I would rank facebook below google and
Microsoft, as stewards of open source projects.

~~~
benatkin
Hadn't seen this. Wow! [https://www.nginx.com/blog/nginx-
joins-f5/](https://www.nginx.com/blog/nginx-joins-f5/)

------
eric_b
I dunno if I'd be long $ESTC at this point. Elastic took $162 million in
venture funding before IPOing. In the last year their Operating Income was
-$43 million. A lot of that is sales/marketing related, but at some point they
need to make a profit. This open distro is certainly not going to help.

As someone who's worked at many big corps on Elasticsearch installs, I've been
worried about them pushing away from OSS for some time. Many big corps want
some of these "Enterprise" grade features Open Distro is offering, and if
Elastic doesn't already have a foothold in them, I wager they'll be using this
instead of paying.

As an aside, how come people still think these Open Source/Core commercial
businesses are viable? We've seen time and again that once commercialization
occurs, the OSS stuff goes right out the window. (RedHat, Docker, MongoDB,
Sentry etc)

~~~
kapilvt
Just commenting on the aside, we should be careful about painting with an
overly broad brush. Each of these companies and organizations is different, at
least two you've mentioned are fully opensource afaik not open core (Redhat
and Sentry). Redhat continues to do tons of opensource work across many
ecosystems, kubernetes, linux kernel, gnome, etc. The Redhat sw model isn't
open core, because what they sell isn't proprietary afaict its a support with
updates model, that they also distribute under a different brand then the
opensource (fedora/centos/openshift okd). Additionally we're talking about a
multi-billion dollar company, seems like a success to me. For Sentry, their
more focused on Saas platform, but afaics an on premise distribution model
doesn't reference a commercial product thats differential to their opensource
repos aka its not open core, afaics. [update] sentry on premise install docs
[https://docs.sentry.io/server/installation/](https://docs.sentry.io/server/installation/)

~~~
eric_b
RHEL, as far as I know, is not available as a nice installable package without
a subscription, at least to run in production. CENTOS was started as a
separate entity to make bundling the RHEL source in to easily installable
distributions more straightforward, as Red Hat made it difficult.

As far as Sentry, I'm referring to the fact that they have not cut a release
of their on-premise in some time, and features have started to diverge, though
they assert they will reach parity with on-prem "at some point"
[https://forum.sentry.io/t/when-will-a-new-version-be-
tagged-...](https://forum.sentry.io/t/when-will-a-new-version-be-tagged-for-
sentry-open-source/5622/10) (Though again, like RedHat, the source is
technically available...)

My main point is that OSS is at odds with making money.

~~~
kapilvt
Easily installed isn't part of the opensource definition, any more than source
available means opensource. Calling companies that are pure opensource, open-
core is misleading and detracts from the conversation, imo. Centos was a
binary distribution because Redhat made redistributions of _binary_ RHEL
harder to distribute(images/logos/binaries/names via copyright/trademark), but
that was part of their business model. The source however was available under
opensource licenses, and most of that was from separate upstream communities,
aka its linux distro (though they have lots of other products). Re sentry, if
I can pull the repo and get the same bits under OSI approved license, its
opensource to me. Again opensource isn't about release management practices
and nice installers (although for sentry its just a docker run/build away).
They might be part of good community of practices for sure though.

I'd say opensource business models are different and potentially more
difficult, but there are plenty of companies selling/supporting/building
opensource. The delta seems to be most vc backed companies sponsoring
opensource have different expectations on growth/value extraction from
customers. aka open-core is not really about opensource, its about selling
proprietary products, so I'd prefer not to mis-label those who aren't doing
that.

~~~
eric_b
We can argue all day about following the spirit of something vs the letter of
the law, but all four of those companies have direct business incentives to
make it difficult for those wanting to use their software for free to do so.
No surprise then that all have taken steps to dissuade free users or make
things more challenging. You're arguing about labels, which is beside the
point.

------
speedplane
Pure software open source projects should not accept VC money and expect to
stay purely open source for long.

------
patwolf
I was always surprised one of the cloud providers didn't release a cloud-
native full-text search service. A lot of projects could benefit by a simple
text search service. I always found Elasticsearch to be a little clunky and
found myself wishing there was an alternative. While I don't like to be locked
into a cloud vendor, having a simple text search service might be enough to
sway me towards a particular vendor, even if it's proprietary.

Amazon's Elasticsearch service does simplify installation and management, but
honestly I wish it were even simpler. I don't want to even think about
CPU/memory sizing or the number of shards.

~~~
thekyle
Well Amazon offers CloudSearch which is a proprietary full-text search engine
native to the AWS cloud.

[https://aws.amazon.com/cloudsearch/](https://aws.amazon.com/cloudsearch/)

EDIT:

Just to expand on this, I also found that Google Cloud and Azure both have
similar proprietary services.

[https://cloud.google.com/products/search/](https://cloud.google.com/products/search/)

[https://azure.microsoft.com/en-
us/services/search/](https://azure.microsoft.com/en-us/services/search/)

------
itaifrenkel
Amazon is forcing elastic to make money from their hosted offering, instead of
their proprietary x-pack modules. Professional Services don’t scale as
software do (compare AWS with Rackspace) - so I’m ignoring that income stream
for now. Basically, they are trying to force Elastic into the AWS red ocean.
Elastic, in response, would have to make a technology leap in order to best
solve their customers problems. Examples include: Integrated data prediction,
anomaly detections, clustering (online machine learning). Seamless multi
region cluster deployment (with some consistency guarantee)...

------
mosselman
'Distro' makes it sound like a linux-distribution with, in this case,
Elasticsearch installed on it. From what I read on the page and on the aws
blog this is not the case though?

~~~
alexose
I thought the same thing. After some searching I realized that "Open Distro"
isn't actually a linux distro after all. This is a non-fork release of
Elasticsearch with the proprietary parts stripped out.

~~~
sam_lowry_
Thanks for figuring it out. Shame on the publishers of the OP website for
making it deliberately unclear.

~~~
tylersmith
In what why did they deliberately make the meaning unclear? The middle of the
page says "An Apache 2.0-licensed distribution of Elasticsearch". It's saying
in black and white that it's a distribution of Elasticsearch, not of an
operating system.

In what way is being deliberatly unclear?

~~~
_msw_
Hi, I work at AWS on compute services, but spent _a lot_ of my life building
Linux distros.

For those of us with Linux distro building experience, this may be initially
confusing. But calling a curated collection of software a "distro" has been
used for decades outside of Linux distributions.

For example, there are multiple distributions of TeX, like TeX Live, MacTeX,
etc.
[http://www.tug.org/interest.html#free](http://www.tug.org/interest.html#free)

See also the Anaconda Distribution for Python and R:
[https://www.anaconda.com/distribution/](https://www.anaconda.com/distribution/)
(not to be confused with the Anaconda that I know and love:
[https://en.wikipedia.org/wiki/Anaconda_(installer)](https://en.wikipedia.org/wiki/Anaconda_\(installer\))
)

------
ousta
Well what about enterprise support? When customers buy an enterprise solution
at elastic they get support right? what would entitle them to move to this
offering.

~~~
otisg
There are other companies that offer enterprise support for Elasticsearch
(e.g. [https://sematext.com/support/elasticsearch-production-
suppor...](https://sematext.com/support/elasticsearch-production-support/)) as
well as training an such. No loss for ES users there.

~~~
ousta
well it is not same. vendor support and third party services (aka support).
you can influence the product through support and through being a direct
customer.

~~~
otisg
You would think so, but that is not how it works with Elastic and lots of it
customers. I have heard that from N of their (ex-)customers. I get why this
happens for them. There is a certain vision and goal, and anything that
doesn't align is a nuisance. This is part business part philosophy, culture,
and brand you want to build in and around your business.

------
olavgg
Question, can I checkout the code from Github and build it with ./gradlew
assemble as Apache License 2 or do I need to delete some modules before doing
that?

------
otisg
My Twitter followers include a lot of Elasticsearch and Solr users. It will be
interesting to see what they think of this from their POV as users...
[https://twitter.com/otisg/status/1105146138921979905](https://twitter.com/otisg/status/1105146138921979905)

------
dman
The cloud that creates an Itunes like solution for OSS, creating a business
model for OSS and earning the respect of developers will make bank like Apple
did. Am hoping for one of the cloud players to adopt this strategy.

Disclaimer: Views expressed above are my own and do not reflect any opinions
from my employer.

~~~
vajrabum
You mean like yum install or apt get? If so, then yes but your prediction is a
little late

~~~
dman
I mean something where the Apache authors get paid based on Apache usage by
cloud customers (as an example).

------
bd103dall
Found this perspective interesting: [https://thenewstack.io/what-the-fork-
amazon/](https://thenewstack.io/what-the-fork-amazon/)

------
all_blue_chucks
ITT: people shocked by the fact that open source licensing really is open
source.

~~~
whoisjuan
\- It's Open Source for everyone* -

*except for Amazon.

------
javadevmtl
P.s: I think people are confused. This isn't a fork? Basically AWS should have
came out and said. "Hey folks we created brand spanking new OSS plugins with
sepcific funxtionality for Elastic" and you can use them!

------
unethical_ban
I notice that ODFE (as I am trying to make the acronym a "thing") is packaged
for Docker and RPM. It would be nice if they provided the plugins/extensions
separately, or otherwise had build-from-source instructions.

------
drkrab
How is elastic (the company) going to make a business going forward? I would
expect that selling these enterprise add-ons (or similar ones) is a
significant part of their revenue.

~~~
baud147258
It depends on how much revenue elastic is getting from their hosted offering
and support contracts.

------
a012
What do they use for Alerting? IIRC Alerting with Elasticsearch 6 is available
on Enterprise edition, not OSS. And what the OS under it actually?

~~~
sciurus
They're using [https://github.com/opendistro-for-
elasticsearch/alerting](https://github.com/opendistro-for-
elasticsearch/alerting)

The main selling point of the project is providing open source alternatives to
features that Elastic has kept proprietary.

------
ignoramous
I see this as a net positive ([https://xkcd.com/927/](https://xkcd.com/927/)
notwithstanding): Elastic.co, imo, has iron-grip (in terms of licensing and
features that it builds) on XPack and Elasticsearch. This incredible
technology that rode on OSS wave is increasingly controlled by a for-profit
organization that also acts like one. This is very similar to how Google dealt
with numerous forks of Android by essentially moving key pieces to
GooglePlayServices and other closed-source apps in addition to responding with
shutting out manufacturers and SoC providers that weren't OHA signatories
[0][1]. Amazon felt the burnt of that [2].

Apart from including/forking other community plugin projects (elasticsearch-
sql, SearchGuardSSL) in open-distro [3], AWS seems to be investing to make
this more "cloud-native", as well: This observation comes from
PerformanceAnalyzer (released) and IndexStateManagement (request-for-comment).

\- IndexStateManagement [4] is simply "never send a human to do a machine's
job" automation of common yet frequent tasks required to keep elasticsearch
humming along. If used in conjunction with AWS's/GCP's/Azure's managed-cloud
offering, this feature might be an amazing sell for a lot of enterprises that
want to set up things and have them running with as few dev-ops/engs as they
can.

\- PerformanceAnalyzer [5] could track multiple signals from servers running
open-distro and may perform reactionary actions to either shed load, throttle
requests, shift workload and so on, something similar to what MeltWater did
[6].

[0] [https://arstechnica.com/gadgets/2018/07/googles-iron-grip-
on...](https://arstechnica.com/gadgets/2018/07/googles-iron-grip-on-android-
controlling-open-source-by-any-means-necessary/)

[1] [https://techcrunch.com/2009/12/22/google-open-when-
convenien...](https://techcrunch.com/2009/12/22/google-open-when-convenient/)

[2] [https://www.theguardian.com/technology/2013/oct/16/amazon-
se...](https://www.theguardian.com/technology/2013/oct/16/amazon-set-to-
launch-htc-built-smartphone)

[3]
[https://news.ycombinator.com/item?id=19361234](https://news.ycombinator.com/item?id=19361234)

[4] [https://github.com/opendistro-for-elasticsearch/index-
manage...](https://github.com/opendistro-for-elasticsearch/index-
management/blob/master/RFC.md)

[5] [https://opendistro.github.io/for-elasticsearch-
docs/docs/pa/...](https://opendistro.github.io/for-elasticsearch-
docs/docs/pa/reference/)

[6]
[https://news.ycombinator.com/item?id=18413862](https://news.ycombinator.com/item?id=18413862)

~~~
praseodym
The IndexStateManagement plugin is basically the same thing as Index Lifecycle
Management (ILM) which was introduced in beta form in Elasticsearch 6.6. ILM
is under the Elastic license though, so it’s not free-as-in-freedom, but the
feature is included in the free-as-in-beer basic license.

[https://www.elastic.co/guide/en/elasticsearch/reference/6.6/...](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/index-
lifecycle-management.html)

------
ggm
I currently run stuff in elastic. I am now thinking that postgres:JSON may be
better for me.

------
bd103dall
Found this perspective interesting around COSS vs FOSS:
[https://thenewstack.io/what-the-fork-amazon/](https://thenewstack.io/what-
the-fork-amazon/)

------
ousta
It is hosted on AWS as a service hence not open source.

------
james_s_tayler
The whole Amazon takes an open source product and monetizes it almost reminds
me in a way of someone getting rich off an asset flip on the unity store. Just
a big, giant, corporate version.

------
viburnum
OK this sounds like bullshit. Can somebody explain what's going on?

~~~
benatkin
Amazon doesn't like the direction that Elastic is headed, so they've forked
ElasticSearch. They don't want to maintain and document it all by themselves,
and they do want for their users to be able to run it themselves (especially
for running it locally or on CI), so they've open sourced it.

~~~
markbnj
Well to be fair they explicitly said it wasn't a fork, and that any work they
do on it will be pushed upstream. The cynical view, I guess, is that the
commits they push will be providing free features that compete with Elastic's
paywalled features.

~~~
benatkin
I can't find anywhere where they said it isn't a fork. I searched the article
for the word _fork_ to make sure I didn't miss anything. They said it wasn't
an _internally forked version_ meaning it isn't a private fork, and they said
that their _intention_ wasn't to create a fork.

I'm not completely sure if anything that's core to ElasticSearch is forked.
The repos on GitHub are modules not including one that's called
"elasticsearch" or "core" or "main" or something like that, but some seem
pretty essential.

As for myself, I like to call an alternative distro a fork, and I disagree
that their intention isn't to create a fork, based on my idea of a fork.
They're free to have their opinion on what constitutes a fork, but I don't
think they're an authority on it so I'm gonna stick to my current view of what
a fork is...

~~~
markbnj
The statement was made on the AWS blog, which was linked from an earlier post
of this item I think. Anyway: [https://aws.amazon.com/blogs/aws/new-open-
distro-for-elastic...](https://aws.amazon.com/blogs/aws/new-open-distro-for-
elasticsearch/)

> This is not a fork; we will continue to send our contributions and patches
> upstream to advance these projects.

------
oldpond
I find it very ironic that AWS is behind this. Whenever I buy books from
Amazon I am amazed at their broken search engines. Just search a topic on
Amazon and then try to sort the results. LOL.

~~~
tracker1
IIRC, most of that infrastructure is/was Oracle based and much of it going to
another DB. I'm not sure if Elastic is even in the mix, or how well it scales
to Amazon's levels.

~~~
gtsteve
Given the age of the offering, it might actually be the CloudSearch [0]
product that is used by the Amazon.com search team.

FWIW, I've always found the Amazon store search to be pretty decent; normally
I don't use a website's built in search system, opting for Google instead.

[0] [https://aws.amazon.com/cloudsearch/](https://aws.amazon.com/cloudsearch/)

------
hguhghuff
In the old days, big companies spent gazillions to buy companies when they
wanted to add their software to their product lineup.

Amazon just takes it regards of whether or not the original developer is happy
or hostile.

This is going to redefine open source.

~~~
whoisjuan
I don't understand this rationale. So it's Open Source for everyone, but not
for Amazon because is Amazon, so fuck Amazon?

You can't have it both ways. That defeats the whole purpose of FOSS.

~~~
anth_anm
Then maybe FOSS was a mistake.

~~~
analognoise
FOSS is a huge mistake, and has been for years; it's the fumbling combination
of starry-eyed Utopian visions and faux-community applied to code, with no
feedback loop from reality.

Good software is paid for, by someone. The divorce from reality comes in when
people realize that Big-Co (in this case Amazon) is actually abiding by the
license - what should cause zero cognitive dissonance is instead reverberating
against the ideals that FOSS never "grew up" to encompass.

The dream of the WWW is dying, computers are truly no more "free" than they've
ever been - but they're so much cheaper that we don't really care, or even
notice. A few will notice the disconnect, but cannot work together to solve it
- they're too busy polishing whatever "freedom" Totem stands in its place,
oblivious to the foundation built on sand; ironic, given sand's place in
semiconductors.

~~~
paulcarroty
> FOSS is a huge mistake, and has been for years;

Stupid conclusion. Without FOSS you're forced to pay $99.99 for text editor
and email client, and it's just beginning.

Open collaborations works IRL, not only in your "utopical dreams".

~~~
analognoise
Most used email client: Gmail. Freedom: 0, and you are the product.

Notepad, originally released in 1983, free as in beer since Windows 1.0 in
1985.

I'd rather pay for a text editor and email client than be the product.

Open collaboration is a joke - it's always a very small group of highly
technical people, paid by someone, only useful if it can be funded as a loss
leader for some other part of the business. The ideological foundations are
untenable at best as very little of the actual requirements of computing
(graphics cards, processors, ethernet PHYs, etc) are in any way "free" \- this
is what I meant by people stroking their Freedom Totem, and why the whole
thing is built on sand, and that computers are too cheap for them to notice.
People focus on whatever their pet project is, rather than band together to
actually FREE (as in freedom) computing.

We all gave up well before the Freedom dream was achieved, and in its place we
got free text editors, email clients, and a surveillance based web. It's
depressing.

------
softwarelimits
Dear Ladies and Gentlemen, it would be of extraordinary interest to I believe
many people in our industries if anybody with some production experience on
the topic would like to write a short comparison "Elasticsearch vs. Solr",
thank you very much for your attention!

~~~
joking
In most cases it's just a matter of preference. Elastic has some extra tools
like kibana an was more focused on having statistics and aggregations working
well, while Solr webui is a lot more complete than the elastic one and the
full text search and relevance bits are more explained. Internally both use
lucene, so you are going to be able to do the same things with both.

I worked extensively with both and I prefer the way of administering Solr.
Anyway I use Elastic when it involves logging and statistics thanks to kibana.

------
jawns
> At AWS, we believe that maintainers of an open source project have a
> responsibility to ensure that the primary open source distribution remains
> open and free of proprietary code so that the community can build on the
> project freely, and the distribution does not advantage any one company over
> another.

Sections like this make me wonder if customers should demand that these
announcements be delivered on video, rather than in print, so we can see
whether the spokesperson can say it with a straight face.

