Hacker News new | past | comments | ask | show | jobs | submit login
Amazon Elasticsearch Service Is Now Amazon OpenSearch Service (amazon.com)
81 points by daigoba66 44 days ago | hide | past | favorite | 109 comments



Remember that in this story, as strange as this is to say, Amazon is the good guy. Elastic changed their products from a FOSS license to a proprietary one, and Amazon did exactly what you're supposed to do in that case: fork and continue development of the last FOSS version.


>Amazon is the good guy

This is exactly what Amazon wants you to think. That's why they're calling it "OpenSearch" and not "AmazonSearch".


They're literally keeping the source open. Call Amazon out for other stuff, not this. Enough with this good vs. evil attitude. People and companies are more complex than that and have varied interests. They do not fall neatly into one camp or the other.


I’m not so sure, they are also continuing to build a walled garden elk hosting alternative and already invested a lot of development into open distro for elasticsearch.

People often run this software on their compute, so obviously eliminating proprietary license complications makes that an easier option for their customers to deploy, and redirects some of that licensing money into the customer’s aws spend.

Don’t get me wrong, I’m happy there is a FOSS fork, but at the same time it feels like Amazon again has eaten another company’s lunch while feigning innocence.


You are not so sure that the world is not black and white? I assure you it's in color. Perhaps we do live in The Giver and only a few people realize this after all.


I agree, it's in color. In this case green.


That's just as myopic. We all do many things daily with no expectation of monetary reward.


This was never about what we all do. The topic was Amazon OpenSearch.

If you are arguing that Amazon has no expectation of monetary reward, then I have a bridge I'd like to sell you!


It's still myopic to say everything Amazon does is for the greenback. Engineers at Amazon likely pushed for OpenSearch because it provides personal growth to work on open products. Working on closed source may mean you can't bring as much knowledge to another job. It's likely less people will use your work and that may be less satisfying. You may like to switch jobs for reasons other than money. And to retain employees, Amazon may agree to allow employees work on open source.


Without a solid and forward-looking understanding of the internal cultures of both Elastic (the company) and also the specific team(s) at Amazon that will be managing the future of Elasticsearch and OpenSearch, respectively, I think it's too early to tell who will end up being the better steward of the search commons here.

Also worth noting that other open-source search engines exist out there; it's entirely possible that other alternative technologies will become more attractive, depending on how all this plays out.


> Without a solid and forward-looking understanding of the internal cultures of both Elastic (the company) and also the specific team(s) at Amazon that will be managing the future of Elasticsearch and OpenSearch, respectively, I think it's too early to tell who will end up being the better steward of the search commons here.

Amazon is the better steward of open sourced Elastic Search, hands down, because nobody else is doing it. I'm not even sure that stewardship of the technology's proliferation is TBD. Didn't Elastic close the source because they feared Amazon was taking too much of their potential revenue? Elastic gambled that only they could carry the technology forward, and they lost that bet. I'll grant it's possible Elastic could become more competitive with Amazon+OpenSearch somehow down the road, however it appears that right now Elastic is on its heals while Amazon is driving forward. It's also possible that another closed-source search engine becomes popular. That said, Elastic is built atop Lucene, and as long as an open sourced version of Elastic exists I'm not sure why you would start from scratch or with something else.

> Also worth noting that other open-source search engines exist out there; it's entirely possible that other alternative technologies will become more attractive, depending on how all this plays out.

Which open-source search engines do you think are competitive?


> Which open-source search engines do you think are competitive?

I work on an open source search engine called Typesense: https://typesense.org/

I'm biased, but I'd like to think that when it comes to site/app search, Typesense is quite competitive to Elasticsearch. Of course, ES has tons of features and configuration options, which is a blessing and a curse in terms of complexity and learning curve, so I wouldn't want to claim feature parity. But in terms of getting a good out-of-the-box search solution that works well for most use-cases and is easy to deploy and scale, I'm hoping for Typesense to be that go to.

I recently put this comparison matrix together: https://typesense.org/typesense-vs-algolia-vs-elasticsearch-...


> it appears that right now Elastic is on its heals while Amazon is driving forward

Perhaps - what are the commit/accepted contribution rates for the two projects like at the moment? (I'm not saying that's a perfect metric, but it could provide some information)

> Which open-source search engines do you think are competitive?

MeiliSearch[1] is an impressive-looking candidate. Perhaps two of the stronger innovations of Elasticsearch were horizontal scalability and the ability to pass documents to it over-the-network (basically, the ability to curl a JSON document and instantly have it be searchable -- and I'll admit/credit that Solr may have gotten there first with XML documents). Those two features make it easy to get started, and easy to scale up if needed (a common architect's concern).

[1] - https://github.com/meilisearch/MeiliSearch


Maybe. That one distinguishes itself from Elastic and hasn't set its sights on big data.

> Elasticsearch is very powerful when it comes to massive datasets; it can quickly scale horizontally and allows one to build complex queries. The flipside of that is that nothing is easy. Even if all you need is simple search functionality that handles typos, synonyms, or filters, you're looking at hours of training, documentation, and configuration.

Spending some hours learning a new tool doesn't seem so bad to me when I'll be working with it as part of my product.

> For big tech companies dealing with enormous logs, Elasticsearch might make sense. But when it comes to medium-size datasets (i.e. less than five million rows) and smaller apps, it simply isn’t right for the job. Nonetheless, since no open-source alternative has historically been available, developers still go for Elastic as their default solution and wind up losing a disproportionate amount of time on setup and training.

> MeiliSearch is not made to search through billions of large text files or parse complex queries.

https://blog.meilisearch.com/why-should-you-use-meilisearch-...

I'm not sure I'd like to assume that my dataset will never grow. This is kind of like using training wheels. Some may find it helpful but I don't think it will grow to the widespread adoption Elastic has seen without a change of vision.


When we say company we tend to subconsciously level them to any other business,

But there's absolutely no way a company at the scale of Amazon could have got there without hiding some skeletons in the closet so it's good for the society in general to view ~~companies~~ Mega Corp. which has more money(read power) than several countries combined together always with suspicion.


Why not just call them out for the bad things and support the good things? Assuming everything will inevitably go bad at some point is (a) false and (b) misses the point: While Amazon carries the Open Search torch right now, at any point in the future everyone else can contribute.

They do deserve credit for stepping in to help even if it does support their own profit margins. Profit in itself isn't evil.


> Why not just call them out for the bad things and support the good things?

Not just calling them out, Even punishing them(fines) for their mistakes is not even a slap on their wrist at their scale we've seen this happening time and again with all Mega Corps.

There's just no incentive to not do something evil.

So extraordinary scrutiny for every single action seems to be the optimal approach until the governments decide to break Mega Corps. into smaller companies which can actually get hurt from punishments.


They're a huge company. If you want to attack you need to be able to identify where they are acting badly and strike at that weakness. Calling the whole thing an evil empire isn't going to make any progress: your attacks are easily fended off.


> If you want to attack you need to be able to identify where they are acting badly and strike at that weakness.

Can you give an example?

>> Even punishing them(fines) for their mistakes is not even a slap on their wrist at their scale we've seen this happening time and again

If this wasn't true then there wouldn't be sub-list for every wrongdoing spanning decades[1] i.e. If the punishments achieved it's purpose we wouldn't see continued offenses under same category.

[1] https://en.wikipedia.org/wiki/Criticism_of_Amazon


> Can you give an example?

Forcing employees to pee in bottles is pretty bad.

> If the punishments achieved it's purpose we wouldn't see continued offenses under same category.

Hah, the notion that punishments always work is funny. Good one. Try that on your kids. Look, making a punishment bigger may or may not help. I don't know about that. You can't just fine them out of existence for no reason, however. Bezos isn't Jack Ma and this isn't China.


> Forcing employees to pee in bottles is pretty bad.

Yes it is(Like dozens of other things listed in that Wiki), But I didn't ask that. I wanted to know how do you suggest we strike at 'the weakness of Amazon' to fix it (Like you suggested) that it treats it's employees with dignity and respect after that?

Because IMO if there's anything, It could have been done when Amazon's mistreatment of its workers surfaced way back in 2001.

>Hah, the notion that punishments always work is funny. Good one. Try that on your kids.

That's the point of punishments, isn't it? Just calling them out is convenient but does nothing.

> Look, making a punishment bigger may or may not help.

That's the whole point. They're too big for any punishments, Hence qualified people have been arguing that private companies shouldn't get that big(It undermines the basic tenants of democracy reg accumulation of power); But unfortunately those who can take action about it are also getting big thanks to $AMZN.

> however. Bezos isn't Jack Ma and this isn't China.

>> Calling the whole thing an evil empire isn't going to make any progress: your attacks are easily fended off.


> I wanted to know how do you suggest we strike at 'the weakness of Amazon' to fix it (Like you suggested) that it treats it's employees with dignity and respect after that?

You call them out for the individual wrongs like those listed on the Wikipedia page. Vote with your wallet. Debate and discuss.

> They're too big for any punishments

I don't agree with that. No company survives forever.

> Jack Ma & China

The US is not going to fine Amazon out of business like the CCP did to Jack Ma. The CCP is evil, not the people. Their crackdown on people's ability to freely debate and discuss just shoots the country's most valuable asset, its people, in its own foot over in over in order to maintain footholds for a relatively small group controlling the party. Any creative ideas are squashed. If the CCP would let up they would give rise to a new generation of ingenuity in China. Of course some CCP members might face criticism for that. It's up to them to decide whether or not to move in that direction. Nobody likes to be on the receiving end of criticism and you need to be strong to take it.


> You call them out for the individual wrongs like those listed on the Wikipedia page. Vote with your wallet. Debate and discuss.

It's been happening at least since 2001, Starting with layoffs and now it has to come to -

> Forcing employees to pee in bottles is pretty bad.

> No company survives forever.

I agree, 'East India company' just survived for 273 years and people who ever affected by it are still suffering several generations later.


The East India Company was rightly criticized for holding trade monopolies that were later abolished. It's fair to consider whether a company the size of Amazon also has some monopolies and to ask the FTC to investigate or act.


And it's exactly correct. Amazon got burnt by Elastic. Doesn't matter if it's $100 billion company.

And mind you, Elastic is no tiny startup. It has 2000+ employees.


It’s one of those things. Until Amazon does this to you, you will love that company no matter how many people they will screw over.

Mind you this company basically is a monopoly in multiple domains. I’m afraid of the world where we align with the villain on something only because it’s convenient for us.


Not everything Amazon does is evil. And not everything Elastic did is good. I highly doubt a lot of people in this thread would sign off that Amazon generally is a good company; it's just that in this case, they're on the right side, in part just because Elastic made it easy to be.


> I’m afraid of the world where we align with the villain on something only because it’s convenient for us.

We're not. We're aligning with them because they happened to do something legitimately good. Being a villain doesn't mean that literally everything you do is bad.


I’m curious though, did Elastic’s license change impact you in anyway? Were you planning on offering Elasticsearch as a managed cloud offering? What exactly is the legitimate good Amazon did here?


I for one was. We have a platform whereby we need to expose logs and metrics to our clients, elastic/open search is a great option for this, unfortunately, being a non-opensource SaaS, elastic's license precludes it. Different discussion as to whether we should pay elastic as we can't afford to, currently.


> Were you planning on offering Elasticsearch as a managed cloud offering?

What can make me certain I won’t ever need it? Ecosystems without monopolies tend to be more robust.


We wanted to have multiple service providers for Elastic. So if one company goes bad or raises prices (be it Elastic or Amazon or something else) we can move to a different one.


No, it doesn't directly impact me, since I'm not planning on offering Elasticsearch as a managed cloud offering. But just because something doesn't benefit me personally doesn't mean that I shouldn't consider it a good thing. For example, I'm not a child and don't have cancer, but I'd still consider it a legitimate good thing if someone donated money to St. Jude Children's Research Hospital.


They changed the license after Amazon took all their revenue.


> We have experienced significant growth, with revenue increasing to $608.5 million in the year ended April 30, 2021 from $427.6 million in the year ended April 30, 2020 and $271.7 million in the year ended April 30, 2019, representing year-over-year growth of 42% for the year ended April 30, 2021 and 57% for the year ended April 30, 2020

https://s2.q4cdn.com/265747582/files/doc_downloads/2021/Elas...

Revenue for FY2018 and 2017 was $160M and $88M respectively.

Is that what Amazon taking all their revenue looks like?


The fact that they grew despite Amazon doesn't mean that AWS Elasticsearch wasn't a viable competitor. It just means that at the moment it was not an existential threat.

However, when competing with a literally A in FAANG, one must be very careful about writing them off.

To compare, I knew a company that 10 years ago made really great business intelligence integrations into Excel. They'd operated for 15+ years and were a major local employer. They thought Palantir was going to be their competitor, but then Microsoft came out with Power BI.

At first, revenue grew because Microsoft's marketing shined a great spotlight on the field (plus MS had only entered because the field was growing in the first place). Then as the situation matured, MS's capacity to offer integrations to the rest of the stack became a key sales point.

Like AWS, the point of investing in their stack isn't one particular product but the fact that all the billing, support, and integrations happen via one company.

You can even accept it if any particular product is deficient (e.g., some say that about Teams or Skype), but in the case of an open-source product like Elastic--the value of the software itself is 100% the same.

AWS was going to eventually kill Elastic. Not just because of all I wrote above, but also because Elastic's got a significant part of its revenue by selling Elastic as a distribution on top of AWS. Anyone would want to cut out the middleman and simply use AWS's version even if it lagged behind the current Elastic.


I'd have more sympathy for that viewpoint if Elastic didn't start off by using the open source community before switching to a proprietary license. The never would have gotten to the level where they mattered if it wasn't for being open source, and turning their back on that community makes them the bad guys.


Isn't elastic still open source, just not in the way the open source diehards like?


A small history tidbit is that AWS has been offering ES service since 2015, while Elastic launched Elastic Cloud in 2016, after acquiring Found (a ES SaaS company) in 2015.


I don't fault Elastic for choosing to change their license. I fault them for their choice of the new one. They should have moved to the AGPL instead, like Grafana did under basically the exact same circumstances.


yep - not going AGPL is just greed


You talk like they're entitled to that. When you make software open source, the social contract is that you accept free contributions, in the form of commits, issues, and code reviews, in exchange for being held accountable by the community to ensure you provide the best product. If another company provides a superior service and support, that's part of the risk. If they want to go back to closed source, they can't be surprised if everyone jumps ship.


Doesn't matter. If you don't have the courage for FOSS, don't do it. And you also won't get the good will FOSS gives you.


> Amazon is the good guy in this story

That is so twisted.

Elastic did all of the hard work. Amazon came in and took it.

Open source is fair when there aren't giants plucking it out of the hands of workers and co-opting it into their massive platform options with god-tier leverage. You can't compete in that world.

We need licenses even stronger than the AGPL to require that platforms running the code are open all the way down to the billing stack.


You could similarly say that Apache did the hard work with Lucene, then Elastic came and took it. Except, of course, if you think that simply hosting is not worth any money, but then Elastics own paid offering is void as well.

I really don't fault Elastic for disliking the situation; someone making billions with you work while you're still counting coins sucks. But they reached their size due to OpenSource and AWS, so painting them as the profitless victim really does not show the whole picture. And their handling of the situation so far was horrible; they've even gone so far as to embed code in their client libraries so that they don't work with Amazons offering.

So yes, in conclusion, Amazon really has the moral high ground here.


Elastic is welcome to license their code how they want. If they want to write a license that says “to use our code, everything down to the billing stack must be open source”, they’re welcome to. But when they license their code and then throw a tantrum when Amazon uses the code in line with the license, I’ve got no sympathy for them.


This is one of those rare situations where I can see both sides and I'm thankful I don't have to make the decisions Elastic.co did. Most open source companies don't have to worry too much about competitors grabbing their code and forking it because they have the expertise needed to improve and support it and there isn't much incentive for customers to leave them. But this is Amazon we're talking about. They can package up the code and charge next to nothing for it (not that they are doing that) and make their money on compute and bandwidth charges.

That said I wish Elastic had chosen a better license and I wish they wouldn't charge for basic security options like SAML, RBAC, and encryption.

edit: And IIRC this all started because Amazon added authentication and RBAC to open source ELK and tried to submit the changes back to Elastic, which rejected them.


>> That said I wish Elastic had chosen a better license and I wish they wouldn't charge for basic security options like SAML, RBAC, and encryption.

What better licenses or monetization strategy would have allowed them to charge enterprises who use it for more commercial purposes, and allow the individuals/casual users to use it for free?


AGPLv3. Off the top of my head, Grafana and RStudio already do that and seem to have had the desired result. I’m sure there are others.


One could argue AWS brought much more attention to Elasticsearch than Elastic could have ever hoped for on their own.


Did Elastic do the hard work, or Lucene?


There's so much on top of lucene that I'd say both


Especially considering ~11 of the Apache Lucene committers are Elastic employees as well as 10 members of the Apache Lucene Project Management Committee.


Disclosure: I work for AWS where I build infrastructure services that sit low in the stack. I am a relative novice when it comes to Java-heavy codebases like Apache Lucene, Elasticsearch, or OpenSearch. But I know a thing or two about Open Source and Communities.

I think it is hazardous to count the number of committers and PMC members that are part of a project that is built at the Apache Software Foundation. It is helpful to keep track of these things to make sure that the project keeps its independence, and is not unduly influenced by the interest of any one company or organization. Committers and PMC members earn their position based on their own efforts as individuals. They don't represent their employers.

The current Apache Lucene PMC chair just happens to work for Amazon today. A number of folks that used to work for Elastic now work at Amazon. But over-focusing on that, and keeping careful track of what company contributes how much to ASF projects, is bad for community health in my personal opinion.

See also Doug Cutting at https://twitter.com/cutting/status/1359160158723711002


I think it's reasonable to expect that corporate citizens worth billions of dollars would do their fair share of open source contributions. I'm curious why you think this is bad for community health.


It is totally reasonable to expect for-profit corporations to be good stewards of shared resources, such as the public goods that make up the Free and Open Source Software (FOSS) commons. I think it is important to all users of FOSS that the software is well maintained, and is able to grow to meet the changing needs of users. We often call achieving a system where sufficient value created by FOSS can be captured in ways that funds its ongoing development and maintenance achieving a "self-sustaining" system. One view on sustainability in FOSS can be found in this Apache Software Foundation blog post [1].

Companies should also be respectful of the communities that produce and maintain software, including understanding the philosophy and governance that communities have adopted. For the Apache Software Foundation, this is being mindful to not unduly influence the project, such as to try to control it through employing a majority of the people who work on the project, or otherwise weaken the independence and autonomy for the community that builds and maintains the software. [2]

I've seen situations in the past where companies employ an open-source marketing strategy that ends up encouraging unhelpful activity in open source communities. This can also happen when companies measure the performance of a software developer based on metrics such as "number of patches accepted in an open source project." This kind of thing is generally unwelcome in open source software communities. See this thread on HN [3] on the Linux kernel development list, TL;DR - "Please don't waste maintainers' time on your KPI grabbing patches."

[1] https://blogs.apache.org/foundation/entry/the-apache-way-to-...

[2] http://community.apache.org/projectIndependence.html

[3] https://news.ycombinator.com/item?id=27629366


Indeed commit count is a silly metric. I reminds me of the bad incentives of the hacktoberfest


There is a ton of work on top of Lucene. And Lucene is still OSS. So yes what Amazon is doing is stealing Elastic’s work.

I’m sure this will change how every other company is the OSS world will operate from now on. CockroachDB for example is already preventing a monopoly like Amazon from taking over.


> There is a ton of work on top of Lucene.

1. All of which is useless without Lucene. In fact, it wouldn't have been possible without Lucene. Thus, by your own twisted argument, Elastic "stole" Lucene's work.

2. Not all of the code in ElasticSearch is written by Elastic. Co. It's not their sole codebase. They have no monopoly on it. Maybe if they never took in contributions from others, but they did.

> And Lucene is still OSS.

OpenSearch is still OSS. The ES base that OS forked from is also OSS.

> ... a monopoly like Amazon from taking over.

Now I know you're being dishonest. Amazon never had a monopoly on ES, it still has no monopoly on OS. It's Elastic (like CockroachDB, that you mention) that are trying to enforce an artificial monopoly. Of themselves.

If you want to find out whether ES is really the "little guy" or the "monopolist", try providing a hosted service of their latest release and see how they react. It doesn't matter if you've contributed code to ES before, it doesn't matter if you're the most prolific contributor to Lucene.

It doesn't matter if you can build a better version of ES or provide better service. Elastic Co. (and CockroachDB Inc.) will simply not allow you to compete.

----

For once, try pointing the finger fairly, at everyone. Try judging everyone with the same set of laws/morals. Hold your heroes to the same standards, at least, as your villians.


CockroachDB is doing the right thing and I bet ES would have done it much sooner if they had any clue that Amazon is going to screw the small guy.

Also your arguments doesn't make any sense. Lucene is still OSS and OpenSearch is not Lucene but built of Elastic code. They didn't rebuild everything from scratch but literally used the code Elastic wrote.

No one argued OpenSearch is not OSS.

And sorry - if you think Amazon as a company is not screwing the small guy, I feel sorry for you.


Elastic also screwed plenty of small guys to get to where they are. And they're no longer small.

> Also your arguments doesn't make any sense.

Thank you, I used your twisted logic to arrive at them.

> Lucene is still OSS and OpenSearch is not Lucene but built of Elastic code.

And Elastic is not Lucene but built of Lucene code. So why is it okay for Elastic to build a proprietary package using OSS code, but not okay for AWS to build an OSS package using OSS code?

> They didn't rebuild everything from scratch but literally used the code Elastic wrote.

1. OpenSearch is more than ES Community Edition. OpenSearch does have some code of its own; significant enough to matter. Code that was considered basic featureset that Elastic refused to add, even rejecting PRs. What do you do when your PRs that add significant features are rejected? Especially when the reason is, "because we don't want you to compete with our proprietary packages"? How FOSS is a project, really, when it refuses such PRs?

2. Like Elastic didn't rebuild everything from scratch but literally used the code (as a library) Lucene authors wrote. So why is it morally okay for Elastic to profit off of OSS work they didn't do, but not for AWS to build OSS off of OSS?

> No one argued OpenSearch is not OSS.

You called it 'stolen'. Why is it not stealing when Elastic builds a proprietary package off of OSS code, including code that was donated specifically to be included in an OSS package, when you claim it is 'stealing' to make an OSS package of the same license and with copyright notices preserved from another OSS package.

> Amazon as a company is not screwing the small guy

I know exactly where and how Amazon as a company is screwing the small guys. This is not one of them. This is a field so full of strawmen it is difficult to see where it ends.

> I feel sorry for you.

Please keep your sympathies to yourself. You need it more than I do right now. If nothing else, then for the simple task of judging your heroes with the same yardsticks as your villians.


Amazon did contribute code back.


If I was a mole for Amazon working at Elasticsearch, I can’t think of a better way to accelerate the shift onto AWS’s offering than breaking client comparability with previous versions of Elasticsearch. If a customer needs to upgrade the client for some reason, they are forced to update the server as well. Maybe they’ll just update the client and server to OpenSearch while they’re at it so they don’t have any more forced updates in the future.


like that didn't happen


As far as I'm aware, this is the first major "hostile" fork AWS has done. In fact, it's probably one of the largest ones overall. It will be really interesting to see how this plays out in the long run.


Google certainly didn't support Amazon forking Android into FireOS. I believe that was just about when Google began migrating tons of APIs from the Android OS to Google Play Services.


It's good fodder for the antitrust suit.


While this is tangentially related to Amazon's fork of Elasticsearch, it's more related to Elastic's trademark infringement suit against AWS's use of the trademarked Elasticsearch name in their managed service, especially egregious given AWS's attempt to portray it as a "partnership" with Elastic in a since-deleted tweet:

https://mobile.twitter.com/ppgosavi/status/11791819009690378...


I doubt Amazon cares about that suit. If anything, it's elastic that hugely benefited from calling their product after countless Amazon products that have the word elastic in them.

Amazon can show that EC2 predates the company by half a decade atleast


So many people in this discussion take sides. The only thing I see, is divergence. APIs will now evolve separately, and compatibility will break soon. I think this competition and diversification will be good for innovation in search. But it’s also going to be confusing for a while. If you use Elasticsearch in Amazon, you better decide very quickly whether you want to hitch your wagon to the OpenSearch future. Migrating is going to be a pain in a year or two.


This didn't happen with MySQL/MariaDB split -- they are still mostly compatible.

Granted, it looks like Elastic is trying to break compatibility pretty hard (see the new version checks in clients for example) but hopefully Amazon will release universal connectors, it certainly in the Amazon's interests.


Good point about MySQL/MariaDB. I think this is different though because search engines are at a big pivot point to include approximate-nearest-neighbor dense vector search (it has before been only sparse vector search for Lucene based platforms).

Specifically, this feature https://github.com/elastic/elasticsearch/issues/42326#issuec... will be a big change for Elasticsearch. OpenSearch might try to mimic the API, but implementation details here will matter a lot, since this type of search is really picky when it comes to performance/recall balance. OpenDistro has already been working on their own version: https://opendistro.github.io/for-elasticsearch/features/knn.... ...so will they switch their API? Perhaps - but the results are going to be very different.


AFAIK when it comes to nearest-neighbor searches Elasticsearch is catching up with OpenDistro k-NN, and not the other way around.


Perhaps, but the point about divergence stands! Elastic won’t be copying the OpenDistro KNN API.


It is pretty clear to me that Elastic is planning to build their ANN features differently than OpenDistro's k-NN implementation, or other plugins modules that extend Easticsearch in similar ways. They now will build on the Apache Lucene capabilities that were collaboratively built "upstream" by a number of individuals, some that work for Amazon and some that work for Elastic.

From the linked issue, it seemed that they were originally planning to develop this as a proprietary feature of Elasticsearch, without contributing the functionality to Apache Lucene, but then changed direction when the Apache Lucene developers (some of which are currently employed to do such work by Amazon) started to build its approximate nearest neighbor (ANN) vector search capabilities. [1]

It's great to see folks that work for Elastic collaborating and building on what is in Apache Lucene to extend the utility of ANN with Hierarchical Navigable Small World Graphs (HNSW) [2]! From this, I think it should be possible to implement an Open Source version of the functionality with a compatible API, if that is something that OpenSearch users seek.

[1] https://issues.apache.org/jira/browse/LUCENE-9004

[2] https://github.com/apache/lucene/pull/250


"it should be possible to implement an Open Source version of the functionality".

Elasticsearch is still open source...just not in a license that Amazon can take advantage of.


Why would most companies need to migrate if they're happy with the features in a past version? Why not stick with it (assuming there are no major security flaws?)


Well, how far is Amazon going to take OpenSearch? Elastic is all in on innovating and building their flagship product forever with significant dedication. AWS just needs something good enough as an offering to go with the rest of their OK cloud services.


I think most people can see both sides of this conflict. But, as you yourself pointed out at the end of your comment, finding a side to pick quickly is important.


I changed from AWS managed ElasticSearch to Elastic.co a couple of months ago. Should have done it sooner.


If the elastic product were better, they shouldn't have bait-n-switched the licence to non-free.

They would have competed on the merits of the product. Looks like whoever is making these decisions is not familiar with what made them popular with Devs in the first place.


Nah, they compete on merit and they have the clear better product. But AWS has the inertia and massive user base.


Why? Tech or on principle? We are using Amazon ES but leaves a lot to be desired.

But I don’t know how much of that is just ES and how much is Amazon


What pain points do you see with Amazon ES, if I may ask? I'm considering migrating to it.


We are doing the transition and everything is just so smooth. Adding new nodes is easy and fast. There's an actually good and consistent UI.

Last time we tried to upgrade our cluster version on AWS we ended up waiting for weeks until it randomly updated mid day on a busy day and broke everything.

It's a ton of small things that really make a difference.


You can upgrade instantly on AWS, it doesn't take weeks. A lot of problems with AWS Elasticsearch Service are caused by corporations trying to squeeze a dollar out of a dime.

T2 and T3s with no Master Nodes, let alone three, leads to a lot of problems.

When everything lives in MMAP'd memory and you are nickel and dime'n it with 4GB and one CPU. You're going to have a bad day.


Yeah that was for sure not the case for 3 m5.large master nodes and several r4.2xlarge nodes.

The update was instant, what wasn't instant was the wait for the update. IDK if it was the maintenance window (which was set up for a weekend) or some bug, but we DID end up waiting weeks until it randomly updated.


How do you manage user access to the management console?


Good. I wonder if some of this mess could have been avoided if AWS had used a different name to begin with.


Morale of the story: licenses matter. Get them right.



Use a copyleft license. If you're not, you are just giving free labor to billion dollar companies.


In this case, which copyleft license would help them?

If they use GPL, it makes no difference as Amazon can still run it but if they use AGPL orgs would blacklist this because they don't want to be forced to open-source the rest of their code for their "elasticsearch integration"


AGPL worked fine for Grafana for this purpose.


Too early to tell. People might still be running the older non agpl version


> One of the most important reasons is the freedom to use that software where and how they want.

True for me, I agree.


In OSS, what a company giveth, a company cannot taketh away.


This is consequence of open source.


Are you claiming this is a bad thing? Your statement doesn't have a whole lot of meat in it without elaborating a bit.

There were a lot of decisions and changes that were made that led to this happening, and we have no idea how it ends up.

I personally am interested to see what happens in the elasticsearch versus opensearch space. I want to see how this experiment plays out.


This is the consequence of closed source. If Elasticsearch were still open source, then Amazon wouldn't have had any reason to do this. Elastic is getting exactly what they deserve.


If they would have been closed source from the get-go, Amazon would have had a far harder time cloning their product or even creating a service based on it to begin with. So yes, this is a consequence of them starting out open source.


If they would have been closed source from the get-go, they'd have even fewer customers than they do today with Amazon "stealing" all of them.


Possibly, I didn't refute that (and I actually agree). However, the point that this fork would likely not have happened without them being open source is completely valid.


Open source didn't go far enough in this case. It should have mandated the platform itself - from the billing pieces, to the hardware imaging, user management, and more - was all entirely open source.


Open source can't do that, literally. Any license that tries to impose "copyleft" on the whole platform like that will necessarily fail to meet the Open Source Definition.


You're arguing semantics, not intent.

Just call it something new if definitions matter so much. "Fair Play Source", "Fair and Open", etc.

Fuck open source the way Amazon likes it. Look at how the giants abuse it to crush smaller innovators. It's not rewarding the original authors. It's turning them into labor slaves whose collective work is captured and monetized, with little hope of fair competition or reward.

Might as well call it "open slavery" and imagine Amazon as some Jabba the Hut creature.

The work was intended to be open. They just didn't imagine the giant Jabba slug eating the whole pie and freezing them out. Unintended, yet fatal, consequences.

The DOJ should look very closely at this behavior. It's anticompetitive af and wholly against the spirit and intent of the original authors.


Amazon is working on a fully open version that you can fork, edit and host yourself. Elastic closed their product and is changing their client libraries to not support the open version. In fact, this whole thing started because Elastic got envious of Amazons revenue, which is not quite the open source spirit.

I mean, I get why Elastic is not happy. But in this case, neither intent is pure, but at least the result on the Amazon side is.


> But in this case, neither intent is pure, but at least the result on the Amazon side is.

Indeed. Amazon may be doing the right thing for the wrong reasons, but that's still better than doing the wrong thing for the wrong reasons, which is what Elastic did.


RIP Elastic




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: