
Opening up search is an ethical imperative - softwaredoug
https://opensourceconnections.com/blog/2020/01/23/opening-up-search-is-an-ethical-imperative/
======
cptaj
All platforms after a certain size need to be regulated as utilities.

When thousands or hundreds of thousands of companies live and die by your
platform, its economic suicide to leave it in the hands of one single private
entity.

A few things need to happen in order to guarantee fair play:

\- Transparency: Your process can no longer be secret. People need to see
whats going on to ensure no foul play is going on.

\- The owners of the platform can't participate economically in it. Amazon is
the obvious example. You don't want to be kicked out of your own platform?
Don't open it to others. Its a simple matter of conflict of interest.

\- DUE PROCESS: You can't have some minimum wage employee following a
flowchart in call center making life of death decisions for companies living
in your platform. There needs to be a proper and accessible appeals process
with independent arbitrage if necessary. And above all, innocent till proven
guilty. You can't fucking shut down my company and THEN I can go through the
lengthy appeal. No company can survive that.

Too expensive? Tough shit, go wipe your tears with your trillion dollar market
cap.

I'm sure there's a ton more that needs to happen. These are just the ones I
can think about.

~~~
biggestdecision
The world economy runs on Google/Microsoft/Apple/Amazon. Imagine if any one of
them were to fail, it would be catastrophic.

That said I'm not sure I trust the current American political climate to step
in and regulate, and I'm not sure they have a right to... The influence of
these businesses extends far outside the borders of the United States.

~~~
tomrod
Adtech is a pretty big bubble. Failure would be disruptive. But from a long-
term perspective I'm not convinced adtech is desirable or needed. Let's go to
Luna and the icy moons instead.

~~~
pb7
As long as people who have next to nothing (a large portion of the world) can
access valuable resources for free in exchange for looking at advertising, it
is quite necessary. Even those who do have disposable income in first world
countries are not interested in paying for a subscription to resources they
rely on a daily basis. Ads are a necessary evil.

~~~
tomrod
Perhaps. But I contend that they are ineffective.

------
WClayFerguson
Lucene is the best search technology, in my humble opinion, and is amazingly
fast. MongoDB has it built in (full-text search, based on Lucene) and I'm
using it in the project I'm working on.

------
aaron695
What would be an interesting project would be to create an automated framework
to compare search engines.

Have dataset/s and see where the different technologies, win, fail and IDK.

Solr's Phonetic Matching I always thought needed improvement, it would be
interesting to get a report on those for instance.

~~~
jedimastert
> compare search engines

What metrics would you use? I would be fascinated to know.

~~~
pieterk
User retention? ;)

Color me interested! There’s so many layers to measure: from natural language
understanding to pagerank-style relevance ordering, filtering duplicates to
presenting the different permanences of content (breaking news vs books on the
same topic).

~~~
aaron695
> filtering duplicates

Yep. It's interesting. How do you deal with versioning. The same document 10
times on a CMS might overpower the search results. Does uniqueness rank
higher? There you need a human judge at which search engine / search engine
defaults is better.

> natural language understanding

I'd like to know if any search engines do this. I feel like it's out of scope.
If they use the words "natural language understanding" I think they are
running a scam and mean statistics.

