Hacker News new | past | comments | ask | show | jobs | submit login
The Web Began Dying in 2014 (2017) (staltz.com)
370 points by pcr910303 61 days ago | hide | past | web | favorite | 263 comments

In addition, what's changed between 2017 and now is that back then Google was still admired by the tech people. "Google is evil" sounded ridiculous only a few years ago. I've been ridiculed by my colleagues and fellow engineers every time I said it. But I think it began to change with the introduction of AMP, a growing list of products discontinued for no apparent reason, failures of some of the new products often practically at launch, and finally the decline of quality and coverage of Search (more spam and more older sites dropping out).

Google like no other tech giant always relied on the credibility among the tech people. Whether it's a new end-user product or a new programming language, it would go full avalanche with minimal marketing effort.

Losing our support, on top of poor product management and inability to innovate will be more and more difficult for them to handle. Rolling out poorly designed and poorly targeted products is already a sign of a disconnect from reality. They are slowly becoming less and less relevant and most importantly, less respected by the tech community. In a way it's the Microsoft of the 2000s when the revenue figures still weren't bad but the signs of the decline were apparent.

Yet unfortunately there's no competition to Search on the horizon and "google" is still a commonly used verb that will be very difficult to remove from the language. Microsoft seems to be happy with Bing's position today, which looks and feels like Google Search and maybe internally even mimics all the stupid and "evil" things Google Search does judging from their search results.

I think it is time (if not a bit too late) to try to challenge Google Search with something new, a new perspective on how the information on the Internet can be indexed and searched. I bet a lot of people are constantly thinking about this, but the window of opportunity may be now while Google is putting a lot of their time and effort on ML/AI. I doubt you can compete with them by building a ML-based search, so it must be something else both in terms of technology and possibly even the UI.

I don't know what the bottom line here is. Google is not dying any time soon, but the time to challenge them could be now.

I'm refusing to implement Google Analytics on my startup's product. Because it's slow, but mainly because I don't trust Google enough to let them put whatever code they like on my page.

I'm running into serious flak from the marketing team because they don't know how to use any other tool. GA is the industry standard, apparently, and every site uses it, etc etc. They're even hinting that not using it could affect our SEO (I haven't seen any evidence for that, but if so it just reinforces my desire to not give Evil Google our traffic).

Even a couple of years ago, this wouldn't have been a problem. I trusted them to do the right thing. Now. No.

Google is the new M$

Surprised nobody mentioned Matomo Analytics yet:


It's FLOSS, comes both self-hosted and as a paid service and it's pretty good. Perhaps good enough for your use case?

Or, we could stop sending people programs to track themselves on their own computers. Just use your server logs. I don't care how much of an inconvenience it is, you should never have started doing any client-side tracking in the first place.

Great point! We wrote about how we're doing it on whotracks.me a while ago, if anyone is wondering how that could work: https://whotracks.me/blog/private_analytics.html

this is useful, thanks... couple questions:

1. Do you grab the country from the IP address before anonymising it? (I know VPN's are a thing, but we would like to know where our customers are coming from)

2. Why encrypt with a daily key instead of just hashing the IP address and storing the hash? Do you ever decrypt the IP address?

3. How have you found CloudWatch? I'm logging all requests to our own database, because it's so simple to do. What's the bonus for Cloudwatch?

Hi Marcus,

Thank you for your questions.

Disclaimer: I also work on WTM from time to time, so I hope can answer these questions to some degree :)

1. WhoTracksMe is served via CloudFront, it only get's the IP that CF reports. To get the country, there are two ways that can be used. IP => Country mapping via a Database or additionally, because CF has multiple edge locations, if needed the you could see which edge location served the request, that will help you narrow down the location at a country level. Ofcourse, in both cases if the user is using VPN, you would only records the VPN location.

2. The problem we want to avoid is same IP with same hashing algorithm will produce same value. Which can then be used to co-relate user activity across days. Second, depending on what hashing scheme one is using, rainbow tables can be used to get back the original value. Therefore, to avoid, we used the approach of daily key. Now that I write, we could also use some hash + daily_salt. This should give the same guarantees. - Needs to be checked.

3. In this setup we are not using Cloudwatch but Cloudfront, which is the CDN provider from AWS.

1. Ah, OK, that's useful. I was thinking that dropping the last byte of the IP address would still be able to give us the country via database lookup, but anonymise enough. But getting it via CF (or similar) would be a good alternative.

2. Chances of a hash collision on IP address is pretty small (i.e. statistically insignificant). But adding a daily salt would decrease the odds, for sure.

3. Sorry, my bad, I meant Cloudfront, but it's Sunday night here and there has been wine ;) Same question: how has Cloud{{thing}} worked out for you? Is it worth the extra complexity?

> 2. Chances of a hash collision on IP address is pretty small (i.e. statistically insignificant)

The problem, as I understand GP, is not so much the fear that two different IPs might collide. The problem is that seeing the same hash, you know that it refers to the same IP, and as such you can correlate users/sites over time (a privacy invasion).

That makes sense, kinda. Though I'm a little unsure why correlating http requests over an hour is not invading privacy, but doing the same over a week is?

But if I do that then how am I going to convince investors to give me so much funding I can sell my company and retire?

What's the huge objection to client-side tracking over server-side tracking?

20+ years ago, we wouldn't discussing the ethics of server/client based tracking with a straight face, because that shit is creepy as fuck.

Now, everyone takes tracking for granted as if the option of not being tracked is some sort of unachievable goal.

That's the issue. It isn't tracking A as opposed to tracking B. It's tracking period.

I need to understand why my customers are not clicking on the "subscribe" button.

I'm not selling adverts to them. I'm not capturing personal information. all I'm doing is trying to understand who my customer is and why they're not buying my product.

Why is this unethical?

And why is it OK to analyse server logs, but not OK to use javascript, to do this?

> And why is it OK to analyse server logs, but not OK to use javascript, to do this?

The data in your server logs is data that people sent to you on purpose (by visiting a page on your site, or filling out a form on your site). If you collect data about people by sending them code to execute and send data back to you, they're not meaning to send you that data; you're doing it behind their back.

I'm glad you try to use data you collect responsibly, but if people aren't meaning to send you data, you're not entitled to it.

Users are free to block the execution of client-side scripts though we all know what that means: much poorer to non-functional UX in many cases. Plus, to an extent browsers can control what can be collected on the client side, e.g. location, microphone input etc. (I wish there were similar restrictions on the keyboard and mouse, maybe some day we'll see that too)

But I'd agree that letting the users use a public resource on a condition that they should allow arbitrary scripts to be executed on their side can be considered unethical.

We have a Privacy Policy linked right from the landing page that states we will collect user behaviour information to improve our product.

If you walk into a shop, you have an expectation to be observed (both to check you're not shoplifting, and also to see how customers behave). They don't tell you that you'll be observed, it's just one of those things you expect when you walk into someone's shop.

We effectively have a notice on the front door that says we will be observing their behaviour while they're in our shop. They have every opportunity to walk away. I really don't see how this is unethical.

> If you walk into a shop, you have an expectation to be observed (both to check you're not shoplifting, and also to see how customers behave). They don't tell you that you'll be observed, it's just one of those things you expect when you walk into someone's shop.

The problem is that this is a poor analogy to computer networking. I send you an HTTP request; you send me an HTTP response. I'm not visiting your shop; we're sending each other messages.

My expectation is that you won't follow me around, observing how long I look at certain parts of your message, gauging my reaction to different parts of your message. That this is the norm on the web is an unfortunate reality, but it doesn't make it okay.

Sorry, but we'll have to agree to disagree on this. Visiting amazon.com is exactly like walking into a shop, and nothing like "sending messages to a server".

I totally understand that that's what's happening behind the scenes, but that's like saying "I vibrated the air in specific harmonics" rather than "I said yes".

Taking your walking into a store analogy, what you are actually doing is taking an xray of every potential customer, going through their pockets, recording every receipt from their wallet and then uniquely tagging them with a barcode all in the very moment that they touch the door handle to enter. And that's assuming you are not trying to be evil.

Could you please disclose the name of your company so I can be sure to avoid it like the plague?

The extent to which visiting amazon.com is like walking into a shop is the extent to which Amazon puts up a facade to make it feel like that. That we're exchanging messages is not just what's happening behind the scenes, it's the essential truth of how communication happens on the net.

If I were visiting your shop, it's totally reasonable for you to check to make sure I'm not stealing anything. But I can't steal anything of yours by sending you messages! (Setting aside the possibility of me hacking your server, but that's not what you're sending me javascript to check on.) Hopefully this expresses more clearly why I don't think the shop analogy holds water.

If "sending messages" is too primitive of an abstraction, we can imagine exchanging letters in the mail, or buying a newspaper, or something like that.

(BTW, I'm sorry for the unconstructive sister comment here - I definitely don't condone it.)

Like a complicated privacy policy people won't read. (not your fault) The average person also doesn't understand the depths of what going into a shop now means. Walking into a shop they might expect they are on CCTV sure, but not that they are being tracked via wifi, customer counters, and whatever else retail establishments have come up now to datamine their customers without consent.

Maybe its time for shops to have EULA's covering the entire door before I open it.

> The data in your server logs is data that people sent to you on purpose

That's not true: people have no idea what is sent and they're not choosing what's sent (with the exception of some forms.)

It's OK to collect display width, or estimate the time on a page even though people didn't send it to you on purpose.

I need to understand why my customers are not clicking on the "subscribe" button

all I'm doing is trying to understand who my customer is and why they're not buying my product.

Then ask them. The same way companies did for hundreds of years before javascript tricks made people lazy.

These things can be achieved by simply talking to your customers. Why is that so hard to do?

Also, what makes you think that your interest in knowing your customer's behavior is more important than your customers' right to privacy?

> Then ask them.

Startup 101 "Mom Test" stuff. If you ask people what they think, you get their intentions. If you measure their behaviour, you get actual data. Talking to customers is excellent, but measuring what they actually do verifies that they're not lying to you because they're nice people.

>Also, what makes you think that your interest in knowing your customer's behavior is more important than your customers' right to privacy?

I don't understand this. If I ran a physical shop and watched my customers browse to understand their behaviour, no-one would think this was unethical, intrusive or violating their privacy. Why is it all of those things online?

Or is your objection just to the Javascript? If I "violate their privacy" by analysing server logs it's OK, but adding javascript to measure behaviour is "bad, hmmkay"?

I already responded to your shop analogy in another comment, so I won't do that here. But -

> Or is your objection just to the Javascript? If I "violate their privacy" by analysing server logs it's OK, but adding javascript to measure behaviour is "bad, hmmkay"?

If you misuse any data you collect from anywhere, you're being a jerk. We're talking about collecting data in the first place. If you collect data by having a webpage visitor execute code on their own computer in the background and send data back to you, they clearly aren't meaning to send you that data, so you shouldn't do it.

> Startup 101 "Mom Test" stuff. If you ask people what they think, you get their intentions. If you measure their behaviour, you get actual data. Talking to customers is excellent, but measuring what they actually do verifies that they're not lying to you because they're nice people.

Startup 101 is anathema to treating people well. That some data about people could help you run your business better doesn't mean you're entitled to that data.

Again, we'll have to agree to disagree.

I respect that you have this point of view, though. In that I'm glad there are people out there fighting this fight. It needs to be fought, and we need to be questioning these boundaries.

Thank you for forcing me to question mine.

Cool, likewise! I appreciate the conversation.

So, if you have a main street shop, it would be ok to have detectives monitor people as they enter the store (EDIT to add: or, actually, as they're window shopping), take notes as to what they buy, surreptitiously check their ID, and then follow them home (or place a bug on their car) to find out where they live, what car they drive, etc., so that you can better understand "who my customer is and why they're not buying my product"?

If they're not buying your product and not subscribing, they're not your customer yet.

technically true, yes. /s/customers/potential customers/g

What I do when am not on your site is my business and not any of your business.

I totally agree. And we're totally uninterested in what you do off our site.

I'm purely talking about tracking your behaviour on our site. We want to know if you spent 20 mins reading our "about" page and then looked at our pricing page, and then vanished.

In terms of the rest... country is probably interesting, language preference is interesting (when do we need to translate from English because we have enough customers not speaking English), platform is probably interesting (purely so we can prioritise development to match our customer base). Where you got referred from is probably interesting, I guess.

But all of that we'll be getting from the http request and ip address rather than any malign tracking cookie nonsense.

Then suppose it would be best to avoid the tools and modalities currently being exploited by those who are so keenly interested in what everyone does all the time. Because who has time to figure out the few non malignant players in this vast space of parasites?

>We want to know if you spent 20 mins reading our "about" page and then looked at our pricing page, and then vanished.

I don't want you to know this sort of information. It's too invasive.

I concur.

I’ve been using self-hosted Matomo for https://www.cannonvoice.com since I launched a couple of weeks ago. Man this is some neato software! On top of that it was super easy to deploy to the K8s cluster that runs the app. +1 would recommend.

To folks saying to “Just use server logs”, it’s simply too hard to connect the dots unless you really know what you’re doing. Digging through my nginx logs to find that the majority of my users are dropping off at the second step of sending an invoice would be nearly impossible. Oh I could connect those logs to my api’s database, now I need a place to aggregate that data together. Maybe I could send all of that data together to elasticsearch and query with kibana. Now I need to learn how To set those things up.

Or I could use Matomo for some analytics real quick

Afaik Open web Analytics(https://github.com/padams/Open-Web-Analytics/) can use embedded code in PHP pages for analyzing.

GUI is nice, currently trying it out but not with php code but with embeded js like other tools use.

> To folks saying to “Just use server logs”, it’s simply too hard to connect the dots unless you really know what you’re doing.

No it's not, just stop overarchitecting your stuff.

> Digging through my nginx logs to find that the majority of my users are dropping off at the second step of sending an invoice would be nearly impossible. Oh I could connect those logs to my api’s database, now I need a place to aggregate that data together. Maybe I could send all of that data together to elasticsearch and query with kibana. Now I need to learn how To set those things up.

You can log requests/responses however you want on your server, it doesn't have to be an nginx log. Just put whatever request data that matters to you into your database immediately after you send a response.

> Or I could use Matomo for some analytics real quick

"Not being a jerk to my website visitors is too hard."

I don't know, it seems like an arbitrary line to me. If I logged everything about your visit in a file, then wrote some code to analyze and reconstruct a picture of you as a user, would it be OK with you because I call it a server log?

When I hit <enter> in a URL bar, or click on a link to a webpage, or fill out a form on the web, I know that I'm sending a request to a server. That's why I'm taking that action in the first place. But if you send me a program in a webpage to collect data about me, I'm not trying to send you that data, you're doing it behind my back.

You can still certainly be a jerk with data you collect in your server logs. (For example, I'd be pretty angry if you sold your log data to google or some other advertiser.) But your server is your computer, and the data in server logs is the data making up the HTTP requests that were explicitly sent to your server. That's your data, not mine.

Many government sites uses matomo in Denmark. It's good.

thanks for the tip :)

thanks for mentioning

Oblivious marketing ppl bullying tech guys into doing stupid things. Such a classic.

Unfortunately I have a hunch aswell that not adding GA does indeed negatively affects SEO.

How would it be possible to prove/disprove this?

Adding Google Analytics is probably good for escaping the "Google Sandbox." There's clearly some sort of review process (algorithm or manual review) before a website starts ranking well in the Google SERPs.

Experiment with your own web sites? Create two of them with and without GA, possibly with different content because Google seems to be able to detect duplication, then try to promote both (this is the difficult part of course) and compare.

It's hard to do any kind of objective comparison like this when it comes to SEO. It takes months to see any effect and even then you will have to speculate.

You could try to remove GA from half the pages on an existing site one day and see if those subsequently drop more, if you have enough pages to experiment with.

There is some evidence that older pages (>10 years old) drop out if they don't use any of Google's plugins. I think it was Tim Bray who was one of the first who noticed [1]

I also found another case of a lot of un-googleable reddit pages from only 5 years ago. They just don't exist on Google anymore even though they are linked from e.g. peoples profiles who commented there. Someone decided that Reddit is not important, or maybe there's some broader exclusion criteria in place.

[1] https://www.tbray.org/ongoing/When/201x/2018/01/15/Google-is...

I didn't read the comments, but the article didn't mention anything about Google plugins, did it?

I suppose it's possible Google used GA stats and maybe even content fingerprinting to determine if certain pages should be indexed, or reminded from the index, and how often to crawl them if indexed.

True, I think it was in various discussions here on HN including maybe under Tim Bray's article too. Same kind of rumors were circulating regarding AMP. There is still no direct evidence that Google's web plugins matter, but at least it seems like spammy websites that are fully equipped with Google's stuff get favorable ranking. And vice versa, you won't see high ranking spammy results that are not crammed with GA, Google Fonts and what not.

Yeah that's a tough one because there aren't many "legitimate" competitors to GA other than clicky from what I know. Sounds like an opportunity.

There have been a lot of discussions on Google Search quality lately on HN. (eg here: [1])

I wonder how much of the negative sentiment towards Google is partially motivated by subconscious dissatisfaction with their main product. I think I'm probably guilty of that.

Others have mentioned that the web environment has changed a lot and I've come to agree. PageRank style algorithms are probably pretty useless in an era of low effort reposts, aggregators, optimizing for clicks and a large SEO industry.

In addition, most of the user-generated content is now inside "walled gardens" like Facebook, Instagram and Reddit, and is either impossible to index, or much harder to quantify and rank than websites linking to each other.

There has to be something better out there, but it will be a difficult challenge.

Ironically, the thing I would appreciate most is a manually curated, Wikipedia inspired, hierarchical directory.

[1] https://news.ycombinator.com/item?id=21515181

When I started increasingly using DDG a while ago, it was primarily because I (1) thought it was "good enough" and (2) I wanted to preserve some diversity in the search ecosystem.

But by now, DDG is simply better. An important use case for search for me is to copy and paste URLs for discussions etc. Google will still show original URLs when I hover over their links, but when I COPY them, I don't get the URL, but some redirect link through Google.

At best, those are ugly and confusing when pasted into text. At worst, for e.g. image URLs, naïve processing may not longer recognize them as images because the suffix is buried deep in the redirect link.

DDG gives original URLs, As Sir Timothy Wanted Them.

Additionally, for image search, I get the impression that yandex gives more relevant results, although I don't have hard data to back that up.

> Google will still show original URLs when I hover over their links, but when I COPY them, I don't get the URL, but some redirect link through Google.

Google is doing that for a very long time. I use this extension to get rid of it:


> Ironically, the thing I would appreciate most is a manually curated, Wikipedia inspired, hierarchical directory.


Fun fact... This is how search engines started. One of the first steps in early SEO was to get your site listed on a manually curated web directory. VLIB, the first web directory, still operates at http://vlib.org/.

List of Web Directories, Wikipedia, https://en.m.wikipedia.org/wiki/List_of_web_directories

Uh, yeah. Jerry Yang when he was a VLSI student at Stanford, and I was a lowly semiconductor designer at Motorola Japan used to talk about these things, so ... yeah: I was being sardonic.

EDIT: He used to say to me: "[WhateverMyNameIs]! Nobody is going to pay a penny per newspaper article or a nickel for a song!". And I'd say: "Jerry! Nobody is going to pay to advertise on your personal curated bookmark list!". Turns out that one of us was right. And it wasn't me.

Looks like VLIB is rotting though - the first link I clicked (Electrical and Electronics Engineering) seems to belong to a domain squatter now.

More like the now defunct DMOZ, I'd say.

> More like the now defunct DMOZ

DMOZ was a good idea, but the execution was biased because curated lists invite partiality. You might build something better only if you could mitigate this natural tendency.

>Ironically, the thing I would appreciate most is a manually curated, Wikipedia inspired, hierarchical directory.

Those are subject to the limitations of hierarchies [1]. They don't work well beyond a quite small data set. And for a directory curated by someone else, you would need it to be someone that organizes things in exactly the same way as you would. Tagging and faceting can help a bit with that in the early stages of growth, but ironically you ultimately need a way to search the contents of the hierarchy.

[1] https://wiki.c2.com/?LimitsOfHierarchies

"back then Google was still admired by the tech people" "Losing our support"

why do you generalize 'tech people' as 'we' ?

I'm in tech and I've been for 30+ years. I still admire Google.

Sure some tech people don't like Google anymore or even never have.

But I wouldn't generalize my case nor should you generalize yours.

And it's not all or nothing: I want to believe most tech people are mature and educated enough to not treat Google as a single block, hate everything they do or love everything they do. There are nuances and shades of gray in life, as we all learn when we grow up.

In the end, what matters is the silent majority not the chatty loud ones that have a personal grudge against some big corporation.

That "Google is evil" is highly overrated and over hyped. I always question the agenda of people promoting this.

> But I wouldn't generalize my case nor should you generalize yours.

The way I read statements that you replied to is with a silent prefix "based on the experience of folks I work/talk with". If one disagrees (i.e., sees different views prevalent), chime in and say so. There must be a way to express opinions that are between "that's my personal view" and "statistically significant, peer reviewed validation at this link".

As a personal data point, I see the same thing (around me) as the parent: opinions of Google tanked. 10 years ago an employee leaving for Google would be discussed with "wow" and "best of luck" feeling, while today it is mostly "well, at least he will be well paid" and " I am not yet ready to sell my soul". Just my 2c.

I want to jump in to underline your point in a couple of ways. One has to be able to make by-and-large generalizations from experiences and guesstimations. We should already know when we read that kind of thing that by-and-large means there are going to be one-off exceptions. So we don't need to hear about those exceptions unless they're intended to dispute the bigger point.

Every time I hear the contrarian anecdote in the form of: "well, I personally don't include myself in that generalization" my first question is whether that's intended merely as a report of a one-off experience, or if it's intended to be treated as evidence in favor of a different generalization. Virtually every time I see this type of Contrarian Comment, it's mute on this most important question and just confuses everybody.

But secondly I think the assessment of Google's reputation going negative is, by and large, a valid one. Though I might say the seeds of that started with the closure of Google Reader and the turn away from supporting RSS, which signaled turning away from open protocols more generally.

That's exactly how you have to read things to take posts in good faith, which is one of the few rules 'round these parts.

Similarly, I'm trying to eliminate the "I think" prefix from writing. It shouldn't be necessary to write "I think Google is evil". Of course the writer thinks that. They wouldn't have written it if they didn't. They can very safely just write "Google is evil".

If I write "I think Google is evil", I mean: "I think Google is evil, but I acknowledge that it's possible for other fairly reasonable people to have a different opinion."

If I write "Google is evil", I mean: "I think Google is evil, and I consider this an established fact that reasonable people shouldn't be disputing."

These are different propositions, and I might want to say either of them.

A policy of not using qualifiers like "I think" because of-course-it's-my-opinion makes it harder to make that distinction. I don't think the advantage of punchier prose is worth that loss.

(Note the "I don't think" at the end; other people might reasonably see the tradeoffs differently. Note the absence of "I think" elsewhere; I don't see much doubt that there is such a tradeoff being made.)

A reasonable lesson should already think it's possible for other reasonable people to hold different opinions, and not feel threatened by them. It's only in this bullshit really of the internet that we decided to start interpreting the things other people say as "potential facts in need of disputing" rather than just "the utterances of peers". The former mode implies a level of deference that we don't take on in public, face-to-face conversations.

I was more concerned about his usage of 'the tech people' terms.

Like you, I think when someone saying 'Google is evil' he implies 'I think Google is evil'.

But someone saying 'back then Google was still admired by the tech people" or "Losing our support" is more ambiguous. Like if all 'tech people' or a majority of them think like him which or he represents the majority.

tl;dr: I'm fine with him saying 'Google is evil' but not with him saying 'the tech people think Google is evil'.

Yes, the implicit "I think" still applies. "I think the tech people think Google is evil".

I don't know how to say this lightly, but taking personal affront to statements of opinion is not a sign of the insult of the opinion of the speaker, but a sign of the insecurity the listener has in their own opinions.

In other words, relax.

People who think carefully then call Google 'evil' may as well be the same people who used to call it 'good'. Corporations aren't good or evil. They are powerful.

Being good is many things, but a key part of that is restraining the self from acting improperly. Corporations can't do that, they will sooner or later do whatever makes money. Or they get replaced sooner or later by a corporation that will. At best, there are stylistic differences around the edges that are simultaneously important but minor in the grand scheme of things. Sooner or later a corporation will be thinking primarily in terms of 'profitable' and 'not profitable'.

Calling Google 'evil' is trying to articulate something else - Google are powerful and dangerous. Any entity harvesting and directing the flow of information at that scale is. They have an unprecedented capability to cause harm; and sooner or later it will get used by someone to do exactly that.

But Google ten years ago seemed good, full stop. All of their behaviour seemed to be a net positive for the rest of the world, even when done in self interest.

Now, it's easy to see some behaviour which are not.

Even assuming there is a slide toward evil for corporations, and I'm not sure there is, it still seems correct to say that a company is at a given state of "goodness" in a certain time.

How about insidious? It's the banal sort of evil that doesn't realize the harm it's causing, or doesn't see a way to survive without causing that harm.

Well, maybe, but that's the human condition also. All of our actions — even the most well intentioned ones — can have unexpected and even disastrous consequences.

> Corporations aren't good or evil. They are powerful

That meant to imply that google is neutral? The strong nuclear force is powerful that doesn’t mean anything on its own. It can be harvested for good electricity or bad bombs. Of course the consensus on this is almost universal, the consensus on google is much more ambivalent. Corporations may well be good or evil

Yes, Google is not neutral, it definitely have a clear, intentional agenda. But it's non-moral (not imoral, or evil) as are all corporations. It just uses it's power as a way to complete it's objectives. This is way different from being neutral.

In a republic, the government - representing the people - is supposed to have the biggest stick. Huge companies with their own agendas are throwing a spanner in the works. And they basically only care about money. So the bigger they are, the more inherently evil they are. And it becomes even worse with transnational companies whose headquarters are basically out of reach.

Objectives can be evil

Certainly, what I mean is that it's actions can be also be seen as good under certain moral code.

when ppl say google is evil they mean its leadership

The thing about Google is they have a monopoly on search. The risk for them doing good at the expense of additional profit is much lower than almost any other business. They won't be easily replaced. They could choose to be good but they don't.

> DDG/Bing/Yandex will list the exact same pages that Google gives as a search result most of the time, and sometimes better ones.

I've switched to DDG about a year ago, and this just doesn't match my experience. Results are frequently worse. It's worth it to me but makes it hard to recommend to everyone still.

There's probably a wide range of how people use search; works for you doesn't mean works for everyone. As an example, your recommendation to use stackoverflow directly for programming queries works for you and is nonsense to me -- I basically never land on stackoverflow from my programming searches. The most privacy-conscious person I know continues to use google news search to his own dismay because he doesn't find anything else comes close for keeping tabs on what people are saying about products he works on. And my wife claims DDG is basically non-functional for her search needs, which have very little overlap with mine.

"Losing our support" means a process, it doesn't mean everybody hates Google now, of course it's not the case.

> I always question the agenda of people promoting this.

My only agenda is my growing frustration with using Google Search and the lack of competition. Web pages can be excluded now for not using any of Google's plugins for example. I see how Search is increasingly becoming a tool for plain old profit maximization. It is generally up to Google to do whatever they desire with their own products, however the belief that Search is fair and "democratic" is lost now. No matter how technically competent, Google is lost for me as a company I can trust.

So, where is my fair, convenient and comprehensive search of information on the Web? There's none on the horizon. This is very worrying.

As an anecdote, I installed Vivaldi on a new laptop recently. It uses Bing as the default search engine. Previously I would always change this to Google, but this time I thought, "Hah, Google has been shit for a while now, why not give Bing a try?"

Comprehensive means centralized. This provably cannot be fair. We can’t have everything. I use bing for difficult searches (google can only understand 1 or 2 words it seems) and yandex for image search. Not bad

Does it, though? The Internet is pretty "comprehensive"...

> I want to believe most tech people are mature and educated enough to not treat Google as a single block, hate everything they do or love everything they do.

Unfortunately, as long as it is a single economic entity, it is a single block. All of its parts strengthen and finance the same entity.

If the different parts want to be treated separately, they must become separated.

I consider it basic logic that if you do not want something to be under a control of a single entity, you shouldn't continue fuelling an entity whose growing power will ensure exactly that. Isn't this far less judging and more nuanced than "Google is evil"? The motto is basically just a catchy shorthand for that.

> I want to believe most tech people are mature and educated enough to not treat Google as a single block, hate everything they do or love everything they do.

They weren't mature enough to do that for Microsoft or IBM back in their days of dominance, why would you expect that level of maturity now?

"1. for the money, 2. for the..." is how i understand the "no evil" thing. just like people used to write micro$oft. it's funny i have not seen that one in a while. but yea, the most important factor is perhaps money.

>But I wouldn't generalize my case nor should you generalize yours.

That's fine, but there also exist general sentiments in any group...

Google has attempted to build products suiting a regime that murders people for organs and operates concentration camps for over a million people. Evil enough? They have barely, supposedly, engaged in a very small climbdown thanks to the laudable efforts of some insiders, clearly against the leadership's desires.

Imagine if they scrubbed search results for child immigrant detention camps, or My Lai, etc. in the US

So, you don't happen to work at Google, do you?

The typical Googler would have a more cynical view of Google.

> there's no competition to Search on the horizon

This is bunk, and it's time for this myth to die. DDG/Bing/Yandex will list the exact same pages that Google gives as a search result most of the time, and sometimes better ones. If you find that the pointless endless-scroll listing of millions of search results (in 0.333ms!) that Google and also DDG gives you isn't helpful, you can also try Startpage.com.

What are you searching for? Programming-related questions? Then go to stackoverflow.com directly (where you'll end up in 99% of cases when you search with DDG or Google anyway on these topics). Tip: to search SO, I've found it works best to create (but not submit) a new question with as much info in the headline as possible; the results that SO brings up to check for duplicate questions are excellent search results (btw have they changed it recently? I don't remember working it as good as it's now). Requires a StackExchange account though.

Are your searching for review of tech gadgets or christmas gifts? Then many people on HN admit to include "reddit" in their search terms. Then why not go directly to reddit.com for your search.

Are you searching on YouTube? Then indeed Google (YouTube itself) works best, but the real solution is to quit your media consumption habbit of spending your waking hours on monopolists video streaming sites that don't benefit creators. For just music without pointless and data-intensive video and ad noise, there are several alternatives.

Maybe it's time for new topic-related meta search engines and link aggregation sites.

In the end there isn't much new content worth searching for, thanks to Google and Facebook extracting almost all (70% in 2018) ad revenue, and the demise of the federated web in favour of 2010's portals such as github. I can assure you that the modest amount of new content worth reading can be easily summarized and searched by basic, non-personalized search algorithms; no ML required.

Going to Google to search things is really asking for the site with the most ads and trackers on them, is something that no self-respecting "Hacker" should be doing, and is a habbit you can quit at the turn of the decade if you haven't already. Because it's literally equivalent to asking "Ok Google, fill my feeble mind with shit".

I really don’t think OP meant “there is no way to avoid using Search”. The point is that nothing that you described is a serious competitive threat to Search.

You might get a lot of gratification out of avoiding giving Google your data but the reality is most traffic is coming from people who will choose convenience.

I don't want to use neither Bing nor Yandex either, and AFAIK DDG isn't a proper search engine, but takes all of their results from those three. With the recent "collapse" of Qwant, what am I supposed to use now?

What happened with Qwant? I've used them from time-to-time (and love their OSM-based map viewer), so I'm curious.

Can you explain your feelings on YouTube in more detail?

I find that I am often looking for information in video form (generally because I am trying to learn how to do some physical skill or to learn about something that is innately movement based such as dance). Although I often use Bing's video search in order to not just get YouTube results, the majority of them end up being on YouTube anyways.

In general I haven't found a good way to search for video content so it seems whether irrelevant whether its on YouTube or not since I cannot find it either way.

Am I misunderstanding you?

> Am I misunderstanding you?

Perhaps. As I said, YouTube rules video and video search, there's no point arguing that. It just bothers me that as a consumer I have to go through ads the revenue of which doesn't benefit creators, and that lots of content apart from genuine video stuff such as memes, carpentry/hobby, and dance and niche content as you're saying, could be much better (and with incomparably more effective data use and accessibility) presented on a fsking web page rather than on 1960's linear TV shows with ad interstitials, noisy video suggestions, etc.

It looks like Bing SFW search is only Youtube, same as DDG. Only video search site I've seen lately is Petey Vid which searches the expected Vimeo, Dailymotion, but also Instagram, Twitter, Peertube, and a bunch of random non-US video sites.

I just tested, and Bing Video Search returns non-Youtube sites even with strict SafeSearch.

This is easiest to check by using the "-site:youtube.com" parameter, but they also show up for certain queries (try "news") or if you scroll far enough down the page.

> Then why not go directly to reddit.com for your search.

You’ve clearly never tried to use the reddit search. It is almost guaranteed NOT to give you meaningful results.

The benefit of using a search engine is knowing that it will give you the results you want, and not having to rely upon the sites’ own search implementations.

That obviously does not imply that one needs to use Google specifically, but I commonly will add “reddit” to a search query on any search engine to get the results I’m looking for.

Most of the time I'm searching for very very specific and niche technical topics, and while Google's result quality used to be the best, and it has significantly degraded since then, its competitors are unfortunately still far behind...

In the end there isn't much new content worth searching

True, and the old content that's actually worth reading seems to be getting more difficult to find.

> "I think it is time (if not a bit too late) to try to challenge Google Search with something new, a new perspective on how the information on the Internet can be indexed and searched. I bet a lot of people are constantly thinking about this, but the window of opportunity may be now while Google is putting a lot of their time and effort on ML/AI. I doubt you can compete with them by building a ML-based search, so it must be something else both in terms of technology and possibly even the UI."

There was a social search engine, Aardvark (circa 2008). Google acquired it in 2010, then discontinued it off in 2011. The design is discussed in their paper below. There was also ChaCha -- another social search engine.

> Aardbark wikipedia entry: https://en.wikipedia.org/wiki/Aardvark_(search_engine)

> "The Anatomy of a Large-Scale, Social, Search Engine": https://web.archive.org/web/20110216155450/http://vark.com/a...


=== Update ===

I've found some HN posts about Aardvark and the acquisition:

> Aardvark: Now All Your Friends Are in the Answer Business: https://news.ycombinator.com/item?id=677178

> Aardvark Mulls Over A $30+ Million Offer From Google: https://news.ycombinator.com/item?id=980607

> Google Acquires Aardvark For $50 million: https://news.ycombinator.com/item?id=1118132

>> In addition, what's changed between 2017 and now is that back then Google was still admired by the tech people.

That’s not quite right. There are plenty of tech people who stopped admiring google much earlier (shuttering of google reader was probably what keys me into where they were ultimately heading, and I’m sure many others saw things much earlier). On the other front, G$ is neither entirely evil or good. It is a very large company. There is still good, field advancing work and valuable contributions to the open source community coming from them. One the other front, there are absolutely perverse motivations and capabilities to break the open internet coming from them.

It’s a fine line to tread. They are like the Bell labs of our generation, for better or worse. Great tech came out of it, and breaking it up will kill that great tech R&D part. Then we will have to wait for the next disruption. I’m not in a position to say which is better, but this is how I see Google position currently.

It worked for Microsoft.

They started as liked hackers.

Then they went full throttle with monopolistic practices, corruption, patent trolling, litterally and publically insulting the competition...

And now, they just use a tiny fraction of the huge amount of money they made during that time to wash their image: better PR, involved in open source, etc.

I'm not worried for Google. People will forgive and forget everything like they do with most politicians.

They just have to keep accumulating benefits, and then hire poeple to "make the brand great again". Works everytime.

I’m no fan of Google these days (I use DuckDuckGo) but I’m not sure I can entirely fault them with the decline of search quality. Spamdexing is burying the web in mountains of garbage. Malicious SEO and fraud have exploded and continue to grow at an accelerating rate.

What do we do about all this? I don’t know. I want to see a new kind of search engine that can mine through all this crap. I would build it myself but I’m too busy with school right now. I would like to see an experiment with static websites that have no JavaScript and no cookies. The kind of thing that was the original promise of the web: a collection of information connected by hyperlinks. How we get a search engine to achieve that, I don’t know (it may take other tools as well), but I think it’s worth a try.

I feel like Wikipedia is doing a pretty solid job of holding onto that dream: no shouty content, pure information that's maintained by those who are interested in it, and a fairly well outfitted battalion of bots that fend off adversarial edits

Wikipedia is pretty good for its specific purpose but it’s far from the promise of the open web. Wikipedia has a number of authoritarian moderators who will revert people’s changes for opaque and arbitrary reasons.

Maybe if you took all wikis as a collective then you could get something a bit closer. There’s still the matter of finding them so it’s not ideal. Then there’s the matter of ads. Wikipedia doesn’t have them but other wikis are rife with them.

> I would like to see an experiment with static websites that have no JavaScript and no cookies. The kind of thing that was the original promise of the web

You might find wiby.me interesting then

I like your parallel between Microsoft and Google.

To me, it also implies Google probably won't ever lose the search crown directly, the same way Microsoft never lost it's OS crown, to Linux or MacOS.

However, Windows is dwarfed by the Android install base.

True but I think the parallel between Google and MS (of the 2000s) is more interesting in terms of the vital signs in their organization rather than on the product level. Because Windows was only improving from release to release, while Search is getting shittier and shittier (sorry, can't find a better word for at least my own frustration with using it today). So Search may in fact die a quick and sudden death at this pace. Theoretically replacing Search can only take one browser update: offer a choice but rearrange the options, for example.

However, what Google and pre-Nadella Microsoft share is let's say living in a bubble of a tremendous success of one product that's also the main source of revenue (MS Office, and Google Ads), plus inability to innovate. This is an institutional/organizational symptom more than anything else.

Old HN'ers should remember PG's essay[1] so maybe now is the time to reassess Google's place in the world.

[1] http://www.paulgraham.com/microsoft.html

Yeah, I noticed a trend in Google search just sucking (being less relevant) more and more, but tbh same happened to Windows as well.

However I do find Google's lack of relevancy way (compared to itself few years ago) more stark.

Isn't Google making more and more money from non-search products? e.g. Google cloud products, YouTube.

It seems they are trying to differentiate, see also Google+, stadia etc, just not being very successful.

Doesn't seem to be the case [1] The article that comments on this chart says other sources (non-ad ones) look promising but it more sounds like a speculation because I don't see real signs of that.

[1] https://www.theatlas.com/charts/n8ea9MPE0

why not? it's about ~9B in 2016, ~11 in 2017, ~19B in 2018 and ~10B in the first six months of 2019, with the last quarter being the largest in the past.

Sure, it's less than the ad business but it's doubled in 3 years, surely that's a healthy rate of growth?

That’s revenue not profit.

How is Windows OS these days? I haven't used it since the early 2000s. I think the last version I used was... XP? Everyone at my work uses a MacBook.

Some of my children are in their early 20s now and have never used Windows. It has always been OSX/MacOS, iOS, Ubuntu, Android, or ChromeOS. I don't know if my kids are typical or not but if many of the rising generation don't grow up using Windows, will they want to use it when they are adults? Will Windows hold on to its OS crown?

Windows still owns gaming which is going to keep it relevant to the youth despite anything it does poorly for some time to come.

Just look how many front-page carousel blockbusters on Steam still only support Windows.

I game on macOS and things are slowly getting better, but macOS is still mostly relegated to Aspyr ports. I'd be reluctant to claim that Windows is dying to youngsters until the state of gaming radically changes.


Do you have Proton (Valve's supercharged Wine distro) on macOS? Here on Linux, Steam's game coverage has improved significantly because of Proton

Gaming is definitely the big use case for windows. Also, an ok os if you don’t want to pay through the nose for top end upgradeable hardware or run a Linux desktop or hackintosh.

Haven’t tried desktop Linux in 20 years, hopefully it’s gotten better. Back then it was common to have to seek out and compile your own drivers to get network, audio, printer type drivers working lol.

If they work in any large company in most industries, then yes, they will use Windows.

At scale, Apple is too mercurial to be a sole source supplier. Chrome is interesting, but there isn’t a ton of adoption outside of schools.

I work for a Fortune 150 company with tens of thousands of employees. Every meeting is a sea of MacBooks. I can't remember the last time I saw a Windows machine at work.

I have a Macbook as well as a Windows laptop. I honestly find Windows to be a smoother experience than OS X. Windows management and file management have always been better on Windows, but it is now also much faster than it used to be. A recent update has made search very good.

At this point, all my heavy duty productivity work happens on Windows. The Macbook is mostly used only when I'm working away from my desk.


87.76 percent.

Not everyone is going to pay a massive premium for Macintosh no matter its alleged advantages.

What are the advantages anyway? I used it for a while and came away mystified. Is it all about the appearance? They spend so much effort on its appearance that I suspect that's a big part of it. It does look and feel much nicer than a typical laptop.

From a practical point of view, it felt like playing Candyland. Big silly icons bouncing around, stuff like that. The Unix shell is nice, but you can get that cheaper and better elsewhere.

If macOS wasn't UNIX-based it wouldn't have taken off for professionals to use. I quite enjoy the much better screens and my eyes are less tired when sitting at a Mac, but again, if it wasn't for UNIX, it wouldn't matter.

I ve come to find these arguments about pc vs mac are neverending and very polarised. They re not very rational either. Just like all arguments about aesthetics

The UNIX shell is not just nice, for people like me it's crucial. And mind you we are talking about Windows vs. macOS here, not macOS vs. everything else.

gaming... that's sadly what keeps kids on windows. and huge contracts... what keep companies on windows.

Great points. The idea that Google was evil had been out there long before 2014 of course, and perhaps some of that tin foil hatism was prescient rather than ridiculous.

> a new perspective on how the information on the Internet can be indexed and searched

I wholeheartedly agree! In order to fuel the brainstorming with some divergent ideas:

- Queries answered/improved by real people (you didn't find what you wanted with "X", now people can suggest alternative search terms to help you find it. Whoever helps, gets a reward)

- Reputation-based search based on reviews (can be networked [as in social network] as well to put "trusted" reviews/hits at the top)

- Expose the query-language for your search engine, allow anyone to build a UI around it, link them together somehow so it's easy to jump between views

- Highly curated search engine, have a team of people insert pages manually after review

- Adjustable ranking. Some people might prefer links to are used a lot on Twitter, or from a special blog they like

Mix/match the above ideas, add/remove some, experiment and deploy prototypes for people to try. I'm sure there are better ways of doing search than what Google has right now.

(if you feel like contributing and adding more ideas, please use your lateral thinking, instead of convergent to disprove the ideas above)

Everything you mentioned is possible but needs a lot of product management level refinement. Also I'd be wary of all kinds of adjustable and adaptive stuff since they create a bubble that may be difficult to escape should you become curious about things you don't know. Adaptive stuff discourages curiosity and it's one of the things I started disliking about Google Search among other things.

There may be some room for thinking about bringing unique content above non-unique one. Chit-chat on myriads of forums is probably not interesting unless you are looking for something very specific in which case you'd use the query language. It would also de-rank all non-unique marketing fluff that can be seen everywhere. Of course this might open an opportunity for spammers to de-rank good sites by copying them or to boost others by adding artificially unique content, but then combined with the human review process it might yield net positive results.

Voting... I don't know, Reddit's /all is a result of popular voting. Do we really want the Internet to become all kittens, memes and showerthoughts? :)

But I'm thinking the stereotype of what a search engine UI should look like could also be reconsidered. Google has made a lot of interesting gradual improvements (Wikipedia, IMDB etc. snippets, maps, calculator, dictionary) while leaving the core UI almost intact. For example, when skimming through the search results, do you read the web page titles or the content with your search terms highlighted? Etc, I think there's some work to do for UX folks. Let alone that Google's highlighting functionality has declined considerably to the point of being almost useless most of the time.

Web directories failed though, repeatedly and especially the curated ones. Google was seen as better because the algorithm was opaque.

I think a good mobile version of Apple spotlight might do more damage to Google than Bing/DDG.

Improving voice assistants in particular, like with Siri, could damage Google unless they’re not careful, since these assistants go beyond search and act as an operating system, to control IOT etc.

Text search is valuable and so on, but I can imagine that voice becomes the primary mode of interacting with the world (internet, etc) in the future. In that world without screens as the gateway, where are the gateway ads?

Back in 2015-7 or so yes, you'd be immediately dogpiled on here and reddit and most other sites for daring to criticize Google. I was trying to warn people about them and was always banned or downvoted to some shadowban realm immediately.

Google isn't the actual problem though, advertising is. The backlash needs to be primarily about ads, not the companies that just lie repeatedly in order to bend all tech infrastructure towards the needs of advertisers.

We've never banned people for "daring to criticize $bigco". That's half of HN. If we banned you, it was likely because there was something else abusive about your posts.

Ads are the worst, but aren't the only issue. Apple and (older?) Microsoft are problematic too without being advertising businesses.

> Google is not dying any time soon

One way to kill it is blocking all their domains either in /etc/hosts or using pi-hole. But I imagine using the web would be slightly broken as Google host various font libraries and javascript libraries which if blocked, could break many sites.

Use https://decentraleyes.org

"Protects you against tracking through "free", centralized, content delivery. It prevents a lot of requests from reaching networks like Google Hosted Libraries, and serves local files to keep sites from breaking. Complements regular content blockers."

Thanks for recommending that :)

"Google is evil" has been a given for close to a decade now, and certainly since Snowden.

unfortunately there's no competition to Search on the horizon”. What about duckduckgo? I’ve been using it roughly for a year and i’m happy with it.

DDG mostly uses Bing, Yahoo and Yandex (anonymized). To be honest apart from some obviously spammy web sites that rank high only on Google, their search results don't seem to be significantly different. My impression anyway. And so we don't even know whether what we see via these search engines is the real Internet or we all live in a bubble created by them.

I could be wrong but I think DDG is actually backed by Bing.

The tech industry should not take it too personally / hard: any forces that earn prestige will soon become surrounded by similar challengers

I think geeks overestimate their influence. Google’s end user products were not successful because of a few nerds.

1. Chrome was successful because it was hawked on Google’s home page.

2. Android was successful because Apple never played in the lowend market.

3. Gmail was successful because it was better than the alternatives and had more storage space.

> the window of opportunity may be now

Google has secured a lot of top AI researchers. Depending on the dates when their options vest, we may see a lot of them leave.

What about DuckDuckGo or archive.org? Wikipedia?

People used to sit on a couch and watch TV for hours and also spent hours on the old phones talking about the most exciting shade of nail polish. Shopping is also fun.

Now all that activities largely moved to Internet. Thanks to omnipresence of smartphones the hours dedicated to those activities had also increased. Of course share of general Joe Doe's website got smaller as said website does not always cater to few most popular people's activities.

That does not mean that the total amount of Joe Doe's websites decreased. It is growing, just not as fast as the amount of my cat videos.

Sure big companies may try to subvert the Internet but I think that creative type of the populace will always find ways.

I agree and I also believe that it's important for people that care about these things to be vocal, stand behind and support people and companies that are doing good things. Promote what they're doing to an audience that doesn't know about them, buy their products, share their work... only that way the web may "recover" and a larger percentage of internet users may actually discover and spend their time on the more independent stuff.

Except that it's drowning in noise. The SNR is definitely dropping.

Well I would say that web search takes good care of SNR. So far no troubles finding interesting stuff.

"Finding interesting stuff" maybe. But finding a lot of the things I used to find some years ago is just not possible anymore.

Somehow searching changed as well as the ecosystem of the web itself. The big global players dominate nearly every inch of the web and many sites look like clones of each other.

There is a lot more effort being put into censorship and also more threats to ordinary users in my opinion.

When I loved to use the web it was like a big playground and it seemed that the most people there where extremely curious and open-minded.

Today there is so much hate - no real discussion (this community is a rare anomaly considering this argument) and I think it's not about discussions for these people who spread hate. Also it's all about the money. In the earlier days people used to write blogs and create sites just for fun but today everyone wants to get some money or measurable attention (likes, views, followers etc.).

Constantly you are being bothered by advertisement which gets more invasive from time to time, cryptominers and other malicious software that creeps into your system if you just click on the wrong link. Don't remember that from the beginnings and I'd love to experience that again some day.

All you have to do is be the change you want to see in the world. There's plenty of us out there still living the dream of the 90s. Your descriptions sound like you're still stuck on the big platforms where money rules everything. Just stop. Start hosting your website from home. Code it by hand. Don't care if anyone but you likes or sees it. It's the best you can do.

I've been there & have done that, thank you ;)

For my own services it's okay to host them from home but "the web" just doesn't work like this anymore.

Do something big and you'll want to get things like DDoS protection - talk to customers about projects and just realize that they won't scale on your mini server at home while the internet connection is much too slow and unreliable.

I have started like you describe it and I learned some things on the way but the industry and the web have changed. Of course there is tiny companies who can still get their static website from a freelancer guy but this guy has a lot more responsibility and tasks to fulfill nowadays.

In my opinion back in the 90ies you could play around in a naive way without getting hurt or hurting anyone seriously. Today it's about a lot of cash, there is more laws, censorship and other malicious entities (go back in time and tell the people about bitcoin-mining in their browser and DDoS attacks against centralized services like CDNs that bring down half of the WWW).

Put a server on the web that isn't hardened properly and have a look at the logs - this is some new level in comparison.

>Put a server on the web that isn't hardened properly

It's easy. Just don't do/enable dynamic things. There's no need for anything more than a static webserver in 99% of cases with personal websites. And static webservers ship very secure by default. There's not a lot of remote exploits in nginx or the like. It's much more risky to use a web browser with javascript enabled.

I'm wondering how far back you're going, because when I think of "the old web" I think of being intentionally misdirected to a worthless website that had temporarily won the SEO fight and "Punch The Monkey And Win A Prize" flashing ads. Not to mention popups, pop-unders, and all manner of ad-spread malware taking advantage of under-policed ad networks coupled with near monoculture of OS.

Today's internet experience is much improved

I am going back to the 90ies.

In my opinion it was much easier to block those popups etc. as they've not been as invasive as the current malware.

Of course back then there was already stupid things as well but it scaled and got a lot worse I think.

I suppose you talk about the experience as a user? Because as an admin etc. things got a lot more complex since then.

My particular need is finding particular stuff. Information.

The Google used to be very very helpful. Now I have to McGuyver it into giving me what I am looking for. Now, my Google-fu is ok, and I can do it - but it used to be trivial.

Maybe this is an artifact of the "web is too damn big!", but I suspect that Google itself is going through both organizational (not necessarily algorithmic) growing-pains, but learning what to censor and what to promote in this complex world, and we are the crash test dummies.

Which is sad - to me - because I used to like it when they just gave me what I asked for.

" when they just gave me what I asked for"

was when the web was for hobbyists and companies that could afford a substantial investment. Before the web was 99% professional profit-driven spammers (both automated and human).

Sadly it become a common occurrence for me lately to be working on researching problems and google giving me tons of results that only match a single word of my queries

Thank you, this was an amazing insight.

FTA: "GOOG and FB ceased competing directly, focusing on what they do best instead."

“People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices” - Adam Smith

Some things never change.

I think Smith is almost right, a _conspiracy_ is likely in the broadest sense but he's wrong to suppose it will necessarily be against the public.

So, the CAB is maybe a useful contrasting example. The CA/Browser Forum (CA/B or CAB for short) was a meeting that happened as a mixture of conference calls and email lists back in 2011 or so. This is where Extended Validation (the green bar) comes from, but ultimately CA/B metamorphosed into a standing (in the sense of ongoing, not you can't sit down) meeting, still going on today, for the Certificate Authorities and the Browsers (today in practice the Operating System vendors, but with Mozilla standing in for the Free Unixes) to get together to figure out what the rules are.

Because of what Smith notes, the CA/B's rules are explicit that discussion about prices, product offerings and so on is prohibited, if you want to conspire to raise prices you'll have to do it somewhere else.

But plenty of _good_ things can be agreed this way. Contrast the many years it took to get rid of MD5 from certificates (which happened before CA/B existed) despite collisions being known - against SHA-1 where CA/B was able to co-ordinate industry wide discontinuation of SHA-1 with only a handful of minor defections before even a single proven collision of the hash was published.

More broadly I'd argue that the IETF, and certainly the thrice annual IETF physical meetings (the most recent was in Singapore last week) are exactly "people of the same trade... meet together" and you could characterise plenty of the work done at many Working Groups (obviously DPRIVE and QUIC but I don't see any reason you couldn't make this claim for KITTEN for example) as a conspiracy, just that it's only "against the public" if you think global surveillance was what "the public" wanted...

Why else would you meet with a competitor? :-)

Most of my friends are co-workers, or other people in the industry, including competitors (more the field/sales types than the corporate/management types, because you see them when you are bidding for a client's buisiness).

One of my friends used to be an employee. He quit. He then forked me. He then undercut me (or, well, my prices).

Somehow, I don't know how, maybe because I am such a sterling man of honor (ed: throwaway prince Andrew reference), we are still friends. But I've got to tell you! In terms of price? It's kind of still a race to the bottom. (At least for the vanilla, not the value-add, product)

So - even though Adam Smith has it right (he often does!), the public does not. We are enjoying shared stories of battle, over large accounts, and the blow-by-blows during large-account acquisitions go, and toasting one another's accomplishments, after battle, but ... the prices end up going down. It is the nature of competition.

Sometimes I wonder how you balance that Adam Smith quote with "Tragedy of the commons".

I don't think it's accurate to count stuff like youtube and google cloud as part of the %. It's gigabyte heavy but attention light. Just raw data about cat videos.

Compare it with say whatsapp traffic - that has a crazy high amount of attention crammed into each gigabyte.

Doesn't really change the message/conclusion but the metric used here seem flawed.

I had the same reaction. Bandwidth is only one metric of interest. I think time spent is a better proxy to the true metric of interest, which you rightly called out as attention.

There's a fundamental error, or at least assumption, in this piece that really needs to be called out. The author states it this way:

"Any website aspiring for significant traffic depends on Search and Social traffic."

This is stated in the context of a discussing of what the web was and might have been.

Let's be clear: when the web was tiny, having "significant traffic" wasn't a particularly difficult target to met. Get yourself on the netscape "What's New" list, and that's more or less the end of it. But the web (really, the internet) is now unimaginably huge compared to the days referred to at the beginning of the article. Having "significant traffic" now is not a realistic goal, or even a worthwhile goal, unless you define "significant" in a way that isn't strongly related to simple numerics.

If you're interested in and/or aspiring to the sort of web that seemed possible/desirable in 1994, then you're not really interested in building sites that attract "significant traffic" in the sense of huge numbers. Attracting the right people is a much more important goal, and that still relies on the same basics of "how to write a good web page" that existed in 1994.

In short, the dominance of "the trinet" in driving web/internet traffic only restates what could have been before computers: most of what humans do is related to commerce and mass entertainment, and attempts to do other stuff may succeed but will never be as visible or dominant as those two.

>The Web’s diversity has granted space for multiple businesses to innovate and thrive, independent hobbyist communities to grow, and personal sites to be hosted on whatever physical servers can host them.

A few months ago I was feeling the same way, especially about personal websites. I tried to find StumbleUpon as that seemed like a good alternative to see random unique websites. However, I was saddened to see it had shut down or transformed itself into a kind of Pinterest like clone.

Given that StumbleUpon was gone, I tried to quickly recreate the core functionality - click button get random website, at https://stumblingon.com - there are still a bunch of small, personal, interesting websites out there. We just need a good way to find them.

First result was a coolie cutter website selling seo services The creative web is truly dying

I don't see anything wrong with trying to sell services. I wouldn't want a generic, fraudulent, big business, or cookie cutter site in there though.

If you recall what website it was (or check the history after reloading stumblingon) and let me know the URL I can reconsider whether or not it belongs.

This is pretty cool. Thanks.

>Internet Service Providers (ISPs) are probably soon going to dictate what traffic can or cannot arrive at people’s end devices.

This was the money quote for me. FANG may kill the web, but the idea that ISPs (in my opinion some of the scummiest corporations out there) get to decide what content they serve me and what they don't has the potential to kill the Internet as we know it.

Yes. The author of the article buried this point under all the Google, Facebook, Amazon commentary. Yet, it’s the key point.

Also, this article got me thinking that maybe public research should get to work on the AR/VR aspects of the web so the key infrastructure of the web continues to be publicly owned.

How can they even do it with encryption and CDNs everywhere? It's just not possible. The last trace is unencrypted domain name in TLS handshake and it'll be gone soon.

> The last trace is unencrypted domain name in TLS handshake and it'll be gone soon.

How can they respond to dns queries if they don't know what you're asking for?

I think you could potentially use a DNS independent of the ISP? But ultimately the ISP will know what IP you're trying to reach due to the nature of their position.

I can use as my DNS resolver. So all my provider can see is encrypted connection to then encrypted connection to some which belongs to cloudflare and serves plenty of different websites.

ISPs/governments and CDNs/clouds could start partnering/synergizing/colluding. Shake some hands and write some APIs to share user data with each other in real-time. Even make it a 3GPP standard.

Naw, with the coming of 5G it will destroy their monopolies on the communication lines which should open the door to more local competition.

No. The modulation and time sharing built into 5G only offers 15-20% increase in bandwidth available, MHz for MHz, when compared to 4G LTE. This is not a significant increase. It is not at all like the 3-4x increases in bandwidth, MHz for MHz, that the jump from 3G modulations to 4G LTE did.

Almost all of any potential increase in total speed will have to come from using new parts of the electromagnetic spectrum. To that end the FCC is kicking out old users from the 3 GHz range and auctioning it off to the mobile telcos. This includes both the traditional backend media distribution role and completely eliminating the amateur radio alloacations (ref: https://docs.fcc.gov/public/attachments/DOC-360941A1.pdf).

But even with stealing bandwidth from you, me, and the normal media companies it won't replace an ISP with wire. Why? Every single wire or fiber has as much bandwidth to itself as the entire electromagnetic spectrum useable in freespace. And you don't have to share. Wireless, no matter the modulation, can never replace wired ISPs simply because there's not enough spectrum for everyone to use it at once.

Additionally, the 3-4 GHz spectrum really doesn't have that good of propagation characteristics. And mm-wave, well, that's about as effective as a light post light in terms of coverage area.

No, 5G will not destroy any monopolies. That's silly.

Smaller cells do wonders. A LimeSDR with <200$ of general-purpose CPU/assorted components to work between the USB3 SDR and Gigabit Ethernet (and some <50$ antenna + GPS-clock) is enough for a small 5G cell, where small primarily refers to single-channel and limited reach due to ~4mW output power (on each of the 2 antennas).

The reach can be increased for <50$ to at least a few hundred meters radius.

I don't understand, are you advocating for hobbyists to make their own 5G cells, or carriers?

Why not use Wi-Fi at this point?

I assume Namibj is suggesting a switch from our current hub-and-spoke ISP model to a peer-to-peer mesh networking model.

Radio technologies have strengths and weaknesses. While WiFi works great in a home, other radio types are better at transmitting data long distance. The best solutions combine a mix of radios to leverage their strengths.

Unfortuantely it's not the radio technologies that allow the cell network to work. It's the cell network paying the big bucks for height above terrain on various towers, buildings, and the like.

This is the problem mesh networks have to solve: height. Everything else is cheap and easy. But without height above terrain and line of sight it simply won't work.

Not really. I mean that cells are cheap, so the increased density offers higher speeds (less sharing) and possibly new organizational structures due to a lower entry barrier.

I'm advocating that cells are cheap, so the density can be increased by a lot. This might enable more local competition, but the main benefit of LTE is proper QoS support.

That assumes that you'll have a healthy offering of wireless providers. If you're from Europe, then I understand that's the norm there, but in the US, it's the same story as your local ISPs. For home internet, you'll be beholden to whatever companies own the towers near your property.

But what makes wireless telecom giants any better than ISPs?

I think their point is more competition is better regardless; every one of them will try to find some way to make themselves seem better than the other. Though the problem with that regardless is that from a business perspective, there's a very finite amount of spectrum, and it's being gobbled up exclusively by four (maybe soon to be three with the pending Sprint merger) companies.

From a technical perspective, there's definitely been mixed reactions to the effectiveness of 5G technology. That's another topic in and of itself, though.


Over the air carriers have much broader latitude to block and prioritize traffic than the wired ISPs do. They also have an absurd national security justification to build out 5G local infrastructure without any input or regulation from local authorities.

5G isn’t a substitute for fibre.

Could you elaborate on this?

But somehow it's the FANG that decide what to serve/monetize, not the ISPs.

I've always the held the somewhat cynical position that this sort of stuff would push [the interesting] people towards other, decentralized Internets. I'm a big fan of new projects like IPFS and DAT, and GNUnet had a great call to arms on their about page (which seems to have been moderated a little) [1] saying very similar things to the article. Maybe the internet of things will save us after all, by finally providing the vast, dense network of interconnected wireless devices that is needed to provide a robust structure to a really distributed net.

[1] https://gnunet.org/en/about.html

> [the interesting] people

I don't think those people have to leave. We just need to collectively realize that, just like with old media , there is a 'mainstream', people who mostly use the internet (the facebook crowd, who are now the bulk of the web traffic), and the more eclectic crowd who care about the internet (largely, the pre-FB mass of the internet).

These are two distinct crowds. Considering that the FB crowd contains mostly people who did not use the internet before, and don't really know its culture, it's going to take a decade or more until this mass of people is replaced by younger internet-natives. Until then, the net is living in a transitory phase

That's interesting, I like the idea that digital natives will have more of a vested interest in caring about the internet, but I'm not sure if it will play out, especially if there continues to be such a strong focus on things like Instagram and tiktok which are largely disconnected from the net at large.

I am learning Japanese. I am spending an increasing amount of time searching for things in Japanese, and am starting to notice how often a 'trinet' website even comes up on the first page. Virtually zero. I wonder if the 'the web is dying' mentality is only restricted to English? I would be surprised if English websites made up even half of all internet traffic

Actually, the web began dying in the late 90s.

The eternal September was 1993. I remember that change well and the earlier flavour I remember nostalgically. And it was tied to the rise of the web. There's a good argument to say the web killed the internet.

But all things change. Every phase has good and bad points. IMHO

I'll take 2000-on to the 1993 web/net any day.

I get:

* an up-to-date encyclopedia

* all the music referred to by said encyclopedia (Napster)

* an impressive number of books referred to by said encyclopedia (Demonoid)

* 2010s bonus-- a lot of the scientific articles cited by said encyclopedia (Scihub)

* 2010s bonus-- a searchable cache of stupid questions to do an end-run around the pain of interacting with the asocial gate-keepers who wish the internet wasn't so full of people (StackOverflow)

* an archive of all the stuff you're nostalgic for (archive.org and Google's copy/paste of usenet junk)

My internet is like O'Brien to your Winston from 1984-- its mind includes yours.

Edit: clarification

OK. But most of that stuff is noncommercial, and not part of the neo-AOL/Prodigy thing that the Web has become.

In my alternate history, the Web would have all that. And so much more like that.

Yes right after Mosaic 1.0 was released. RIP

Hard to tell if you’re joking. If you do actually believe this, I assume you also believe that e-commerce broke everything?

No, I'm not joking.

It's not so much e-commerce that broke everything. Although there were many who did argue that.

For me, it's been the shift from self-published (and indeed, self-hosted) to Google, Facebook and cloud services. And the tide turned in the late 90s.

Edit: When I first got online, it was AOL, CompuServe, EcoNet and The Well. I was on all of them except AOL. And then we had Usenet and websites. And networks of websites. The prospect of citizen-based journalism. But now we're back to something a lot more like AOL.

It has never been more easy to self publish ... via Twitter and Instagram. But the real problem is that you only have 1 hour of free time, spread out in five minute intervals during the day. Being an adult sux.

Those are all centralized. And trivially censored.

I'd rather have stuff like Aether, NotABug, Tor onion sites, and so on.

This is NOT self-publishing.

When making home made food, do I also need go grow the wheat myself? eg. do I need to run the server at my house, or can I use a service? Where do you draw the line?

How easy is it to move your profile from Twitter/Instagram to another host ?

or something a lot like Prodigy...

Damn, forgot Prodigy.

I was mainly a CompuServe geek.

If by the web you mean booting up an entire JVM to enable drag and drop

I mean static HTML.

Why that is the web anymore than a javascript enabled version makes no sense.

It's arguably rude to be running code on someone else's machine without explicit permission. Except for websites, indeed, it's generally illegal.

That's why prudent folk use tools like NoScript.

The next consolidation will be cleaning up the plethora of apps for mobile. User's interactions can be simplified with a few well-designed multifunctional apps. WeChat probably points the way in this regard.

Interesting article that's somewhat ruined by a click-baity title that it can't support. I first got on the web in 1994. There was far more difference between 1994 and 2014 than from 2014 to 2017. And that's the crux of it: just because it's different doesn't mean it's dying.

The difference between 1994 and 2014 seems more quantitative though. Here we are talking about the ownership of the referral traffic and how those companies who own it try to benefit from it. Nothing that happened between 1994 and 2014 comes close to it in terms of the impact on how people see the Internet and by extension how they see the world today.

I agree with most of what you're saying, but disagreeing with this part of it. The change between 1994 and 2014 is the difference between the internet growing to global dominance in everything from social connections to . . . well, is there anything about our lives that hasn't rested somehow on our usage of the internet since well before 2014? It seems to me that the internet of MSDOS 3.10based computers and dial-up modems to online shopping and the creation of billionaires is more different than what's changed from 2014-2017.

It is true that online shopping and online ads alone have created a whole new industry in a very business sense of the word. The change happened some time towards the end of the 1990s.

However, imagine you discover a new continent with vast resources and invite everyone to share the land and start producing and trading. When all land grabbing is done you begin to notice the rise of some powerful players who buy out or simply squeeze out smaller land owners. At one point your whole continent and the traffic within it becomes either directly owned or dependent on these giants to such an extent that you simply die if you don't play by their rules. Your continent now more looks like a conglomerate controlled by only a few powerful corporations rather than a free republic. And we can't even be sure they didn't make arrangements between them behind the scenes. So it's the ownership that's different now, which is nothing new really, we all kind of know about it.

What is "Web"?

Side note: both FB and Google cannot be accessed from China.

... But they both had basically equivalent local counterparts

Uhh... I think it's http. Or maybe it's just any data that a web-browser can handle. Definitely a few OSI layers above the internet.

No question about HTTP. However I'd argue that when HTTP is used for weird purposes that aren't directly related to serving hypertext pages (often to go around draconian firewall policies), or to run arbitrary code (JavaScript), then it's not the Web anymore. (Yes, this means that "Webapps" is an oxymoron.)

Yes they can by using a VPN for example.

The VPN can access them, but not from china.

At least in 2018 I was able to.

I was trying to say, if the VPN was in china, people in china wouldn't be able to access google and facebook through the VPN.

Google is interested in building an AI that is designed to turn people into its products.

What do you do with a product when you're done with it? What do these people become?

All you need to do is search google for weight loss advice or anything that could remotely have a commercial interest and what pops up? Pages and pages of irrelevant clickbait. If you ever wanted to build an encyclopedia of nudges, persuaders, and other such exploits of the human mind, now would be a good time.

The entire game is forcing you to take actions, like answering "yes" to a sales guy over and over to build the habit so you'll agree to the sale. It's taxing on the mind, and like television, it got to the point you had to see 30 minutes of garbage ad's for 20 minutes of entertainment, that it's no longer worth it.

The "Trinet" is dieing because it has become useless; something far more useful will come along to replace it because people are viewing the recent infestation of marketing people as damage and are routing around it.

Wait but goog fb and amzn have been aggressively trying to capture global markets. They have been increasing the footprint, does not that count for % increase from 2014 to 2017 ?? Does not make sense ..point about web is dying is opinionated. Today I can watch Netflix, order from doordash and connect with my family through FaceTime. All via web.

Yep. Author is romanticizing the "democratic" web of the time before facilitating companies simplified the process of getting and being online. The reality I remember is that it wasn't a democracy; it was a technocracy and the barrier to entry was high. It still is; hosting one's own full-fledged web server that is secure and reliable has only grown in complexity in the past decade, improvements like Docker notwithstanding, and it's no wonder average users have flocked to Facebook, Google, and Amazon for simplicity of access.

To take Facebook as example, people seem to often forget that the primary feature Facebook offers the average user is simple low-cost blogging of small thoughts and reliable auto-aggregation of those blogs. No fussing about with running one's own RSS feed or managing RSS subscriptions.

If the "web is dying," it's dying like a wooly mammoth---ill-suited to a changing climate. The mentioned companies created an ecosystem where the fundamentals of communication, location, and commerce can be accomplished without the technology getting in the way. It's no surprise users flock to them.

Don't know about doordash, but Netflix and FaceTime are clearly NOT part of the Web. (They are still part of the Internet of course.)

Great analysis; highly-recommended reading.

I want to bring attention to the followup article Staltz wrote about what we might do about the problems he enumerated in the OP:


Small pet peeve: why do some people write GOOG, AMZN, etc instead of Google and Amazon when the context has nothing to do with the stock market?

I'm guessing to easily refer to the parent company (not sure if these collections are still called companies, english is not my native language). Instead of GOOG one could probably use Alphabet but differentiating between Facebook and FACEBOOK might be a bit confusing and for Amazon that might just be to be consistent with the rest of the article.


So what's changed in the two years since?

Traffic is just so hard to measure if you equating page views or hits, or packets with relevance, economic value, importance, etc.

In 1994 I was junior in high school and took as many years of auto-tech as possible. In the end we were rebuilding engines and air-conditioning units. And yeah, our high school had welding, cnc, woodworking, auto-cad, and even lumber-jacking as electives.

But I took a lot of auto-tech, three years of it. And the funny thing is I have never gotten my license. I'm nearly 45 years old and have never been able to legally drive a car.

But I was living with my sister at the time who was 18 and our landlord had a weird crush on her. Our landlord also owned a ton of properties in Eugene. And he would have me mow as many lawns as I wanted for 8 bucks a hour. Most of his rentals were to elderly people. I was making double the minimum wage and had no shortage of work. But I also had a girlfriend and skateboarding and bowling league to attend to. I did the minimum to maintain to healthy balance. If I needed new shoes I would mow a extra lawn. Skateboarding is hard on shoes. I would go through a pair a month.

But eventually I found myself doing auto stuff for people after mowing their lawns. I would get scheduled for oil changes, filters, plugs, brakes.. Eventually I was mostly doing auto maintenance for people and I charged significantly more. I had a lot of cash for a kid in high school with a Operation Ivy T-shirt.

Enter the internet.

My buddy Billy went to the University and could get me a cheap computer. This was back when Apple had serious discounts for students. I got a Performa 6214, Stylewriter 1200, and 14" Multisync monitor for about $900. A significant discount.

And Bill gave me his dial-up info into his university account. He gave it out to a lot of people. They didn't check if multiple people were using it. Back then we mostly looked up stuff about how to make LSD and mortal kombat fatality codes. It was Amazing. And a lot of very slow loading scans from porno mags.

But a person that was exploring this new thing that was non-commercial with me had a forward-thinking father. He was a real-estate agent and thought the internet was cool. So he wanted to put his listings on the internet that nobody would ever possibly use for looking at houses. Images were brutal since dial-up was the norm. And we used tables since css wasn't a thing back then.

But we made a commercial site. A site for a real-estate agent. And we hosted it on the university's of Oregon's site since we didn't really know how to get a domain name or find hosting. But it was pure HTML.

He paid us well to update the site with new listings. But then the dot.com thing happened and I got a job at a movie theater and wouldn't touch the internet until 2002.

edit :: I forgot to add that I think I might have been responsible for making the internet worse.

Helping someone do legitimate commerce online isn't making the internet worse. Quite the opposite.

That misses the point. I wish the internet was never used for financial gain. Just the sharing of information.

> financial gain

> sharing of information

... These features aren't really extricable from each other in any society where money exists


I mean in general, not for any specific agent involved in the sharing of the information. But where information is flowing, someone is turning it into profit.

I am 100% certain that people have made significant revenue facilitated by their ability to look things up quickly and efficiently on Wikipedia. I know it has certainly helped me do my job. The fact that that revenue hasn't flowed directly to Jimmy Wales's institution is because nothing about commerce guarantees that everyone involved gets a cut of the money flowing.

Imagine how much money they would make if they started running ads though.

The web has never been particularly robust.

Why on earth is he naming them "GOOG", "FB" and "AMZN"?

It is ticker symbols of their stocks in the market.

Oh help, you can't be serious that he uses those on purpose.

It's a neat trick: it emphasizes that they are amoral companies required to do nothing but make money, and simultaneously that they're only companies like the thousands of other gyrating tickers on the stock market. This same author frequently uses the term "Microsoft Github".

Thank you, it's framing. Now that makes sense.

It's a quite common practice to refer to companies by their ticker symbols. I see it all the time and it has never struck me as odd or noteworthy.

I've never seen it before.

It's a virtue signal that he's in the know about the stock market.

Does that help create credibility in some circles?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact