I think one of the fundamental things that make search work well about 1-2 decades ago was that web sites would link to each other, and that those links could vaguely correlate with reputation. There were link spammers, but there was actually a some decent organic content as well.
What's happened since then is that almost all the normal "people linking to things they like" has gone behind walled gardens (chiefly Facebook), and vast majority of what remains on the open web are SEO spammers.
Why has blogs and articles stopped linking to things? I'm reading a restaurant review site, and they won't link to the restaurant. The chef name is a link to a list of all articles tagged with the chefs name, rather a wikipedia link or something useful that can tell me who that person is.
Average websites goal is now to keep you on them as long as possible. According to some metric folks, the longer you stay on a website the more money you spend there. Linking to another website destroys that metric.
Also if you are going to make a purchase somewhere, any website would try to get a cut of the money you spend by actually sending referral links to the product. So small websites that do not allow this service will not get linked so much.
On a metalevel it is thus that links or connections between items are information. Information is money. And as soon as that became evident links and connections also became more scarce.
Yup and developers have been allowing the marketing and product teams to break the back button as well opening every external link in a new window instead so users have to keep something open to their site. You always had middle-click to do this, but now it's being forced on users.
I just noticed even the goobers at GitHub break the back button when you click a project link too. I don't know why people champion this brand when they have dark patterns and shoehorn 'social' functions into the proprietary platform.
Bang on. Saying that "there isn't anything out there anymore" is missing the point: Google's algorithms created this situation, intentionally or not. Before Google, people linked to what they wanted and communities would naturally cluster around topics of interest. Google came in and made reputation into a currency which effectively destroyed all these communities through incentivizing selfishness.
"When a measure becomes a target, it ceases to be a good measure"
-- Goodhart's Law.
Google's algorithms didn't create this situation; people chasing high Google rankings did. Had Google used completely different algorithms yet became equally dominant, people still would have poured their hearts and souls into getting higher rankings.
Basically, an application of the tragedy of the commons. Or: "why we can't have nice things".
But that's taking for granted that Google would have become dominant. Perhaps if they hadn't chosen the algorithm they did then they wouldn't have been as overwhelmingly successful. Instead, I could imagine a world in which there are multiple search engines and none of them are all that good. In fact, that's the world I remember from before Google existed. Search was bad but communities were strong and life was good.
Then Google came along and we all found it a lot more convenient than the bad search engines we were used to. And of course, we all know where that led. In some sense, Google built an 8-lane superhighway and bypassed all the small towns.
We all traded away paradise in exchange for convenience. Now we have neither.
On the glass-half-full side of this: we're getting those communities again! Here on HN, on reddit, for certain topics on various social media (there are pearls there too), on Mastodon, various blog authors, Ars Technica, Quanta, etc. [1]
It's just fragmented - i.e., catering to a specific group. Because if it isn't, it's awesome for 5 minutes and then monetization rot sets in.
[1] None of these work for everyone; conversely, all of these are seen as great things by some and have people who prefer that one thing over others for its quality.
The trouble is, you are no longer "surfing" the Web, you are digging through your RSS feeds and links to interesting sites, fediverse subscriptions etc,.that's not good UX, perid.
>Google's algorithms didn't create this situation; people chasing high Google rankings did.
You're technically right. You'd be more right if you said people chased the highest spots on search engines for the widest breadth of queries.
If there were implicit alphabetical ordering of search results I guarantee you'd end up a bias toward A's, Z's or otherwise in people trying to get top spots.
How is transparency worse than smoke screen that we have today? For example healthy and good websites could rank according to good content, good optimization, variety of multimedia content, decent design and UI etc. You can't have too much of good things and qualities. That would be something like writing a too good book or making a too good product.
Because the rank algorithms are subjective heuristics, not absolute metrics. All rank algorithms always have been. It started with the link metrics, then people started gaming that. It's been a signal/noise war ever since.
It's also dangerous to ask for the exact criteria because they are ever changing. Google et al don't want to be prescriptive about what a good site is, they want to recognize what a good one is. You make a good one, they'll figure out how to recognize it.
They can't sit down and publish "The Definitive Guide to a Good Website". That's just not their role and it will be out of date before it's published.
A big problem is that a lot of community content went behind Facebook. Instead of creating webpages or forums people started using Facebook pages and Facebook groups. This is the main reason I have been anti Facebook for over a decade. Not because of privacy reasons as many are but because I saw that Facebook will put the web behind it's closed doors. Even today some of the best reviews about any product or service are usually in enthusiast community forums. But a lot of that activity has gone behind closed doors of facebook and now reddit. Most of the current thriving forums are those that pre existed Facebook.
That's why mechanism design [1] exists as a field of study. The whole idea of that field is to provide the proper incentives to steer the participants towards your objective. Yes, considering they will try to "game" the system however they can.
I'm pretty sure google could do strictly better (i.e.: better in all reasonable accounts) than they do now if they focused on the users' experience instead of revenue for a couple terms.
Paul Graham says Google doesn't want to follow anybody down that road (human intervention in search). But ISTM the problem is that even though they don't, they can just throw a giant pile of money at it if they needed to crush a competitor. Also, VC will refuse to invest in anybody doing it because Google.
I wonder if punishing presence of advertisements would filter out most pages that are SEO'd to the max and instead promote "labors of love" type pages.
This is an interesting idea because it would create a type of non- or anti-commercial SEO that could counteract the commercial one. However, Google would never do it because they sell most of the ads that would be (not) hosted on these sites.
Wouldn't it be reasonable from Google to show how their ranking algorithms work so all webmasters and content creators know how to behave on the web. Now we have black box that's causing confusion and is misdirecting websites and web users.
I don't think so. If the problem is people gaming the system, making it easier to game isn't going to improve the situation. It's not going to put good content creators on a level playing field with spammers, because good content creators simply don't care as much as spammers about search engine gaming.
But people are already gaming the system, and on any search from product reviews to code snippets, I see SEO-optimized spams populating the top results. Good content creator don't have the time or the technical inclination to reverse-engineer the ranking algorithm (or to brute-force by creating tens of thousands of sites and see what sticks in a giant multivariate test).
Knowing the actual rules might give them a fighting chance, since the bad guys already know these rules anyway.
Some would even say it killed the web by centralizing all the content in the hands of a few [0].
Which is the direct consequence of everybody optimizing to better show up on Google/Facebook/Amazon/Microsoft and ultimately even migrating all their hosting to these companies.
That will get worse yet most likely. Younger people no longer produce public text to the extent they did prior to the the smartphone heavy era. Supply of that blog style content will continue to dwindle as the producers age out. I'm sure there's a stability point it may reach, of course, because some tiny percentage of people will always want to write long-form.
Younger people TikTok, they Instagram, they chat in private conversations with eachother, they occasionally post short messages in walled gardens like Facebook, they YouTube, they listen to music, they watch Netflix & Co. That's what they do. They do not persistently write LiveJournals, Tumblrs, blogs. That pre video/audio-focused era is over and it's not coming back (even if there's occasionally a bubbling up of hipster fakery centered around how cool it is to write text).
I heard an interesting theory the other day: blog viability declined because Google killed Reader. Which indirectly ends up poisoning Google's biggest well, since blogs are an important source of relevant cross-domain links.
I'm somewhat skeptical, it seems a little too poetic to blame Google's ultimate downfall on a decision that was notably hated at the time. But it's plausible. If you want it to be a conspiracy theory, you can posit that killing off independent blogs was the intent, to convince bloggers to migrate to Google Plus.
I'd believe it. As an IT consultant, I interact with a lot of people who are semi-techs themselves- mostly small business owners who are used to wearing a lot of hats, and also the type to have been motivated to run their own personal blogs about diving/photography/conlangs/quilting/gardening/whatever their personal hobbies are.
Ten years ago, the majority(!) had at least something up and running, where they would post essays, thoughts, whatever came to mind.
Nowadays? All gone. All! When asked why, the answer almost always is along a mix of ever-increasing negative feedback and harassment from randos, and aggressive automated spamming of their forums. Loss of the pseudo-anonymity plays a large role as well. Many have deleted years' worth of work, simply because they are afraid of someone trolling through their posts to find something to harass them with.
I was never a blogger myself, but I am sad about the change. There was a lot of good stuff out there for a while, and sometimes it just plain made me happy to read someone joyfully nerding out on a favorite subject of theirs.
I think a lot of people are still writing this kind of content, but you have to look elsewhere for it: Reddit, Facebook, Twitter; to name the obvious ones. It’s also harder to find, but you can find all kinds of personal content written in comments and posts on these sites.
I realize that this is a hard thing to 'prove', but I am personally certain that the amount and quality of such things has dropped significantly from a decade ago.
Not to zero. You can still find things tucked away in a post on reddit or the like. Almost never, as far as I have experienced, on Facebook or its ilk, as the affordances are different. I genuinely think there has been a loss.
It used to have positive utility, as before you were acquainting with people you would literally have had nil chance of acquainting with before.
Now?
Nope. Putting anything out there is basically just doing the rest of the world's Open Source Intel for them. Maybe it isn't the Net that changed. It's just there's way more sharks out there that can't just leave well enough alone.
I frequently append site:reddit.com to searches for a niche search term these days. I think a lot of people who would have blogged or commented on blogs are posting there instead.
I wonder if they'll do a walled garden after their IPO. I've always found the site pretty useless, outside the 'old.reddit.com' version. On the bright side, maybe this will open up space for one of the federated clones to grow.
Which ones. They can index their own, but for the others only the public stuff. Facebook has a lot of things private so nobody can see them except your friends. (they are by no means perfect, but a lot of things are private and only seen by friends - most of it isn't of interest to a search engine anyway but comments of the form "I love X product" could in a perfect world be indexed as a sign of what people find good)
> I find that claim surprising considering how many more people there are simply using the internet at all.
Most of these many more people are mobile users, where creating long-style text content can be quite bothersome.
What ain't bothersome, with a smartphone, is taking pictures and videos to slap filters over them, alas that's why we are where we are with TicToc, Instagram and Twitter dominating large parts of the web.
It's even noticeable in a lot of online discussions with text outside of these communities; The average length of forum posts feels like it's gotten way shorter over the decades. People have less attention to read anything that looks longer than a few sentences, often declaring it a "wall of text" based on quantity of text alone.
Imho it's a big part of what drives misinformation; Doing any kind of online research on a small phone screen is extremely bothersome compared to the workspace an actual computer/laptop, particularly with multi-monitor, gives.
There's also the difference in attention; When I sit down at my laptop/desktop, I actively decide to spend and focus my attention on that task and device.
While smartphone usage is mostly dominated by short bursts of "can't do anything else right now", I don't chose to take out my phone and surf the web, it's something I do when I'm stuck in some place with nothing else to do and no access to an actual computer.
But for the majority of web-users [0], that smartphone access to the web is all they know, which then ends up heavily shaping the ways they consume and contribute to it.
> TicToc, Instagram and Twitter dominating large parts of the web
For my part, I'm glad these fora aren't indexed well; I don't want my search results dominated by single-sentence posts and photos. In particular, I don't have accounts on any of these services.
I'd be happy if search engines would decline to index sites behind paywalls. Links to Medium, Substack and Washpo are very common, and if the first thing I see is a popup demand for payment, that browser-tab gets closed.
I wonder if it would be possible to have a big filter button “commercial” or “non-profit” or something along those lines. So you get results that are not deemed commercial or are.
Don’t know how hard it would be to know which is which. Maybe non-commercial : don’t run ads, don’t sell a product or service and provide information only.
The original Google algorithm was a clever hack but it relied on web being a hypertext, and links being used contextually.
The algorithm made linking valueable. So instead of writing hypertext people tried to create isolated sites and boost their rank. Remember rel="nofollow"?
I agree very much with this. It seems that between the walled gardens and also people being so reluctant to have “their” audience leave their site/page/etc the discoverability of the web has dropped dramatically.
That's an interesting observation. IMO, we stopped linking to good content because Google was good at finding it. Now Google is suffering, and we need to go back to doing more links.
Yup, early Google relied on a lot of unpaid , unseen human intervention and choices. I ran some weblinks and curatorial sites during the search wars, and PageRank could only work because there were people behind the sites choosing links based on their usefulness to their audience.
I wish FB would be more open, but since they have all this walled garden info, are they well placed to start a competing search engine? Would be interesting if their activity could help filter out seo hackers.
FB search seems to have gotten worse and worse. Unless I can remember the specific Group where I saw something, it's very unlikely that I can find it again. And they know which posts I've been highly engaged on...
What's happened since then is that almost all the normal "people linking to things they like" has gone behind walled gardens (chiefly Facebook), and vast majority of what remains on the open web are SEO spammers.