Ask HN: Why isn't Google indexing information about the AT Protocol?

[iameli]

I've been observing this for a while, where very basic queries about atproto stuff doesn't show up. But yesterday I found a query that makes it VERY obvious: "list of public atproto relays". Here's DuckDuckGo:

1. https://firehose.directory

2. https://atproto.at/relays

3. https://atproto.wiki/en/wiki/reference/core-architecture/relay

4. https://pulsar.feeds.blue

5. https://leaflet.pub/12022731-ae4f-4a13-9f7a-5738b7a83c2e

Of those results, Google only has 3, the only one on the list that... doesn't have a list of public atproto relays. None of the other sites are present anywhere. Trying not to assume malice instead of incompetence here, but it's really ironic that one of the ecosystems with the strongest ties to the open web, with users creating dozens of new websites every week, isn't getting indexed.

[entropyie]

Maybe they should have used a name that wasn't already taken...

https://en.wikipedia.org/wiki/Hayes_AT_command_set

[UI_at_80x24 → entropyie]

Lol, that's what I thought this was about too.

Why don't these people/companies think about how hard it will be to search for their $thing !?

[wolvoleo → UI_at_80x24]

Yeah like /e/ os. That's hard to search for because the slash is ignored and the e is only one letter.

[flexagoon → wolvoleo]

All the results for "e os" are about /e/OS, and that's its full name, so you wouldn't just search for "e". D language is a better example — its just called "D", but searching for that doesn't give results about the language. On the other hand, for "C" and "R" the top results are all about the languages. "Go" is also a pretty bad one, they insist it's not called "golang" but you basically have to search for that to get any useful information.

[wolvoleo → flexagoon]

Yeah but when e/os first changed their name it was much harder. I have a feeling these results have had some human help.

It used to be called eelo but some prick thought it sounded too similar to his one man consulting company and threatened to sue. In the beginning it worked much better to search for the old name.

[wmf]

Google search has generally rotted. Can't get promoted for maintenance.

[therobots927 → wmf]

That’s best case scenario. Worst case is that they’re censoring the internet on a massive scale but have rolled it out slowly enough that no one noticed

[alightsoul → therobots927]

Here's a way to fix that: post the info you're looking for on reddit. Google will index that instead. Happened here: https://www.reddit.com/r/LocalLLaMA/comments/1umokol

You could post links instead of the actual info to avoid centralizing info on reddit

[gnz11 → therobots927]

No, they want you to search multiple times to show you more ads.

[bilbo-b-baggins → therobots927]

They are. There’s been a bunch of studies already showing how Google skews its results to align with oligarch interests.

[iameli]

I'm now getting _this HN article_ as the top Google result for "list of public atproto relays". Weird.

Yes, I "used an incognito window". Yes, I checked on multiple devices. Yes, Google is personalized. Maybe if you search "list of public atproto relays" from East Timor you get lists of public atproto relays. But you sure don't from Seattle right now.

[Suppafly → iameli]

>I'm now getting _this HN article_ as the top Google result for "list of public atproto relays". Weird.

I find that happening all the time when I google stuff discussed in reddit threads, the top result is often the thread I just came from, despite whatever quote or lyric or whatever having existed for years outside of reddit.

[alightsoul → iameli]

Yes happened on reddit too here: https://www.reddit.com/r/LocalLLaMA/comments/1umokol

[EnglishMobster]

Can confirm that I can also reproduce this. Interesting.

[alightsoul → EnglishMobster]

Here's a way to fix that: post the info you're looking for on reddit. Google will index that instead. Happened here: https://www.reddit.com/r/LocalLLaMA/comments/1umokol You could post links instead of the actual info to avoid centralizing info on reddit

[alightsoul]

Here's a way to fix that: post the info you're looking for on reddit. Google will index that instead. Happened here: https://www.reddit.com/r/LocalLLaMA/comments/1umokol/meituan... You could post links instead of the actual info to avoid centralizing info on reddit

[runningmike]

I got 2300 results. DuckDuckGo Uses google. Google is very personalised. They use many dirty tricks to give personalised results. Use another box and check the results again.

[Peanuts99 → runningmike]

Doesn't DDG use Bing?

[wolvoleo → Peanuts99]

Yes it does.

[iameli → runningmike]

Are any of those results a list of public atproto relays?

[runningmike → iameli]

Yes.

[iameli → runningmike]

Could you link it? Curious to see how buried it is for me. Personalization is one thing but on my results I'm getting Tor relays, Nostr relays, IPFS relays, and Fediverse relays and I haven't yet seen a list of atproto relays. I'm on like page 6 now.

[ThePowerOfFuet → iameli]

I'm not the same commenter, but here's a Kagi share link so you can explore their results:

https://kagi.com/search?q=list+of+public+atproto+relays&r=no...

[danabramov → ThePowerOfFuet]

Kagi's are good, yes. The issue is with Google's.

[PaiDxng]

That result set is pretty damning: Google finds the architecture explainer but misses four pages that actually answer the query. Whatever the cause, atproto clearly has a discoverability problem.

[whateverboat]

https://imgur.com/a/OS7P9jh Seems working for me.

[iameli → whateverboat]

In the AI overview yeah, but do you see the sites below? I wonder if it's being tuned to prefer generating its own "list of X" implementations and downranking other peoples' lists.

[fragmede]

https://firehose.directory/robots.txt isn't one, but comes up in Google despite that.

https://atproto.at/robots.txt is good and findable with a direct search

https://atproto.wiki/robots.txt is empty but findable on Google.

https://pulsar.feeds.blue/robots.txt is 404 and thus not indexed by Google.

https://leaflet.pub/robots.txt is good. A logged in google search for this finds this for me.

Tell the people at https://pulsar.feeds.blue to fix their robots.txt if they want to get indexed.

[seanhunter]

Use incognito/private browsing mode to get a genuine sense of whether something isn’t in the index or whether google decides that even though you’re asking for it you probably don’t want it.

[danabramov → seanhunter]

I can reproduce the same in incognito mode.

[ChrisArchitect]

Spare us the conspiracy; atproto.at? firehose.directory? Why would google index any of those sites with no authority/age? Maybe the owners didn't submit them to give them some credibility to start. Maybe the results vary.

[iameli → ChrisArchitect]

Google has indexed both of them, actually, you can search successfully for those two domains directly. They just don't come up when you search for "list of public atproto relays". Any other theories?

[inigyou → ChrisArchitect]

"Credibility" for Google seems to mean being around for 5 years or having over 10 million dollars. They no longer try to surface the long tail of websites.