Hacker News new | past | comments | ask | show | jobs | submit login

Any decentralized system that is more efficient if centralized, eventually gets centralized by efficiencies of scale.

Email is federated, but everyone running and managing their own email servers is too costly, so for consumers, it migrated to large sites. Web sites were federated, but everyone being web master for their homepage was too difficult, so we get GeoCities, and Yahoo Clubs, and later, Squarespace and Wix, etc. Then blogs were decentralized, but everyone self hosting and authoring blogs was too much, so then we get Wordpress.com and Tumblr, and Twitter, and FB, etc. (even USENET eventually developed super-large hubs like uunet)

For a federated, decentralized system to work and resist centralization, it has to be the case that running a node is dead simple, cheap, and out-of-sight/out-of-mind. It also can't be the case that hosting on a more powerful cluster, colocated with other nodes, gives you large benefits or cost advantages, otherwise, it'll just get centralized again.

Even cryptocurrencies fail this. They have terrible efficiency, but at least they were supposed to be relatively flat, instead of centralized and hierarchical, but instead, a majority of the hashing power is owned by a few large entities, so in effect, back to large financial players controlling much of the power.

I think one day we'll discover some ways to decentralize things in ways that resist re-centralization, but in the mean time, beliefs that you'll achieve cyber/crypto-anarchy by clever protocol design and the federales won't be able to rubber-hose-cryptanalyze you, is a dangerous belief that diverts us away from demanding the government and society agree to the goals of freedom. If everyone wants unfreedom, underground internet usage is a slim consolation.




> Email is federated, but everyone running and managing their own email servers is too costly, so for consumers, it migrated to large sites.

I think this is what federation looks like when it works. Yeah, everyone uses Gmail, but it's also ok to use another service. It still works and delivers a pretty similar experience. Your messages make it out of the Gmail garden.

> Web sites were federated, but everyone being web master for their homepage was too difficult, so we get GeoCities, and Yahoo Clubs, and later, Squarespace and Wix, etc. Then blogs were decentralized, but everyone self hosting and authoring blogs was too much, so then we get Wordpress.com and Tumblr, and Twitter, and FB, etc. (even USENET eventually developed super-large hubs like uunet)

Geocities was one of many roughly equivalent services. It was sort of a social network, but it was mostly just free hosting. Anyone could still link to anywhere and being on geocities wasn't a requirement for being on the web.

Squarespace is a pretty good product and not what I would consider harmul centralization, because it's still interoperable with other websites and they don't prevent people from hosting elsewhere. The web is still federated.

Something different has happened with social media. Different social media services aren't interoperable. I can't message you on Twitter from Facebook, or tweet at you on LinkedIn. You must stay in their walled garden to interact with their users. They're at such a critical mass that almost everyone is in their garden.

It seems like federation is a happy middle ground between decentralization and centralized walled gardens.

Maybe something like Mastodon will some day become so popular that Twitter is forced to become part of the federation and compete on features and marketing instead of vendor lock-in.

With a federated version of any type of social media clone, eventually new entrants will prefer to join the federation instead of trying to make a new walled garden. It wasn't really an option for Hotmail to only let you interact with other Hotmail users, because the federation was already established.

There was tremendous demand for social media services, but no federation to provide those services when the demand hit and explosive growth happened, so now we have a centralization problem.


It’s becoming less OK to use a small service. Gmail and office365 will often classify mail from small provider as spam, with no feedback to the sender.

I was using s small British provider, and almost by accident discovered at some point in time that emails I sent to gmail addresses in the prior 2 months had been marked as spam, unless they were a reply something sent to me (so it wasn’t true that everything got lost - only stuff I originated).

I gave up and moved to FastMail because I don’t have the time to deal with this (and solving it once is no guarantee it won’t happen again)

So, email is federated but effectively not decentralized - unless you use one of the central services, you are at risk from being obliviously blackholed at any second (that happens to large providers too, but there’s safety in numbers - it will be discovered sooner by someone)


> I was using s small British provider, and almost by accident discovered at some point in time that emails I sent to gmail addresses in the prior 2 months had been marked as spam, unless they were a reply something sent to me (so it wasn’t true that everything got lost - only stuff I originated).

Looks like Google will have to keep the 5bn "fines for stupid things we do to EU citizens and companies" line on their budget for years to come.

I don't think this will fly when the relevant authorities finds out (a nice mix of protecting local business and punishing an annoying company that have ignored the rules too many times).

And frankly I think that is great!

Hoping Google will sooner or later return to their roots as Internet superhero.

Meanwhile get ready for more fines, I'm especially looking forward to the almost-destroying-browser-competition-on-desktop fine that I've been waiting for a while. : )


> I don't think this will fly when the relevant authorities finds out

I've seen public entities switch to Office 365 in order to "get rid of spam" and thereby losing a lot of incoming legitimate mails in the same process. A fact that is worsened by users using the "Mark as spam" button to delete legitimate emails from their inbox (a practice that is horribly common).

Going from that to billion-EUR fines to Google and Microsoft is a very, very long way.


I get spam from legitimate companies, probably because they don't validate the email addresses their customers give them, and thus incorrectly assume that I have a business relationship with them. I do not, thus it's spam.

Companies need to validate email addresses.

I get AT&T bills, OnStar notifications, etc. I have never used any of their services. Someone else probably signed up with my email address.

Additionally, other companies are violating the CAN SPAM Act by hiding the unsubscribe ability behind a log in. Nextdoor does this.


I don't know a single person that uses the mark as spam option to "delete legitimate emails". In fact, almost everybody just leaves those emails in their inbox after reading them. A small fraction of users use the archive function. I do however know lots of people (me being one of them) who use mark as spam for unsolicited emails. And no, spam is not legitimate, no matter how hard online marketing people are trying to make it so.


You should try working at an abuse desk for an business providing email services at some point. :)

You are, of course, absolutely right re. spam and rightfully so. On the other hand, I regularly get spam reports re. 100% legitimate mails, often in the middle of longer email conversations and so on, and where one side of the conversation marks entire email threads as spam.

We always follow up on spam complaints and the one thing we do get often in cases like that is that the other party only wanted to delete the emails and didn't realise there were other consequences.

Yes, there's a lot of people that knows how to use the "Mark as spam button". I still, however, maintain the position that a lot of other people don't know how to use such features.


If you automatically get the newsletter of a company where you have signed up on their web service, you would mark that as spam?

I would unsubscribe and delete, but have not really thought about marking as spam or not yet. Just interested in some input there.

If I would get it from a company where I have not signed up the answer is of course obvious.


In theory, I agree with the parent post and the sibling reply. If I didn't ask for email, then it doesn't belong in my inbox. In practice, I do what you do - unsubscribe and delete. If I get any more mail after that...

When I get the old "you'll be removed from our list within 10 days" message, I generally reply to it with a quick message: "Please remove me immediately or your next message will be marked as spam". Many times the sender has complied with a personal response provided someone saw my message, even for some large companies with "no-reply" emails.

For smaller companies - especially the mom and pop type operations - I'll usually give them a friendlier notice. "Hi, I unsubscribed and I'm still getting emails. Please remove me as I'd really rather not mark your messages as spam". I usually get a friendly personal response with apologies and a removal.

In all cases, after the company has been warned, I don't think twice about sending them to spam.


Yes, I would. Buying a product off you (or signing up for your free/promotional service) doesn’t mean you get to spam me, IMO.


I thought this too while using a small provider, as my messages were frequently not delivered to Gmail or Office365.

Then I discovered my provider didn't set up SPF and DKIM correctly so emails were failing authentication.

There are many factors affecting email delivery beyond SPF and DKIM, and it can be difficult for a small provider to get everything right.

So if you're having delivery issues I'd start with that, not by assuming without evidence that the big guys are intentionally classifying spam just because it's from a small provider.


This.

Been running my own mail for a long time, and with one big exception[1] and occasionally spotting misconfigured systems, any time I've had delivery trouble it has been my fault.

If someone is bouncing you, make sure your ducks are in a row. One easy thing to do that won't catch everything but will help is to send mail to a Spamassassin-protected account, and look at how it classifies things.

[1] The Deathstar has been bouncing me for a long time, and their postmaster appears to be /dev/null. Given that I've had exactly one person using them I wanted to mail, they can bite me.


Yes, this.

Except sometimes, like when Comcast looks at your DMARC policy that specifically says "p=none" and decides that means "reject all mail from this domain". Gotta love those guys.

Edit: s/spf/DMARC


Even if you setup SPF and DKIM, you still run into issues when your domain does not resolve to the same IP as your mailserver with Gmail.


That's what SPF does.

It allows the owner of a domain to specify which mail servers are authorized to send mail from the domain.


Did you ask the provider about it, and if so, what was their reply?

Being small may not have been their problem. I run a single-user server and haven't had any trouble with delivery to Gmail.


I did, they had no idea why, they weren’t on any Blacklist, and google did not respond to their or my inquiries.

It was only google, only non-response emails, so it is a google specific thing and not general IP reputation.

Switching to fastmail, otherwise exactly the same (domain, equiv dkim and spf setup) solved everything.

Sure looks like I was on the scoring edge, with fastmail being better (or some special trust arrangement) than my previous provider. Which is fine if I had gotten feedback and had a way to resolve this - but practically I didn’t.

Edit: I had a catchall on that domain forward to an account on gmail. It was suggested to me that may have been the problem. But i still do from fastmail, and everything is fine, so it’s more likely that overall reputation or special fastmail arrangements were the issue. Either way, i’m far from being alone with this issue - as evidenced even i this subthread. It is federated but not practically open to everyone, which is an implicit attribute most people associate with decentralization.


> I was using s small British provider, and almost by accident discovered at some point in time that emails I sent to gmail addresses in the prior 2 months had been marked as spam, unless they were a reply something sent to me (so it wasn’t true that everything got lost - only stuff I originated).

Same here. Perhaps I should switch to a larger provider too.


Report it with relevant authorities.

This is anti-competitive behaviour.

Especially clear since, as GP points out mail gets through if it is a direct reponse, meaning they knew very well it was a legitimate address.

This is abuse of bulk mail filter to squeeze out local competitors.


That sounds like a stretch even by EU standards when a acting openly envious. Google can warn or ban their misbehaving users for repeated spam violations. They can't ban people on other people's domains. "Legitimate addresses" can still send spam and it could have been caused by ignorant users who think "junk" is for non-immediate deletion and use it to keep their inbox tidy. In fact they /did/ deliver it but prefiltered.

Even if they could kangaroo court it for standard business practices they wouldn't since they don't want to be known as "the dumb fuckers who banned domain based spam filtering" and face pressure from annoyed constituents now receiving piles of spam.


> "Legitimate addresses" can still send spam

It seems clear from the description above that poster is someone recipient wants mail from.

As for the ignorant users using spam button I've seen that once since I started working.


> Report it with relevant authorities.

Happy to do so, but which are the relevant authorities?


I have lived in Norway more or less my entire adult life so I don't really know about UK but here it is konkurransetilsynet.

I tried searching for it though and this seems relevant:

https://www.gov.uk/government/organisations/competition-and-...


You local antitrust regulator.


I've encountered services I can't even register to since I have my own domain name, so I had to register a gmail.


If you can list them we can consider what to do about them.


(Edit: just realised GP was probably not referring to email providers, but I’ll leave my question here with apologies)

Microsoft, Apple. I’d move my mail to one of these to get away from Google but I can’t find a way to do so. I already pay for Office 365 and iCloud but I don’t get to bring my own domain for email. Unless I’m missing something.

I don’t pay Google for my email on my own domain because I got in early enough on Google Apps for your Domain or whatever it was called.

I’m happy to pay a reasonable price - I just don’t know whether it’s possible.


Office365 only offers "use your own domain" for business type accounts. $8.25/month. https://products.office.com/en-us/compare-all-microsoft-offi...

Fastmail is a relatively cheap option. Yandex mail will do it for free.


I like the idea of IPFS. I also came from Yahoo and the GeoCities and YahooClubs remained autonomous communities for the most part. Y!'s model was to integrate but not to fully take over the communities but let them reside where they were. That model changed over the years as monetization became a problem.

I've also tried Mastadon but their interface leaves something to be desired. The experience is not there yet.

I do agree on interoperability -- and while Zuckerberg is doing this with Whatsapp and Messenger, we should really take this across most messaging app.

I fear for increasing centralization ONLY because they have the resources and money to do this. This will not change unless we develop a consortium that has the numbers to make it work.


>I think this is what federation looks like when it works. Yeah, everyone uses Gmail, but it's also ok to use another service. It still works and delivers a pretty similar experience. Your messages make it out of the Gmail garden.

That's not true "federation" the way the OP meant. What's happened with email is that it's turned into an oligopoly: you can have email from Gmail, Apple, Yahoo (if you're elderly), or Microsoft, and that's it, unless you have a corporate address for work. If you want to run your own personal email server, forget it: you'll be considered a spammer and your emails will go into the bit-bucket.


> I can't message you on Twitter from Facebook, or tweet at you on LinkedIn. You must stay in their walled garden

Speaking of Facebook and whatsapp, a while ago I read that you couldn't post links to some other social networks. Is it true and if so which sites are affected?


Pavel Durov (founder of Telegram) once reported about Telegram links being censored on Whatsapp.

https://twitter.com/durov/status/671357796730834945?lang=en


> For a federated, decentralized system to work and resist centralization, it has to be the case that running a node is dead simple, cheap, and out-of-sight/out-of-mind.

Imagine your ISP bundling a full node together with your modem etc. That would be a start.

> I think one day we'll discover some ways to decentralize things in ways that resist re-centralization

Centralization unfortunately is still at the core, but we should strive to add more nodes so we can achieve more distribution. The more distributed the system, the less corruptible it would be.

As much as I hate censorship and blocking of content, I believe there is a place for it, for example, you don't want kids seeing pornography and the likes but I do hate it when they do it for political reasons, blocking campaigns or even conspiracy theories, I believe the mind should be able to explore and be critical of everything, this means being able to see/watch everything and analyze.


good decentralized architecture is pretty well undermined by ISPs capping upstream bandwidth. you already can't host a popular website from home - your upstream bandwidth can't handle the traffic


Another problem of decentralized is search, even if the web remained decentralized in hypothetical world, each one hosts their own web-site being their own web-master and blog-hoster, searching in the decentralized-web would still be centralized. Why tho? If there was many decentralized-search-engines (YaCy is one or some such name), someone would still make a meta-engine which would be the one people use, since each one wouldnt want to lose out connections/indexes of any other search-engine, again leading to centralization due to cost and scale as you say.


We laud Google for being our goto for everything, but it surprises me that search hasn't fragmented into specialised fields. Building a Google just for farming for example, seems like it could be better at serving that niche than gigantic Google, and people definitely have enough headspace for a few more websites if they provide value.


Well yeah, but existence of Google and similar warps people's usage of the web. Search is not just typing something into search field. You can search by surfing, by going through a link dictionary, by going through links on websites.

People say, that most people are connected by i don't know how many direct relationships (6?). The same may have been true for web pages.

When the search engines were much worse, people were using webrings, pingbacks, whatever - I remember going through lists of links at the end of paper magazines. Web was optimized for this kind of searching. It was called surfing for a reason.

Search engines like google hugely distorted the incentives on the web. In the name of seo and adsense people did crazy things to the web (as an ecosystem). There has been an entire cottage industry of individuals creating websites specialized just for getting pagerank for other websites, or for capturing audiences via other tricks (mirroring other websites or public/national databases), or for making the website a soup of boldened keywords.

Anyway, there are specialized searches all around, if you look. Product searches, torrent searches, Shodan, whois, transportation, pretty much anything that's annoying to search for in Google has a better alternative somewhere.


Google has one killer feature: It (barely, but) understands. It can find sentences that use different structure but meaning is similar. It can find result using synonyms instead of the original words. And so on. Google also understands simple plain language questions ("how do i do ... on ford mondeo mk4" works perfectly - but only Google can do that). It is even able to detect irrelevant results - e.g. it really filters just MK4 mondeos when I am looking for something, but I also had MK3 and putting MK3 into the query really did filter MK4 results away. I never had such success and productivity with any other search engine, and I really tried to like DuckDuckGo.


> It (barely, but) understands. It can find sentences that use different structure but meaning is similar.

Can't say I see meaningful difference between DDG and Google on this anymore.

DDG keeps getting better and Google has been on a downward spiral for some time.

> It can find result using synonyms instead of the original words.

DDG does this as well. And unlike Google they aren't driving me crazy by ignoring my double quotes and verbatim searches.


I have to agree with the other commenter, Google understands semantics, DDG just replaces words, working even worse than without it in some cases. I seriously hope they will improve it soon (and I donated some money to them), I'd love to switch.


i'm sorry but ddg fails very rapidly on anything that requires semantics


> Email is federated, but everyone running and managing their own email servers is too costly, so for consumers, it migrated to large sites.

I'm pretty sure the vast majority of email servers are corporate servers, not in FANG datacenters.


Don't count the servers. Count the people. Only people who work in tertiary industries tend to have corporate email accounts, and they rarely contain personal data, only data owned by their employers. Many businesses have switched to FAANG services, though, because having an IT department is costly for businesses, and has largely gone out of fashion.


I'm pretty sure the vast majority of email servers are corporate servers, not in FANG datacenters.

3-5 years ago I probably would have agreed with you. Today I'm not so sure. And even among the companies running their 'own' email servers, the physical hardware is very likely in a Microsoft/Amazon/Google datacenter.


"I'm pretty sure the vast majority of email servers are corporate servers"

Many are moving to Office365, and to a lesser degree, Gmail.

Microsoft is making it financially difficult to justify running your own Exchange on-prem.


The way to decentralise without the risk of inherent centralisation is to go p2p. If everyone installing an app runs their own instance and doesn’t need to pick a server, then the efficiency gains of a server drop off massively.

This is why we’re hoping to steer Matrix from being decentralised-at-the-servers to being decentralised-at-the-clients.


There is a major hurdle in getting people to install their own servers though.

We've seen this with OpenBazaar. It makes p2p pretty simple out of the box but if it's not in browser or a mobile app many won't even try it once.


Something like Sandstorm[0] might be able to solve this, since then a person would hypothetically only need a single server (with a nice web interface) that they install all their decentralized apps into. In practice I think modifying the apps to be compatible is too much overhead. But I think something in the same vein could work.

I'm sure @kentonv has some ideas on what the main roadblocks to adoption are.

[0] https://sandstorm.io/


Cryptocurrencies are p2p yet centralized, bittorrent is p2p, yet a few search-engines and trackers compared to clients.


>yet a few search-enginescompared to clients

Bittorrent doesn't have a search engine, so of course they are not decentralized.

> yet a few trackers compared to clients

Well, not if you count the DHT (the decentralized tracker).


well, you don’t gave to have a tracker to seed p2p, and they don’t have to cause centralisation. search engines can and should also be decentralised :)


Is there any write-up regarding Matrix plans on this topic, please? Genuinely curious.


Seconded, having the full Matrix way of architecting data synchronization available from any endpoint is something that can truly change the field.


A swarm of insect-like drones maintaining a mesh network for client devices to tap into, and tree-shaped data storage towers connected with each other via fiber-optic roots (or a fungus-like substrate [0]) in a global network with a standard protocol for everyone to query/read/write.

A worldwide Bionet, built from an amalgamation of semi-self-sustaining, solar-powered organics, electronics and EM/radio.

[0] http://www.bbc.com/earth/story/20141114-the-biggest-organism...



> Email is federated, but everyone running and managing their own email servers is too costly, so for consumers, it migrated to large sites.

Well, federation is the sweet spot between distribution and centralization. I've run my own email service for quite some years now, it is just fine.

A lot of people have bad experiences running their own mail senders though, because they do not go through the checklists and deliverability tests to make sure things come through. Google's SMTP service will only accept mail over TLS, most good email services will reject mail from servers which do not have a matching rDNS on their IP, most will reject senders who have not set up SPF and DKIM, some will reject those which do not have a DMARC policy (including a reporting address).

If you use any good email deliverability testing tool, and iron these issues out in your configuration, you can fairly reliably deliver emails.


I don't see any decentralized services making a dent until the residential internet infrastructure improves by an order of magnitude. There is such a huge disparity between upload and download bandwidth that I fail to see any immediate future for decentralized or p2p replacements for things.


I suspect one thing that would help that is if upload bandwidth intensive activity grew in popularity. Also a matter of who they want to reach and why for demand. In terms of use cases decentralization is currently still in Cypherpunks (doing it for its own sake) and "connoisseurs" (doing it because what they want isn't available otherwise regardless if it is accessing paid content but bypassing geofences, infringing downloads, or banned content). I would like to see decentralization be viable as well but it is fairly niche and a solution looking for a problem.

Currently media is downloading more than uploading even for video games. Client trust issues (and not allowing arbitrary cheating) encourage centralization to at least domain servers for the sake of validation and finding other players.

I think something ARGy like say Pokemon Go but strategic (you have fixed resource pools and open knowledge of everything thus cheaters can instantly be flagged as not playing the same rules). But that would also be niche.

One thing that really could use decentralization is IoT (along with actually making it worth useful and security).

Of course the whole domain is sadly a hot mess of greed often about gathering as much data from their customer, screwing them over with needless connectivity requirements and considering actual functionality an afterthought and security an afterthought to that and then wondering why it isn't catching on. That is a whole other rant.


> There is such a huge disparity between upload and download bandwidth that I fail to see any immediate future for decentralized or p2p replacements for things.

I believe there are valid technical reasons for this, at least for coax connections; fiber not-so-much. At the same time, I get what you are saying.

...and this is how they want it. In short, they keep the "last mile" from allowing true "peering".

The best you can do - and this is to mainly get around the TOS statements that (technically) prevent you from running servers (as well as physically blocking you from running anything on port 80 and 25 mainly, plus probably others) - is to purchase a business plan instead of a consumer plan.

Though I am sure they will try to (or do) block this if the installation address is for a home and not an office (then again, what about small home offices and such?).

I don't know a good way around any of these issues, as it seems all high-speed providers do similar things (most are aimed at prevention of spam and other similar reasons). Even the WISPs I've seen in my area have weird terms (one seems really great for the rural area they serve - until you come to the clause about not being allow to run BT because of "multiple streams"; I tend to wonder if they only allow a single user to use a browser - on a single page - or what).


And ipv6 deployment without NAT.


Good point. But one way the internet is being decentralized is along national boundaries.

The chinese, russians, south koreans, japanese and even the EU are now looking to protect it's turf.

Wouldn't shock me if the internet truly becomes a network of national intranets. I suspect if the EU or any major player truly turns their nation into an intranet, it will accelerate the process with every nation controlling their own network and protecting their own interests, companies and information.

Though highyl unlikely, if china ever becomes a dominant player on the internet, I could even see us closing ourselves off to protect our internet.

I guess this is the price of success. It attracts political attention.


Sounds like nobody wants to do sysadmin work. If spinning your email server or web server would be as easy as opening a google or yahoo account then federetion would be a lot more affordable.

But not even professional sysadmins want to admin and maintain servers (at least for personal use).

Why several companies are trying to build self-driving cars but so few are building servers that you spin up and they're secure and work? We need "self-driving computers".


This is a good insight and generally true.

> Any decentralized system that is more efficient if centralized, eventually gets centralized by efficiencies of scale.

I’d add “or by network effects” since these aren’t “efficienciences of scale” (related to increased marginal value not decreased marginal cost) and are clearly at play with some (many!) centralized systems.


I wonder if we could build a decentralized system of a 6th dimension?

Based on the US military doctrine, we've got 5 dimensions: air, sea, land, space, and cyber. There are 4 dimensions designed by nature and 1 dimension designed by humans. And we're able to engage differently (e.g., at worst, becoming 'domains of warfare') in each of these dimensions.

Wouldn't it be neat to create a completely new 6th dimension, which is independent of the 5 we have, with distinctive features for its system? I'm curious if the brain-computer interface, like what Elon Musk is doing with Neuralink, could be that new dimension?


Exactly this. However, it is still possible to create a virtual decentralized network on top of a federated/centralized network.


as there been any research about waves or oscillations between centralization and decentralization ?

computer started as one processor and distributed terminals, then cpu spread to terminals, then back to server/clients, then p2p. Lastly consoles are threatened by centralized gaming datacenters ..


One of the best comments I’ve seen lately. Congratulations, and keep up!




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: