From my own adventures in colocation, i came to conclude that data centers are Narnia, in the worst possible way: you enter to perform what feels like 30 minutes worth of work, and somehow 8 hours pass before you see the outside world again.
I am simultaneously impressed and depressed by data centers. I still get the same frisson I used to get when I was 20, when I enter the space, with all the security theater, and the noise, and the blinkenlights.
What depresses me now though is thinking of all the energy that it takes to power and cool a DC, and for what? So some sloppy code can hog the CPU 100x more than it needs to? So we can train AI models because everyone else is doing so?
I feel its high time we had a green software revolution, where we strive for code efficiency not just as an intellectual exercise but because it might help save the planet. And save the bottom line of companies that are paying to run their stuff in these environments.
I'm glad someone else feels the same way, it can be kind of isolating to want to be efficient at the expense of speed. There are important features to be shitting out, dont-cha-know.
It feels the same way a neglected pothole does: the city doesn't want to send a crew out for half a day (uh oh it might cost a couple thousand bucks!) to fix a pothole that five thousand cars each and every day are running over, accumulating wear and tear, discomfort to drivers and riders, day after day, week after week. No itty-bitty investment to fix the problem that might benefit so many others, similar to how we could invest a little bit of time (city money) to fix a little inefficiency here and there (that dastardly pothole). It just makes too much sense.
Some of them are very human accommodating, and seem to understand that today's engineers might be tomorrows customers.
Others seem to go out of their way to create anti human environments. Hot aisle is literally 38cm wide? Doors wont open? Lifting tools are in an undocumented location? Rack uprights randomly unscrewed? Specific permission required to open a roller door so you have to wait 3 hours for an escalation request to be processed before you can reach the guy on the other side. 24/7 access conditional on the 1 security guard for 3 data centres being available to let you in.
My favorite was the one that had declared bankruptcy with our hardware still inside. No staff(despite 24/7 staff guarantee), but we had some limited after hours access pass.
Between the pass, some keys we "found" + friendly landlord disabling the alarm system for us, we got in, saw fans operating in place of the aircon, dismantled our rack and bailed.
>Northern Virginia — home to the largest concentration of data centers in the world
Beside other factors, it seems there is decent concentration of nuclear power plants in the area.
And it seems we'll see even more close ties between those technologies - Oracle talks nuclear reactors as a part of the future data centers, and MS partnered to start a Three Mile reactor specifically for MS datacenters.
I guess I only use the cloud for my email. No social media either. The climate aspect of these data centers frightens me that I took all my stuff off the cloud and store it locally. Even my email I store locally with thunderbird.
RE climate argument, your marginal impact on the environment is likely going to be worse with storing stuff locally vs. in the cloud. Even including the extra security/maintenance work that you don't need locally, the data center is still going to have more efficient hardware that's managed better, and gets all kinds of economies of scale, including efficiency (cooling) and utilization (via virtualization, if you're just using a service or VM and not colocating).
I say that as someone who prefers local-first and avoid cloud dependencies like a plague.
My photos are on my internal SSD drive. I don’t know how who having them on a cloud server is better ecologically. I don’t access my old photos really, sort of like a photo album under the coffee table.
Taking less or no photos is kind of my thing now anyway.
So you're decreasing your ecological impact by simply not doing what other people do.
In other words it's not that you're local vs cloud. You're using local-S3-glacier and other people are using cloud-S3 + cloud-EC2.
re: the climate aspect, it's actually better to have some services like email operating in a data center. Think about all the idle no-op cycles your home email server is burning through. A million individual email servers will consume orders of magnitude more electricity than those same million+ accounts hosted in a data center. Efficiencies of scale.
I searched and cannot find the answer in their archives: has the Washington Post, or any other nationally prominent newspaper, ever written coverage from this standpoint, but about an oil refinery?
One thing that could explain this dichotomy is that writing such an article for oil refinery would be expensive task. Because data for such research may not be readily available online (not focussing on intentions). Just a guess. You will find more articles about things that are
easy to write about. If an article requires lot of field work, on foot research, talking to many people and synthesize and piece together disparate information in a coherent manner, don't expect it to be commissioned.
I at least expected a description of the conditions inside data centers that portrayals in movies in TV shows always seem to get wrong: how _loud_ and _cold_ they are.
Honestly, the Internet as a decentralized global network of computers suggests otherwise. There really shouldn't be any, or very few, datacenters, people should host their own stuff from their own machines, from their own homes.
Unfortunately, culture and technology took a different road.
It might be hard to look beyond status-quo, but it USED to be easy to _SEND_EMAIL_ from your own machine into the world, sure it came with problems, but they could be overcome.
It USED to be easy to host a website on your home computer, simply ask for a static ip, and forward a port or two on your router.
All of this was taken away from us in the holy name of convenience (and corporate greed).
It's really an economy of scale thing, and is not that surprising.
The economy of scale is separately at the level of hardware procurement, security, interop, bandwidth, access to electricity, and probably other things.
This isn't a normative claim. Simply that the tendency for these types of things to centralize as complexity and demand increases is natural.
I think it's tough to say that it was taken away, or that there's anybody in particular to blame.
The hardware market moves fast, the capabilities needed evolve fast, etc.
It's still 100% possible to host your own stuff. Many people do, but it's obviously a vast minority. But it's not like there are licensure requirements to host your own website. Some things are disallowed by consumer ISPs, but you can also generally upgrade that allow it.
It's worth noting that when it was simple and unrestricted, it was also vastly more expensive and far fewer people participated. Part of the reason why internet is so cheap now (on a $ / byte basis) is that new offerings were designed that restricted people on the now available cheap plans from doing things that are very expensive.
I think I disagree… desktops are pretty inexpensive in the grand scheme of things. I mean most of us could host our stuff on a raspberry pi or something like that. It isn’t like we’d individually need to handle a ton of traffic.
Rather I think the problem is that most server (software) was pretty poorly written, to the point where it requires professional administration and maintainence. Although, this did create a lot of IT jobs. I guess there isn’t anyone, really incentivized to architect things better.
You could run your blog off your machine at home, much like you (hopefully) run your home automation from a machine at home, not a cloud instance.
Serving your collection of videos from home is a bit ore taxing, should anything become popular. And if you produce popular videos, you want to handle subscriptions and comments, accept payments, fend off spam, and weather the occasional DoS attack. Things quickly snowball into a full-time sysop job.
This is to say nothing of businesses that need hundreds of boxes. Putting them all in the office building means you have really hard time moving, e.g. because you grow and need more space, even as we ignore the question of redundant power and internet links in the building.
Datacenters most honestly deserve their place. They make life easier for many people, for many reasons. They are not the only thing that should exist, but very certainly they are not a mistake or aberration.
People don't do this for reliability reasons. All you need is a static IP, a machine at home and a internet connection. It's fine if you don't care about uptime, but people aren't running businesses that way for a reason.
> It's still 100% possible to host your own stuff.
It's absolutely not.
A minority of people have direct access to a network that will let other people reach any service they want to host. You need to carefully select your location to get that. And a lot of people don't even have direct access to a public address.
And that's not even going into the monopolized federated protocols like email, where you can only do anything if big-tech allows.
>All of this was taken away from us in the holy name of convenience (and corporate greed)
Corporate greed will serve up whatever people will pay for. I (and nearly 100% of people) am willing to pay for the convenience of not running my own email service, so the greedy corporations which sold that solution succeeded while others failed.
Also think of Capex vs Opex. I don't want to drop $4k on an at-home PC to play with new AI toys, but I am willing to spend $20 per month paying for ChatGPT and Claude AI access.
Me browsing a website for my local pizza joint should be a connection from my pc, through a couple of switches/routers, to the PC in their office.
As it stands now I have to route out of town to the nearest datacenter hosting their web platform. Then I need to make more requests out of town to every third party analytic, font, and framework their platform includes. Not to mention the associated energy cost of running all of that code.
Every one of those requests triggers a cascade of dozens of database updates, replications, and algorithm updates across many geographically diverse datacenters.
(Yes I am aware that 'distributed' means many different things depending on the context and is largely a meaningless word)
Restaurants are a good example of where this breaks down, though, I think.
There are cash-only bars with mechanical registers, and food trucks with no electric hookups, and small third-party kitchens in bars and corner stores that all have websites or at least social media profiles with hours, menu, location, etc.
These businesses have no interest in running a web server. They may not even have internet access onsite beyond employee cellphones.
Exactly right. No one is bemoaning the lack of restauranteurs constructing their own buildings or farming their own food. Why do we want them to host their own website?
You're absolutely right, and yet the romantic in me still thinks there is hope.
I have an excellent example where some friends used to run a wifi mesh in my city. They handed out mesh nodes to small business owners like restaurants and corner shops. These devices were tiny, and only required power since they were mesh nodes.
The point here is that these nodes lasted for years. Sometimes we went around and restarted, or updated firmware, but by and large they worked so well that they were forgotten by the business owner.
This was 10 years ago, today a local web server, or similar device, might work in a similar way.
One idea would be to integrate it into their internet package. They get a router that gives their shop wifi, at the same time it also runs their ordering system.
I think it's possible, anything is possible if you're motivated, but large businesses don't like independence. They prefer that small business and end users are dependent on them.
> Me browsing a website for my local pizza joint should be a connection from my pc, through a couple of switches/routers, to the PC in their office.
I don't think you have a very good picture of the energy costs involved. Half of the energy costs in the total worldwide collection of information systems are in wireless links at the edges of the network (mobile networks and wifi). So if you traverse two wireless links on the way to your pizza joint it is likely that you have caused the worst-case energy usage. An optical link to your nearest cloud datacenter is going to need less energy.
Not to mention the now-always-on nature of the pizza place's PC, which can't be as efficient as a multi-tenant cloud system, has terrible AC/DC conversion losses, spectacularly bad HVAC inefficiency, etc.
Except not every pizza joint can afford their own network engineer, IT Security officer, hardware guy. Small firms that try to roll their own often fail because these things become neglected.
They already franchise in so that the franchiser can perform these central tasks in a more efficient manner.
The franchise looking at the costs of on prem vs hosted vs colo vs cloud will probably find that on prem is the least efficient way they can go about it.
IIRC direct sending of mail wasn't removed in the name of convenience, it was removed in the name of "99% of incoming mail is junk, can we permanently reject the worst offenders?"
What I would like from the cloud is absolutely positively guaranteed lifetime backups of everything I deem important enough, and heavily encrypted to boot. Everything else cloudish is... extremely optional.
I like to see articles about the hidden systems that make so much of the modern world work, I think it's a positive thing to allow common people to have that perspective if they're interested. However, when I do see stories about datacenters and the like, I just really sort of despair, to be perfectly honest.
I see how little the industry has changed and innovated since the 'modern' datacenter crawled out from the legacy telephone switching centers. You still see the marks (read: scars) left on the old in the new: same sized racks in width, asinine cold and hot aisles shuffling hot air around blown about by fan after fan after fan screaming away endlessly until they throw their bearings. The energy moved about among so many different interfaces, air/water/air/refrigerant/water/air and around and around and back again. It's the same feeling as railroad gauges echoing the ancient wheeled carriage ruts in the hard dirt because we can't be arsed to make anything better.
You'd think that with all of the developments of omg distributed computing that every tech dweeb loves to bandy about that they're experts in that there would be a stampede to, you know, actually make use of some new technology, moving the data and compute closer to the user (and don't talk to me about edge facilities, they're just more of the same), reduce the blast radius of the idiocy of centralization, pare down the eye watering waste of facilities like this. But no, it's still client-server everything, maybe put a copy in another region if you're feeling kinky, but the promise of decentralization and distribution hasn't really materialized. Such a large percentage of people now have phenomenal power they carry with them at all times, but they're just dumbed-down gateways to these huge black holes of resources because the control needs to remain held in the same hands as the capital that continues to build them larger and larger.
I guess what I'm trying to say is that in an ideal world that uses the technology that it develops in a sane way, these places really shouldn't exist anymore, like another comment suggested. The fundamentals are all there, it's just the incentives are so out of whack it's comical. Build bigger and bigger facilities and collect more and more data and fritter away cycle after cycle and maybe you'll increase your year-over-year by a couple dozen basis points. And since the demand is so insatiably fueled by this feedback, nothing will change. You could have minuscule neighborhood nodes scattered around a city that sips resources in comparison that does all of this in concert with the portable supercomputers we all carry, each benefiting a small handful of people at once but the beast must. be. fed.
