Half a petabyte in storage and 3 TB transfer a day? Shit. That's nothing, unless you're "saving money by using the cloud".
While I feel for them, they put themselves in this position. Even when someone gives you something apparently for free, you have to consider what might happen when it disappears, and it seems that nobody thought about it, or if they did, they didn't care.
People often learn how expensive "the cloud" is the hard way.
3 TB a day is roughly 36 MB/s, so yeah it's pretty much doable. 10 Gbps connection is already common, and it's even possible on 1 Gbps connection if people don't mind some saturation during peak time.
> People often learn how expensive "the cloud" is the hard way.
Yeah, people really go a long way around just to get back to on-premise and colocation...
> Yeah, people really go a long way around just to get back to on-premise and colocation.
Outside of pet projects I'm not sure many people are doing that, but happy to read any research. From all I've seen the numbers show an overwhelming move to the cloud, and a few grumpy cats annoyed that the pricing structure is different.
I've read the article from DHH he makes some decent arguments, but as I see it that article is mostly notable because most people don't self-host. At least in my corner of the tech scene (ie EU startups) there are probably hundreds of companies hosting in the cloud for every company choosing to self-host and I haven't seen a shift in that trend yet. If anything it seems to be shifting more towards the cloud.
True, the wake up call is mostly in articles and social medias, not in the trend, in the same way most people use Windows despite most technical writers using Mac or Linux.
Personally I've been self-hosting forever, and that's the only reason I could stay afloat. There is a whole category of business that is simply not profitable if you use the cloud.
I may be wrong, but I feel like “startup” no longer means “bootstrapped for the lowest amount possible” (the model that created sv) and now means “vc-backed for the lowest amount of dev-effort possible”.
I'm curious. What would you do, host on-prem? Even if I had the hardware, as a pure software person, I have no clue how I'd even start setting up a physical location to serve data on that scale. And even if the people on the NixOS team know, it sounds like their time would be better spent working on NixOS than building & maintaining a data center (even if small). Bare metal storage services seem to be on a similar price level as lower-cost cloud services like Cloudflare or Backblaze, but I'm very interested if you happen to know a cheaper alternative!
Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally? Optimally you'd want a distributed CDN, but that doesn't really sound feasible without the cloud. Would you still use a cloud CDN, or just serve directly from your own server?
It is astonishing as to how marketers from the three big clouds have successfully convinced people that there is no middle ground between operating your data center and using one of their cloud services.
Here's what you do:
1. Buy two servers equipped with a large amount of storage. There are many providers[1][2] that offer this. (You could technically do it with one, but two allows you to have high availability and prevent data loss in case of accidents, imagine `rm -rf *`.)
2. Use Minio to set up erasure-coded storage on one of the servers, and replicate objects[3] to the other server.
4. Use Cloudflare (because of their open source sponsorship program[4]) to serve the objects from the primary server, and when performing maintenance tasks, fall back to the secondary.
You could also serve objects directly out of the primary/secondary, but using Cloudflare takes care of low-effort DDoS attacks by hiding the origin, and provides some caching which helps prevent hitting the egress limits of the server provider.
It's not trivial to set up but once you do it's much cheaper than AWS. It's even cheaper if you can find the machines in the server auction. There are some 150 TB machines on there right now for 174 euro/month.
While I totally agree you can easily handle this with Ceph, I would never do it with 2x replication on spinning rust. You can maybe do 2x replication with SSDs if you really know what you're doing and you have data replicated on multiple Ceph clusters.
Also, your math does not quite add up if you want to store 425TiB. You'd be filling up the Ceph cluster to 95% which would be impossible (you'll start losing OSDs way before that).
If this was me, I would definitely do this on Ceph, but I would opt for a more traditional erasure coding setup of 8+3. For that, you'd need 12x SX134s as a minimum starting point.
I suppose you could start with an EC profile of 2+2 or 4+2...
I do have a serious question on Hetzner:
Do they offer any private networking options? That page only lists a single 1 Gbit connection. I don't think that would work well with Ceph. It's doable, sure, but I wouldn't want to support such a cluster, not with 160TB raw per box.
Yes, perfectly doable, but the expertise and time needed to do it (without fatal mistakes) and keep it running 365 days a year is non-negligible. Doing this by unpaid volunteers is more than challenging.
This is, in my opinion, a falacy perpetuated by the marketing departments of "big cloud". Don't forget that pre-2006 there was no S3, and we managed just fine. They are not challenging skills to learn, the zeitgeist has just convinced itself they are not worth learning
It's condescending to think that someone can't reasonably choose to pay for a service they are capable of doing.
Cooking isn't hard either but I still go to restaurants, and don't go around laughing at patrons telling them that they never bothered to learn how to cook if they are there.
Do you eat every meal at a restaurant though? That would not be sustainable. To roll with your analogy it would be like the hospitality sector persuading the majority to ditch home meals entirely
The problem was your assumption which is condescending. I gave you an analogy to make it clearer. Taking the analogy in the second paragraph apart doesn't change the point in the first - it was simply to help illustrate it, but it stands on its own.
It's not condescending at all. You are saying they could reasonably pay for it, but they quite obviously cannot (hence the existence of this whole post). What I'm principally challenging is the assumption that this required effort is substantial. I'm making the point that it is not, certainly (in my view) compared to the effort to run an entire operating system project and all of the ancillaries that come with that. This is an area they have chosen to pay for that (again, in my view) they don't need to. They have gone to great lengths to architect a system that allows deep pinning of very specific versions, it's not a stretch to think about the architecture required to deliver that. Instead they have stopped that line of thinking at [insert S3 here] rather than carry on solving. A direct consequence of that now is a $9k/m hole they need to fill in their budget. I'm being critical, sure, but not condescending. Overall, it frustrates me seeing opinions that operating infrastructure is difficult and therefore must automatically be avoided. In this particular example avoiding operating said infrastructure potentially is putting the project in the red and at risk, but nowhere on the NixOS call for help does it suggest they have considered operating this themselves. And there in is the crux of my point. The zeitgeist has turned against operating infrastructure, and yet is frequently caught out by the cost of this decision.
> They are not challenging skills to learn, the zeitgeist has just convinced itself they are not worth learning
Next time you see someone paying for something you think it's easy to learn, that the "zeitgeist told them not to learn", go up to them and say what I quoted and you'll see real fast if it's condescending or not.
The difference in situations here is literally that on the one hand large corporations have spent money to market the idea that their services are essential because the alternative is incredibly difficult. On the other hand, McDonalds, KFC, Burger King, "Big Restaurant" etc have not spent large quantities of money trying to persuade the populace that cooking at home is incredibly difficult. It's possible to sell someone a convenience without also telling that person that it's the only real option and this is what the person you're replying to is highlighting.
It's not condescending, it's the reality of the situation. Yes in the examples you've described it would be condescending but in the situation we're actually talking about there are actually large numbers of people who genuinely believe (as a result of successful marketing campaigns) that the equivalent to cooking your own meals at home is incredibly dangerous, difficult, expensive and error prone.
> On the other hand, McDonalds, KFC, Burger King, "Big Restaurant" etc have not spent large quantities of money trying to persuade the populace that cooking at home is incredibly difficult
They haven't because it's a ridiculous proposition. The same way cloud providers would be laughed out of the room if they tried to say the same thing and they didn't actually have a better product with trade-offs that are genuinely better for some use cases. The fact that you believe that everyone using cloud products is either too lazy to learn or has been fooled by marketing because they can't think for themselves, unlike you who see the light, is the condescending part. Mind you, on the face of overwhelming data showing most companies have, or are moving large parts of their infrastructure to the cloud, so everyone else are dummies either too lazy or too stupid to see what you can see.
> The fact that you believe that everyone using cloud products is either too lazy to learn or has been fooled by marketing because they can't think for themselves,
I never said anyone was too lazy, just misinformed about the benefits, and it's a more far fetched statement to claim that marketing doesn't work than it is to state that it does work.
As evidenced by some people in this thread claiming that not using S3 would be basically impossible (despite a large number of popular distributions literally doing exactly that) there are clearly plenty of people who have fallen for the marketing.
Moreover, you're the one implying that falling for marketing is something only people who can't think for themselves do, that's also not what I'm saying. I think everyone is susceptible to marketing, and in this case, clearly a lot of intelligent people who are otherwise capable of thinking for themselves have fallen for this blindspot.
The original post did not come off as condescending to me, but instead just highlighted how BigCorp cloud marketing departments have an incentive to sell developers on “infrastructure is tough, just pay us”. (I say this as a consumer of AWS services and fan of their products).
Eating in the restaurant once in a while is fine. Lage range of choices delivered quickly. But is it sustainable? Price higher, usually less healthy and not tailored to special needs.
In any case one does it professional IT done right is costly but cheaper than e.g. IT without offsite backup in case of a disaster.
> And IT departments in some random company were much bigger.
Not sure about that. There appear to be just as many people in IT, it's just that they're not that technical anymore, mostly producing policy documents and pushing tickets around. The actual work is outsourced to the cloud, SaaS products and outside consultants (for duct taping the mishmash of SaaS products together). But that may be just my lacking understanding of my strange little part of the globe.
We do this at work, albeit much larger scale than this and I'm the only one who spends any time on upgrades, replacing failed disks and so on. Honestly, apart from upgrades every 3-ish months I don't spend any time at all on this, it's all chugging along without intervention.
Things don't fail as much as they used to, at least not from using something mature as minio and of course hardware can fail as they do specifically spinning rust, which is non trivial to replace if you have hot-swap capable servers in the first place.
I don't possess any decade worth expertise in either minio or running this kind of setup so personally I believe you are exaggerating that this is even worth calling challenge at all.
So I’m sure you’d be more than happy to volunteer your services to NixOS and be responsible for the setup, including when there are issues (which will inevitably happen regardless of tools and technologies used), right?
I'd volunteer to do it, if they paid for hardware, colo and transportation for initial install. After things settle down, it takes no more than half an hour or so every month.
Pretending to know how much work something is when it's not your area of expertise is a bit silly.
The money that are paid as the exorbitant AWS egress fees could be used to pay a competent SRE to handle that. I don't think that running the package mirror would require a full-time dedicated paid position. Initial setup, maybe some heavy restructuring operations as storage requirements grow, maybe some firefighting, but not the daily operation.
It still happens. You need storage a TB or two with unlimited (or at least very high) traffic? That's not an issue. Even I could help being a mirror from home. But I wouldn't be surprised if there would be some free tiers already from cloud provider.
Half PB is a bit more than that, and I guess it is expected to grow... in 2023 it is not realistic to throw such storage around for free (+mirrors). It will be realistic in a few years considering that the storage prices have been in free fall for the past 30 years, but that time hasn't came yet.
The falling cost of storage is at least partly the reason hosting a few thousand packages takes up hundreds of TB. "Storage is practically free" from the perspective of each of those projects. By the time .5 PB is a trivial cost, they'll need 50 PB.
Yeah NixOS has done some technical choices to require tons of data downloads, even if there is little modification of the content of the packages. Regularly my computer has 10 Gb worth of downloads because some high level package got upgraded and even if it's dynamically linked, the lower level packages still get re-downloaded from cache.nixos.org. It doesn't matter because storage is free right?
You do all of this and you still don't have nearly the availability or durability of data you get from S3, nor people on call to fix it once anything happens to it.
It's like comparing leasing a Mercedes-Benz with full service and all inclusive insurance to building your own bicycle. Some people don't need the car, sure, but it's not the same thing.
But they're complaining about the cost of leasing the Mercedes, so they need to lower their expectations if they want a cheaper solution.
Although having to spend some time to attain good enough availability/durability is not a big deal when the rush to launch was over long ago. Unlike in your example, you can turn the bicycle into a car.
All it needs to do is to get you the data faster than the compilation itself - most of the time - to be useful.
I also wonder at the original thinking, given the use case. Cache usually has high ratio of download to upload and doesn't need high reliability. S3 doesn't excel at the costs of fetching the data from it, at all.
It only seems like that to people who don't know any better.
Do you ACTUALLY think you can communicate with people at Amazon who truly know what's going on? If so, why do we see countless stories about people who have problems with Amazon, Google, et cetera, who can't communicate with them? Do you know of some special, magic way to get the skilled people to talk with you that the rest of us don't?
You act like being able to call Amazon, even if you could get someone on the line who knows something, is somehow better than having even just one good systems administrator. All of your ridiculous assumptions go right out the window when you have a sysadmin build, test and deploy your own hardware that you (via your sysadmin) have complete control over.
Security? You can't secure Amazon. You literally can't. If you wanted to, you'd have to buy the company, then you'd have to evaluate your whole S3 staff. If you hire a systems administrator, you have to vet your systems administrator and anyone else who has admin access, not a collection of tens of thousands of employees across the planet, any of whom could be stupid enough to visit the wrong web site using the wrong browser on the same computer as their keys.
Availability? My side projects have better uptime than Amazon.
Durability? Two geographically disparate locations and LTO, plus many mirrors, make things plenty durable.
When you're not a systems administrator, you can be forgiven for not understanding the big picture, but not for acting like you do.
Yes it is very simple, you pay for support. Maybe I don't know any better, working professionally on top of AWS for almost 10 years and growing usage from a couple of thousands a month to half a million a month (while remaining profitable the whole time). Maybe, just maybe, people disagree with you even if they are informed.
And for the point in question, you don't need to talk to anyone, the point is there's thousands of datacenter, networking, app-level layers of staff all working to make service stable for you without you even knowing. I don't need to talk to anyone if an S3 server, rack, or even full DC goes down.
Honestly, we are an open source project, not a company. If we have the money, of course, we would prefer to spend it there and do better and more with it.
If this is threatening in a serious countenance, our growth and ability to conduct our project. I'm not sure if we need all the 9s of durability of S3, given that people already lost all their buckets on S3.
We do not have the money to store 5x or 6x the contents of the cache.
So we much prefer to have control, and we have enough skills to run all that shit easily, the problem is that the skilled people are in rare availability and are usually working on harder problems than running that. So ultimately, this is a balance problem.
On the technical side, garage (https://garagehq.deuxfleurs.fr/) does multi master replication by default, so is probably better for this use case. Still with S3 API.
If `minio server /mnt/sd[abcd]1` and the occasional `apt-get upgrade` and changeover of DNS records is so much work, maybe you should just spend 100x for S3.
Seriously, there is a large amount of comments of people who seem terrified of this or throw out "but but updates and failing hardware". We run multi PB storage on prem at work and it's just fine. Not some massive team running it.
> Even if I had the hardware, as a pure software person, I have no clue how I'd even start setting up a physical location to serve data on that scale.
3TB/day is about a third of what you can serve out of any run of the mill PC with a gigabit ethernet connection. There are 22TB drives and individual servers with 24 drive bays, which exceeds 500TB. So you could do this with one machine connected to a gigabit fiber connection, and the machine would cost approximately what they're paying monthly (almost all of that for the drives).
In practice a 24-bay chassis is silly and you might use a separate storage array etc., but that's just details.
> Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally?
At 3TB/day you're not Netflix. Just serve it directly from your own server.
It's not silly because it doesn't exist. It's silly because it weighs a hundred pounds and makes design trade offs to achieve a level of density you probably don't even care about if you only need one of them. The people who want that thing are the people who are going to fill a rack with them or more instead of several times that many racks with something else.
In practice if you need ~24 drives you get a couple of 2U array containers that each hold 12 drives and then buy a 2U server or two to plug them into and don't care at all that you used 8U for 24 drives instead of 4U for 72 because your 42U rack still has 34U to spare.
> Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally?
As a simple example 1 of the cheapest hetzner baremetal nodes gives you 500gb nvme or 2tb HDD storage with an unlimited gigabit Connection in Europe. This sums up to i believe 10TB of data per day.
Given the provided numbers, a single 50€ instance could handle the load, even if it's located in Europe. CDNs make sense for systems which are sensitive to high latency, or you're pushing multiple TB per hour, which this just isn't.
I wonder how unlimited such cheap unlimited connections really are. In the simplest case insufficient capacity in the provider network might become a bottleneck. They might notice that you are the noisiest one in the whole neighboorhood and either ask more money or throttle you. Offers that are too good to be true typically don't last very long.
I have quite a few servers with hetzner. I don't know what the exact criteria are but once you hit around 250TB/month per server with multiple servers for (likely) multiple months in a row then they will tell you to chill out or they will terminate. I've gotten it and I've seen a few others on lowendtalk mention it too.
I've been using hetzner for a couple years and have always used them for my high bandwidth services but I have only gotten that notice once despite being a very heavy user, and around the same time everyone else did around a year ago. Maybe they're gonna start to care? With all the price hikes and all. I ended up just getting more servers to level it all out.
Thats true, and I didn't actually want to suggest using the cheapest server available as a singular node either.
I just wanted to put the given numbers into the scale of the cheapest baremetal servers available on the market, to illustrate that this is not an amount of data you'd have to consider for scaling. Realistically speaking, you'd want to have more then 2 nodes so you don't have to worry about rebooting/updates etc
I’ve run some fairly network-heavy services (900mbps+ on a 1gbps connection, 24/7) at both leaseweb and hivelocity; they never complained about the traffic
Actually, for someone building an OS, having to face very practical challenges in the server space would likely be extremely beneficial. Dogfood, people, dogfood.
Surely you know what colocation is, don't you? Suggesting someone has to build a datacenter for it is like saying that programming is time consuming because people have to write their own compilers.
If someone asked me to do this, I'd colocate in two locations (likely Europe and west coast USA) two full copies of everything in 1/4 rack. After physically building, testing, installing, then testing some more, which likely would take a month, it'd be a part time job to monitor and babysit for the first few months of use, then it'd be an hour a month maintenance job.
CDNs cost money and come with tons of problems. For non-interactive static files, they're not necessary.
If it requires constant attention, you're doing it wrong :)
> Optimally you'd want a distributed CDN, but that doesn't really sound feasible without the cloud
CDN is a good choice when you ether need to sever small volume (i. e. not 500Tb) with low latency, e. g. to reduce web page load time by hosting static assets (js/css/img) close to the customer or when you have really huge traffic (e. g. Netflix, but they decided public cloud is not cost effective for CDN and built their own).
For NixOS mirrors latency is not important, traffic is not huge (withing capacity of a single commodity server) and just two mirrors - in US and EU would be close enough to majority of users.
Well, having a $9k per month budget you could hire an entire rack in a premium data center location (let's say Telehouse London-Docklands that hosts London internet exchange - You can connect to it in a very cost effective way). This would still leave 80%~90% of the budget (depending on power use) for hardware/software etc.
But a full rack is a huge overkill. How about a quarter rack for few hundred $ per month?
What about connectivity? Well, as mentioned Telehouse Docklands hosts LINX so you can get an unlimited transfer gigabit right into the Internet backbone for a couple hundred $ (mind I dealt with them few years ago, but I doubt prices changed that much - probably they're lower if anything). Also there is a cluster of data centers in docklands so they compete on price heavily as well as resellers that often have good deals.
This is all just location and connectivity. How about hardware? Let's say they wanted proper support with their hardware not diy so they were looking for a "solution provider". Send a request for quote to 10 of them including fujitsu, ibm, hp etc. Then tell them you're looking for a "charitable organisation discount". Unfortunately we're missing the most important metric to price this "solution". IOPS and if its mostly small or large files (random or sequential Access), but they host most of it in "infrequent access" s3 so let's assume it's not a lot, just to find the lower pricing boundary.
So let's say a pair of 2u servers with two hba cards each and 4 12 disc jbods with 22tb Sas drives a total of a petabyte in raw storage or half a PB mirrored (not that they would run it mirrored). A system like this from a big name (it's rather low performance, but we're after cheap right, their s3 is likely very similar) might cost $80k. Over 5 years of lease that might be ~$3k per month plus their hosting and connectivity it's about half of their $9k.... Then there is software, but that's a separate thing. Also one could DIY such system for half the price, one could add SSDs etc. This is just an example. To design this properly one would need usage data from their s3 and few days of work to put it nicely together.
> mentioned Telehouse Docklands hosts LINX so you can get an unlimited transfer gigabit right into the Internet backbone
First, look at the LINX network map. They are not just in Telehouse Docklands.
Second, with all due respect, do you know anything about how the internet works ? "internet backbone" huh ?
LINX is a peering exchange. A connection to LINX gets you nothing. Sure you can peer with the route servers. But you won't find any Tier 1 routes on the route servers, and the Tier 1s will also refuse to peer with you because you're too small or you don't meet their ultra-restrictive peering policies (for example, good luck getting BT to peer with you !).
Most people are better off getting a good connection to either a Tier 1 or a high-quality Tier 2 network and leaving it at that.
> Also there is a cluster of data centers in docklands so they compete on price heavily
London Docklands is the most expensive place in the UK to colocate, and probably in the Top 5 most expensive globally. Its prime real estate, all of the big-name colo facilities know that and will charge you accordingly, ESPECIALLY Telehouse !
And trust me, you don't want to go anywhere near a "cheap" Docklands facility, they are cheap for a reason.
The days of £300 a month for a full rack in Docklands are long gone my friend.
There are many other perfectly good places both in the Greater London area and elsewhere in the UK and Europe where you will get more bang for your buck.
The Googles of this world only really build their own data centres in the US and maybe, if you're lucky, one or two other places elsewhere.
But generally, everywhere else in the world, and absolutely certainly in London, they're in the same buildings as everyone else ... except they rent entire floors and not just a few racks of course !
I would split it in pieces and create a DIY CDN (like lots of Linux distros have). From the numbers I've seen in the article, the big problem is the storage, not the transfer, although they say that the space can be reduce by garbage collecting the less used caches (although not ideal).
Note that they are using Fastly as CDN in the front, but that's not important for them (monetary wise at least). The big part of the bill is just the storage part.
With little detail about their data I would chose one of these architectures...
Debian (or other distro) mirror style:
- Split the cache storage in pieces (by architecture, release or other categories)...
- Deploy a bunch (10-30) of cheap servers in affordable hosting around the world. The most popular data is distributed in more servers (for example, the x86_64 build caches for the latest version of the packages)
- Get some people to volunteer with small mirrors
CDN style:
- Get a couple or three medium/big sized (in disk, not CPU) servers for storing all the cache, for example, with Minio, so they don't even have to stop using S3 storage libraries.
And then either:
- Keep using Fastly
or
- Distribute tiny servers with Varnish or the like
- Get some people to volunteer tiny edge cache servers
Using the pricing from your other comment and the numbers from the parent, if I estimate correctly, Storj comes in at about $8k/month? That's cheaper than AWS (considering there are no CDN costs), roughly on the same level as Cloudflare and Backblaze, but still more than the entirety of NixOS Foundation's (monthly) incoming funds in 2022.
According to https://discourse.nixos.org/t/nixos-foundations-financial-su..., they have 450TB and 29.5TB of S3 egress/month. By my math, assuming the Nix Foundation goes in through our front door, talks to no sales people, and pays list price, that's $2k/month.
You would replace the Fastly CDN with Storj, correct? Your link mentions 1500TB of Fastly traffic, which would cost $12k with Storj. Not sure if this is related to parent's 3TB/day number, but I went with that when I estimated the $8k.
I think the best cloud solution might be something like Storj + Fastly or Cloudflare end-to-end. But I'm really curious how much could be chopped off if you tried to solve this on-prem.
Storj + Fastly is genuinely a pretty good combo. There are certainly some things Fastly does that we don't do (besides being more specifically a CDN, they also have compute@edge, etc). We have a good relationship with Fastly (e.g. https://docs.fastly.com/en/guides/log-streaming-storj-dcs)
But yeah, I don't know how the Nix Foundation is getting their Fastly traffic paid for, maybe that's in/out of scope. Certainly they should keep Fastly if it makes the overall thing cheaper. Having a distributed origin (us) to dedicated edge cache (Fastly) makes for a strong distribution story.
Its nothing to do with "start-up" or "open-source project".
As I clearly said. Its Dummies 101 ... if "somebody" is giving me money, then only an idiot doesn't consider what happens if that "somebody" stops giving me money.
What you're describing is premature optimisation. They made the right call going with the solution that required the least effort. Now that that solution is no longer viable, they can spend resources addressing it. Not fixing problems that don't exist is the right call.
I see people usually use the term "premature optimisation" to mean "I wouldn't think about this thing, so no one should". Also people really take that one quote and consider it gospel, forgetting that "premature" is contextual and subjective.
Meanwhile I think it's a bit silly that they didn't have a plan B for 425 TiB (!) of data and are scrambling to figure it out now. It's not about "fixing problems that don't exist", it's called "planning ahead for a scenario that is practically guaranteed to happen" and has high risk for the future of the project if they don't have such a plan.
They created essentially a SPOF where "failure" was losing the financial goodwill with some company.
They had assurances that they wouldn't just have it ripped from them, are being given time to find a replacement, and are engaging multiple avenues to do so. They are having absolutely no downtime from this process. You're making assumptions that aren't warranted at all to make a point that really just seems like an excuse for you to be angry at random strangers. I'm honestly not sure why you are so upset (and rude) about this.
I think that is "something" for a huge amount of business operating around the world.
For everyone here saying its trivial and they can do it cheap.
Get together and form a non-profit organization that offers
super cheap infrastructure to the NixOS foundation.
With a 99.99% SLA and around the clock staff ready if something
goes wrong.
Which I think if you just co locate at a datacenter means going
to the physical racks when shit breaks.
(Or pay for "remote hands" and "remote spare parts".
Or just pay for managed hosting.
I think that sounds like a lot of work for volunteers who have
other jobs as wel.
Yep, the thing is they're using Fastly as CDN, moving 1.5PB a month, and its cost doesn't seem to be a problem for them (I don't know if its covered by Fastly or they have a good price). The gross of their problem is the storage, and S3 is probably the most expensive place to put raw storage.
If they start paying the bill, their best option is moving out. Either to a cheaper provider (be it R2, B2 or whatever) or self managing it (Minio...).
And seeing the $32K price for the migration to R2, and guessing that's transfer costs, I would garbage collect and move only hot data and let the rest be rebuilt during some months.
There's still open source hosting providers that could provide the bandwidth and maybe the storage for such a project, but I doubt many of them could provide something like S3, it would either be git, https, ftp or maybe rsync.
Building something like S3 yourself on Swift or Minio isn't easily done, the management of these solution can be tricky, especially for multi-petabyte systems.
S3 most likely got them going and scaling really easily, but then at some point the cost comes back to bit you. Still I don't see any good alternatives right now. I don't completely disagree with your, but the overhead of building an S3 like storage solution upfront would have actively prevented the NixOS project getting off the ground.
> While I feel for them, they put themselves in this position.
I agree entirely.
I feel the same way about OpenBSD when they come round begging for money.
I'm like "sure, great cause, but can we first have chat about your totally energy inefficient hosting of multiple racks full of ancient power-hungry kit in Theo's basement".
Money doesn't grow on trees, and especially in this era of inflation and high energy prices, I'm afraid I don't think it is unreasonable for me to expect the organisation asking me for money to get their affairs in order first.
Well they're hardly going to find a colo that'll take those machines, so you must mean that they should stop building for them?
They've said before they hardware heterogeneity has been useful in bug hunting, and at worst it can be viewed as a heating expense for the fairly prolific guy.
> That's nothing, unless you're "saving money by using the cloud".
I've helped a number of clients save money by using the cloud. It's not that hard when you use the tools for the purposes they are suited for. It's not the hammer's fault that it does a poor job of pounding in that screw. At some point the "craftsman" needs to take the time to learn what their tools do and what they are best used for.
Cloudflyer (https://www.cloudflyer.io/) offers free migrations from S3 to Storj. Storj is $4/TB/mo, $7/TB egress. It's decentralized storage, so the base functionality has CDN-like performance (no Cloudfront needed), and you don't need to pay extra for multiregion redundancy.
I jumped on the Nix Foundation Matrix to try and help them directly, but for anyone reading this thread, Storj might be able to cut a zero off your storage costs. Check us out!
This statement basically makes me unable to take this seriously.
At "Egress Bandwidth $20/TB" it should pay off to buy up all the cheap hetzner servers and convert them to storj nodes. They would pay themselves back after 2 TB and then you are off making profit.
But wait, there is more:
They charge 7$ for egress bandwidth... How can they pay 20$ to the one providing the bandwidth. Oh, right they don't actually pay the operator, they give you a crypto token.
So all in all paying for storj feels like supporting yet another crypto scheme.
Certainly there are differing opinions about cryptocurrency in general.
In terms of "does the math work", you're absolutely right that charging $7/TB and paying $20/TB to our storage node operators is not sustainable. But it shouldn't be a surprise that this was an intentional early subsidy to grow the supply side of the network. In an open system like ours, you see these business dynamics, whereas you might just not see them otherwise.
It it a bit mean to anyone who has invested in equipment to supply to the network and used the subsidised numbers to work out if it will be profitable. This should be clearer on the website.
This seemed interesting, and I had a quick look at a few pages on the Storj website. The website looks like its geared towards the enterprise segment. At the same time, I didn’t see any prominent link to a comparison page. Comparing with Backblaze B2, Cloudflare R2, etc., would be quite useful.
On integrations, I see there’s a tech preview for Restic. Any plans to support Borg? Is there a platform roadmap available publicly? I’m primarily looking as a single user to store larger backups. The Storj pricing (only for storage, not for egress) seems to be a lot lower than that of rsync.net (even with the special HN Reader’s discount).
The platform roadmap is available here: https://github.com/orgs/storj/projects/23. Good question about Borg. We are supported by rclone natively, and I've seen some success with people combing Borg and rclone. But that's certainly something we can look into more.
Very quickly, the roadmap page you linked to seems to be available only for GitHub users (and probably people who are also given permissions on your repo?). I only see a long list of “You can’t see this item” cards.
I really like the storj concept - distributed storage, local encryption.
Comes from tahoe-lafs I believe, except you don't need to find 10 buddies.
There is one main reason I don't use it besides web share:
Despite being OSS, uplink-cli is not available in linux distribution
packages, which
- makes me uneasy about stability/maturity of the platform (e.g. do you not provide packages because you don't have a stable API, and if so, should I really rely on you to keep my data?)
- it increases friction - either build from source or manually copy binaries; and you have need to do this on all your machines and servers; and it means no automatic updates
- binaries/packages without trusted update channels may be against corporate/personal policies, and you usually want your storage accessible on all/most of your computers
- if available in non-distribution package repositories (didn't see in quick-start) - it would still create trust issues
Have you considered getting your cli packages in distributions?
If I could just `apt install uplink-cli` or similar, I would already be using it.
As a customer of Storj, this isn't something you'll need to worry about. We're integrated with Stripe and take credit cards and invoices. We will also take STORJ token if you want.
If you are looking to provide storage space to the network as a storage node operator, then your nearest reputable exchange should do the trick.
I think to serve files over WebTorrent / BitTorrent with a (HTTP) web seed stored across blob stores at Cloudflare R2 / Backblaze B2 / Wasabi is a better model for a distributed CDN.
We're hoping they respond to some of our reach outs for a deeper discussion! But if for some reason you're reading this here, NixOS Foundation folks, my email is in my profile!
One of my colocated servers has 60TB of space to spare and has a 10gbe unmetered uplink on the provider's IP-transit blend.
It is easy to configure traffic shaping on the box so that any unused upload bandwidth can be used without any impact to existing applications.
I would be happy to seed NixOS torrents. There is an extension to torrents where files can be added to a torrent after a torrent is released that might make sense to look at.
With NixOS the main problem is that a lot of local "packages" are configuration files with a lot of secrets. It is very easy to end up seeding your VPN and email passwords this way.
I use agenix when I can, but I'm internally wincing at the thought of trying to explain it to friends I'm getting into nixos. It's not exactly usable.
/nix/store isn't world-readable. It's machine-readable, and most machines are single-user. I'll explain the caveats, but I can't tell them never to put secrets there; the predictable outcome then is they won't try nixos.
There are unfortunately lots of software packages that don't offer a convenient passwordFile option in nix and the easiest way to get running is to specify a password in a way that ends up in the nix store.
I knew of a few options like that, but didn't realize it was so widespread. I wish Nixpkgs had a policy against writing modules that way. This creates a policy nightmare for companies that might use Nix. Alternatively, we need ACLs for the Nix store, or at least some store paths, at a minimum. :-\
The reason usually cited for why not BitTorrent for software packages is because it makes public what package versions you have and haven’t downloaded.
I wonder if it’s possible to easily seed portions/parts of giant files. A little bit like another project mentioned here but just mediated with torrents and volunteers.
Google and some napkin math tell me that this kind of workload should not be on the cloud.
A 4U or 8U rack server with 1 PB of storage and plenty of cores: ~$40K
Colo facilities in the Bay Area: $400-500/mo
Compared against the status quo of $9K/mo, the ROI ends up being 5 months per box, for twice the storage capacity they need. Set up multiple of those boxes around the world to the extent that you care about redundancy/distribution, and they still end up far ahead in a few years time.
I am no fan of the cloud, but with that many disks, failure rate goes up. So you also need someone to go to the DC. All the disks will bought around the same time, which means failures start to cluster up as well.
The machine also needs to be replaced every $interval. After 4 or 5 years, keep it up is more hassle that replacing it.
A single machine patches and will need to reboot. What should happen to people trying to access the store?
Keep replicas help, ensuring a petabyte or even half a petabyte of small files is in sync across multiple sites with various link speeds is not as trivial as rsync.
Most datacenters offer remote hands, so there is no need to travel to the DC for those kinds of things (just ship them the drives and have them replace it).
One can also easily add extra spare hard drives to the pool that ZFS or mdraid can automatically put into production in case of drive failure, so there is no need for active intervention after each drive failure.
And if any downtime for upgrades is unacceptable one can go for distributed storage solutions like ceph, seaweedfs or minio.
Or simply 1:1 mirror everything to a second machine (which you want for disaster recovery anyway) and switch over during the maintenance period.
And since for this specific usecase the server(s) would only serve as CDN origin, a brief downtime might even have very little actual impact.
As is, Nix would have to spend $108k/year to continue to host from S3. For comparison, OpenStreetMap's entire budget for infra/ops is 169,197/year (FY2023).
Of the cloud storages, AWS S3 is the most expensive for both storage and bandwidth.
A half petabyte at other places such as wasabi would cost 35k a year, including bandwidth.
Open street maps by comparison in 196GB compressed including full history. 20 years of full snapshots would amount to only .2PB, or 20k a month to host on cheaper storage providers.
I don't think using OSM's budget was the best example.
Sounds about right. You might possibly be able to bring the server cost down to like $20K if you really turn every stone looking for good offers (for a storage server like this you can probably go with a refurbished machine; the demand on the hardware is really nothing to write home about), but you should be paying hundreds per month, absolutely nowhere near $10k/month.
"We will cover egress and migration fees for customers migrating over 10 TB of data from US, Canada, and Europe regions, and storing it with us for at least 12 months." - https://www.backblaze.com/b2/c2c-migration.html
Also free egress from Backblaze to CDNs (Cloudflare or Fastly).
Hopefully France is successful in suing AWS for their anti competitive Egress fees. Either way we've avoided any potential land mines like this by switching to Cloudflare R2.
Why just AWS, Google leases dark fiber to the other telcos, it basically runs the entire internet. That it charges egress fees at all is kinda nuts. They make 100% profit (more) on your egress from GCP.
> Oh man, there's potentially a $32k migration fee to pay for S3 egress, as well.
I've known about this for a while, but never bothered to ask. Why is that kind of fee even legal? Is there anything pro-social or valuable about that kind of business practice?
Why should it be _illegal_? AWS egress fees are clearly documented, it's not like they are some kind of hidden deceptive fee.
Companies are free to impose whatever fees they want. You can simply just choose to not have business relationships with them. Monopolies complicate the question, but AWS is not a monopoly.
Because it may be viewed as an anti-competitive business practice intended to prevent customers from switching providers.
Yes, AWS’s customers knew what they were signing up for, but AWS’s competitors and potential competitors have a right to enter the market and compete with AWS, and they can’t do that effectively if AWS is hoarding all the customers with punitive switching fees.
Sounds like a libertarian trope. Their house, their rules.
AWS and all the clouds are too big. They structure prices to milk the most of their captured customers. They all price their products within parity, so on the whole there isn't a huge difference in moving. This also makes a nice natural churn between them. The clouds themselves offer almost no innovation in the computing space.
You know who the real rubes are? Everyone else, chip manufacturers, storage, etc. They all made commodity things using shared protocols and people are just pissing it away. And this sht replacing everything is worse than the stuff getting recycled, hardware, ideas, all of it.
> Sounds like a libertarian trope. Their house, their rules.
Well, yes. I support regulation when it makes sense, but this situation isn't it.
> They structure prices to milk the most of their captured customers. They all price their products within parity, so on the whole there isn't a huge difference in moving.
You've got to make your money somehow. And expense structures for various large providers are pretty similar, there's no shady conspiracy here.
> Companies are free to impose whatever fees they want. You can simply just choose to not have business relationships with them.
Maybe in libertarian paradise, but this line gets parroted a lot for something so patently untrue. Even in the US there are plenty of laws that rightfully limit the freedom of vendors for the benefit of consumers.
You maybe conflating limited circumstances of enforcement and other countries such as those in Europe with ideals. If it's a US company, then there is little meaningful regulation. "Laissez-faire" and "the invisible hand" are the prime directives.
Why would that be illegal? That’s just normal egress fee x GB stored. This is known in advance so anyone storing so much data in cloud is willingly putting themselves in this position
The big clouds - AWS, GCP and Azure - all have utterly ridiculous egress fees compared to every non-"cloud" hosting service and compared to what ISPs charge for these kinds of connections. For example, AWS charges $90/TB while a typical price is more like $1/TB.
It's almost certainly done on purpose to discourage you from switching away, and in the hopes that you won't notice until you deleted all other copies of your data.
The question is why egress costs this much in the first place though. All cloud providers pay for the aggregate bandwidth, but its cost seems substantially lower than what they charge for the traffic volume:
Also to note that on many smaller VPS providers customers rightfully pay for the bandwidth, not total traffic. So this charging-by-traffic appears to have no other explanation than predatory practices.
In case I wasn't clear, my argument is that egress is most expensive for someone trying to migrate away from the system (assuming a big dataset with relatively low traffic). When comparison shopping at the beginning of a project, people often neglect the cost of someday wanting to switch. So this can be hidden vendor lock-in.
because in peering agreements it is convention to only charge for only ingress or egress, whichever one is bigger. Most AWS servers use more egress bandwidth than ingress bandwidth so AWS only charges for egress.
I know we want to jump to hating aws but is there any info on the breakdown of this?
Like who is charging the amount? Is it all aws or is there a cloudflare portion? Part of it is paying an engineer's time? How much data is being transferred?
If it was pure egress fees then it's roughly ~350tb of data?
I don’t know much about NixOS but I imagine the data in those buckets exists elsewhere? There could be a community effort to push data to the new provider and avoid the egress fee (or some of it)?
The only thing I wonder is how much space it would use if the nar files were stored without .xz and in a store that does smart deduplication. Because of the way Nix works, there's going to be a lot of e.g. ELF files that are nearly identical except for the rpath.
See this post in the original discussion regarding flokli's work on tvix-store, which provides a completely different storage model, while still being able to provide NAR compatibility by producing them on demand:
The tvix project itself is quite fascinating and worth a look. It reminds me a bit of the Gentoo times when Paludis came around and blew Portage's performance out of the water - but tvix seems to care more about compatibility with the existing ecosystem instead of being a competitor.
tvix-store is neat and I look forward to what it might bring in the future (along with the rest of tvix) However there are many deduplicating storage systems that exist right now. Switching to one of them could potentially be an intermediate solution, though compression would have to be solved; perhaps a low -memory version of zstd for compressing in transit would make sense.
It might be worth a try to run Caddy, which supports zstd out of the box (the only webserver which does so to my knowledge), in front of an uncompressed store to see whether on-the-fly encoding/compression is feasible.
I've never heard of LogicBlox but they were sponsoring $9000 a month? That's crazy, normally even big sponsors will give a once-off $5000. Well, well done to them
The author and lead maintainer of Nix used to work there for some years, as well. Maybe he pushed for it, or maybe they saw both things as similar kinds of investments in their stack.
That seems like the perfect use case for IPFS. I can even bet that if the people at the NixOS foundation played their cards right, they could get the Filecoin bag holders to pay for the development of this new feature.
Unfortunately it currently isn't, because nix packages are not content addressed. They are "input addressed", which means that when requesting the contents of a derivation, you have to trust the source to give you the right bytes.
There are efforts underway to make things content addressed. This doesn't solve the trust problem -- you still have to trust someone to tell you which inputs map to which content address -- but it does help with the distribution problem. Mirrors no longer have to be trusted. Trust and distribution become decoupled.
If you went "meta-nix" and had a nix package hierarchy that was just constructed out of operations and inline literal binary blocks, then the results of those purely functional operations (splice, insert, f(n), swap, etc) could be content-addressed and deduplicated.
Views of purely functional data transformations of f_tran(n)(addr_start:addr_end), but if you symbolize it you, you make it generic. The problem is that Nix isn't Haskell enough.
It also sounds like if they stored the cache on a deduping file system that their storage costs would plummet.
> If you went "meta-nix" and had a nix package hierarchy that was just constructed out of operations and inline literal binary blocks, then the results of those purely functional operations (splice, insert, f(n), swap, etc) could be content-addressed and deduplicated.
How do you build with an arbitrary build tool then? You have to call out to gcc/any other compiler in the process.
IPFS in not magic you can just sprinkle on things, say “p2p! decentralised!” and be done.
you still need either pay someone to “pin it”, which is more expensive than dumping it on s3. Pinning services (that’s a thing) are all sketchy (as the big ones are all somehow connected to NFTs) and if you do it yourself it’s actually harder than just storing it
As a NixOS user (who doesn't even use the binaries), I would happily pin NixOS packages!
The point of IPFS is to decouple the names of resources from what's serving them. It's exactly what is needed for distributed libre software projects where many are willing to donate some resources, but few are able to donate all the resources.
I had missed that the storage fees were significantly larger than the traffic fees.
I still don't know if Tahoe is appropriate though. It's been a while since I looked at it, but doesn't it work with a fixed set of storage nodes? That wouldn't work for nodes coming/going where people can join organically.
There has been work ongoing to support this in Nix for a long time, actually. Maybe this will help push that effort along, or direct more resources towards it.
Yeah, it doesn't make sense for the team to keep hosting every single state of the distro, just to boost building some random dudes' old/peculiar setup. It's almost like Nix team is providing an artifact hosting service. Nix team should expire old versions from their repository, and let people pin what they need on IPFS.
Can someone please enlighten my uninformed self why NixOS needs to store this mindboggling amount of data and why it would be damaging to the research community specifically if it were to be lost? 9000$ a month is a LOT.
Nix works quite differently compared to other distributions. If you have eg. the hello package in Debian it specifies that it is dependent on the glibc C library with some version range. You can install another version of glibc and the package manager will consider hello's glibc dependency satisfied as long as it is within the version range.
In Nix on the other hand, a particular hello package/binary will be dependent on the glibc that it is built with. That doesn't mean just the same glibc version, even if you'd change something like glibc's build flags it'd be considered a different version [1]. One of the benefits is reproducibility. If you build hello from the same nixpkgs revision as I do (barring any impurities), you'll have exactly the same hello with the same transitive dependencies.
Now, with hello this may not matter as much, because it's a simple program and glibc is fairly stable. But now imagine that you are doing some machine learning project and you want to provide a reproducible machine learning experiment. Building on a typical packaging ecosystem (eg. pip) is like building on quicksand. There are all kinds of subtle behaviors that can change and influence your experiment (eg. you machine learning library rewrites some compute kernels, and the small numerical differences pile up after a large number of layers). Though usually it ends at: my system has Python 3.11 and this experiment requires an old PyTorch version that was only built for Python 3.5 and earlier. The Nix approach gives you the benefit of pinning almost the whole world - your experiment and all its transitive dependencies.
Suppose you have a package repository like nixpkgs or Debian that contains tens of thousands of packages that are continuously updated. In Debian, if glibc is updated, it gets rebuilt and you are done. glibc has a stable ABI, so packages that were built against the previous glibc version don't need to be rebuilt. In nixpkgs if glibc is updated, every package that uses glibc needs to be rebuilt due to Nix' use of exact dependencies. And this is true for almost all non-leaf packages --- a package update requires all packages that are (transitively) dependent on that package to be rebuilt. And you need to store all these builds, so that users can download them, otherwise their machines need to rebuild the packages on every update. This is how you end up with a gigantic binary cache.
[1] This is a bit different now that Nix supports content-addressing for non-fixed output derivations.
What's the compressibility of this? As a whole it looks highly compressible but with something like s3 and everything being its own file, it may not be getting utilized.
sure, but at 425TB i'm guessing they're storing either everything in full or everything in full and uncompressed.
and compression can do a lot for text files like sources etc. for example (last time i did that) the linux kernel source code was about 1.2GB uncompressed, but ~60-70MB in tar+gz format.
> I guess that reflects the fact that debian is a distro from another time, when you had to be mindful of the resources you consumed.
I am inclined to argue that the fact NixOS is having this problem and discussion goes to show that you *still* have to be mindful of the resources you consume :)
While public cloud resources may seem infinite to some, at least the depth of the pockets of those shouldering the resulting costs should not.
“That would incur a fixed cost of around $32k for the migration.”
They should consider Backblaze b2 if they want s3 api compatibility. Backblaze currently offers to cover the migration costs assuming some spend commitment.
As one of the TVL folks, we aim for cache compatibility (i.e. output path equality), so the NixOS cache itself is highly relevant to us, too.
Would we choose to host it on S3? Probably not, but that's where it is right now and we have to be realistic about whether or not that will change. Personally I don't think so, though even on S3 there are still some optimisations to be done.
Just to clarify, the missing component in my mind is in mirror advertisement. The need for manual compilation and review is a hindrance to casual participants, we need a config option that you just toggle (maybe a configuration struct with eg. bandwidth limits) to opt-in to IPFS or GNUnet distribution of store items present on your system.
It does. Nix has an extremely powerful (and flexible) caching system. Not to mention popular caching systems like cachix that are easy to integrate into your system.
Unlike most other packaging systems though, Nix packages are rebuilt when there's changes to any of their dependencies. This results in a huge amount of storage data used in their caches compared to other systems. Not to mention the compute power required to rebuild those packages.
For example, let's say something like `glibc` updated with a new patch. Every single package that has `glibc` as a dependency needs to be rebuilt, along with every package that has any of those packages as a dependency. This could easily cause the rebuild of tens of thousands of packages, even if they are only dynamically linking to it.
OpenSuse tumbleweed also has to do this. I don't know what happens with NixOS in such cases but with tw we have to download and install all those new packages too. There has been 2 times in the last year we had to download 2-3 TB of packages.
While we're talking about opensuse, perhaps there's a way for them to exist on the opensuse build system?
Guix's graft system is a hack specifically made to get security updates out ASAP. They should really only be seen as a last resort because it circumvents the primary benefit of using Guix as a functional build system (pure, reproducible builds). I'm not sure if Nix has a similar system.
IMHO I'd prefer to see compile-time caches like ccache or sccache be better integrated into Nix while still providing the guarantees that a functional build system provides. That alone would save massive amounts of compute power when rebuilding packages, as you'd essentially only need to recompile the files that change in a package. Not only on continuous build systems like Nix's Hydra, but also when just rebuilding packages locally. This won't solve the storage cost issue, but would definitely alleviate compute costs.
It's possible to bring something like `ccache` into your system's Nix sandbox. However, that change in the sandbox means all your packages have different hashes and you lose the ability to pull from upstream caches. The result would be rebuilding every single package when trying to update your system. Not a fun time.
The preferred way to do this is to override `stdenv` with `ccacheStdenv` on the individual packages that would actually benefit from the compile-time caching. Such as for large packages that you already have overrides for, or for nightly packages that get updated extremely frequently and don't have the chance to be cached upstream (such as firefox-nightly). The NixOS Wiki has a pretty good article on this: https://nixos.wiki/wiki/CCache
I personally use this for any packages I override in an overlay to reduce build times significantly.
This seems to be about a cache of compiled stuff where it'll cost $32k to move the data out of S3. So... rebuild the cache somewhere else and drop the original data?
I think this is technically an option, but IIUC it'll have plenty of its own hurdles. The cache is populated by the results of builds run by NixOS's hydra build farm over a period of many years, and hydra stays busy enough that it's hard to imagine this approach being viable without standing up a separate cluster of build machines dedicated to the task. To complete it in a meaningful timeframe, the cluster might also need to be quite a bit bigger.
(NixOS hydra obviously has some spare capacity or it'd never catch up, but I would be surprised if it has enough idle capacity to rebuild everything over any meaningful timeframe without also significantly affecting velocity on everything else.)
Calling on the community to dedicate temporary build resources might work--but the systems need to be trusted, so it can't quite be Folding@home...
I guess if there's a bit of Pareto in the distribution of build output sizes, there might be meaningful savings in rebuilding the packages that take up the most storage and paying to move the small stuff. It might also be possible to triage the sources in the cache--re-fetch all that are still available from the original source or a mirror, and then just export the ones that either couldn't be fetched or could be fetched but the contents no longer match the hash.
Probably does take a lot of compute to recreate the cache, but doing it in order of requests works pretty well as a heuristic for most important first, gradually backfilling when idle. Adding machines to the build infra seems like less of a loss than paying egress fees too.
Overall seems plausible that this data store isn't just a cache of things that can be recreated at will. The sibling comment about source being unavailable (or still available but at a different location) for the earlier builds is interesting in that respect.
There has been a controversy around the server cache.nixos.org. The signing ed25519 cache key is from the year 2015 and there is no plan to rotate the key. It is apparent that a key cannot hold for all these years.
I have been made aware of the matter when somebody on the matrix claimed that there is no plan to rotate the cache key. He/She said reasons behind that are explained by Eelco D. in https://www.youtube.com/watch?v=yXPQdZh8o7c. Unfortunately, I couldn't confirm this. That person afterwards claimed that the video is cut and that part was removed intentionally.
If that's the case, it might be the reason why ppl stop paying for the cache server costs. Furthermore, if the foundation attempted to hide the facts by cutting the video and re-uploading, for me, this is a matter of dishonesty and somebody should take the responsibility for it. I believe this person is Ron E., the current director of the Nixos Foundation. But in any case, this claim should be backed by finding the original stream of the Nix 20th Community Panel recording.
For the rest of us, until some cryptogeek fixes nix to support certs and key lifecycles, the solution is to build a local cache with hydra and flakes. There are many ppl keen on the subject, you can look for one on the matrix.
Just order 25x BX41 Boxes with 500 TB for 1015€. They do not support S3 protocol, but have plenty of other protocol options (FTP, SFTP, HTTPS, rsync, mount as network drive, ...).
While I feel for them, they put themselves in this position. Even when someone gives you something apparently for free, you have to consider what might happen when it disappears, and it seems that nobody thought about it, or if they did, they didn't care.
People often learn how expensive "the cloud" is the hard way.