Is it very important that they have someone to restore service at 3am? This isn't a hospital here; it's more like a library. My library is closed at 3am every day!
> Is it very important that they have someone to restore service at 3am?
Depends on the service, but any effort to maintain this daft "gosh, what are they worried about, it's just the internet archive" line of thought is going to depend on either trivializing the needs of their staff, their users, or their computer systems. Good on them for taking things more seriously.
(3:00 AM on the west coast is work hours for some of their volunteers, users, and contributors across the globe)
> My library is closed at 3am every day!
A library will have IT staff on call. Staff need their computers to work when they come in in the morning, users need the infrastructure working in order to do pretty much anything - and why the hell shouldn't people be able to use an online borrowing service in the middle of the night?
They can use an online service in the middle of the night, but as a citizen, I'd be upset if I learned my public library were paging someone in the middle of the night if it weren't working. That's incredibly inappropriate. It can wait.
Like I said in another comment, computers generally don't just randomly stop working, but no, a library worker doesn't need their computer at 9am so badly that they need an on-call IT worker (and presumably some kind of middle-of-the-night monitoring) to ensure it before they get in. Patrons don't have some incredibly urgent need to borrow a book such that we need 5 nines of uptime here. Or any SLA at all. It's fixed when it's fixed.
This attitude is how you waste a ton of money overcomplicating everything while making working conditions terrible for literally no reason. We should not "take seriously" things which are not serious just for fun. Nothing bad happens if the library computer is down for a couple hours beyond maybe some mild disappointment, or perhaps manually writing the data on cards like we did 30 years ago for a bit.
Hell, most banks in the US are "closed" for maintenance online for several hours every weekend (or at least they were as of a couple years ago) and no one cares. It's fine.
I work on systems that are in the critical path of 24/7 manufacturing lines, so I'm familiar with things that actually do rapidly cause problems if there's an outage. But that's not every computer system in the world.
And if they have people across the globe, then problem solved; they don't need a 3am on-call.
> a library worker doesn't need their computer at 9am so badly that they need an on-call IT worker (and presumably some kind of middle-of-the-night monitoring) to ensure it before they get in.
A single worker probably doesn't, yes. In contrast, hundreds of people being unable to do anything when they come to work in the morning is not a terrible reason to call someone and have them login for a few minutes.
Back to the IA, it seems to me our hundreds of thousands of users, employees and volunteers in time zones across the globe will be unable to access the service until we fix this is a decent reason for an institution that takes its own mission seriously to have an on-call. But you do you.
> Hell, most banks in the US are "closed" for maintenance online for several hours every weekend (or at least they were as of a couple years ago) and no one cares. It's fine.
(this is my industry) Some of their online services are closed, yes. The hours they'll be closed are typically posted ahead of time to minimize the inconvenience to everyone concerned and to display an understanding on the part of the institution that downtime is taken seriously and to signal it is not the result of carelessness. In contrast to online services, most institutions consider it a major failure if something like the ATM system is down for more than a few minutes.
> This attitude is how you waste a ton of money overcomplicating everything while making working conditions terrible for literally no reason.
That's remarkably backwards. The need for humans to be on-call is a result of systems being overcomplicated, not a cause.
If it's only a few minutes, you could just as well fix it at the start of the day. And if it's not, oh well? People can still read books. The librarian can still help them to find a book. If they want to manually track until the computer is back up, they could even still do checkouts.
If those users just say "oh, okay, I'll check back in a couple hours", then okay. Again, this is not hundreds of thousands of people without power or water or something. It's viewing or uploading to an archive. Like, if you're a user, do you really want someone woken up at 3am because you can't use the wayback machine right now? Because "they should take it seriously"? There's a person actually having to deal with that because... you really want to get that one Grateful Dead concert video from their free archive right now? I'm happy to say that yeah I hope their engineers aren't on-call after hours. I would much prefer that they get a good night's rest rather than fix it for me. I hope they're a cool place to work without delusional, psychopathic management.
Perhaps put another way, is there any computer service now that doesn't need a guarantee? Or is everything just critical now?
My point isn't that on-call is overcomplicating, it's that inventing and chasing an SLA often leads to expensive complicated systems (which, ironically, can often directly lead to lower uptime), and on-call has a human cost.
At this point I think you’re just committed to the bit but you cannot possibly believe what you’re writing, as it’s a blinkered analysis and truly misanthropic in its effect. A metropolitan library system employs hundreds of people in a variety of roles and if we can prevent them from having a frustrating and wasteful start to their day, it’s worth an on-call person’s time.
> and on-call has a human cost.
In the context of this thread, we’re talking about this job description:
Senior Datacenter Network Infrastructure Engineer Full Time San Francisco, CA, US 30+ days ago Requisition ID: XXXX Salary Range: $140,000.00 To $200,000.00 Annually
It’s not like we’re talking about indentured servitude or something.
> Perhaps put another way, is there any computer service now that doesn't need a guarantee? Or is everything just critical now?
That’s an interesting strawman you have there. I thought I killed it upthread when I wrote “Depends on the service.”
No, I quite strongly believe it actually. I likewise find it hard to believe you're serious, so I'll raise my question again: if not even a municipal library (which, I'll remind you, aren't even open the majority of the time and we all get by just fine), what is an example of a non-critical service in your mind? Maybe the online checkin for a specific Great Clips? It depends on what? Pointing out that "it depends on the service", and that a library clearly isn't critical was my whole original point.
Misanthropic to me is making every IT job require getting up in the middle of the night to save a 100*epsilon of theoretical frustration lest we be perceived as lacking the requisite seriousness. And when someone points out that actually a given case is really not a big deal, we're met with "but hundreds of thousands of people might think 'oh, darn' for a moment!" And for a library, IME librarians tend to be really chill, so I suspect they'd, like me, attest that actually they'd prefer you not work at midnight on their account. Like I said, as a citizen funding and using my library, I sure hope they don't treat their employees that way.
> a municipal library (which, I'll remind you, aren't even open the majority of the time and we all get by just fine),
At least half of a modern library's lending happens digitally. But let's put aside the example of the library.
> what is an example of a non-critical service in your mind?
I thought the example above of hundreds of people being unable to start their work day until someone solves an especially serious IT problem was universal, not unique to a library system's network. It's relevant to the Internet Archive, and even relevant to...
> Maybe the online checkin for a specific Great Clips?
Maybe not? Their entire WAN being down, affecting dozens of stores, possibly yes?
I'm not sure what's difficult here. Paying a person to be available to deal with a problem serious enough to justify the call makes sense. Sending trivial, less serious problems that can easily wait until business hours their way at night makes less sense.
Taking those calls isn't necessarily a bad job, but somehow that assumption is baked into what you're saying.
> Misanthropic to me is making every IT job require getting up in the middle of the night
This is really interesting to me. I wonder how many more hours of downtime The Archive would have experienced in 2025 if no one was ever woken up out of bed to fix it. (If anyone was)
I also wonder the cost differential.
Got me thinking about these incredible beautiful museums that are closed all night long… you’d still want someone there early enough to catch if the roof leaked so nobody missed their opening-hour tours. Back to libraries, bummer if I can’t renew my book online one night but would set reminder to try again later. Could affect my perception of my library’s tech prowess! That’s a risk for TIA! (Though I experience many a delay and downtime… perhaps my fault for using iCloud Private Relay like I bet scrapers sometimes abuse… but a few more hours overnight here and there wouldn’t be catastrophic for my current perception.)
Mainly would want overnight support to be cheap since stuff’s donation run. - Love TIA! An internet-billionaire should make history becoming a billionaire donor today (hope one’s reading this :) ).
World class art museums have some staff on site 24/7, FWIW.
> I wonder how many more hours of downtime The Archive would have experienced in 2025 if no one was ever woken up out of bed to fix it. (If anyone was)
Wasn't 2025 the year a kooky pro-Palestinian hacker collective decided IA was a good target for their idiocy? There was a great deal of overtime involved there.
Archive is likely one of the top traffic destinations on the internet. Your local library is likely not.
Maybe they want to hire someone full-time is so they don't have to have a policy of whenever they feel like it?
Having a reliable service is likely one of the reasons why their donors donate in the first place. Why do organizations and wealthy people still want to donate if the service goes down for 2 days and no one cares? Why would this kind of non-profit attract top volunteers in the first place?
High traffic doesn't mean high urgency. New York Public Library is huge and looks to have even more restrictive hours than mine. They're closed on Sundays!
People donate because they believe in the mission, and most of the money probably goes to the actual storage (making the cost of an employee probably pointless to care about, of course).
It would attract top volunteers in the first place because they'd think "I want this to exist, and perhaps they need some help." No one's going to think "wow the wayback machine was unavailable for two days? What a bunch of amateurs." Actually even if they paid me $200k, I'd assume they have some priority issues if they wanted me to be on-call and couldn't convince me of why that's necessary. That'd be a huge red flag for some nightmare mismanagement.
