God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.
It's a pretty well-known fact for years that tarsnap is basically a one-man show, and yet Colin has managed to provide fantastic service so far. Sometimes having ppl who built the service also managing it is actually a big plus, compared to other services where you first have to fight through outsourced & underpaid support that's limited to template answers, only to finally get some "engineer" who got that job 2 months ago and is more clueless on their system than myself...
And to be frank, I've seen plenty of mission-critical services at $bigco which may have had a team of engineers working on them, but the core functionality was maintained, understood, and supported by effectively one senior engineer. If anything went wrong, the supporting junior staff might have been able to fix reasonably simple stuff, but there was essentially one person who understood the system deeply enough to handle problems of any real significance.
Early in my career, I became the second person able to support and operate a system that was public facing and responsible for billions of dollars of activity that mattered to many individuals and stakeholders. The entire team retired over a period of six months, after giving the folks in charge a year or more notice. After about 12 weeks, I was the sole guy, training a 4-5 new people.
We’re all probably using a service like this. As demonstrated by Twitter, well engineered systems can persist, even without proper care and feeding, until they don’t.
I hate to bring this up, but what about the bus factor? If Colin is physically unable to continue maintaining the service and something like this happens again, how will anyone be able to get their data out? It's not really a concern about the service Tarsnap provides today
There's an old Sys Admin saying (perhaps from Allan Jude of ScaleEngine) that goes something like "if your data doesn't exist in at least three places, it doesn't actually exist at all..."
That is to say, if Tarsnap is the only place you've keeping sensitive/important data, then you're "not doing it right" as a backup. Things happen... your hard drive can die suddenly, and a data center bursts into flames all on the same day.
I feel like ovh will never stop earing about this. This has been, frankly, a traumatic event for many sysadmins I believe, and one that was shared by many from the same source, which is quite different from the standard variation of "that time when I erased the production database" (looking at you gitlab, but also at myself!). I mean, at this point it's between a legend and a warning tale and I don't know what else to call it. A bad Wednesday probably.
> I feel like ovh will never stop earing about this.
To be fair, they deserve it a bit as they got up in flames twice .
Indeed, after the first fire, the geniuses over there collected all the UPS and batteries they could find from the DC and stored them all in a pile in a closed container... where they predictably bulged, failed, sparked and eventually triggered another fire after a couple days.
Why the scare quotes? I would expect any well-experienced power user to know a complicated system better than a fresh engineer two months into working on it, with no previous experience on the system. Especially if the power user is an engineer themself.
You really shouldn’t if that’s a major concern for you and that is a valid concern. For the same reason I’ll never use PurelyMail otherwise it’s perfect.
I know you didn’t ask me — but I don’t think Colin can answer differently other than saying that he is training a family member or friend to take over if needed.
I am really confused by the communication thread and am interpreting that the toddler is somehow in charge of running the infrastructure as a joke. Yet I can’t see it as either a joke or serious.
I’m a native English speaker but sometimes I swear I’m losing grasp on communication in the Internet age and am sincerely trying to understand this all.
My toddler runs https://rangerovers.pub and it mostly holds up okay. He's not great at yaml because he can't really read so the significant whitespace is a problem, but he knows how to run the backups and ensure the mail handler isn't choking on all the Russian spammers. We try to limit his screen time though so he's only on for the 15-minute maintenance window. The Aprilscherz frontend for Docker is a big help.
Are you suggesting that those who build enterprises don't have time for kids? Seems plausible, but is the difference in lifestyle so consistently prevalent as to be stereotypical? Elon has 10!
Might take a while. Tarsnap has never had an employee without a doctorate. She's a very bright girl but I'll be surprised if she gets her doctorate before 2040.
tarsnap natively protects against inadvertent or malicious deletion or corruption — old tarsnap backups are immutablez The low-cost competitors (restic, borg, etc) seem to have this feature as an afterthought, and they make it surprisingly difficult.
(FWIW, S3 can be somewhat straightforwardly configured so that old data is effectively immutable. Google Cloud Storage’s similarly named versioning feature appears to be far weaker.)
Yep, S3 is reasonably easy to configure for immutability. I personally use restic to send (encrypted) blobs to https://www.borgbase.com which has append-only mode and monitoring to warn me if some backups didn't happen.
borgbase is another “little” service that I use and like just like tarsnap and to some extent rsync.net. And they also have an excellent gui app Vorta (it’s FOSS; borgbase dev is the maintainer).
Even large organizations can have fairly regular availability issues. I appreciate the noted flaws of "single point of failure", but I also see orgs where 100s of people have access to the infrastructure, make a change, and then it breaks something. I wouldn't do business with an org just because they have many people, that won't mean they're operationally sound, at least not to my expectations.
I mean, you called it crap and then said it's super important. That's what hoarders say.
Subjectively you may feel that your data is super important, but objectively it probably isn't.
When people talk about 'super important' (totally a technical term), I think of things like DB backups in software companies, backups of financial reporting for firms, etc. Not your tax return from 2008.
My nginx config is not super important. My old reports written for study are not super important. My pirated movie copies are not super important.
These are examples of data that I could easily live without. Where losing it would either be a matter of re-doing old work, or just forgetting about old and minor things.
>What is this mythical unimportant data that people still want to back up?
I have lots of stuff like this. Often it is easier to just back up an entire folder than go through sub/sub folders separating stuff into: important, not very important. Storage costs are low enough to just backup everything (almost). Also, one often doesn't know what may be important/useful in future. For example a couple of years ago I had this huge buildroot system (600gb) to build firmware images for a single board computer I spent quite a while to put together. The project I was doing it for got cancelled so I had no need to keep it. Still I wish I did, as I'd love to be able to tinker with it now, but 600gb is not a trivial amount to store so it got deleted. Most of this data was pulled from various online resources that don't exist anymore too.
What's the morale of my story? If you have a fast internet connection (I don't) backup "everything" to cloud. Then find "really important stuff" like the pictures of your children etc and back it up again to a different cloud.
If you're in a middle of nowhere on a slow LTE connection like me, building a nas box is not a bad idea for backups.
Anything that you stashed just for convenience, but you could re-download or re-generate it if really needed, or simply live without it... frankly, like 90% of stuff on my disks fall in the category "I'll read/view it one day", which in reality I'll probably never have time or patience to open ever again.
I really used to enjoy formatting my machines about every 6 months.
Well I used to until macOS kinda went off the rails a bit. Now it’s mostly an exercise in running my arch script for my thinkpad.
Being stuck between operating systems is kinda a mess though, makes backup and file sync in general really hard. But everyone’s gotta have their own cloud, right?!
Why can’t I just put a cloud under my bed and forget about it?
> Why can’t I just put a cloud under my bed and forget about it?
Just buy a Synology NAS. Keep default settings, set up a few user accounts, tweak a few things here and there, enable encryption, install Active Backup on all your devices, done.
There are many cheaper/more open options for self-owned NAS storage, but contrary to a Synology they're definitely far and away from "and forget about it".
It's not about suing, but defining expectations about how you can rely on a service.
For example, my team has people across the world for HW bringup, so we can't allow our code hosting or CI to be down for more than a few hours. Of course, backups have different uptime requirements, but as for everything, it's a tradeoff between features, of which an SLA is one.
Tarsnap's features are granularity of cost, reliability of storage, and encryption, but not 99.999% uptime.
> It's not about suing, but defining expectations about how you can rely on a service.
Meeeeh, my ISP cut of around 100+ fiber connections in my town and spend three weeks fixing it. My neighbor have business line, there's an SLA on those that among other things, require them if reestablish his connection within 3 - 5 hours. It took them over 500 hours, so that SLA is useless for anything but forcing compensations.
The problem is that the SLA should give an indication of available resources, but in reality it's mostly a contractual thing for most companies, they'll pay the "fine" or refund a customer if they fail to hit their SLA and that's about it. Tarsnap most likely have better availability than many midsize competitors simply because it's just one person who really cares about it. Doesn't help if he's hit by a bus though.
SLAs can be meaningless like that. However the better ISPs have in place a backup system that doesn't use the same fiber/wires. Sure the backup might be a radio or satellite feed and so be slower, but it will get/keep you online. This costs are lot more per month though, so if you are not paying for that service your SLA will probably just be we give you a free month (which hurts them enough that they will do some things to prevent downtime, but not enough that they put redundant fiber paths in the ground)
A company could... if you have N users and you pay M for storage per user and downtime cost you X then it could be that a discount of Y means (M - Y) * N = X
Agree. You get a discount if something breaks. But SLA really only works for larger services where the cost of fixing something is small when compared to the discounts.
I know it's a joke, but I think if an SLA involved putting a CTO in stocks and throwing eggs at him then that'd encourage me to sign up for the service. Especially if the video of it were posted after every incident.
Instead we get refunded some pitiful amount when our business is seriously disrupted for an extended period of time.
I haven’t checked the pricing in a long time but you can use Tarsnap also if you have to backup only 7.3kb (okay I might ne exaggerating here but you get the drift) and pay for only that much. You can’t do that with Wasabi et al.
Also it’s really simple and does what it says it does, nothing more, nothing less. In today’s everything convoluted and bloated world this is a luxury imho. The GUI app is also quite good and functional. Support is prompt (that is if you need it).
You don’t have to worry about file being deleted just because your machine didn’t connect or backup for some time even if you keep paying (hello Backblaze) etc. I mean there’s no circus, melodrama , and cliffhangers involved.
I personally would never use it backup my entire laptop, due to price alone. But I have a subset of VVI files and Tarsnap is one of more than one backups for those files. So for that use-case Tarsnap is perfect for me, so far.
Backblaze has a policy of allowing backups of external disks, but the disks have to be connected at least once every 30 days, or they'll delete the backups. I understand they want to avoid abuse, but the lack of any grace period, or ability of support to ad an override, really soured the service for me.
Huh, I had to start paying them $2 more for my nonexistent PC I think, but otherwise was fine. I have only 1 TB of total storage on that PC though, so maybe that’s the reason.
God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.