Hearts out to all the GitHub SREs who get to fix this, and all the SREs at GitHub’s customers who get to spend the next week convincing management and their peers that moving git in house won’t result in fewer outages.
In general, paying GH tends to be much cheaper than employing a team of SREs that'll keep your on-prem Git instance up and running 24/7 (esp. including things like backups, disaster recovery, and similar good practices necessary for any enterprise-ready SaaS).
Something tells me that if you can schedule upgrades "outside of business hours", your scale is small (because in a global company, it's always "business hours" somewhere), which makes the cost argument even more prominent.
If you work in a large business that needs git but doesn't have backups and has no disaster recovery plan at all you're going to have much bigger things to worry about than a git repo being offline.
I really like it. I keep source code, 3rd party libraries, images, binary build output and installers. I can easily go back and debug and crash a customer ever sees because I have all the binaries and debugging files.
It depends on your needs and team size. For few tens of dev that need a versioning system with some sugar, a self hosted gitlab can acieve better uptime.
> It can achieve better uptime, if you ignore the downtime for upgrades, the downtime for configuration errors, the downtime when the disk fills up...
Besides the last one, the others you can schedule when developers are not actually working on something, or give a headsup so developers can be prepared in case of errors.
In the case of GitHub, Microsoft deploys changes whenever they want, whenever you want it or not.
> In the case of GitHub, Microsoft deploys changes whenever they want, whenever you want it or not.
That's because GitHub is used by millions of people every day all over the world. Business hours for them are around the clock. No matter when they update, if it breaks something, many people will hear about it regardless of when the break was deployed.
Yeah, this is exactly my point, their needs don't overlap with your needs, hence sometimes it's better to self-host, because then it's 100% your needs that gets taken care of.
As I said, it depends on your needs. I don't know if you have ever managed a small gitlab self-hosted installation, but it's really trivial, you set up it the first time, and you forget about it. Also, the upgrade process takes few minutes to complete.
Btw, you cannot compare a scheduled maintenance window with an unplanned downtime ... maybe in the middle of an important deployment.
Not add new features every week? Seriously, you cannot tell me that you cannot run a minimal version of a git repo service much more stable than that. I blame the need for new shiny stuff.
Ya'll make me laugh. Don't add any features and "This thing is dead, nothing new in years, find an alternative" But a company actively adds new features their customers are asking for and you're upset that they're adding new features.
And we all know self-hosted software never goes down.
Although I guess the benefit is that the CTO gets a psychological lift of looking over Jeremy from IT's shoulder when he tries to fix the issue rather than being at the mercy of some engineers which the company can't fire.
People like to make it sound like running your own server/service is impossible - completely forgetting that's how everything was done not that long ago.
Some wiz-bang new graduates calling themselves Engineers probably haven't a clue - sure, but for everyone else, you probably still have folks employed that did just that on a professional level.
It's not hard - and often it's less expensive. Particularly for a turn-key paid product like Github Enterprise or GitLab.
The fact that GitHub being down can have such an impact means that Git’s decentralised features don’t go far enough. Why aren’t issues, PRs and all these other little things we depend on also stored in a git repo? Or something similar that can withstand GitHub being down.
I set aside time today to go through my notifications and I can’t because of this. Notifications are trickier to decentralise but still, ugh.
"Everything goes in the repo" is one of the main features of the Fossil [0] SCM which is built and used by the SQLite team. It's a neat idea and I wish something like that had caught on.
It's not too late! At one point, git was the new kid on the block while everyone else was using something else (or copy-pasting directories/zip files for versioning).
All it takes is more people using it, and people here on HN are more suitable than others to start playing around and build their projects with it.
That's why he said that Git's feature set doesn't go far enough. Most projects need an issue tracker and a tool for discussing proposed changes. These things are out of scope for Git, but since people needed them, they congregated around a centralised service that builds around it. Of course, decentralised alternatives for issue tracking and pull requests exist, too.
This is why Linus uses email. It’s the Unix way — Git is just another tool. It doesn’t need to be a website, and an issue tracker, and a change proposal database, and version control software, and a social media website, and pull request tracker. It just needs to be a version control software with some decentralization.
If todays developers were tasked to build the World Wide Web again, we would all log into some oAuth portal.
> The fact that GitHub being down can have such an impact means that Git’s decentralised features don’t go far enough
Github was created to literally bring central server to a decentralized system. Yes, when that central server goes down, that's a problem for its users. It has nothing to do with Git.
You can, if you'd like, send Git patches in emails, or via snail mail, and you won't be influenced by any outages outside of individuals' laptops (which will affect only those individuals). Also, good luck managing your 250 microservices that way.
heh, this was my argument back in like 2009 when everyone was all, "DECENTRALIZED!! We have to take months to migrate from SVN to git cause DECENTRALIZED!! You don't one a single point of failure do you?!"
all I could say was, uhhhh github is a single point of failure. Unless you're mailing diffs around like Linus (which nobody I knew did), it's 6 of one, half-dozen of the other.
there are lots of great things about git and especially git over e.g. svn (especially merges) but I never bought the decentralization argument the way it was spearheading the conversation back then.
With git I can create, edit and merge branches locally. If GitHub is down I can push to another remote I can send diffs. I can calmly carry on working until GitHub is back. If GitHub is down too much I can migrate to gitlab or gitea.
I would say the decentralised features of git work for me. And they are no overblown.
Sadly, people seem to push their entire development flows into GitHub Actions, so things like deployments happens on GitHub's servers, and then their servers go down and you suddenly can't deploy anymore... Or writing their documentation via GitHub issues/PR descriptions/comments and now you cannot access that either...
If your SVN server went down, you couldn't even commit. At least with got, you can do everything as normal except push and pull, and that's only if you don't have multiple remotes (which, admittedly, very few GitHub users do).
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
remote: fatal error in commit_refs
To $REPO
! [remote rejected] main -> main (failure)
error: failed to push some refs to '$REPO'
I think we should applaud this when it actually happens. Far too many services are terrible for this. While I'd rather the service not be down, it makes me feel a bit better if they don't lie about it.
It's funny when a company like Amazon, which many would think is on the cutting edge of automation, implements their status page as a manually updated static page.
While I understand where you as a person are coming from, and my below thought is a tangent, I think this sentiment is like the origin story of editorialized headlines.
If one gets dopamine/validation/click thru revenue from others consuming/clicking on media one shared,and further feel "unfairly gamed" when someone else gets that payout, it's a short time before one starts subconsciously appealing to the human lizard brain and sensationalizing media.
Everything is down. How can this happen? Many of these services are unrelated, like even the brand-new Codespaces and Copilot are completely down. Do they all have a single point of failure? (Presumably the git server)
I'm not really sure Github prioritizes the high availability anymore. These outages might be a way to motivate the customers switch to Github's self-hosted enterprise solution which is the revenue stream for Github. If they're not losing users because of these outages, (I'm sure they can find that out) these outages really make them more money in the long run..
As a previous GitHub Enterprise Server customer on a site with hundreds of users, who's spoken with a number of GitHub employees about this, I do not think that is true. They would far rather you were on GitHub Enterprise Cloud. They view Server instances as a necessary evil for people who have compliance or security requirements they can't handle with the Cloud version, but it is fairly obviously a pain in the neck for support, and attempting to attain feature parity. They have been trying to bring their Server offering closer in architecture to the Cloud version since forever (e.g. moving to Nomad a few years ago to orchestrate it all as containers, so you can shard the services out over multiple hosts similar to github.com, etc.).
Last time GitHub went down was less than a month ago: [0]
At this point maybe it is time to self-host rather than to continue to tolerate this since it's guaranteed to goes down every month. [1]
Even open-source projects like RedoxOS, GNOME, KDE, ReactOS, etc are doing just fine without worrying about GitHub's unreliability as predicted years ago on not centralizing everything to GitHub. [2]
> At this point maybe it is time to self-host rather than to continue to tolerate this since it's guaranteed to goes down every month.
Not a single one of these outages has caused me enough pain to even consider moving to another hosted provider let alone self-hosting and the burden that goes into managing all that stuff.
Can we stop with this whole "omg a major platform that gets used by millions of people a day can't guarantee 100% uptime, we need to all manage our own infra!!" mentality?
Seriously I've been in GitHub all morning and decided to take a break and read HN and I was JUST learning about this now... It was resolved before I seen this thread too.
The Source is what gives a hacker his power. It's an energy field created by all coded things. It surrounds us and penetrates us. It binds the galaxy together.
nc bofh.jeffballard.us 666 # I am surprised this is still working given its age; the CGI version at https://pages.cs.wisc.edu/~ballard/bofh/ doesn't work anymore (but the page is still up)
