
GitHub was down again - originof
https://www.githubstatus.com/incidents/phnch1rww464
======
arilotter
I was seeing PRs failing to update & webhooks failing to trigger upon pushing
code for 30 minutes before GH's status page acknowledged anything. I'm
surprised they don't have monitoring in place that would catch webhooks
failing within minutes of the failure beginning.

~~~
malux85
At large bureaucratic organisations there's often political implications of
changing the official status, so often it lags behind reality until it cannot
be swept under the rug anymore

Not saying it's right, just an observation

~~~
KingOfCoders
As a CTO responsible for an often failing eCommerce website that lost millions
when down and which I took over, I fell in the same trap of trying to sweep
things under the rug.

Until I decided to no longer do that and my life improved considerably.

~~~
malux85
Yeah it's very frustrating - especially if your customers are technical, they
are seeing the errors and the status page says everything is fine.

I've seen status pages and error counts tied to bonuses, which only caused a
giant mess of bad incentive alignment and internal lies, customers are
unhappy, developers are unhappy, management are lying to upper management,
it's so much easier to focus efforts on real problems and just be honest and
improve. Thank goodness I dont work there anymore (cough cough Google)

~~~
penagwin
Do these companies not have live error reporting and tracing? Like surely
github got alerts that things weren't working? Why don't they just hookup
their status page and their alerts? Or is it a political/relationship thing,
and they want to have a human give out the status page updates?

This could have been caught with a cron job and some curl requests :\

~~~
mjayhn
In all honesty they're typically just banking on people not noticing it and
are trying to make it as little of a fuss as possible and get it up before it
gets to twitter. The problem is when it's not just a small blip and they
haven't addressed it and it goes mainstream and is still down, it just leads
to concerns about transparency.

Building infra I have to work around all sorts of 3rd party services going out
or having blips throughout the day, docker registries, caches, bgp, etc., it's
totally an expected part of infra design but not every team has the time or
need to build in the resiliency. I see tons of outages that never get reported
or IMO aren't reported adequately enough.

With that said, I'm no angel, I get all my service down notifications through
slack, so when slacks down..

------
silasdavis
GitHub actions downtime is becoming painful for us. Having been lured on there
with 10,000 included minutes which they shortly thereafter dropped to 3,000 I
feel aggrieved paying for overages incurred from actions regularly shitting
the bed.

~~~
carstenhag
Also having outages at Azure DevOps Pipelines every other month or so it
seems. And that's paying - for hours there's no mention on the status page and
we are stuck there, not being able to merge PRs or release our app in the
standard way.

~~~
atraac
This is weird because we haven't encountered any real issues with agents in
Azure DevOps pipelines. I think we maybe had a single downtime in last 6
months. They recently removed .NET Core 2.2 SDK without any notice and broke
our builds but that's another thing.

~~~
carstenhag
Maybe it has something to do with us using the macOS hosts/agents? For
deprecating things, I know that they sometimes do brownouts, see
[https://devblogs.microsoft.com/devops/removing-older-
images-...](https://devblogs.microsoft.com/devops/removing-older-images-in-
azure-pipelines-hosted-pools/)

------
bob1029
Here we are again. Me taking a break on Hackernews because all my webhooks and
pull requests are fucked and I have no idea where my devops tools are relative
to what the real state of affairs is. I have pretty much had enough of this.
It is too disruptive to our process. It is causing fragility and loss of
confidence in our build pipeline.

At this point, we would probably be better off just bolting some lightweight
git solution onto our devops tools (which are 100% custom in-house developed),
rather than fighting with some more-durably-hosted offering of GitHub, et. al.

Anyone who posts that "but you cant make it more reliable than microsoft" line
is not thinking about the dependencies between systems and the considerable
impact incurred on a service just by virtue of it being a publicly-accessible
platform without any cost barrier to entry. Sure, bringing it in house might
bring additional difficulties, but I think I can eliminate a shitload of
existing difficulties if we moved from webhooks across the public internet to
a direct method invocation within the same binary image.

~~~
dabeeeenster
We've been self hosting Gitlab CE for a couple of years. Its been great. No
downtime, upgrades are seamless, fast, works.

~~~
bob1029
Gitlab is probably at the top of the list of candidates if we go down this
road. I don't necessarily need it to be in the same binary as my devops tools,
but certainly no further than localhost or another machine on the same
network.

------
zelly
[https://gitea.io/en-us/](https://gitea.io/en-us/)

[https://git.zx2c4.com/cgit/](https://git.zx2c4.com/cgit/)

[https://about.gitlab.com/install/?version=ce](https://about.gitlab.com/install/?version=ce)

~~~
nine_k
The first two links miss the idea a bit, I'd say.

I don't often need a web interface to a git repo. I can pull and do everything
locally.

What I _do_ use GitHub for is (1) code review and approval process, (2) CICD /
actions, (3) releases to push stuff out.

The branch / tag / file browser is a nice addition, but it's not key.
Rendering README.md is almost as important, if not more.

~~~
redsky17
Gitea has a lot of those features already: [https://docs.gitea.io/en-
us/comparison/](https://docs.gitea.io/en-us/comparison/)

Issues with a green 'X' means they link to a feature on their issue tracker.

And, as far as I know, they are working on integrating CI/CD right now. They
already have support for other non-integrated CI/CD platforms:
[https://docs.gitea.io/en-us/ci-cd/](https://docs.gitea.io/en-us/ci-cd/)

------
niftylettuce
There have been at least three major outages, e.g. git clone of a repo, in the
past week alone. All three of which have been unreported (and NOT shown on
their incident page), but I have email confirmation from GitHub support of
these issues. It's almost time to switch to Gitlab. I have hundreds of
repositories, organizations, and packages to transfer, while it will be
daunting... I need reliability. I have several paid GitHub orgs and accounts
as well.

~~~
ShorsHammer
To be fair they've been busy fixing the issue of slavery nomenclature in that
time too. Respect where respect is due, important issues are being tackled
here, you can't do everything at once.

[https://twitter.com/natfriedman/status/1271253144442253312](https://twitter.com/natfriedman/status/1271253144442253312)

~~~
forty
I cannot say if this comment is being sarcastic, but for the record I found it
hilarious. Thanks :)

------
rollulus
GitHub used to have a pretty cool status page, with all kinds of real time
graphs. Does anyone know what happened to it? Since it makes me really sad
that this status page is a plain lie, I had to visit HN to get the
confirmation that they are having issues again, and that it just wasn't only
me.

~~~
originof
I read an article about it, look for "status page evolution"
[https://nimbleindustries.io/2020/06/04/has-github-been-
down-...](https://nimbleindustries.io/2020/06/04/has-github-been-down-more-
since-its-acquisition-by-microsoft/)

~~~
kohtatsu
From the first two graphs it looks like they are a lot less liberal about
using "down" instead of "warn".

~~~
hinkley
The best triage policies I've ever gotten to work with had severity and
priority separated.

Severity went something like this (sometimes the numbers flip which always
confuses at least 20% of the team about whether things are almost normal or
people are hunting each other for sport).

1: data loss

2: some workflows blocked

3: some workflows unavailable w/ workarounds (ie other routes)

4: Everything else except

5: Irritations

Having a UI break but the underlying functionality is still working is not
good but people can still do their jobs, if more slowly. It's important to
classify these separate from S2 and S4. There is urgency but don't panic. Go
eat lunch or have your planning meeting, then go fix it. If data is getting
lost ain't nobody doing nothin' until we figure it out, and then some people
can go back to work but don't interrupt the people still working on it.

I think the problem is that so many metrically dysfunctional people, to the
point of cliché, have rationalized that an S2 means that only 20% of our
customers can't do their jobs so we are degraded but still working normally,
when really a yellow status should be at S3, while S2 should be at least
orange although those affected will be upset that it's not red.

Over time that 20% will shift around to most of your customers. Eventually
several times, and then you'll wonder why everyone is talking trash about you
on HN. It's not like that many people were affected!

------
DevKoala
The company I work for moved to Gitlab because we were pessimistic on GitHub
in the past few years. I don’t really have a strong opinion on which is better
though, I still keep my private repositories on GitHub. However, I feel that
Microsoft will start feeling the pain soon as more people in the development
community get sour on GitHub.

~~~
ldiracdelta
Why do you think they will feel the pain ever?

~~~
DevKoala
Github had been in growth mode up until the acquisition. If Github stops being
the nirvana for developers that it once was, it will be another dark mark in
the history of MS acquisitions. Moreover, considering how sentiment influenced
the stock market is at the moment, continued news of one of their products
having outages could easily shed a considerable amount of Microsoft’s
valuation, ~1%. The say stocks only go up nowadays, but when everything goes
up, whoever grows at the slowest rate is really going down. I‘d assume that
the Microsoft executive team won’t be happy with the new perception of Github.

------
cameronfraser
Why is outage history pre-acquisition removed from their history? If you try
to go back in time it seems they only retain history up to a couple months
after the acquisition. Is this just a 2 year retention policy or something
being swept under the rug?

~~~
ketzu
I can go back all the way to 2010:
[https://imgur.com/DsSKcFV](https://imgur.com/DsSKcFV)

~~~
mardifoufs
Wow, it used to be so much more detailed! I get they probably can't have that
level of "casual" disclosure now that they are so big, but man the current
status updates just feel so... useless and unhelpful in comparison.

------
Animats
What's the easiest way to duplicate all your Github repositories, with
history, somewhere else?

Ideally, I'd like to have two synchronized repositories, for no single point
of failure, organizational or otherwise.

~~~
monokh
Run git on a personal server[1]? It's not as complicated as you might think.
Probably much more usable to setup gitlab.

Then set up the alternative remote on your repos.

[1][https://www.linux.com/training-tutorials/how-run-your-own-
gi...](https://www.linux.com/training-tutorials/how-run-your-own-git-server/)

~~~
rovr138
I posted this on another thread. If you only want the commits, something like
this works,

    
    
        ssh user@git.example.com
        mkdir project-1.git
        cd project-1.git
        git init —-bare
        exit
        git remote add alternate user@git.example.com:project-1.git
    

All you need is SSH

~~~
judge2020
All you need to get all commits and all tags/branches:

    
    
      git clone --mirror https://github.com/you/repo
    

Push those to another server:

    
    
      git remote add new https://gitlab.com/you/repo
      git push --mirror new

------
miguelmota
Where does github publish post-moderms of downtime? I only see things like "We
have deployed a fix and are monitoring recovery." in the github status history
which doesn't provide details.

~~~
spatulon
These posts contain some details about incidents over the last few months:

[https://github.blog/2020-07-08-introducing-the-github-
availa...](https://github.blog/2020-07-08-introducing-the-github-availability-
report/)

[https://github.blog/2020-05-22-april-service-disruptions-
ana...](https://github.blog/2020-05-22-april-service-disruptions-analysis/)

------
originof
The Latest commits don't appear in the commits tab

~~~
jbergknoff
Yeah, I'm seeing the same. The branch reflects the new commit, but the PR open
for that branch does not show it.

~~~
spockz
I’ve had this before. Closing and reopening the PR does seem to do the trick
sometimes.

------
varbhat
For free Gitea instance, You can try
[https://codeberg.org](https://codeberg.org)

------
rvz
Do I have to repeat this over and over again? If these non-profit open-source
projects [0] are able to self-host a git solution like GitLab, Gitea, cgit or
Phabricator instance somewhere, surely your team or open-source project can
too.

Even a self-hosted GH Enterprise would suffice for some businesses but this
would be overkill for others. I even see the Wireguard author using his own
creation (cgit) to self-host on his own git solution for years. [1]

This is problematic since many JS/TS, Go and Rust packages are on GitHub,
which many developers rely on. Thus, it would be risky to think about tieing
open-source project to (GitHub Actions, Apps, etc).

[0]
[https://news.ycombinator.com/item?id=23818020](https://news.ycombinator.com/item?id=23818020)

[1] [https://git.zx2c4.com](https://git.zx2c4.com)

~~~
nurettin
Wait, will you really repeat this until you no longer see a "github down" link
on HN frontpage?

That is dedication.

------
kchoudhu
That's it, I'm launching SubversionHub.

~~~
fluxsauce
Come on, blue ocean.

CVSHub.

~~~
njsubedi
Why not MercurialHub?

------
zymhan
How do they not have a single update in over 1.5 hours? This is ridiculous.

------
yizhang7210
Ugh the last month has been pretty difficult. Hope they get better soon.

------
neurostimulant
So that's why my automated build wasn't triggered ~4 hours ago. I was like "no
way github is having issues again, they were down just the other day, it's
probably just docker hub's fault". If they decided to publish a blog post
about these series of outages later, I bet it would be pretty interesting.

------
donatj
It's been having issues all day. Wanted to show a coworker some changes I was
proposing but the site wouldn't show the changes I'd pushed to my pull
request. Ended up just having him pull the changes.

FWIW the git backend always seems rock solid in comparison to the front end
they have displaying it.

~~~
throwanem
I'm not sure this time. I had a PR update and kick off a build half an hour or
so ago, only to see the build fail because git couldn't parse what it got from
the clone operation.

------
rydre
I really want to move to GitLab but its UI is atrocious... like too mobile
phone looking on desktop

------
mikewhy
noticing issues on GitHub, CircleCI, and Launch Darkly.

------
may4m
I had a problem with github a while ago when I tried merging a PR to the
master branch, the merge commits reflected on master but the PR was still
open.I would repeatedly click the merge button but the PR wouldn't show as
merged

------
mlang23
Likely unrelated, but I recently noticed that GitHub stopped updating my
activities overview for july. I definitely pushed commits, but they are not
noticed. Anyone else having a similar issue?

~~~
NoobTW
Are you pushing to master branch?

~~~
mlang23
Yes, I was. However, I was pushing to master of a fork. Maybe thats the
reason.

------
marcinzm
How is GitLab like in terms of downtime? I looked at their status history page
and I'm seeing a lot of incidents but it's hard to figure out what it actually
means.

~~~
s_dev
Thats why the self hosted options are there -- and why GitLab has a
competitive advantage in this sense.

Cloud solutions are great -- however they have a golden rule, don't go down
ever. This is seriously damaging to GitHubs reputation.

~~~
hn_throwaway_99
> Thats why the self hosted options are there -- and why GitLab has a
> competitive advantage in this sense.

GitHub has GitHub Enterprise

> Cloud solutions are great -- however they have a golden rule, don't go down
> ever.

Oh where oh where can I sign up for this mythical unicorn cloud service?

------
talkingtab
Running your own git server is trivial. I have been doing it for years on a
very cheap digital ocean instance. Set up ssh keys, lock it down with ufw,
done.

If that is not enough, run your own instance of gitlab.

If that is not enough use Gitlab.

Microsoft is going to attempt to make a profit on Github. That's okay, but
based on past experience and current issues, their business model is lock-in
not service.

I suspect the same is true for NPM.

------
stunt
At the current rate, Whatever you host will have a better uptime than Github.

------
drcongo
They're probably using the Facebook SDK /s

------
revskill
Scaling Rails is hard ? Github needs to move to CDN, static site deployment
instead.

------
juped
Git is distributed.

------
MattGaiser
Did Microsoft adopt Scrum?

------
josefrichter
The Microsoft Effect

------
iso947
Don’t host yourself, it’s impossible to meet the reliability of the
professionals

~~~
wizzwizz4
No, it's not. Apart from scheduled downtime when nobody's using it (e.g.
restarts in the morning to update the kernel), it's not that hard to beat
GitHub's uptime for a small Gitea instance. My power's on more than GitHub is
up.

A UPS and a tethered smartphone would get me three nines uptime-while-anyone-
needs-it, which is _well_ in excess of what I need.

~~~
gourmetennui
I think the OP was being sarcastic.

