
GitHub was down again - pupdogg
GitHub is down once again!
======
epoch_100
While I use (and mostly like!) GitHub, this is yet another reminder that
centralizing so much infrastructure—from package managers to CI pipelines to
static websites and more—around one company is a very bad idea, and will
likely bite us in the end.

~~~
jonsolo
Is that actually bad? A chain is only as strong as its weakest link, so if my
system depends on many sites then even one outage can take me down.
Centralizing like this can actually _reduce_ risk.

~~~
Epskampie
Depends on how it's done, if it's one authority with many mirrors it can be
more reliable. Take for example `apt` in debian, you can always find a mirror
online.

With git's distributed nature, this would be very much possible.

~~~
dyingkneepad
Will you also mirror CI infrastructure, issues, pages, documentation, API-
consulting scripts, etc?

Also, pushing to a mirror is not exactly the thing you want.

What makes apt easier is that pretty much everybody is just downloading from
the source or from a mirror. When the source stops, mirrors don't get updated
and everything still works. That's very different from the usage model that
people have with Github.

Gihub/Gitlab have extended the Git usage model very much, they are not just
git. You can't easily migrate away from them to another git offering (and not
even very easily between both).

------
tylfin
I would really like to see a postmortem for the recent outages. While I doubt
it's all the same root cause, it would be nice for some messaging around
improving resiliency.

I think the outages could be related to:

1\. Github mobile just went public so they may of changed the scaling params
to keep up with expected increase in traffic 2\. The new notification system
seems to be a lot heavier, and they could still be catching up to the changes
3\. They were somewhat recently acquired by Microsoft so maybe they're
migrating to Azure to reduce expenses

Whatever the cause, 11 days with outages out of 90 is pretty rough when you
rely on Github for project management, a central hub for viewing CI, and all
VC concerns. Feel bad for the smaller companies that wholly adopted GitOps and
are blocked deploying hotfixes during these outages.

~~~
jlgaddis
> _I would really like to see a postmortem for the recent outages. While I
> doubt it 's all the same root cause, it would be nice for some messaging
> around improving resiliency._

They recently posted a "post-incident analysis" [0] about several "service
disruptions" in February that were all a result of database issues.

Obviously, I have no idea if today's outage is related.

\---

[0]: [https://github.blog/2020-03-26-february-service-
disruptions-...](https://github.blog/2020-03-26-february-service-disruptions-
post-incident-analysis/)

------
twistedpair
Geeze, so many outages/impacts (12+) in the last 90 days.

The status page looks like a Pez dispenser.

[https://www.githubstatus.com/](https://www.githubstatus.com/)

Usually companies put in place a release freeze or "Code Purple" when there
are such demonstrated problems with releasing stable code.

------
chacha102
Fortunately, Github is built on a distributed system that doesn't require us
to stop coding just because we can't access a central server.

Distributed systems for the win!

~~~
YetAnotherNick
What is distributed here? If github is down no one can see what you changed,
not even the build servers. Git is distributed in the sense that every client
has complete history, but not in the sense that my laptop can act as
replacement of github(like torrent).

~~~
chrisseaton
Git supports lots of protocols for sharing your changes - even email!

~~~
YetAnotherNick
Diffs can be shared over mail, right. Can I do a `git pull` via mail?

~~~
ddevault
Yes:

[https://git-send-email.io](https://git-send-email.io)

You can also do a "request pull" over email, which if it sounds confusing it's
because GitHub wanted it to:

[https://www.git-scm.com/docs/git-request-pull](https://www.git-
scm.com/docs/git-request-pull)

------
chkaloon
Why is this post flagged? Seems like an appropriate HN topic

~~~
dang
The more repetitive a topic gets, the more users flag it. 'Is down' posts are
repetitive as a category to begin with, and 'X is down again' posts are
repetitive along two axes. There was one of these a few hours ago, before it
got flagged:

[https://news.ycombinator.com/item?id=22955595](https://news.ycombinator.com/item?id=22955595)

... which was likely because users regarded it as a quasidupe of

[https://news.ycombinator.com/item?id=22935941](https://news.ycombinator.com/item?id=22935941).

By 'quasidupe' I mean it's not strictly the same story, but the difference
('down again') isn't significant enough for the thread to be substantially
different.

(The submitted title on this one was "GitHub: Here We Go Again".)

~~~
headalgorithm
I understand your argument, but the number of comments on this post shows
people still want to discuss it.

Is it possible to run an experiment. Remove the flag option but highlight the
hide option to users. Users who are not interested in the topic can remove it
from view but do not kill the discussion for those who are interested in it.
Then see if the quality of posts/discussions decline.

------
Yetanfou
It almost feels like HoTMaiL all over again.

For those too young to remember, Microsoft bought Hotmail (forget the silly
CamelCasing, we know that HoTMaiL was HTML + mail by now) which was based on
FreeBSD+Apache and was champing at the bit to use it to demonstrate the
scalability and stability of their then relatively new NT operating system and
the IIS web server. Let's just say that... things did not go the way Microsoft
would have wanted and the demonstration more or less achieved the opposite of
what they intended. It took them a long time to move the frontend to Windows
and an even longer time to do the same to the backend.

They won't make that mistake again but they might succumb to featuritis or
wrongfooted attempts to steer Github-users further and further into the
Microsoft world.

~~~
duxup
I remember signing up for Hotmail and abandoning my isp email as if it was
some sort of internet activism ;)

------
sbuccini
GitHub Enterprise as a quarterly Uptime SLA of 99.95% It's probably worth
checking to see if they've violated it this quarter. The status page says that
their uptime is 99.92% YTD but their support page says that the status page is
not connected to their internal metrics. [0]

[0] [https://help.github.com/en/github/site-policy/github-
enterpr...](https://help.github.com/en/github/site-policy/github-enterprise-
cloud-addendum#enterprise-cloud-uptime-sla)

------
kalev
Be glad you're not on Bitbucket though. Issues happen so often people stopped
posting it to HN.

~~~
bmgoau
I feel Bitbucket has been remarkably stable this year. The difference between
the Github Statuspage and Bitbucket's is night and day. Of the two recent
issues I can see in the last month they look like minor glitches and not
outages. Bitbucket had an unfortunate series of interruptions for a couple of
days in October last year, but Github has been having major outages every few
days for 3 months now.

------
hirundo
Single Point of Failure As A Service

------
reledi
Just switched back to using GitHub again and the timing couldn't be worse.
Hope things are going okay for them, definitely some stressful weeks.

On a side note, the status.github.com page is quite delayed. Following live
updates from Twitter has been a better strategy for confirming that GitHub is
having issues:
[https://twitter.com/search?q=github&src=typed_query&f=live](https://twitter.com/search?q=github&src=typed_query&f=live)

------
qzio
mhmm
[https://www.githubstatus.com/incidents/2y6v4ltq26g7](https://www.githubstatus.com/incidents/2y6v4ltq26g7)
:(

~~~
danra
> Currently observing issues affecting GitHub.com ?.

Yes.

------
johndavid9991
Experienced this last week, too, we worked hard to complete a feature, and we
were ready and excited to merge and push the update to production, then GitHub
was suddenly down. We didn't sleep well that night. We could have done some
workarounds but just too frustrated to do so.

------
SirensOfTitan
Seriously considering moving our projects off of GitHub at this point.

What other services are good? I loved using Phabricator at Facebook, but its
cloud option is pricy.

~~~
chatmasta
In my experience, it's not really git itself that is the problem with
downtime, but rather things like CI and recently, package registry. Last week
I thought I was isolated from the GitHub downtime because I was working on a
project on GitLab. But turns out a bunch of the npm dependencies are hosted on
GitHub. Normally that wouldn't be a huge problem, except I'm specifically
working on CI pipelines at the moment so it's quite frustrating.

~~~
leesalminen
Came here because my Gitlab CI/CD was failing on `npm install` with a 500 from
codeload.github.com.

~~~
chatmasta
And now they've bought npm as well! We're cornered.

------
Epskampie
I'm so used to github being reliable that I was first checking my own machine
when composer wouldn't install something.

~~~
pupdogg
Same here! I'm using WSL2 on Win10 and started getting "fatal: Could not read
from remote repository" errors and thought it's definitely something to do
with my WSL legacy to WSL2 upgrade.

------
chatmasta
Have they provided any insight as to the cause of the recent spike of
downtime? Any post-mortems or anything like that?

~~~
jlgaddis
[https://news.ycombinator.com/item?id=22963336](https://news.ycombinator.com/item?id=22963336)

------
coleifer
Looks like it's affecting git operations for me.

Yikes.

------
arghblarg
Consider self-hosting via gogs.io or gitea, and use them to mirror your key
repo dependencies ...

~~~
Havoc
>use them to mirror your key repo dependencies ...

Is this possible with a self-hosted gitlab?

Trying to get that to push to a cloud repo (GCP/Azure/whatever) as backup.
Managed to selfhost gitlab but at the edge of my technical git knowledge here

------
archi42
I am often a bit sad that we don't have all the fancy pants GitHub features.
But right now I'm quite happy I can still use my git, the CI runs quite well,
and everything else is quite dandy overall =)

------
dbingham
What was that we were all saying about Microsoft being on an upward trend?

Years and years of stability to the point where it starts to taken for
granted. About a year post Microsoft purchase, and here we are.

~~~
adrianvoica
Exactly.

------
Legogris
They must have been anticipating this happening more often, given the
replacement of the raging unicorn with a sad squidcat (or is it a catsquid?)

~~~
cocktailpeanuts
i think they shuffle it up algorithmically. I am getting the unicorn, for
example.

~~~
Insanity
Yup same here

------
dubcanada
What are we at 5 downtimes this week?

~~~
leeoniya
and the new notifications they rolled out are worse in some significant ways
:(

~~~
ivanfon
Not sure if they've fixed anything, but this blog post describes some of the
issues: [https://drewdevault.com/2020/03/13/GitHub-
notifications.html](https://drewdevault.com/2020/03/13/GitHub-
notifications.html)

~~~
leeoniya
i reported a lot of this to them during beta, too.

thank god they added back a button to group by repository. it was literally
unusable otherwise - just a stream of disorganized crap.

the repo name is still uselessly duplicated in every issue line despite the
common heading when grouped. you can no longer "mark all as read" in a repo
without going elsewhere to see the hidden notifs, pressing "select all" and
clicking "done"...so next time you refresh the page without doing that, it all
floats to the surface again. etc etc.

ugh.

------
JeroenKnoops1
:( a lot of colleagues are now sad... :(

~~~
pupdogg
tell me about it! I got an early start today wanting to push deploys before
customer opened...well, now I can't!

