
GitHub was down - benbruscella
https://www.githubstatus.com/incidents/j597fw8kv04c
======
ivalm
This is a nice reminder that you can have multiple remote repos and push/pull
to all of them at the same time. For my side projects I usually use both
github and google cloud source (I use gcp). If one is down the other is still
available and then just resync when service is recovered.

~~~
mpweiher
Additionally, I have a machine on my local network that I push to (and that is
backed up separately).

 _Really_ quick pushes, too :-)

------
a012
I know Github is down because I'm trying to update a kops cluster, but Github
breaks it

    
    
        > kops update cluster
        error reading channel "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable": unexpected response code "500 Internal Server Error" for "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable": 500: Internal Server Error

~~~
breakingcups
See, I like the idea of K8S quite a lot. Enough that I decided to set up a
cluster from scratch to figure out how everything ties together.

I've since grown to strongly dislike how much of the _entire_ ecosystem seems
to depend on random unversioned Github gists or direct links into a raw file
in a repo somewhere.

~~~
IshKebab
NPM is the same. You might think that all of the packages are stored on NPM
servers. Mostly true, but still when you install some of them they'll download
precompiled binaries or whatever directly from Github.

~~~
miyuru
If you have IPv6, remove IPv4 to see how many tools break, because github does
not have IPv6.

Composer is useless in a native IPv6 only servers.

------
lprd
This is slowly becoming a weekly occurrence ever since Microsoft entered the
picture...

[https://www.githubstatus.com/](https://www.githubstatus.com/)

~~~
kenhwang
I remember it being a weekly occurrence before Microsoft as well.

edit: though their historical data doesn't back it up. It's just a 2020 thing
it seems; coincidentally picked up right near the middle of February which
correlates way too closely with the beginning of COVID.

~~~
Insanity
correlation != causation though.

~~~
p0llard
The whole "correlation does not imply causation" thing is completely
misunderstood.

The issue is with the meaning of the word "imply"; when used in the formal
sense as it appears is Classical Logic, correlation does indeed not imply
causation.

In common parlance however, "imply" is often used to mean "provides evidence
for", and correlation can indeed provide (potentially strong) evidence for a
hypothesised causal link; the problem lies in people reading "correlation does
not imply causation", assuming the informal meaning of "imply", and then going
on to reject any notion of causation which uses observed correlation as
evidence.

Pretty much every empirical science uses notions of correlation (in its
various formal statistical guises) to provide support for causation, indeed to
reject such reasoning would be to invalidate huge swathes of mainstream
accepted science; half the battle in these instances is _making the leap from
correlation to causation_ in a manner which is considered scientifically
sound.

------
FBISurveillance
From what I understood, they are having capacity issues with one of their
MySQL masters or something. I've read that they are in process of
sharding/resolving that but it takes time.

To GitHub SRE/oncall people: hang in there, you're awesome.

~~~
terom
That's the picture you can get from their recent blogs:

* [https://github.blog/2020-03-26-february-service-disruptions-...](https://github.blog/2020-03-26-february-service-disruptions-post-incident-analysis/)

* [https://github.blog/2020-07-08-introducing-the-github-availa...](https://github.blog/2020-07-08-introducing-the-github-availability-report/#availability-report-for-may-and-june)

------
luiseduardo
Yeah, our builds are failing for a couple of hours now, and we just use Github
because one of the dependencies in NPM downloads a binary from a public
repository. We're already forking what we can to our gitlab self-hosted
server, but even a simple git clone or even browsing the website can lead to a
HTTP 500 right now.

Lots of incidents lately, but it's becoming increasingly hard to get away from
Github.

------
rvz
I said this before many times and I'll say it again, consider self-hosting
your projects on a solution like GitLab or Gitea to avoid this sort of
situation. [0]

GNOME, Xfce, Redox, Wireguard, KDE and Haiku all have self-hosted on either
cgit, Gitlab, Phabricator or Gitea.

[0]
[https://news.ycombinator.com/item?id=23676072](https://news.ycombinator.com/item?id=23676072)

~~~
sparkling
How about a more low-tech solution? I don't want/need a full-blown web
application like Gitlab, just need somethings than auto-syncs my public and
private repos so that i can still access everything when Github is down.

Any ideas?

~~~
hamandcheese
The lowest tech solution is to just wait until the outage is over. It’s 100%
free!

Seriously, though, any repo that I work on regularly will be cloned to my
local dev environment, so it’s not a hard blocker.

That said, a cron job on a cheap VPS would probably do the trick.

~~~
jopsen
Moreover, GitHub will eventually get these outages under control. Or we'll all
be driven to gitlab :)

------
jonny383
Seriously, again!? I am growing seriously impatient with this. It's been
downhill since Microsoft took over.

GitLab is starting to look good (or even Gitea self-hosted).

~~~
navanchauhan
I might just self-host Gitea on my Pi today

~~~
jonny383
I was going to warn you about SD card corruption, but then I realized even
this is probably more reliable than GitHub at this point.

~~~
navanchauhan
I personally haven't experienced any SD card corruption, even though I am
running a Maria-DB instance, Home-Assistant instance and a Pi-Hole instance on
a Pi Zero with a 32 gig SD Card ( 23 days uptime as of writing )

~~~
_ikke_
I've had 2 sd-cards go corrupt (read-only, any writes were lost). Had a
mariadb instance running on it as well. It will hold for for some time, but
eventually, the card will give up.

~~~
navanchauhan
Oh no! What will you recommend as a long term solution? I have ordered a Pi 4,
should I use an USB Flash Drive / HDD / SATA SSD?

~~~
m0xte
Honestly throw the thing in a bin and use an old mini pc instead. Lenovo
thinkcentre tiny. You can pick them up on eBay for about the same as a fully
equipped pi.

Pi is 100% not suitable for 100% duty work. It’s just a toy.

~~~
kelnos
As someone who's been running a media server and home automation server on his
Pi 24/7 for 3 years now, I beg to differ.

~~~
m0xte
I’ve had 6 so far and all have had reliability issues or weirdness. Mostly
related to SD corruption, crashing or power brown out. The power issue was not
solved by running them off a proper keysight bench supply.

SD cards are quite frankly horrible boot media as well.

------
yuppiepuppie
ITT: Lots of Github hate.

Dont forget that change in software is inherently risky and will result in
bugs, etc. Id rather have a platform that is always looking to make things
better and risking a bit of downtime, than a stale platform that we all know
we depend on.

~~~
kkapelon
I think the issue here is that Github is considered a mature product and it
just works.

So is there any actual pressure to move fast and break things?

~~~
robertlagrant
Yeah it feels they're just being pushed to out-everything Gitlab, to kill a
competitor, before relaxing and stagnating.

------
tdonovic
Getting 500s all over the api, gl whoever is on call

------
sairamkunala
(copied over from other thread -
[https://news.ycombinator.com/item?id=23817794](https://news.ycombinator.com/item?id=23817794))

Github started doing availability reports. Last month's details in the blog
post below with summary of the issue.

Stay tuned till next month for the current outage.

[https://github.blog/2020-07-08-introducing-the-github-
availa...](https://github.blog/2020-07-08-introducing-the-github-availability-
report/)

~~~
spians
Here's the RSS feed of the blog

[https://github.blog/feed/](https://github.blog/feed/)

I am using
[https://github.blog/category/engineering/feed/](https://github.blog/category/engineering/feed/)
for engineering category

------
ashishb
What if we had a smart failover. Use GitHub and GitLab simultaneously. All
issues, all comments, all PRs duplicated. If one goes down you use the other
one in the time being with no interference at all. One can probably then build
a frontend which magically does this failover for ci/CD etc. Isn't that's how
much redundant this should be?

------
bamboozled
I guess it's time Github realizes that they're not longer just relied upon for
git, but for so much more.

------
chvid
Cool thing about git is that it is distributed and there is no single point of
failure ...

~~~
Symbiote
It is. We all have our own copy of the repository, and can still distribute
changes using any of the other methods:

\- A different central server

\- Email

\- A shared on-filesystem copy, e.g. local network drive

\- HTTP or SSH between developer computers (put your repository somewhere
where your NginX or Apache serves it, the other developer can "git remote add
chvid
[http://chvid.example.com/repository"](http://chvid.example.com/repository")).

~~~
sgt
Do you really think the average developer using git has any idea about how it
can be used decentralized?

~~~
rovr138
IMO, more people need to ask themselves how their tools work and why they
exist and find the answers to those questions

------
mro_name
Who would really rely on a single, external, free, no-guarantees service and
not have redundancy to tolerate some hours of downtime?

Make github a mirror (at least source-wise) and you can benefit from it's
outreach without being held hostage. Am happy with that e.g.
[https://notabug.org/mro/ShaarliOS/src/master/doap.rdf](https://notabug.org/mro/ShaarliOS/src/master/doap.rdf)
Inspired by [https://indieweb.org/POSSE](https://indieweb.org/POSSE)

~~~
bamboozled
A lot of people pay for Github?

~~~
mro_name
seemingly not enough :-)

------
m0xte
Yay this crashed our Jenkins instance as well.

~~~
YetAnotherNick
Same mate

------
holler
Hopefully it's just them reverting their latest UI changes

~~~
whatch
What's wrong with them? Maybe it's because I do not use project management
features much, but I like new UI more.

~~~
misnome
My main dislikes are slow, lazy loading, and that you can’t read the last
commit messages without extra clicks of the “...” button. Even in pull
requests I’ve seen new commits show up without the message (and clicking the
expand button, it closed itself after a few seconds).

There have been some other annoyances/changes in behaviour that have bugged me
too, but mostly stopped remembering them because am resigned to it now.

~~~
thecopy
>you can’t read the last commit messages without extra clicks of the “...”
button

You can opt-in to this now. It is a preview functionality, i guess it will be
GA in a couple of weeks.

------
INTPenis
I just had to access my Github stars to find an old app I bookmarked. No dice.
Otherwise I've moved all my current projects to Gitlab so Stars and
contributing to other repos are my two most used features atm.

------
Evidlo
Everything seems up now. Does anyone know if Github pages went down too?

------
mxschumacher
in my build pipeline, I query several different package hosts (npm, pypi,
docker-hub etc) and Github/Gitlab. If any of them is unavailable, the build
fails.

What's the best way to keep my own copy of the packages my software needs (and
their dependencies), so that my build process is less fragile? Ideally, I'd
only have to rely on those 3rd party platforms to download new versions or
have them as a backup.

When relying on my own copy of required packages - can I expect much faster
builds?

~~~
benalfarley
I've used Nexus for a while without any issues.

------
quyleanh
I still don't understand people who always mentions to Microsoft's
acquisition. Until the official statement, it isn't Microsoft failure. Don't
blame them.

~~~
lopis
It's not MS fault, of course, but since MS acquired GH, GH has been much more
relaxed. New features are added all the time, clearly not ready for the
spotlight. It gives you a different comfort knowing your daddy is there as a
safety net.

~~~
tester34
But also they have a lot of new customers/users, so maybe the scale's the
problem? I don't know.

------
jschulenklopper
This can be merged with
[https://news.ycombinator.com/item?id=23817794](https://news.ycombinator.com/item?id=23817794)

------
mindfreeze
Another Discussion:
[https://news.ycombinator.com/item?id=23817794](https://news.ycombinator.com/item?id=23817794)

------
bezmenov
They’re minutes away from dropping below two nines looking at _Issues, Pull
Requests, Projects_. Other services look comparably unreliable.

------
benbruscella
"We have identified the source of elevated errors and are working on
recovery."

A day wrecker!

------
hhas01
Human sacrifice, dogs and cats living together… mass hysteria!

------
Beldin
It is ironic that a version control system engineered to be distributed is now
typically used in such a centralised way.

------
maxbaines
Savage Unicorn, Github who knew.

------
nicc
People are advocating hosting their own Git repos, but wouldn't those go down,
too, and wreck the day even more?

Or, are you guys all devops geniuses better than those who work at GitHub?

~~~
dnet
But those wouldn't take most of the world's public git repos down all at once
just because of a single issue. Single points of failure have a bad reputation
for a reason.

------
quantummkv
Surely it can't be a coincidence that Github is down every other week after
the Microsoft acquisition? Is Microsoft interfering too much? Or did the core
technical expertise leave for other greener pastures in Microsoft or outside?

~~~
pcr0
I believe GitHub's been adding new features much faster after the Microsoft
acquisition. Moving faster can lead to breaking things more frequently.

