
GitHub Outage Currently Ongoing – Issues, PRs, Notifications - dazbradbury
https://www.githubstatus.com/?date=22082019
======
yjftsjthsd-h
Alright, cue the usual:

Monoculture bad

Microsoft bad

Git is distributed but all the things around it that we really need aren't

You can set up git to push to multiple remotes automatically

Nobody is actually using git in distributed mode

Did I forget anything?

~~~
jolmg
> Git is distributed but all the things around it that we really need aren't

Can anyone recommend an issue tracker that is distributed with the repo?

~~~
Griffinsauce
@TODO comments strewn around in your repo.

~~~
jolmg
I was thinking of something that wouldn't be affected by the git history.
There's no point in having branches for issue comments.

Something like:

    
    
      $ git issue add "foo doesn't work"
      Created issue ece5591: foo doesn't work
      $ git issue comment ece5591 "the problem might be with bar"
      Created comment 0191fa1 on issue #10.
      $ git issue comment --reply 0191fa1 "or with baz"
      Created comment f98e783 on issue #10.
      $ git issue show ece5591
      foo doesn't work -- ece5591 Your Name <your@email>
        the problem might be with bar -- 0191fa1 Your Name <your@email>
          or with baz -- f98e783 Your Name <your@email>
      $ git issue push
      Pushed issues ece5591.
      $ git issue pull
      Pulled issues 3ebdc7e, 24cdb90.
    

EDIT: I just gave myself a clue:

[https://github.com/dspinellis/git-issue](https://github.com/dspinellis/git-
issue)

~~~
WorldMaker
The nice idea to having issues in your history/branches is that it easily
inverts the "flow" of branch state information: most issue trackers do a lot
of work to figure which changes affect which issues, and which branches those
changes are in. This often requires a lot of explicit manual work like making
commit notes always include issue numbers for instance and "magic" commands
like "Resolves #10" to auto-close issues.

If the issues are just another source artifact alongside the code in the same
git history a lot of that information flows the other direction. Instead of a
magic command in a commit note to close an issue, an issue closing shows up in
a diff in the commit. Figuring out which issues are still open in a given
branch (what isn't in our release branch yet?) is a simple command in a
branch. Figuring out which issues changed between branches (what's finished in
this feature branch that hasn't been merged to release yet?) is a typical diff
operation.

In terms of documenting the state and progress of an individual branch in a
complex project there can be a lot of magic in having issues tracked directly
inside code branches. It's a fascinating ideal for code documentation to have
issue comments, changes, and workflows side-by-side the code that changed with
it.

That said, issue trackers are sadly not ideal code artifacts for a lot of
reasons, including that it is easy to forget that issues aren't just for
coders, but also stakeholders/product owners/PMs/etc.

------
jmisavage
I kind of wish HN wouldn't show these outages unless they're going on for
multiple hours or at least hide them after they're back up. Usually by the
time I see them and then check myself the site is already back up. Maybe I
just need to read HN more often.

~~~
djsumdog
Yea, there are a lot of outage posts. It does kinda illustrate how many people
depend on these core services though; and how little distributeiness or
elasticity is built into the system.

Linux repos and ISO download sites have TONS of mirrors, sometimes contributed
by companies, but also a lot of universities, government and public
institutions.

Why is one of the largest core open source repositories centralized and
private? Why are we so dependent on AWS, Google and others for large chunks of
Internet services to be accessible?

There is a deeper question here than just the outages; but rather why they
affect so many people.

~~~
starbugs
I have a theory:

Centralized infrastructure tends to be less complex and therefore cheaper.
Economies of scale drive people into using a centralized monolith and then
network effects do the rest.

If you want a decentralized system to sustainably compete with a centralized
one, the decentralized system must provide tangible and obvious surplus value
in comparison to the centralized one or otherwise it is not economical.

In reality that's often not the case. Usability is often worse or
infrastructure is in favor of centralization (e.g. asymmetric broadband
internet connections).

------
krzkaczor
Events like this make me realize how much I rely on my CI/CD pipeline for
working on new features, deploying etc. I am often too lazy to run E2E tests
on my machine if I know that on CI it will take only a few minutes to be done.

~~~
prepend
I feel guilty because I’ve never installed GitLab’s CI process locally and
have only run it for the past few years through git commits. I keep expecting
someone to complain, but so far, so good.

I expect similar poor behavior testing GitHub’s actions because it’s so
convenient to only run through the service.

It is important though that it’s possible to run without CI/cd. If it wasn’t
available I would complain because of the risk of lock-in.

------
olah_1
A peer-to-peer (IPFS) stack for code collaboration.

[https://radicle.xyz/](https://radicle.xyz/)

A Radicle project contains a git repository, plus the associated issues and
proposals.

^ Neat project

------
alexeiz
Funny thing, I didn't even notice. In fact, being heavily reliant on Git and
Github, I encountered maybe one or two issues a year when I couldn't push to
Github because it was unavailable. In those cases, the progress can still be
made locally, my workflow was barely affected. Of course, the deployment was
impossible, but the downtime was never significant enough to worry or do
anything about.

------
smartmic
Git is designed as distributed VCS, so an outage of a centralized server
should not matter too much. So far, the theory. Using Github excessively is
the way back to centralization, nowadays handing control over to Microsoft.
Everybody who is concerned (probably not too much) should have a look into
alternatives.

~~~
csomar
Yea because the alternative (hosting git yourself on your own servers) will
have better availability and few issues (UX just came to mind!)

~~~
xtracto
[https://about.gitlab.com/install/](https://about.gitlab.com/install/)

So the UX point is moot. From there, it is just a matter of how much do you
want to pay for availability (multizone replication, etc)

------
scandox
We're self hosting Gogs on a Hetzner VM and it's been a great experience. Of
course it has less features. But it's simple and fast and I do about 10
minutes sys admin per quarter.

------
enlyth
Resolved, all systems operational, that was quick.

------
fortran77
If you don't the the "community" features of GitHub, we have had very good
results with Amazon Code Commit. I've never experienced an outage, it's easy
to manage, and very inexpensive.

[https://aws.amazon.com/codecommit/](https://aws.amazon.com/codecommit/)

------
u801e
I see posts like this make the front page of hacker news every so often, but I
don't recall seeing posts about things like making list outages for something
like the LKML (Linux Kernel Mailing List) or an IRC network like Freenode.

~~~
objclxt
Perhaps because Freenode has about 0.25% the number of users of Github -
~100,000 vs 40 million?

~~~
megous
No, it's just the US tech bubble. When my whole country and some was suddenly
denied all google services for several hours, including CDNs, DNS, all cloud,
gmail, android app store, gdocs, all everything, it didn't make any news here
either, despite being much more disruptive, affecting 10+ million people, and
being a much more intersting reminder of how much techies make other people
and themselves dependent on this megacorp.

All kinds of stuff broke, including "the entire internet" for people who were
made to use 8.8.8.8 by their ISPs. E-shops that load jQuery from google
stopped working, for no real reason whatsoever, people getting lost because
they couldn't navigate anymore in the real world, etc.

Since then I try to flag all this repetitive nonsense about slack/github going
down for a few minutes, to no avail. It keeps comming back to the first page
of HN. :D

~~~
marcinzm
People care about what impacts or relates to them. That's human nature. Even
your example was "my country" which is something that relates to you.

~~~
megous
That being said, being dependent on Google is not unique to my country, and
almost everyone should relate to that, at this point in history. And the scale
of dependence only shows when it's truly completely shut down.

------
xchaotic
Justyna yesterday I was complaining about GitHub monoculture in the context of
Atlassian sunsetting Mercurial. Outages like this are normal, but the problem
is lack of good competitors.

~~~
nickpsecurity
The problem is bad design and/or integration of components. VMS clusters over
leased lines went years without outages. Record was 17 years at a rail yard.
They also did rolling upgrades across both different OS versions and _CPU
architectures_. The methods are public if any competitors want to match or
exceed them:

[https://support.hpe.com/hpsc/doc/public/display?docId=emr_na...](https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04622808)

These modern cloud services using Linux ecosystem are doing rolling outages
instead. I'm amazed people keep making them dependencies. Maybe their local
systems using similar technology went down more often. Maybe this is an
improvement. I'd still put mine on an OpenBSD or OpenVMS cluster if it was
business-critical with lots of money on the line. I want it staying up, up,
up. :)

~~~
mugsie
Modern apps are designed to deal with part of them being down - it is the core
upgrade strategy.

VMS clusters were (and modern equivalents are) awesome, but they have a
different speed of innovation / development.

It is a trade off - perfect uptime, but that new feature could take a year, or
99.9% uptime, but it can be written and deployed tomorrow.

~~~
nickpsecurity
Their development velocity might have been due to their management or internal
culture. You can get most of those results faster and on a budget. The best
evidence was the startup behind FoundationDB continually improving their
product in highly-competitive space while doing QA like this:

[https://www.youtube.com/watch?v=4fFDFbi3toc](https://www.youtube.com/watch?v=4fFDFbi3toc)

Generally, on top of code reviews, one can get high return with minimal labor
and hardware with a combo of Design-by-Contract, contract/spec/property-based
testing, low-F.P. static analysis, and fuzzing with contracts in as runtime
checks. That's my default recommendation.

In Github's case, they also might have access to both closed and open tools
that Microsoft Research makes. MSR is awesome if you've never checked them
out. Two examples applicable to system reliability and security:

[https://www.microsoft.com/en-
us/research/project/ironclad/](https://www.microsoft.com/en-
us/research/project/ironclad/)

[https://www.microsoft.com/en-
us/research/publication/p-safe-...](https://www.microsoft.com/en-
us/research/publication/p-safe-asynchronous-event-driven-programming/)

Plus some of their other tools in various states of usability:

[https://rise4fun.com/](https://rise4fun.com/)

------
cproctor
I've been having trouble getting GitHub to send me a verification email (after
adding a new email address) for over a week. Anybody else having this problem?
I wonder if it's connected.

------
knorker
Also still ongoing: They don't have IPv6.

------
kome
it's time to switch to SourceForge.

~~~
sergiotapia
I've migrated all of our repos to codeplex.

------
justinator
What's a, "PR". (I'm being serious)

~~~
janpot
[https://help.github.com/en/articles/about-pull-
requests](https://help.github.com/en/articles/about-pull-requests)

~~~
justinator
Thank you, of course - I was wondering if PR was the same PR as in Agile.

