A bit weird. GitHub says they fixed it, but on the other hand CircleCI still considers it as an outage :
> Monitoring
> May 31, 2017 3:08 PM
> GitHub have declared the outage resolved and we are starting to see incoming GitHub hooks. Builds are being triggered again. However we are still seeing failures with the GitHub API. This continues to prevent our webapp > from fetching data from GitHub. We are monitoring the situation and will ensure sufficient capacity for when their service resumes normal operations.
Nitpick: www.statushub.com is less obviously related, and therefore won't be attacked in tandem. If they want to go all the way, maybe just something like www.wesellnikesdiscount.com.
I've seen that, but I've chalked it up to clock skew on the client. It only seems to happen on one of my machines, and only after a couple months of uptime.
Ah, yes. I hadn't looked at it, but now I see that fixed date/times are in the html source, and the "x hours ago" messages are rendered in the browser.
I've noticed this behavior with a lot of services.
I can only chalk it up to something like clock drift between the processing node and the database server.
Irritatingly I can't remember which site it was but I posted something somewhere a couple days ago and immediately after hitting enter the site marked what I'd said as submitted "a few seconds from now". I never fail to be amused that the fuzzy time library being used has code specifically designed to handle this edge case scenario. :D
I can totally agree about relative timestamping in both directions (past+future) - my argument is more about the UX of situations where you're canonically referring to a past event.
Git timestamp can be set with different timezone so technically GitHub can analzye the timestamp (which they do) and compare with the actual clock of timezone. There's only a small problem though: the latency from the clock check server must be low otherwise we would be 1 second ahead by the time we get response and then another second or two after webpage render.
So from a UX we should simply discount time drift +-5 seconds simply says "blah blah just now" and anything larger might warrant as a warning to the author (Github can reject the commit and ask for confirmation). Although re-editing commit is a hassle for changing timestamp. It's more of a "please make sure you computer time is synced next time."
Wow, interesting. This info is definitely filed away, thanks.
I don't quite remember but I think I might've been commenting on something on GitHub when I saw the time glitch.
Initially for a moment I thought "why not just have OCD local NTP tracking?" but then I realized that time glitching around (even at the millisecond level) can be disastrous. One way to solve this is to obsess about keeping up to date with NTP, but instead of updating the time, update a global reference to the offset. Then your time server is simply (system date)+-(saved offset), which should be super fast. And of course this can run on the node generating the HTML.
While parent object ids are provided on the command line, author and committer information is taken from the following environment variables, if set:
GIT_AUTHOR_NAME
GIT_AUTHOR_EMAIL
GIT_AUTHOR_DATE
GIT_COMMITTER_NAME
GIT_COMMITTER_EMAIL
GIT_COMMITTER_DATE
(nb "<", ">" and "\n"s are stripped)
In case (some of) these environment variables are not set, the information is taken from the configuration items user.name and user.email, or, if not present, the environment variable EMAIL, or, if that is
not set, system user name and the hostname used for outgoing mail (taken from /etc/mailname and falling back to the fully qualified hostname when that file does not exist).
A commit comment is read from stdin. If a changelog entry is not provided via "<" redirection, git commit-tree will just wait for one to be entered and terminated with ^D.
DATE FORMATS
The GIT_AUTHOR_DATE, GIT_COMMITTER_DATE environment variables support the following date formats:
Git internal format
It is <unix timestamp> <time zone offset>, where <unix timestamp> is the number of seconds since the UNIX epoch. <time zone offset> is a positive or negative offset from UTC. For example CET (which is
2 hours ahead UTC) is +0200.
RFC 2822
The standard email format as described by RFC 2822, for example Thu, 07 Apr 2005 22:13:13 +0200.
ISO 8601
Time and date specified by the ISO 8601 standard, for example 2005-04-07T22:13:13. The parser accepts a space instead of the T character as well.
Note
In addition, the date part is accepted in the following formats: YYYY.MM.DD, MM/DD/YYYY and DD.MM.YYYY.
You see I got a 3 minute (using local time) and then 4 hours ago because I set the timestamp manually in the most recent commits. So yes you can set time in the future / past.
Message queues and eventual consistency. Unless your request requires something "atomicy", 200/201 response should be a sign of "got the message, will get to work on it when we can".
201 is "request has been fulfilled, resource has been created". It is explicitly not "we will get to it when we can". You are thinking of 202, "request has been accepted for processing", for asynchronous request processing.
I assure you, 201 is used all the time to say something has been done when it's only been queued, regardless of what the RFC says. This is based on real world integration experience.
Http response status codes rarely align with RFC guidelines.
Actually, most of the "relative time" JS libraries can be used for countdown as well ("this event is scheduled in two hours"). Accidentally getting a timestamp that's supposed to be in the past shouldn't break it :)
I just think there should be an intent argument specifiable to the library to indicate that the duration in question refers to an event that has happened in the past.
In such a scenario, the library should mark the duration as happening "just now" and possibly flag a warning or raise an exception.
The reason I say this is that, humanly speaking, "Your message was sent 23 seconds from now" is amusing at best to developers who know what's happening (negative time delta) and linguistically confusing to general users ("was sent" vs "from now").
Hmm. You pay them to uphold a contract. What does that contract say about SLAs and availability? Probably the same as the TOS that I agreed to when paying and those specifically say:
GitHub does not warrant that the Service will meet your requirements;
that the Service will be uninterrupted, timely, secure, or error-free;
that the information provided through the Service is accurate, reliable or
correct; that any defects or errors will be corrected; that the Service will
be available at any particular time or location; or that the Service is free
of viruses or other harmful components. You assume full responsibility and
risk of loss resulting from your downloading and/or use of files,
information, content or other material obtained from the Service.
If you negotiate, you might get better terms and guarantees, for example with github enterprise. You might also have to pay substantially more for those.
I understand, it sucks when github is down. But we all get what we pay for and we all don't want to pay for more. And yes, I do have clients that meticulously mirror all their dependencies from outside sources and spend significant money on this - money that pays off in exactly these situations.
Huh? You don't have to use Github Enterprise (self-hosted) to get an SLA. Github Business, which is hosted on github.com has a 99.95% uptime SLA: https://github.com/pricing
An upgrade from Team to Business is "only" a 2.3x price bump per dev. I have no experience with this though, my team is still of the Team plan and thus suffered from the outage today.
Huh indeed. 99.95% uptime -- AKA: three and a half nines. My quick math tells me that 99.95% uptime equates to a downtime of ~4:23/yr. If github is down for an hour once every few months, I'd say they're likely well within their stated SLA.
My problem with a credit is that it never even comes close to what I'm losing in income. An ISP is an excellent example. I might get a $10 credit for 24 hours of downtime. I'm charging slightly more per hour than that... /s
Maybe switch to bitbucket or other competition for a while?
If the price of the service working (or not working) is disproportionately large compared to the price of your lost business, that's a problem at your end: you needed to calculate the risk vs return for redundancy.
Eg, if your Internet costs $100/mo, but you'd lose $100/hour when it's down during business hours, buy a fallback connection from a competing ISP. ;)
Infrastructure so often becomes a monopoly. I can't pay a competing bridge service to drive to work quicker, I can't pay a competing gas company to deliver gas via different pipelines to my house. And I can't pay a competing electric company that uses different wires.
I actually am lucky enough to live in a city where there are many competing high speed ISPs. But guess what? I've paid for fallback connections in the past and when one goes down, the other goes down, so I go out to lunch and see the guys working on the wires in the cabinet down the street. The wires that both my ISPs share. I suppose I could get a satellite ISP? That latency. True redundancy for infrastructure is actually very expensive in most cases.
Funny thing is Github sometimes makes more sales after an outage because clients want to upgrade to the enterprise edition to host on their own servers.
You sure it isn't zero downtime deployment? But I thought Github runs infrastructure globally? I remember some outage were caused by DoDS, and some were software bugs / bad config.
Probably good idea to do rolling deployment. I will be surprised if they haven't for the kind of top engineering team they are running.
Business idea: github hosting failover. You'd probably need a modified git client, but if you can't push/pull/whatever from github, it transparently fails over to your service which will sync up with github once they've recovered.
A couple times I have configured multiple "remotes" for my local git repo and pushed to both, e.g. GitHub + Google Cloud Source Repository, or even just a bare repo on a VPS.
Was receiving random downtimes when I've tried opening a certain project and its wiki ~1 hour ago. Nothing big and a couple of refreshes fixed it, just minor annoyance.
This is the biggest cop-out of a reply. I hate it. OP has already stated:
>I pay them. My work pays them.
Github is raking in oodles of cash and they STILL can't keep their service up without going down, quoted from OP, "[e]very couple months".
It's not about "making a better one", nor is it about paying for the fancier/premium features; it's about the uninterrupted service, which Github keeps failing to provide.
I fail to see what the problem is. If you don't like their service, then switch to a competitor, or just set up your own git server. It's like this for any vendor: if you don't like the product or service you're getting, you can either bitch and complain endlessly, or you can look for alternatives. One of these choices is more productive than the other.
I think hindsight is 20/20. They obviously did not think they needed to find a competing service; nor build one as they were paying for one. Given that they have been inconvenienced by downtime they may switch or make their own.
> Feel free to make a better one
The GitLab team did that already so I use their service. ;)
We have a pretty detailed post-mortem on the timeline of events for the outage, as well as actions taken after it to prevent anything similar in the future (with links to related issues).
You can check it out in https://about.gitlab.com/2017/02/10/postmortem-of-database-o...
They haven't lost my data so far and they had to restore the production database at least once in their history. All circumstantial, but we'll have to wait and see.
Gitlab does not have features parity with Github. I personally also stumbled upon a bizzare bug that doesn't allow a friend to add me to any of his private repo's. Asked on IRC and nothing much came of it unfortunately.
Err... you're responding to someone who very clearly said they hit a (serious for them) bug which needs fixing. That's definitely not a feature proposal. :D
> Monitoring
> May 31, 2017 3:08 PM
> GitHub have declared the outage resolved and we are starting to see incoming GitHub hooks. Builds are being triggered again. However we are still seeing failures with the GitHub API. This continues to prevent our webapp > from fetching data from GitHub. We are monitoring the situation and will ensure sufficient capacity for when their service resumes normal operations.