Hacker News new | past | comments | ask | show | jobs | submit login
GitHub Status – Incident on 2019-03-12 (githubstatus.com)
57 points by fniephaus 12 days ago | hide | past | web | favorite | 44 comments





I’ve been hearing from friends at GitHub and MS that things are on fire internally. The GitHub employees are in semi-revolt because they think Nat is integrating too fast with MS. Many people have quit or are being pushed out. It may actually be for the best because the post-founder/pre-buyout team was pretty mediocre. Nat is also trying to crack down on some of the more overt activism in the company. They do “eq” screenings for political fit during the interview process and he’s trying to get them to stop. They also have a brand new head of product and engineering. Product was taken from Jason Warner about two months ago and placed directly under Nat. Sounds really bad and I don’t expect much turn around.

> They do “eq” screenings for political fit during the interview process and he’s trying to get them to stop.

This sounds crazy to me. My last company would freak out if the dev team got together and discussed the person's politics and if we thought they should be hired or not based on that. I hope they do stop this.

The more i think about this the crazier it sounds. Imagine a Houston oil consultancy firm discussing a candidate who applied and asking "when we talked about lowering taxes, did the inflection in his voice sound real or feigned?". This is what an "old boys club" looks like.


"the person's politics"

I interpreted it as a how a person deals with internal politics not what their political views are. the latter would most definitely be illegal.


from the OP > It stands for “emotional quotient” and they ask you a bunch of questions about diversity, feminism... They watch your responses for tone and vocabulary and can veto your hire if they deem you insufficiently enthusiastic.

It sounds very much like your opinions on current political topics.


I don't think the latter* is illegal because it's not discrimination against a member of a protected class. That doesn't make it okay or that it couldn't lead to other trouble.

California law provides some protection from discrimination by employers based on political affiliation or activities. NY state and Washington DC have similar statutes.

But it’s not “affiliation” nor “activities.” It’s “is their stance on topic X compatible with ours?”

Still terrible. Still not illegal.


Hi. I don't know of any fires, revolts, or crackdowns. GitHub is independent of MS, and Nat has worked hard to keep it that way. We do have org changes, but I don't things get "taken" so much as "we work together to figure out an org structure that will make us most productive and build a great product".

FWIW I'm very happy at work


>FWIW I'm very happy at work

Aaron, Serious Question; When were you ever unhappy? I am getting the sense Sadness and Unhappy don't exist in your dictionary.

And first time I see a reply without any pun. ( Or Did I miss it ? )


I know several people who have quit recently and more on the way. Can you comment on Nat wanting to do away with the EQ interviews?

> They do “eq” screenings for political fit during the interview process

Wait, what? How is that even legal? Not sure what's the exact meaning of "eq screenings" but anything related to political affiliation should be out of any interview process. At least in Europe discrimination on political views or other belief is completely illegal (with some exception for the church in some countries, e.g: Germany).


It stands for “emotional quotient” and they ask you a bunch of questions about diversity, feminism... They watch your responses for tone and vocabulary and can veto your hire if they deem you insufficiently enthusiastic. The motto is: “eq > iq”. You can imagine why MS might not be keen on this.

This sounds kinda scary. I don't want to have a racist or a misogynist in my team or company but this seems to go way too far.

I think this person is trying to make it sound scary.

Interview responses are monitored for tone and vocabulary everywhere. Insufficient enthusiasm is often a valid reason not to hire someone. It is appropriate to ask questions about diversity and feminism during an interview—those are core values of the company, the first bullet point on their Careers page, even. https://github.com/about/careers


> It stands for “emotional quotient” and they ask you a bunch of questions about diversity, feminism...

Are you saying github is asking questions about diversity and feminism in their hiring process or that an eq test has these questions ?

Because political opinions have nothing to do with eq testing (see http://myframeworks.org/testmyeq/).


Yes they ask about diversity and feminism and call that eq. As another poster linked, you can see that’s their #1 cultural bullet point: https://github.com/about/careers

So they basically give a political Voigt-Kampff test as part of their interview process?

Diversity and equal rights are politicized right now, but that does not mean it has anything to do with political alignment or affiliation.

Holy shit?

> Wait, what? How is that even legal?

IANAL, and my info may be out of date.

My understanding is that political ideology isn't a protected group at the national level here in America. Although states/cities can write their own laws that effectively add to the list. It's not a protected group in Washington state either, I believe, but it is a protected class in Seattle: https://www.seattle.gov/civilrights/civil-rights#whatisaprot... .

Microsoft does have some presence in Seattle (Microsoft Store, Microsoft Reactor?) so I could see some of their lawyers being made quite nervous by even the hint of discrimination by politics, even where ostensibly legal.


Sorry but this a 2 hour old account, I think I'll take this with a pinch of salt and advise others to do the same.

Use Gitlab.

Update 1: Info.

Update 2: Same info Again.

Update 3: The same info again phrased differently.

What's the point of giving regular "updates" when you don't add any information? Is it just to tick a managerial box somewhere?


If you're on a flight, do you not appreciate when the pilot updates you, even if he says "We are still investigating the maintenance issue?"

It's funny because if you think it carefully it's like you think they look at the problem for a while and stop until you ask again for updates. haha Like you need to keep asking for them to work on it. Sounds like an oldschool boss expecting you to be on your chair even if that doesn't mean you're working.

No, because there's literally nothing I could do to change the situation...

Some clients will feel ignored, even if you don't regurgitate the same stuff over and over again while there's no new useful information to disseminate (or not had the time to properly write it up).

At my job, I wrote a small slack bot that'll ping the person who is dealing with the status page/customer impact, it'll ask "I'm posting an automated update in 15 minutes, would you like that or do you have a full size one?" That way, in the heat of a fire, nobody forgets about the client's status page updates. It works rather well.


This is probably SLA compliance stuff (stipulations for how quickly they have to provide notice of issues, how often they're obliged to provide updates, and how quickly they'll fix an issue). These stipulations, obviously, don't say much about the quality of the notice or updates.

So, yes--ticking a box (but a box that will matter for securing and maintaining contracts with larger organizations).


Adhering to the letter of an agreement and not the spirit of that agreement is exactly the opposite I'd want to see from an org when I specify and they agree to x-hourly updates during outages.

It might be the opposite of what the people writing the provisions expect, but the quality of the updates will necessarily have at least some correlation with how long the people most-capable of investigating and fixing the problem spend explaining it to people who are less-capable of either investigating or fixing it.

I guess anyone could try to negotiate further stipulations on content of those updates, but I don't think it takes much to make the requirement so absurd that no one who knows what they're doing would agree to it.

Unless the provider is clairvoyant/crazy enough to predict when the issue will be resolved, there's very little actionable signal here beyond a list of what's down, acknowledgement that the provider knows and is working on it, and the eventual recovery notice.


This could be just a coincidence, but just now I also noticed HN was down as well as several global services (non-tech related stuff) we integrate with that are all timing out. Some deeper infrastructure issues at hand?

Odd side effect -- I've received three duplicate 'Github Explore' emails about every hour this morning.

Yep. Can't view newly open PRs, also random GH related failures in CI

Travis CI git clone over http is borked as a result. https://www.traviscistatus.com/

I wasn't getting 2fa notifications either, haven't tried lately but switched to google authenticator as a precaution against this happening in the future.

Does anybody else feel GH has a much higher than normal incident rate?

Unless they under-report, we shouldn't need to feel anything; we can just check https://www.githubstatus.com/history and objectively look at frequency. Of course, some services do under-report; dunno if GH does or not.

The overwhelming majority seem to under-report these days. I can’t tell you how many conversations I’ve had that go like this:

Me: We’re having issues on our site, and we’ve narrowed it down to your service. Can you confirm that you’ve had a partial outage for the last three hours?

Them: Yes, about 25% of API calls are failing.

Me: Why is your status page green?

Them: Our team has decided it doesn’t warrant a status update.

When this happens, we tend to switch services providers fairly quickly. If you’re not going to be honest with us, we’re stuck wasting our time troubleshooting. We don’t care if you have occasional outages—it happens; that’s why we have fallback providers. But you need to tell us when you’re down.

tl;dr: Your status page is not an advertisement for your “amazing” uptime. It’s to make my life easier and save me time/money. Use it for its intended purpose.


Depends on what you mean by normal but compared to many of the status pages I am tracking, GitHub has a pretty low incident rate (though an uptick very recently): https://statusgator.com/services/github

Nah, seems pretty normal. Compare to Netflix, Hulu, Heroku, etc, which have some outages nearly daily. This was just degraded performance for two hours, and the only more serious one I can recall was pretty recent.

dumb question but, why can't the teams of these super large companies who are probably getting paid top dollar get their deployment cycle right down to the point where they aren't causing interruptions?

Deploy to github pages delayed for more than an hour here

that's a shitty incident report and a sad thread if there's this kind of activism internally while shit is on fire.

Seem working alright... hmmm



Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: