
Jenkin developers accidentally do "git push --force" to over 150 repos on github - jordigh
https://groups.google.com/forum/#!searchin/jenkinsci-dev/force$20push/jenkinsci-dev/-myjRIPcVwU/mrwn8VkyXagJ
======
tjdetwiler
Githubs response:

Good news from GitHub: they have extracted the full list of SHA-1 before the
forced push ! Many thanks to Nathan Witmer :-)

See below the full CSV with the SHA-1. He created as well a branch named
'recovery' that points to the candidate point for restoring the master branch.

Hope this will help to sort out the remaining repos.

Luca.

    
    
      > Hi Luca.
      >
      > Oh man, that sinking feeling!
      >
      > But, no worries: I've gone through each of the repos  listed above and done the following :
      >
      > * retrieved the SHA for the previous `master` before you force-pushed
      > * created a branch called `recovery` pointing to each former master
      >
      > In some cases, these are the same.
      >
      > I can go further and reset the master refs to their former shas if you'd like, or you can recover these yourself. To do so, in each repo:
      >
      > $ git fetch
      > $ git checkout master
      > $ git reset --hard origin/recovery
      > $ git push --force origin master
      >
      > I've attached a csv containing the shas (former master and forced master) for each branch, for your reference.
      >
      > Good luck!
      >
      > Nathan.

~~~
lightyrs
GitHub represents a new ideal in customer service. I hope other companies will
begin to follow their example.

~~~
gcb0
why couldn't they do those steps themselves? ...i'm not really familiar with
github's git.

I am having a really hard time understanding why a distributed VCS with
hundreds of users needs the admin to the central repo to do anything for them.

~~~
cjh_
The central repository is the only point that _knows_ (with 100% certainty)
which ref the branch was set to before the force push.

Each distributed node only knows the ref the branch was pointing at the last
time they fetched, so their certainty on what the branch looked like before
the force push is much lower.

~~~
deathanatos
> The central repository is the only point that knows (with 100% certainty)
> which ref the branch was set to before the force push.

git reports it to you when you do the force push:

    
    
      % git push github master --force
      …
      Total 0 (delta 0), reused 0 (delta 0)
      To https://github.com/foo/repo.git
       + 4d44b63...ad5b147 master -> master (forced update)
           ^          ^
         previous    what you've forced it to
    

Of course, if you lose the output, then yes, the reflog is the only thing that
has it. But force pushing is so rare, and something that (should) is done with
care, I'm puzzled as to how someone "accidentally" >150 repos.

~~~
chatman
If losing this output is such a problem and force pushing is so rare, then
Github could email this output with the two hashes to every developer on every
force push.

~~~
swift
This seems like a good idea. (With a setting to turn it off, of course.)

Ideally git itself should record this information somewhere, though.

------
nathanb
Although the responsible developer's reaction and attitude are both
commendable, one element of his response annoyed me: his continued assertion
that he should not have been allowed to do this thing that he did.

I think it is a truism that systems which allow users to do interesting and
clever things must also allow users to do remarkably stupid and wrong things.

Rather than focusing on how to prevent a user from doing a silly thing like
this, I think a well-designed tool would easily allow the user to undo the
silly thing he just did.

Unfortunately, all too often when a Bad Thing happens, the discussion tends to
center around how to prevent this Bad Thing from happening again. This is
often why those who make mistakes end up being demonized; their mistake has
now limited the actions that every other user of the system can make, because
the system will be modified to prevent other users from making the same
mistake.

~~~
munificent
> I think a well-designed tool would easily allow the user to undo the silly
> thing he just did.

Agreed. From a usability perspective, undo is always best. Especially undo
that the user: 1) knows about, 2) trusts, and 3) is fast. That encourages
exploration and lets users fix mistakes.

> their mistake has now limited the actions that every other user of the
> system can make, because the system will be modified to prevent other users
> from making the same mistake.

I don't think this is a black and white issue. It's not about denying all
users the ability to express X ever again. I think it's more about having
systems that can tell if X is an unusual or heavyweight action and say, "Hey,
you're about to do X, which impacts a lot of stuff and you've never done
before. Are you sure that's what you mean?"

It's velvet rope permissions, not a locked door.

~~~
icebraining
_I don 't think this is a black and white issue. It's not about denying all
users the ability to express X ever again. I think it's more about having
systems that can tell if X is an unusual or heavyweight action and say, "Hey,
you're about to do X, which impacts a lot of stuff and you've never done
before. Are you sure that's what you mean?"_

Yes, but git already does that, by having the user type "\--force". Asking
would eliminate the possibility of scripting the action.

~~~
davvolun
That's not entirely accurate, e.g. "git push --force --yes" would still be
perfectly scriptable.

~~~
bereft_orange
how far does that go though? `git push --force --yes --yes-I-am-absolutely-
sure --by-pushing-this-commit-I-agree-I-am-paying-attention`. Seems like
`--force` is a succinct way to cover those two parts.

------
mercurial
That's some facepalm right here. That said, I do appreciate the reaction of
the developer, taking responsibility for his mistake, starting remedial
actions right away and discussing preventive measures for the future.

~~~
zygomega
And ditto the reaction of the community. No aggressive denunciations or
vindictive chat - just calm and helpful trouble shooting. +1 for the culture
they have created.

~~~
pekk
What are you supposed to do if jerks join your community? Excommunicate them?

~~~
sylvinus
Messing up with git and apologizing publicly is hardly being a jerk IMHO.

~~~
Semaphor
To clear it up, it was a question wondering how one would keep a community as
jerk-free as the one they seem to have right now.

~~~
sylvinus
right, thanks!

------
jrochkind1
Nobody's yet commented on... what is it you can do to force push to ~180
different git repos at once?

Normally, it would be difficult to do this even if you tried, no? 180 repos
would usually be in a 180 different clone directories; what can you do that
somehow force pushes them all at once, even ones you didn't mean to be working
on at all?

In the thread, the person who made the mistake seems to suggest[1] it has
something to do with a 'gerrit replication plug-in'? I am not familiar with
gerrit. Anyone?

Mistakes will happen; but for a mistake to have this level of consequence took
amplification, apparently by the 'gerrit replication plugin'? I'd be inclined
to blame that software, rather than human error (or github permissions or lack
thereof! If you have write access to a bunch of repos, of course you can mess
them all up, no github permissions will ultimately change that. The curious
thing remains how you manage to mess up a couple hundred of them in one
click).

[1] [https://groups.google.com/d/msg/jenkinsci-
dev/-myjRIPcVwU/v8...](https://groups.google.com/d/msg/jenkinsci-
dev/-myjRIPcVwU/v8PWC60lNEIJ)

~~~
StefanKarpinski
> If you have write access to a bunch of repos, of course you can mess them
> all up, no github permissions will ultimately change that.

This is simply not the case. Force pushing is literally the _only_ operation
you can do via git protocols that have a destructive effect on the remote
repo. Moreover, it is generally considered best a practice _never_ to force
push public branches. Thus, preventing forced pushes on branches that are
considered public would, in fact, prevent any sort of real destruction to a
repo. People could still push new garbage to it, but they can't make old
history inaccessible.

~~~
wereHamster
Deleting a branch is also a destructive operation last time I checked. Force
push could also be implemented by deleting the branch and pushing a different
SHA1 into the same name.

~~~
StefanKarpinski
Yes, that's true. Protecting a branch from deletion would be another nice
feature and could be tied to denying forced pushes. Both can be construed as
the branch being "public".

------
StefanKarpinski
Maybe a high-profile incident like this will finally convince GitHub to let
you disallow force pushing to specific branches in repos – like your project's
master branch. I asked about this years ago and iirc was told there were no
plans to implement such controls.

~~~
npinguy
The issue isn't force-pushing.

The issue is pushing to multiple branches.

This is a local developer's Git issue, not a Github issue:
[http://stackoverflow.com/questions/948354/git-push-
default-b...](http://stackoverflow.com/questions/948354/git-push-default-
behavior) but one they've thankfully changed:
[http://stackoverflow.com/questions/13148066/warning-push-
def...](http://stackoverflow.com/questions/13148066/warning-push-default-is-
unset-its-implicit-value-is-changing-in-git-2-0)

Summary: By default Git push used to (they've changed it) push ALL your
branches, not just your current one. It is pretty much the first thing I've
historically reconfigured whenever starting on any new project. This guy
didn't do it, and paid the cost.

~~~
StefanKarpinski
No, the issue _is_ force pushing. While it's nice that git changed the default
behavior to be far less dangerous, people with push access can still force
push – be it one branch at a time in newer gits or all branches in older ones.
It's all fine and well to tell them to change their settings and/or ask them
please don't force push, but people make mistakes. Hell, I want to prevent
_myself_ from accidentally doing this.

Git let's you prevent this easily using the receive.denyNonFastForwards
option, but GitHub won't let you control this option. So we're left just
telling people not to do it and hoping that they listen _and_ don't do it by
accident. That seems like a shitty, precarious situation that could easily be
fixed. One checkbox on the GitHub repo settings page and this story would
never have happened.

[1] [http://stackoverflow.com/questions/13016119/how-to-set-
the-r...](http://stackoverflow.com/questions/13016119/how-to-set-the-receive-
denynonfastforwards-on-a-repository-in-github)

~~~
npinguy
Force push is never a mistake.

You have to explicitly add the force flag, and if you choose to go down that
path, then you better be sure you know what you're doing. If you don't, but
still manually setting that flag to fix the error that your push failed? You
deserve the rain of hellfire that will come down upon you if you blow away
other people's commits.

I know about denyNonFastForwards, and unfortunately I work at a company that
turns this on as a rule on ALL branches, which I don't like because I like
having personal branches on the server (rather than on my personal git repo).
As a result when I want to squash my commits I end up rebasing interactively,
deleting the remote branch, and then creating it again.

~~~
StefanKarpinski
I guess you've never written an automated tool that works with remote
repositories and sometimes has to force them. It's quite possible to
accidentally force push a branch that way – and that appears to be _precisely_
what happened here. Sure, he maybe could have been more careful and avoided
the mistake, but GitHub could incredibly easily allow people to configure
repos so that this mistake _can 't_ be made. Your argument boils down to
"don't make mistakes", whereas my point is that in this case it would be so
easy for GitHub to give repo admins the ability to prevent this entire class
of mistakes.

[Note that using an explicit branch name, as has been suggested here, doesn't
really help since you can easily accidentally force push a branch that way
too.]

The fact that your company has a draconian blanket policy is unfortunate, but
has no bearing on whether or not people should be allowed to _opt in_ to
setting denyNonFastForwards on specific branches.

------
jessaustin
Christopher Orr on the list suggested the following, which makes a great deal
of sense to me:

 _One thing I sometimes do for repos where I have commit access, but don 't
want to push, is to clone the repository via HTTP. That way, any accidental
attempts at pushing won't use my ssh key, and so GitHub will prompt me for my
password._

~~~
encoderer
You can also just override the push URL for any remote, then have a separate
remote for pushes. So you'd have like "origin" for pulls only, and
"pushorigin" for pushes.

------
matwood
Switch to Bitbucket. It is easy to selectively disable history rewriting and
branch deletion on a per branch basis. I've been told that Github can also do
this, but you have to email them.

~~~
mbell
Alternatively use git the way it was meant to be used, as a _distributed_
source control system. It appears that they had _678 people_ sharing access to
these repos instead of working off their own forks. I'm amazed they haven't
stepped on each others toes previously with that structure.

~~~
saidajigumi
> Alternatively use git the way it was meant to be used, as a distributed
> source control system.

Let's not be disingenuous. This is exactly what every git user does when they
run `git clone`. You aren't railing against just multiple committers per
public repo, you're railing against multiple _clones_ per public repo. One
user swapping between two client systems can make this error. Most public
repos of any scale will have multiple committers to spread the maintenance
workload.

Even Scott Chacon himself describes a simple, effective process called "GitHub
Flow"[1] -- just submitting a PR against a branch for merge against master on
the same repo. This is a very common workflow on private repos in my
experience, and a very easy workflow for small teams.

Making all this worse, the default config setting for `push.default` will only
change to "simple" in Git 2.0 -- until then any new host/account that hasn't
had push.default changed in .gitconfig is a bomb waiting for the first
_otherwise legitimate_ `git push --force` to go off. I've received the
batsignal from folks on small teams who've otherwise done all the right things
and still been bitten by this.

Fortunately, git has the reflog[2][3] to save us from this kind of mess. It's
just not readily accessible on GitHub, AFAICT.

[1] [http://scottchacon.com/2011/08/31/github-
flow.html](http://scottchacon.com/2011/08/31/github-flow.html) [2]
[http://gitready.com/intermediate/2009/02/09/reflog-your-
safe...](http://gitready.com/intermediate/2009/02/09/reflog-your-safety-
net.html) [3] [https://www.kernel.org/pub/software/scm/git/docs/git-
reflog....](https://www.kernel.org/pub/software/scm/git/docs/git-reflog.html)

~~~
saidajigumi
Update: there's a critical caveat to remote reflogs:

 _core.logAllRefUpdates_ must be manually enabled on bare repos!

To repeat: the reflog is not available on bare remotes by default unless
core.logAllRefUpdates has been manually set to "true". See [1] for
documentation on that config setting.

[1] [https://www.kernel.org/pub/software/scm/git/docs/git-
config....](https://www.kernel.org/pub/software/scm/git/docs/git-config.html)

------
howeyc
I don't understand, does everyone in the organization have full commit
privileges to all repositories?

~~~
jessaustin
You don't need a release manager, until you do.

------
kohsuke
Man, this is not how we wanted to be on Hacker News...

~~~
emn13
I can only imagine. Good luck! At least everyone's commending your community
on keeping it's cool and making the best of it; so hey, that looks
professional.

------
Aqueous
I'm not sure if it contributed to this fiasco, but GitHub's team system is a
bit backwards and screwed up. You create teams of people, that automatically
have either pull, push/pull, or administrative permissions on every repository
they touch. Then, when you go to the 'collaborators' section, you simply add
those teams of people, without being able to see what access level you are
awarding to that specific repository! It makes no sense. Either the
permissions should be RIGHT next to the team name in the drop down or (much
better) GitHub should decouple groups from the permissions they have, and
permissions should be awarded on a per-repository basis. That you way you are
granting X group of people Y permissions on Z repository. You are always
explicitly granting a permissions level to a specific group of people, and no
permissions are automatically added that you haven't explicitly signed off on
for that repository.

Maybe there's some logic behind it, but I doubt it, and I'm really not sure if
GitHub completely thought it through.

------
blktiger
We've been using the prevent force push hook for Atlassian's Stash on my team
to avoid situations like this. If you really need to force push, you can get
around it by deleting the branch and re-creating the branch.

~~~
jlgreco
I think this is the best setup to run with. There really are few cases where a
non-fastforward push to a public repo is called for.

------
pnathan
Ouch, that hurts. On the plus side, they are looking into process fixing so it
doesn't happen again. Not only that, the flameware level is zero (at this
point). Major props to the Jenkins/Cloudbees people!

Further, because git is a DVCS, so long as someone has a valid recent clone,
the pain is much less than, say, a centralized repo being corrupted.

------
runn1ng
Can someone explain to me what exactly happened?

I don't really know what Jenkin is and what "git push --force" does.

~~~
parkrrr
Jenkins is build server software. It's existence is related to the story only
because it was a Jenkins developer.

'git push --force' pushes your copy of the repository to the remote,
regardless of what the remote has. He basically told GitHub, "Hey here's the
/real/ copy of the repository, the copy you have is totally wrong" and it
overwrote all the history because his copy was a few months out of date.

~~~
runn1ng
Oh. Ovewriting history magic.

One part of Git I never really liked.

------
jonico
Force pushes are dangerous but turning them off completely prevents some valid
use cases like pruning problematic IP or removing large binaries accidentally
committed to a git repo. For those reasons, CollabNet ships with a Gerrit
plugin that does not prevent history rewrites but records them in an immutable
backup refs, sends emails and logs it properly. Recovery can be done on a self
service basis and administrators can still delete those backup refs forever.
The plugin also treats branch deletions like that. More details can be found
on [http://luksza.org/2012/cool-git-stuff-from-collabnet-
potsdam...](http://luksza.org/2012/cool-git-stuff-from-collabnet-potsdam-
team/)

------
kohsuke
By the way, it is "Jenkins", not "Jenkin".

------
Radim
An unfortunate accident. Makes me wonder though -- what is the worst a
developer with full git commit access can do? (without having access to the
actual server)

If someone sets out to mess up a repo ON PURPOSE, he can do force push. Can he
also somehow trigger GC to preclude recovery? Do something else?

What are the attack vectors?

------
Osiris
Sometimes I have had to do a git push --force, usually when I want to amend a
commit I pushed just a few minutes before and I know that no one else has used
it.

However, I always do a --dry-run first on a force push just to verify that
it's only going to update the one branch I expect it to.

------
mey
This kind of situation has been one of my concerns about deploying git
internally in a large corporate environment. What steps are appropriate to
block this kind of update on a central repo? I don't see a human patch manager
scaling.

~~~
qznc
We have often inexperienced students, who do get push permission to our
central repos. So far they all behaved reasonable, however we had one
accident.

The student used Eclipse with its git integration. He tried to push, but it
failed (non-fast-forward). Apparently, Eclipse tries to be helpful and says
something like "If you want to push anyways, use the 'force' checkbox". Not
knowing what a "force push" is, the student followed Eclipse's advice.

Thankfully, the distributed nature of git is very helpful. You can just force
push again from another repo to undo the force push.

~~~
andor
Yes! Eclipse's JGit is afwul. I did a project with 9 others inexperienced in
Git. In the final crunch phase, they managed to delete commits via force push
_multiple times a day_! JGit also completely messed up the history with merge
commits, instead of doing fetch and rebase.

------
timmow
The number one feature I wish github would add is a setting to prevent force
pushing to master, but allow a forced push to all other branches. This would
allow us to use a branching model like
[http://scottchacon.com/2011/08/31/github-
flow.html](http://scottchacon.com/2011/08/31/github-flow.html) but we would
still be able to rebase branches after code review, without the danger of
rewriting history on the master branch. You can do something similar with
client side hooks, but it would be nice if it was supported server side.

------
Pxtl
So when do we do the obvious and start running version control on our version
control history? Git the git? Revert our VC mistakes as easily as we revert
our code ones?

------
filipedeschamps
Funny fact: Jenkins is all about continuous integration.

------
sarreph
It's called... Force Push?

------
smoyer
Everyone needs at least one frowning picture of themselves to post along with
an apology. Props to Luca for the no-nonsense apology.

------
yannk
It would be nice if Github was giving a UI control over force-push behavior.
If you open a ticket they'll manually flip this repository settings on or off
for you, but still... Maybe this is the wake up call: it's dangerous and
doesn't scale?

------
InAnEmergency
I wonder if it's related to this: [https://groups.google.com/d/msg/jenkinsci-
dev/-Yk0UFfSPZc/Gz...](https://groups.google.com/d/msg/jenkinsci-
dev/-Yk0UFfSPZc/GzOu5b1AP7QJ)

------
pearjuice
Can anyone mirror the link? Google Groups AJAX hell is not loading here.

------
Kiro
What does the --force flag do? I don't understand the description at
[http://git-scm.com/docs/git-push](http://git-scm.com/docs/git-push)

~~~
ihateloggingin
It deletes the remote branch entirely, and then uploads the local branch.

------
Estragon
Can the history simply be replayed into the repositories with the erroneous
pushes skipped?

~~~
0x0
Yes, unless others have pushed local commits based on outdated clones in the
mean time; then someone will have to rebase and rework (or just blindly merge
and repush)

------
pearjuice
Total mismanagement of repository access. Surprised this is the first time.

------
justinhj
This link just opens a google groups page with no content on my iPhone :(

------
circa
Leroy Jenkins?

~~~
bradbatt
That's what I came here to type. Well, actually...

Leeeeeeeeeeerooooy Jenkinnnnns!!!!

[http://www.youtube.com/watch?v=LkCNJRfSZBU](http://www.youtube.com/watch?v=LkCNJRfSZBU)

/goddammitleroy

------
DevUps
Ohhhhh... that feeling.

------
JSno
should it be "Jenkins" instead of "Jenkin" in title?

------
Groxx
I predict this will happen a LOT more often with git 2.0 changing the default
push behavior to pushing all branches. If you've ever --force pushed over a
temporary remote branch (e.g. to show something to a coworker), 2.0 will run
the risk of overriding everything. Hooray!

~~~
Axsuul
Don't you mean it will happen less often? Git default push behavior in 2.0
will be 'simple'.

~~~
emn13
But these were lots of repos and not just forks, but the look of it - If
that's the case, I'm guessing some scripting was involved, so the defaults
wouldn't have mattered.

