This is the coordination problem to end all coordination problems.
Is there an example in history of civilization turning its collective back on some scientific pursuit in fear of its potentially negative consequences? It seems close to impossible. There are too many different value structures out there, and there’s too much money on offer to anyone who can give any government or large corporation an advantage over its competitors.
You may be able to convince a few researchers today, but as soon as China has the slightest breakthrough towards X or Y technology, most of those who signed the pledge will be tripping over themselves to get cracking on the problem.
First we should ask: is this a "scientific" pursuit in the first place? Or, rather, are we describing the creation of some mechanisms (based themselves on scientific discovery) that can be used for all kinds of purposes, among them, yes, future scientific practice? Computing leverages scientific discovery, and aids in its practice, but is itself not science.
Second, and at the risk of sounding trite, we can also ask "is there an example in the history of civilization of someone flying before the Wright brothers?" or "is there an example in the history of civilization of a world governing body before the 20th century?" The very promise of democracy, in general terms, is that we should, in fact, be able to make large decisions together on a society-wide scale. And because that is, ostensibly, the highest level through which we coordinate, we can should also be able to direct how our democratic institutions help us achieve those aims. If your position is that money, corporate influence, or government corruption prevent such coordination, you are certainly correct. But these are not "natural law." We have the ability -- however likely -- to deal with those problems. Otherwise, what's the point?
> You may be able to convince a few researchers today, but as soon as China has the slightest breakthrough towards X or Y technology, most of those who signed the pledge will be tripping over themselves to get cracking on the problem.
We are not limited to pledges. But even in that case, it is then up to those scientists who turn a dime to deal with their own potential moral failings.