Sounds great, but the other side is just going to raise Godel's incompleteness theorem in the form of the church-Turing thesis to show that the truth of your argument is formally undecidable ;-)
Seriously, though, the legal issue arises from the fact that the patent is not (or rather should not be) granted for a pure mathematical abstraction, but for a product which applies such abstractions to some novel purpose. Mathematical discoveries are not considered properly patentable because mathematics is held to be part of nature's bounty - a highly debateable philosophical proposition, but there it is. But being able to describe a thing in mathematical terms does not equate to that thing being a mathematical statement and therefore unpatentable.
Let's consider the steam engine, for example. The interplay of pressure, volume, and temperature can be modeled with very simple mathematics, as can the mechanics of a drive train to convert the motion of a piston into work by turning a wheel. You could write down all the math necessary to accurately model a steam engine or indeed a train in a few pages. Likewise, we could argue that that the giant boiler on wheels that constitutes a train is no more than a restatement of those same mathematics, along with a few supplementary remarks about the properties of the materials used to express the primary equations of the system (eg the tensile strength of iron, the viscosity of the lubricating oil and so forth). This is true enough in a philosophical sense, but talk is cheap. Building a working steam engine is a lot harder than coming along afterwards and observing that its characteristics could be reformulated as a mathematical statement.
Now software is different, insofar as we can take the source code or even the object code and convert it back to a set of mathematical equations with considerably greater ease. But again, it's not the underlying mathematical statements for which a patent is (ideally) sought: it's for the black-box implementation that allows someone with zero knowledge of the underlying math to make use of the invention. If you rely on the philosophical argument that all code is inherently an agglomeration of mathematical statements, then you must agree that, say, Farmville or Pac Man are in fact part of nature's bounty as well, due to their mathematical foundation as computer programs. That's an argument which would be extremely hard to sell, either in court or to the general public, and with good reason - most people consider the development of a computer game to be a creative effort rather than something that was sitting about waiting to be discovered like the law of gravity.
One could say that Pac Man was in fact sitting around in 'the space of all possible games yet to be implemented,' but then again one could put someone in prison and say that freedom is a state of mind too, or kill a person and argue that it doesn't really matter because they're still alive in the past. These are interesting philosophical avenues to explore, but only on a voluntary basis. Courts are a reflection of the society in which they operate, and theoretical arguments like this which take no account of pragmatic considerations usually go badly for the proponent.
But let's say that you argue there's an important principle at stake here, and get the court to agree about the fundamentally mathematical nature of computers and the programs that run on them. How then do we deal with the fact that board games are patentable? After all, a board game is just a set of tokens and some rules for manipulating them, whose utility lies in its capacity to entertain or edify the players. If we outlawed software patents tomorrow, then couldn't patent trolls simply switch to patenting 'games' in which their algorithms are presented as rule systems? True, few people would fancy playing Super Bubble Sort or Mem Cache Plus all night, but patents are not granted on the basis of popularity.
Again, I'm not out to defend software patents - I'm just bringing up these arguments to point out why 'programs = math, therefore software patents = invalid' isn't the irrefutable legal argument that so many people seem to think it is. Where to draw the line of what can be patented is a much trickier question than it may at first appear, and if our approach to this Gordian knot is to get rid of the patent system altogether, then one needs to address the problems that predated its creation.