How could a physical object be pure math? Even if you could completely mathematically specify the process of creating the object, that only means you couldn't patent the specification itself - the object would still be patentable.
"In a sense, all possible computer programs for a given architecture are implied in that architecture, even if they haven't been written yet - just like all possible novels are implied by the letters of the alphabet, even if they haven't been written yet."
Surely you realize that all possible patents are implied by the alphabet in precisely the same manner that all possible novels are implied by the existence of the alphabet...? :)
There seems to be some confusion here between the thing that is being patented and the description of the thing that is being patented. The fact that the description is written using the alphabet has no bearing on the patentability of the thing itself. No arrangement of letters in the alphabet can be patented - but that doesn't mean that nothing can be patented.