In the audio version, they mention several times that patent law is founded in the constitution. I wish they would actually read it though, as it's not very complicated:
The Congress shall have Power To [...] promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.
Oddly enough, that is not enough to make patent law mandatory by the constitution: the text you cite has the structure "The congress shall have Power To […] promote X by using the mean Y".
First, it is not written that the congress has to exercise that power.
Second, what if Y does not promote X? What if it hinders X? It is written that the congress is allowed to promote X (through Y), not doing nothing about X (through Y) and certainly not hinder X (through Y).
I understand how one might assume that a patent system is a good (best? only?) way to promote the Progress of Science and useful Arts, especially back then. Nevertheless, I think this was a mistake: they wrote the mean in the same stone they did the goal. (Of course, goals themselves can turn out to be means to higher purposes, but in this case it is quite obvious from the beginning.)
The Congress shall have Power To [...] promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.