I'm an academic scientist. There are still huge problems with Elsevier in science, but there has been a huge amount of progress in making open-access journals more prestigious, which may work in mathematics. My favorite example is the collection of PLoS journals (Proceedings of the Library of Science). PLoS Biology was started in 2003, and in 6 years, they became the highest ranking Biology journal according to the traditional (though perhaps highly flawed) impact factor rankings. This is significant, because there's more money invested into research in biology/medicine than all other disciplines combined, so winning in such a monetized market represents a great step forward.
The best part of the PLoS journals is that anyone anywhere can read them, you can send them to your colleagues, since they're licensed under Creative Commons. And they tend to have the best user interface of any of the journals' online access sites.
There's a downside in that PLoS journals tend to ask for contributions from publishing authors to help cover costs. This is less of an issue for, say, a large biology lab with millions of dollars in funding, than it is for a theoretical physicist, or in your case, a mathematician. I think it would be worthwhile to add a part to grants asking for money for publishing in open-access journals. I don't know how grants work in mathematics, but in biology they're pretty flexible.
For example, the Journal of Machine Learning Research is now the most prestigious ML journal, and it has a $0 budget: it's hosted online on donated servers from MIT (http://jmlr.csail.mit.edu/), authors are expected to deliver publication-ready PDFs via LaTeX, administrative work is done by volunteer editors and students, and archival copies are printed off as print-on-demand by a third party publisher.
The Journal of Artificial Intelligence Research, the top-ranked journal in general AI, runs a somewhat more institutionalized operation (it actually has a few paid staff), but pays for it out of sponsors rather than author fees: http://www.jair.org/
I confess I'm not familiar enough with academic publishing to know why there's such a large difference in cost structures, and why other areas can't do AI-style free-and-open-access journals.
Mathematics grants are relatively small, on the order of $20-60k a year. But senior mathematicians (of which Gowers is most certainly one) typically have larger grants. In mathematics, $1-3k (the publishing fee for PLoS) seems like a rather high barrier -- but I hope the coming years prove me wrong here!
Libraries also have infrastructure in place - they already host their own websites, so running a journal just requires adding another page to it.
The most annoying part would be coordinating the different reviewers, but I think that job would be done by the editorial boards, composed of researchers in the specific fields.
If you look at libraries and academics (and their expenses) under the umbrella of the university as a whole, I have to wonder (and have no way of knowing) what the totals costs to the university would be, in comparing:
- The current model where the library (university) pays large money for publications they don't want.
- The possible model where the journals are free to the world, and the university pays the donation or fee for publication of specific papers.