A hacker-publisher friend does "print-on-demand" the simple way; a high-end laser printer plus a high-end perfect binder and cutter. I think they kept their equipment costs down below $10K. The standalone machines don't do much besides print and bind and I believe cost much more.
This is a solution for a low print run rather one-of-kind printing but it works for him.
In the realm of thesis printing, you pay 10 to 22 cents per monochrome side, plus about $30 for perfect binding (plus shipping, and any surcharges for fast turnaround). So printing Strousup's 900 page "The C++ Programming Language" costs $130+ and printing something slimmer like "The Art of Computer Programming, Volume 4 Fascile 4" costs $40+. New from Amazon those would cost $55 and $15 respectively.
If you can get your prices competitive, there might be an opportunity for you in thesis printing.
If you could get your prices so low that a professor with a ready-to-go textbook in PDF form could get it printed on demand cheaper than going through regular publishers and college bookstores, that would be a big opportunity right there. By all accounts college textbooks in America are very expensive, but neither authors nor publishers admit to making much profit, so presumably there's a lot of inefficiency there. Could be ripe for disruption.
Is it possible that because it's so expensive, you could make it yourself for a much better return on investment?
Is it so expensive that you could burn out three cheap laser printers a month for a fraction of the cost? What if it was one moderately expensive printer every two months?
What if you had a Raspberry Pi managing your print queue across multiple networked printers and you bind them with a surplus GBC machine? You could keep a supply of "nice" pre-printed covers available to wrap around books when they're ready.
One nice thing about this is that you could start small, and scale fairly easily. Use human labour (that's already running the shop) to fill in the gaps in automation.
It might be worth looking at Don Lancaster's web site (tinaja.com) for some thoughts on how to do this stuff before any pre-packaged solutions existed (look for "print-on-demand" resources).
I like bookstores a lot. I hope you'll be able to find a way to make it work for you!
Pick up a publishing magazine, they have companies that advertize perfect bindign equipment for a few thousand dollars that only costs a few dollars per unit to use.
