As an author of fabric.js, I'm glad to see you guys found it useful! Just wanted to mention that fabric's requirement for old Node.js is definitely something we can't wait to get rid of. As soon as we find a replacement to "o3-xml" package, you'll be able to take advantage of newer (and faster) Node versions.
Yes. Good point. It's indeed quite expensive to do the graphic generation on the server.
We use a caching from Amazon CloudFront. They have a great feature 'custom origin'. We just put our heroku server URL as an origin for CloudFront and configured HTTP headers properly. After that we were able to significantly reduce the load on a server.