Build time is important at development time, when I'm programming some new feature, and I have to wait 1-2 seconds before refreshing the page. If those changes take 20 seconds, well, that's a horrible development experience.
So, for now, I'll definitely give Hugo a try (didn't even know) but we do websites for people and we should only care that the website is fast, nice and easy for them.
Static Site Generator: Jekyll
Approximate number of pages: 200
Build time: 3.4 seconds
For our main website (www.processout.com, ~100 pages) and blog, hugo takes about 39ms (on a Macbook Pro, late 2016)