But is it really? If I recall it's been found that typical hunter gathers had more leisure time than we do.
Citation needed. As far as I understand, the reason why civilization doesn't happen without agriculture is that everyone has the same job: getting food. Fundamentally, leisure time doesn't matter when everyone's specialty is "getting food." There's a reason that most of history's notable mathematicians and scientists tended to be in the upper class of their society.
Reading things like this makes me glad I work in a 'critical infrastructure industry'. The company I work for improves the quality of life for its customers.
The amount of leisure time that western civilization enjoys today is only a very recent development (within ~100 years).