Many (not all) professional novelists have a schedule, to ensure they write each day. They force themselves to type X words, or sit there for X hours (they get bored and start writing). But that's work, not leisure.
I think a distinction here is relaxation verses intrinsically meaningful activity. Many people find the latter the most rewarding... but goofing off and aimlessly following our whims, just for fun, has a place too. (Actually this worked well for Feynman).