Some time ago there was a post on HN about the difference between a system and goal driven behavior. A system driven behavior applies a strategy over and over endlessly with improvement at each iteration. The goal driven behavior is a sprint based evolution.
What the OP describes is a system which he acknowledges improves his expertise in the studied knowledge.
My opinion is that system has the advantage over goals in that it is an automatic and permanent drive to stick to some rule. It can be learning, exploring new ideas, do fitness, healthy food, keep a good weight, etc.
Goals require a boost of effort and focus wich can only last for a limited duration. You feel good if the goal is achieved and bad if not.
If you work with goals and sprints, there is a high risk of burnout. So better go into system mode for the long run.
The change of topic of interest is something I saw my self and was also puzzled about. I attribute this to some "fatigue" of the brain where it's learning capacity is kind of saturated. Changing topic of interest, changes the neurons and connections in play which let some of them to rest and activate others. Learning is also a matter of repetition and pruning details. You get this by switching back and forth between topics and forgetting the details in between.
Another benefit of having different topic of interest is that at some point we will make the connections between them. We'll use the knowledge gained on one topic on the other. This is powerful and by the combination of topics explored it will also make you most probably unique.
Sounds like a fair summary, but you haven't addressed what (I believe) is the key issue: OP feels bad.
pain [...] consistently high [...] don’t have the energy or enthusiasm
I think a huge turn off for "goal-setting" is that corporations do it, and also that they do it for the purpose of productivity. However, I'm not suggesting it for productivity, but for satisfaction. The satisfaction of completely something. Of climbing the mountain, completing the quest, bringing the elixir back, solving the problem, creating something new. I'm not saying one "should" do it, I'm saying it's a way to feel satisfaction, instead of pain.
The key idea here is that a goal not something to strive for, but a limit. Not to force you on, but to let know when you have accomplished something - when to stop! Now, if you're a perfectionist, you may say, "but there's so much more! I've barely scratched the surface. I've done nothing! I'm not worthy!"
However, if you had a goal, and you can remember how it looked when you set it (that it did seem challenging), then when you reach it, you can compare where you were, and where you are now. And you are forced to acknowledge the fact that you have accomplished something.
This gives a flood of satisfaction, like it or not.
Now, turning to "sprint" vs. "system". Firstly, I stressed that the goal must be doable. There's minimal risk that you can't complete it, if it is "doable". And if it's too hard, pick a smaller one.
Note also that you can shift topics between goals, just as you can with a system. I'm talking about small goals, not huge overreaching ambitious goals. Baby steps - but steps nonetheless!
Sprints seem very compatible with human psychology: movies, books and music usually consist of alternating tension and release. There's a series of difficult obstacles, not just one, but a series of problems and solutions. People really respond to this.
Curiously, it seems that literal sprints (i.e. running) are also much better for building fitness than consistent paces. And they even have "sprints" in some agile methodologies (not sure if that counts for or against my argument...).
Finally, there's absolutely a magic in programming, that you code something... and something happens! It's a child-like thrill that (in my opinion) it is absolutely crucial to respect and preserve, especially as ones goals become ambitious and abstract. Fred Brooks talks about the "sheer joy of making things" - and compares us both to a child making mud-pies and to God creating nature. (just found an excerpt online: http://cs.calvin.edu/books/processing/references/brooks/myth...)
I would guess that, evolutionarily, the human body and mind is adapted to sprints - and this is what makes goals feel so satisfying to us, just as we enjoy food, water etc. The "system" approach makes logical sense, and what you say about it seems true. It's just that you haven't addressed how it engages with human motivation (but assume it is "an automatic and permanent drive") - which is the issue OP is struggling with.
Lastly - I think OP's personal solution will actually come from the thoughts he has when he feels that pain. Part mistaken, part profound - only OP will know which is which. I don't claim my suggestion is right for OP, just adding another perspective.
The whole thread resonates quite strongly with me.
Many interesting points are made. I was just recently wondering whether I should pursue goals when engaging in activities I do for fun. Like you wrote, it is feels so corporation-ish but then again they seem to motivate us.
Another aspect, however, concerns motivation vs. commitment. When it comes to hobbies I am quite driven by motivation. Unfortunately, motivation often doesn't last long. That's why many people point out that commitment [doing something despite a momentary lack of motivation] > motivation.
And this gets me back to square one as commitment is something I rather associate with corporate culture than leisure time.
[Of course, you could make being commited x times a week a goal to get some satisfaction out of it but I doubt that would work for me. It probably also depends on the activity: I can commit myself to regular exercise but I can not force myself to sit down and write some music 3x/week]
I've heard that action precedes motivation - you force yourself to start, and then you become engaged. I've found this is true for me.
Many (not all) professional novelists have a schedule, to ensure they write each day. They force themselves to type X words, or sit there for X hours (they get bored and start writing). But that's work, not leisure.
I think a distinction here is relaxation verses intrinsically meaningful activity. Many people find the latter the most rewarding... but goofing off and aimlessly following our whims, just for fun, has a place too. (Actually this worked well for Feynman).