The energy and interest you have is a tremendous positive. Don't take it for granted. It's something to nurture.
What exactly is making you feel bad - what are you thinking when you have that feeling? i.e. what thought is stopping you and why is that bad?
[OK, I'm guessing in the following - you'll find the true answer in the above questions] You say you feel bad because of a lack of mastery, but what level of mastery counts?
I think it's the lack of a goal: a defined point where you say "ah! I've done it!", instead of a continuous never-ending slog. There's always more you can do in the bigger picture, but you can choose a goal at a point that in some sense is a completion. Artificial in a way, but also on top of something, a logical boundary.
That is, a specific doable task that can be done pretty quickly; with some challenge, not too much - a level that, from your experience, is definitely doable by you; that is clear and definite, so you have a direction/guide to keep you on track, and so you know when you've reached it (so you can stop, and see how far you've come, and a take a rest to choose the next one).
Finally, it must have some meaning to you - it could be to just learn some technique to some standard... but better is an actual project that you want to do: a simple parser, an animation, a tool that you'd like to have, implement an idea, copy a game or tool or library or framework or language or app that impresses you (or some part of it). But note that it must be doable, by you, in reasonable time. So, if too ambitious (or you realize it's too ambitious as you go along!), break out simpler and simpler versions of it, or sub-parts of it - provided it still has meaning, a definite end-point, and is doable.
Of course, as you say, you are learning stuff in this spiral, and you'll get cross-insights from doing different things. It's just that you feel bad about it - I think having goals would help. [But see the questions above...]
Some time ago there was a post on HN about the difference between a system and goal driven behavior. A system driven behavior applies a strategy over and over endlessly with improvement at each iteration. The goal driven behavior is a sprint based evolution.
What the OP describes is a system which he acknowledges improves his expertise in the studied knowledge.
My opinion is that system has the advantage over goals in that it is an automatic and permanent drive to stick to some rule. It can be learning, exploring new ideas, do fitness, healthy food, keep a good weight, etc.
Goals require a boost of effort and focus wich can only last for a limited duration. You feel good if the goal is achieved and bad if not.
If you work with goals and sprints, there is a high risk of burnout. So better go into system mode for the long run.
The change of topic of interest is something I saw my self and was also puzzled about. I attribute this to some "fatigue" of the brain where it's learning capacity is kind of saturated. Changing topic of interest, changes the neurons and connections in play which let some of them to rest and activate others. Learning is also a matter of repetition and pruning details. You get this by switching back and forth between topics and forgetting the details in between.
Another benefit of having different topic of interest is that at some point we will make the connections between them. We'll use the knowledge gained on one topic on the other. This is powerful and by the combination of topics explored it will also make you most probably unique.
Sounds like a fair summary, but you haven't addressed what (I believe) is the key issue: OP feels bad.
pain [...] consistently high [...] don’t have the energy or enthusiasm
I think a huge turn off for "goal-setting" is that corporations do it, and also that they do it for the purpose of productivity. However, I'm not suggesting it for productivity, but for satisfaction. The satisfaction of completely something. Of climbing the mountain, completing the quest, bringing the elixir back, solving the problem, creating something new. I'm not saying one "should" do it, I'm saying it's a way to feel satisfaction, instead of pain.
The key idea here is that a goal not something to strive for, but a limit. Not to force you on, but to let know when you have accomplished something - when to stop! Now, if you're a perfectionist, you may say, "but there's so much more! I've barely scratched the surface. I've done nothing! I'm not worthy!"
However, if you had a goal, and you can remember how it looked when you set it (that it did seem challenging), then when you reach it, you can compare where you were, and where you are now. And you are forced to acknowledge the fact that you have accomplished something.
This gives a flood of satisfaction, like it or not.
Now, turning to "sprint" vs. "system". Firstly, I stressed that the goal must be doable. There's minimal risk that you can't complete it, if it is "doable". And if it's too hard, pick a smaller one.
Note also that you can shift topics between goals, just as you can with a system. I'm talking about small goals, not huge overreaching ambitious goals. Baby steps - but steps nonetheless!
Sprints seem very compatible with human psychology: movies, books and music usually consist of alternating tension and release. There's a series of difficult obstacles, not just one, but a series of problems and solutions. People really respond to this.
Curiously, it seems that literal sprints (i.e. running) are also much better for building fitness than consistent paces. And they even have "sprints" in some agile methodologies (not sure if that counts for or against my argument...).
Finally, there's absolutely a magic in programming, that you code something... and something happens! It's a child-like thrill that (in my opinion) it is absolutely crucial to respect and preserve, especially as ones goals become ambitious and abstract. Fred Brooks talks about the "sheer joy of making things" - and compares us both to a child making mud-pies and to God creating nature. (just found an excerpt online: http://cs.calvin.edu/books/processing/references/brooks/myth...)
I would guess that, evolutionarily, the human body and mind is adapted to sprints - and this is what makes goals feel so satisfying to us, just as we enjoy food, water etc. The "system" approach makes logical sense, and what you say about it seems true. It's just that you haven't addressed how it engages with human motivation (but assume it is "an automatic and permanent drive") - which is the issue OP is struggling with.
Lastly - I think OP's personal solution will actually come from the thoughts he has when he feels that pain. Part mistaken, part profound - only OP will know which is which. I don't claim my suggestion is right for OP, just adding another perspective.
The whole thread resonates quite strongly with me.
Many interesting points are made. I was just recently wondering whether I should pursue goals when engaging in activities I do for fun. Like you wrote, it is feels so corporation-ish but then again they seem to motivate us.
Another aspect, however, concerns motivation vs. commitment. When it comes to hobbies I am quite driven by motivation. Unfortunately, motivation often doesn't last long. That's why many people point out that commitment [doing something despite a momentary lack of motivation] > motivation.
And this gets me back to square one as commitment is something I rather associate with corporate culture than leisure time.
[Of course, you could make being commited x times a week a goal to get some satisfaction out of it but I doubt that would work for me. It probably also depends on the activity: I can commit myself to regular exercise but I can not force myself to sit down and write some music 3x/week]
I've heard that action precedes motivation - you force yourself to start, and then you become engaged. I've found this is true for me.
Many (not all) professional novelists have a schedule, to ensure they write each day. They force themselves to type X words, or sit there for X hours (they get bored and start writing). But that's work, not leisure.
I think a distinction here is relaxation verses intrinsically meaningful activity. Many people find the latter the most rewarding... but goofing off and aimlessly following our whims, just for fun, has a place too. (Actually this worked well for Feynman).