Rabbit Hole Syndrome is a symptom of having a curious, persistent, independent mind. If there were a drug that cured Rabbit Hole Syndrome, and everyone just worked on Shipping Their Products and Not Asking Questions, there'd be no one left to make the interesting discoveries that make the world better in the long run.

There are very famous quotes from Richard Hamming (see "You and Your Research") and Richard Feynman (see "Surely You're Joking!") about the importance of working on problems that seem trivial at first. Not only do they help you enjoy problem-solving for its own sake, but if you work on enough silly problems, then eventually the odds are good that you'll stumble upon something that other people will later think is really important. Incidentally if you read Thomas Kuhn you'll find that this tends to be how scientific revolutions happen: someone is bothered by some tiny little thing that doesn't quite make sense according to the prevailing theory, starts digging at the little cracks, and finally the whole system comes crashing down.

The major benefit of going down rabbit holes is that is opens yourself to serendipity: sometimes you'll turn out to be right about something for the wrong reason, or discover something that you later realize contains a solution to a seemingly unrelated problem. The more rabbit holes you've gone down, the more they start to connect.

In addition, frivolous research helps you develop a very "bottom-up" view of the world. If you know the details cold, then you are better able to see through high-level BS ("conventional wisdom") and evaluate things on their own terms. You'll find that regardless of what's optimal, most things are done a certain way only because they've always been done that way, and that regardless of what's true, most people believe things only because other people believe them.

Of course there are costs to all this -- in particular, "schleppers" will resent you for being irresponsible, and most people will think you're crazy if you ever explain what stupid little you've been working on lately. Which is why Rabbit Holers should try to should avoid actual responsibilities to the greatest possible extent and absolutely not care about what most people think about their work.

Anyway, to sum up, if you're lucky enough to be in a position to descend rabbit holes without impoverishing your family or bankrupting your company, I say go for it. It strengthens your most valuable asset (your mind), and who knows, maybe one day you'll discover something you can teach the rest of us.

Our education system is broken in that we don't recognize and develop the skill you have. People are somehow forced to do things they don't feel like being doing.

My impression is that it's ADD. But It's sad ADD is called a disorder, because these people have the remarkable ability to be extremely efficient and concentrated in what they like and are interested into, well above the average. These people could be experts and genius in their field, provided they could develop their skill.

If you are building something, you inevitably spend some part of your time writing code that is familiar and boring (let's call this WORK), and part of your time exploring unfamiliar code and concepts (RABBIT HOLES).

Going down rabbit holes is not work, it is a form of learning - in fact, in the world of software development it is the best form of learning because you have a highly relevant problem and supreme motivation to solve it.

At the end of the day, however, going down a rabbit hole is no different than reading a book, or articles on HN. It is not actually work, and fooling yourself otherwise leads to the blurring of what is work and what is learning.

You should always be learning, as much as you can. But you must be honest with yourself, there is probably more work to be done, and you can't possibly get on with it until you pull yourself out of the rabbit hole.

By now it's 9:30pm. It's dark and cold. You realize what your original purpose was: Dinner with friends. That was two hours ago. You missed dinner, but hey, you got some satisfaction.

The above was more of an analogy about Yak Shaving than Rabbit Hole Syndrome, but there are parallels. Like you, I used to be obsessive about details and solving subproblems. I used to come home from work and work on my own side projects for similar reasons as your own.

But then an advisor said something like, "You need to focus. If you want this thing of yours to succeed, you have to focus on making it succeed. Nothing else should matter." So I stopped my side projects and I became so effective at building our product that my employees wondered if I ever slept.

Stay on target. Make it to dinner.

Say you want to try alternative payment processing methods on your site. Your site it tightly integrated with PayPal, and you want to add Stripe. You could (A) spend 1 day to bolt on Stripe with a hack in the HTML, (B) spend 4 days and add enough views to the frontend to make Strip work in a non-disgusting manner, or (C) spend two weeks and refactor the entire backend with a generic interface that will handle an infinite number of payment solutions.

All three lead to success, but you can't optimize for both time and perfection. Set a time budget. Leave breadcrumbs for the next programmer. Instill a culture that says, "Our code isn't perfect, we know this, but we do what we can with what we have." Done is better than perfect.

With forethought, you should have built your system originally with method C in mind. If he had done this at the beginning instead of just hacking it together then all this work after wouldn't have been needed!

1) Apply conscious planning and forethought. Some things are clearly going to be used a lot, some things might not. Use bolt-on hacks where appropriate, invest time for things you expect to be using forever.

2) When the longevity of the tool is not easy to pin down, test it out with a hack. Spend 1 day adding Stripe the hacky way, and keep a slushbucket of time budgeted for revisiting experimental hacks that look like they will last a while.

That's my general approach, anyway. Most of the time, a hack is good enough and anything more is a waste of time. The story is admittedly a little different for me though; for me, software is a tool to do my job- software is not my job.

We have only had to do 2 refactors in that time on 2 controls. That was more about understanding our customers needs more as we progressed rather than not enough planning though.

Previous to that I built a custom ecommerce site for a wholesale distributor. That whole process was more suited to a quick hack over a well thought out decision. That's why I say I agree in principle, so long as the project allows it.

That's how I develop now to be fair and not how I would build a product if I were to build it for myself.

At any rate, I'll tweak the "done-perfect" lever more towards done.

Facebook's motto reminds me of something from a Holman slidedeck: "Nothing great was ever not shipped."

Don't sabotage the larger vision for short-term gratification.

That being said, I guess it depends on what results these random maunderings produce, and it's probably a crapshoot as to whether they are more often than not to result in favorable outcomes. Also, I like your example (obviously :)

Chances are, you'd approach Town Hall with the solution to all their pothole problems and they would ignore it. To make it worthwhile, you'd either have to have earlier signed a contract with Town Hall to figure out their potholes (ie. have a customer) or develop a business based on your solution (ie. develop a market).

So I suggest you to try to build the minimum viable working program ASAP. Along the way, write all the things you would like to improve in something like an Evernote note, just a few lines for every thing you want to address and make better.

Then if you have something working ship it ASAP, don't care about what other people will say about the sub-optimal parts of your work: many programmers trying to achieve perfection actually end with a mess of complexity that does not serve very well the purpose of the software, so there is little to be embarrassed for a programmer for shipping simple software.

In the second pass, refine every part with the same approach: find a solution that within the timeframe you have is better compared to the previous one, but will make you able to ship a new version.

Also when you face a problem, other than reading the existing literature, papers, and the proper way to do it, check if there is an intuitive solution that is comparable as a result (even if maybe not provable or not perfectly optimal) but much simpler to implement.

But IMHO the golden rule is: don't freaking care, ever, about what other people think about your work. Often perfectionism is just a form of insecurity.

I get it now: If either one is taken to the finish line, they are the same (all nodes visited). But when left mid-way (time constraint), breadth first still is something workable while depth-first has a few components perfectly done while the others not at all.

>> Often perfectionism is just a form of insecurity.

Possibly brilliant too. How do I tell whether it is coming from insecurity or from my drive to not avoid shlep?

For instance I prefer developing software over marketing my product ( http://gridspy.co.nz ). This means that perfecting our product's software is the comfortable and easy option when what I need to be doing is picking up the phone and calling people.

1) Try the pomodoro technique, or some other form of time tracking. When you're about to take a side path which may or may not be a distraction, you can decide how long it'd be worth investing in it. Once the timer dings, you can stop and evaluate if it was worth it.

2) A few years ago I started repeating in my head the phrase "real artists ship". Embrace imperfect or partially finished solutions that are viable.

3) Keep a list of things that you'd like to investigate more. I've found that the act of writing down the idea lets me stop obsessing over it. Later when you revisit the list you'll be able to cross off the things that you thought were important, but turned out not to be. By delaying work on these items, you're able to better explore the most important parts of the problem you're solving, and so your future self will be in a better position to evaluate which areas need deep research.

I'm like you. I write fairly vanilla production code, then I go home and work on alien technology. And I used to have the same problem as you: I'd get sidetracked and sidetracked, and somehow I went from writing a fart app to reading about type theory.

So I started skipping the fart app part, and started learning some more abstract theory. The nice thing abstract theory is that it's abstract---it's not incidentally connected to anything, so there are fewer places for you to get sidetracked.

So go sign up for a math course on Coursera, or learn the lambda calculus. They are so alien from your everyday programming experience that you won't have anything to connect them to---until you do. But then you'll be coming at the new topic in the direction of abstract to concrete ("a trivial application of x") rather than concrete to abstract ("there's a greater truth here and I MUST understand it!").

At some point you realize your best days are the days when you delete more lines of code than you write.

But that being said, chasing rabbits is what will eventually make you more skilled than your peers. Which is awesome, except now you'll have put yourself in a perpetual category of being paid less than your worth because you don't fit the same performance evaluation criteria as everyone else.

All anyone cares about is a) are you easy to work with, and do you b) get things done to contribute to profitability.

Continutally stressing yourself out by spending more time on problems than they deserve and eating away at your work/life balance does not contribute to a) or b).

This gets to the crux of the matter. I think we all here have an intuitive sense that being a bit of a perfectionist and regularly chasing a few rabbits down holes can lead to better long-term results. But it's very hard to quantify, certainly it's impossible to quantify objectively; all you can do is be a good salesman about it.

There is another path though—put your money where your mouth is. Start your own business based on your chosen principles and standards of software engineering. It will certainly bring you joy and if your instincts are right then it could even become a competitive advantage as your competitors drown under mountains of technical debt while you nip it in the bud at the first sign of trouble.

If you never find anything and still can't help yourself it's some lethargy/depression ADD brain dysfunction. Make sure to exercise enough to be in shape, eat, etc.

This actually has a label: Maximizers vs Satisficers.

http://en.wikipedia.org/wiki/The_Paradox_of_Choice:_Why_More...

And a little something I wrote about it regarding programming languages:

http://journal.dedasys.com/2006/02/18/maximizers-satisficers...

I think the key is to "choose your battles", and be a maximizer where it really counts, and try and be more of a satisficer for the things that aren't so important.

I'm in the middle of The Paradox of Choice right now, and while the maximizing/satisficing thing applies to many aspects of life—the book was certainly not written about programming—programming is yet another aspect of life where we're confronted with a huge array of choices.

This is especially true in side projects. In the office, there may be a standard set of technologies; you're likely working on an existing product; etc. Additionally, there are probably a number of constraints, including time and product requirements. In side projects, there are hardly any constraints, and every single aspect of your work involves choice. Just deciding what you want to create is one massive choice. Then there are the big questions of "what programming language/framework should I use?" and "what database(-ish) platform do I want to use?" There's a good chance since this is a side project, you'll want to try out something new and cool to build your knowledge and gain experience. Those alone can be huge decisions. Then there are questions of libraries (which library? or build it from scratch?), etc. Heaven forbid you want to start thinking about your choice of IDE (time to learn Vim?) and tabs vs. spaces and every other little decision you can get caught up on.

I'm at least as guilty as anyone else of these things. I spend too much time focusing on the best possible outcome and the best possible technologies among lists of hundreds and thousands—because I want the best and because I don't want to regret any failure—rather than simply settling for something that's good enough and moving on to actually get something done. The result is generally painful and unproductive.

You should definitely set standards for your work—don't produce crap—but try bringing the bar down from "ideal and perfect" to something more attainable like "excellent" or "very good" or even just "adequate." This will help you focus on your real goals and appreciate your own success.

Another useful term/link: http://en.wikipedia.org/wiki/Analysis_paralysis

Something that's helped me: make sure you have multiple rabbit holes available at any given time. i.e. several problems, any of which is interesting enough to tempt you. Then you can make reasoned decisions about which would be the most rewarding to work on, while still giving in to the temptation of rabbit holing.

Another benefit: maybe by solving one problem, you'll discover the other was totally irrelevant, or a special case of the other. (At least for me, "Turns out I didn't need this to be perfect" doesn't carry much weight, whereas "Turns out I didn't need this at all" is quite convincing.)

As a recent example, I just spent a long time on a tiny plugin for CarrierWave (the Ruby/Rails file uploading gem). This took me through most of their source code (always good to read code, and I could probably PR on their project now), new Rails internals and techniques, and mocking and stubbing with RSpec (in the process, I turned up a bug with RSpec, which they promptly fixed; and I have a better understanding of mocking best practices and clean RSpec).

I do empathize with feeling I'm "not getting enough done/shipped" (if you feel that way?). To alleviate it, I try to cut corners and just get something out.

Time-boxing helps me do this -- "accomplish X in 1 hour". This doesn't happen too often, however. I know, like you, I prefer the learning itself; and, I view all my spent time as building experience and a critical mass to accomplish work faster in the future.

Also, I find pair programming helps immensely. I am sensitive to the other person's time, and thus naturally refrain from digression when pairing.

As for motivating myself to "schlep". I'm not sure what you mean -- menial tasks? I do those when I'm tired or having down time.

Finally, as in your case, I do this on personal projects and not on "work". That said, it can make it hard to get your startup and product launched - your own "work". Again, disciplined time-boxing helps. I have not mastered this, and find myself regularly looking for good time-boxing tools.

Most of these tickets get resolved as "doesn't matter / won't fix" two or three weeks later - but myself.

First, it forces me to write down clearly the question that's obsessing me, occasionally leading me to realize that doesn't actually make sense or isn't that interesting -- all the more so because it is published so my standards are higher that if I just wrote it in a text file of mine. Second, it gives me the impression that people will work on it now so I can stop thinking about it for a while (even if no one actually cares about the question). Third, in any case the question is documented so it won't be lost.

In some cases I get insightful answers (sometimes, brilliant answers) that save me the time to hunt down a solution in an unfamiliar field or to work it out by myself, other times I end up replying to my own question because I stumbled on the solution later. In any case, it totally defuses the urge to think compulsively about a problem.

What I mean is, if you see a problem to solve, and you are able to keep working on your current task and to solve the problem later, then do so... just note the problem. This will immediately make you much more focused.

Then at the end of the day, review your list of rabbit holes and try and determine which ones are necessary for the current project, which ones would be educational / you want to do, and which ones can be discarded.

Basically rabbit holes are a problem because they are long and narrow and do not offer an overview of the entire grounds, so before jumping down a rabbit hole force yourself to survey the big picture and to see if you can step over it instead.

It might help to limit going down the rabbit hole too much, by researching what could be improved without actually doing it right then. Save it in a list for later, do the schlep, then when things get bigger chances are you'll actually need to take on some of the challenges on that list. And the more useful they become, the more satisfying it'll be.

I found that to be quite helpful in maximizing your happiness by getting done what you most care about in the long term.

If positive user feedback 'gets you off,' as it were, try releasing something that isn't as perfect as you'd like it to be and see if there's any measurable difference in sentiment. I bet there won't be. Do it a couple more times and hopefully it'll break the pattern.

For the other case, try time boxing yourself so you can get back to the more interesting challenge. Make that other thing 'perfect' if you won't.

I'd very much prefer to pick up an apple that isn't perfect than having no apple at all because of looking for the perfect one forever.

The other approach I'd suggest is at the complete opposite end of the spectrum. Start treating your side projects like a real business, and create some process that connects real incentives to them. There are harder and softer ways to approach this. You can quit your job and now you have the focus in your side projects that you need to ship to get paid. You could also keep your day job and set up some automatic funds transfers into a semi-liquid asset like a CD or 401k, such that you leave yourself enough to cover the basic expenses, but so that the marginal income from your side project becomes significant enough to be motivational. Revenue is a great feedback loop getting you to focus on what matters, and letting go of what really doesn't.

You could also bring in another person, perhaps as a business partner, or simply have one of your friends agree to call you up once a week and have you give them a report. The key here is to clarify and draw attention in your mind to the work that you're avoiding, and the outcome that you want. In this process, it's important to vividly color the positive outcome be emphasizing the real changes it will create in your life.

For example: "This week, instead of shipping my product which will enable me to quit working for other people, spend more time with my family, own the high-performance german cars I've always wanted, and visit 20+ countries a year, I instead spend 6.5 hours working on a sort algorithm, and 5.4 hours fiddling with the presentation styles on the country-list drop-down menu in my app."

I used all of the techniques above when I was starting my first company, and they made a big difference in helping me focus on what was important in getting to market, and not OCD-ing out on things that were, ultimately, minor details.

1. Set goals and fixate on them. Imagine what it will be like to have achieved the goal. Get excited about it and keep reminding yourself. When you're not making progress toward the goal, make yourself imagine the situation where you spend 6 months making unimportant things perfect and never achieve the goal. Imagine all the other goals you won't even be able to set because you're wasting your time.

2. Make other commitments. Make plans to meet a friend for drinks at 9pm. You only have 2 hours after work to get anything done – don't waste those 2 hours! If you can stay consistently busy, you'll notice quite quickly that not using your time effectively will lead you nowhere.

It happened the most while attempting to study research papers.

A lot of times I would be unable to finish reading what I had planned, because I had stumbled upon an interestingly-looking concept and then proceeded opening up another paper on that subject and so on.

And then, of course, having returned back to the original paper, I would have to re-start reading from beginning to freshen up my memory.

Terribly exhausting process, so I can perfectly understand your frustration.

The only solution I found was to un-clutter my computer 'work-space' as much as possible: close any non-essential apps, unplug internet cable and each time I have the urge to stop reading and go research a newly discovered subject, I remind myself that I am only allowed to do it after finishing what I'm currently reading.

Another helpful thing for me is to remind myself what I'm trying to accomplish with whatever I'm doing at that particular time and what my long-term goals are.

For instance, if I'm working on a project for a client, the goal is to get the work done as soon as possible and obviously get paid. I am not working on said project to primarily enrich my knowledge, but to make money.

I can use the time after the project is delivered to draw conclusions from the experience or do further research.

This may sound trivial, but it really does help to constantly remind yourself of what your goals are, it keeps you in check.

For extra effect, every time you have the urge to let your mind wander too much, try imagining the possible consequences of not completing your task (on time).

This can be particularly effective if you're doing client work. Imagine how embarrassing/unprofessional would have to explain to your client/employer that you won't be able to deliver on time because of something that you could've prevented.

Side-note: you have not provided enough info, but you may have obsessive-compulsive disorder, which is nothing to be ashamed of, but you can only 'solve' this with medication, so you would need to see a doctor.

I think you're probably not doing projects to help people, but instead projects that you think would be fun to do. If you start with a burning itch, or a person that needs help, you will probably find it much easier to stay on task. This is not to say that there's anything wrong with exploring interesting technologies, or thinking about "the right way" to solve certain problems you find interesting... It's just to say that when there's a burning need the interesting diversions tend to fall away.

Hope that helps :)

Reading about interesting stuff is deeply satisfying, and so it's something you want to justify so you get to do more of it. Actually getting stuff done is hard.

It's a bit like why so many techies - myself included! - vastly prefer to code new app features rather than working on marketing or sales copy. One is has a definite end point, and we feel comfortable getting there.

The other one can seem like work!

It sounds like at least part of your problem may be perfectionism. I did a bit of research on this a while ago, and it turns out there's a lot of literature on perfectionism and how to manage it. A quick look on Amazon under "Perfectionism" should bring up a few interesting books.

My side projects, in particular, are where I allow myself to roam freely, and I often end up prioritizing unimportant (but interesting) work that is only tangential to the core idea. When I am only accountable to myself, then it becomes too easy to get unfocused.

Perhaps the original question should be rephrased as "how do I keep myself focused when working on side projects, since I'm only accountable to myself."

If it is the former then perhaps you shouldn't worry and just let the rabbit hole lead you where it may.

If it is the latter then perhaps it would help to produce some sort of roadmap. Ideally share this and get other people involved (either people who will work with you on it or who want to use the finished product) and they will perhaps keep you somewhat accountable.

If your side project involves multiple difficult problems then perhaps you need to accept that it is not going to be a 1 person job to get it done too the quality standard that you aspire too.

It's actually very easy to generalize.

The answer is endorphins. The harder the problem that you solve, the bigger the fix you get. Hacky solution is not the solution, you know it, so you don't get much. If you realize and accept this as the cause, the solution to your problem (which I think is very common between the programmers) is obvious.

Whilst the desire to strive for perfection is in any craftsmen, ultimately what matters is getting the product to the customers and solving their needs.

Only after solving their needs should you start to indulge in the craft for what the customer cannot see, the internal quality.

I also sometimes describe this as "be lazy"... don't do anything that doesn't need to be done (for the benefit of the business and your customers).

I think it's actually "have kids" that solves this procrastination problem. With having kids you get so many constraints on your time that procrastination is no longer an option.

BTW, going down the rabbit hole occasionally isn't a bad thing IMO. But if you go down every rabbit hole, it can slow you down.

I think you spend time weighing each design because you're unclear on your final vision of your product. You haven't definitely answered what your values and needs are that you're trying to solve.

This may be a hard question to answer. This is why MVP are neat, you can get a product out to consumers quickly and use their response to develop values and a overall direction.

I find you if you think too far ahead, you don't leave any room for random disruptive opportunities that can occur in each step.

So know where you're going. Figure out and focus on the next step that gets you close to that.

Over time, your rabbit holing will develop in a direction called a 'specialization.' Take note of which areas interest you and what areas have opportunity and rabbit hole that way.

Long term hack: meditation.

How long with each, how much of each... that's going to take a lifetime to learn and figure out.

The problem is that you're interested in the wrong aspects of your side project. You're interested in learning and solving the code problems that come with your project. You have to shift your perspective to the human side of your project. You have to learn to see the user's issues as the problem to be solved.

Let's use an "reviews" site as an example. Your users cant find reviews, that annoy them. What you're solving is not a programming problem; your problem is "make it easy for users to find reviews". That is your problem. Your problems is not what the best algorithm for ratings. None of the code stuff matters. These users do not care about code.

When you've shifted your perspective to the human side you'll start to see hacky solutions differently. A crappy algorithm for ratings isn't a poor solution, it's a perfect solution. You've solved the reviews problem. You have a perfect solution.

The problem is that most of aren't actually excited about the human side of our side projects. This is because we get excited about things that interest us, and what interests a programmer is usually programming. It's not surprising we choose projects with interesting code problems.

Start reflecting on the human/sociological side of things before you choose a project. You might be surprised at what you find. Projects that seemed interesting may suddenly become dull and the one's you thought were dull might suddenly become interesting. A human perspective doesn't exclude building stuff with a code focus, just make sure coders are your audience. You and your users just need to be excited about the same things.

There is this classic saying about "building something for yourself" resulting in the best products. As with most sayings, they left out some important information. They forgot to tell to you to make sure that what you're building for yourself is the same thing you're building for your users.

Make your goals align with your users. Otherwise, you'll find yourself trapped down the rabbit hole.

Some extra thoughts

Don't forget that all your tricks to solve programming problems work on human problems. For example, for a reviews site it's easy to phrase the problem as "make it easy for users to find the best reviews in the most efficient and enjoyable way possible while allowing them to simultaneously book and view and and and....". Break it down into it's simplest elements first, just like you would a programming problem. The simplest element of the problem is "make it easy for users to find reviews"; best reviews are a separate problem.

You're probably good at solving programming problems, breaking them down and being productive. You know all the 37signals posts, all the design patterns, and can quote re-work by heart. Use those principles you have learned and apply them to the human side. Everything your learned about productive programming applies to productive life.

Also sometimes feel I read way too much

One post that really catched my atention that's related was: http://www.ribbonfarm.com/2010/03/18/the-turpentine-effect/

I related to it in the way that I find it difficult to see myself as a super nerd technical robocoder, I need to create too, can't not think about creating and ~wholism~ in this because it's the reason I started to learn it in the first place, so I accepted that

So, well, I'll buy a board, outline stuff and start on them, maybe I'll do an Output Week in that I'll try cranking projects for a week and this is good because of learning and etc like other ppl said here, but learning how to focus is just as much important, so is actually getting-things-done... so it's like "there's no problem in running as long as you know where you're going"

I think the anxiety increases as you don't do stuff and the more you do the more you also learn what's worth doing, no? Before I tried setting out to do a complete project by myself I'd probably accept the invites of "Hey let's start this project, you code!" more, now that I know the work it takes I'm way more picky because I know it doesn't matter if I build a complete skeleton for this but then there's nobody up to even do a fucking design for it and I won't be willing too... so it rings "not worth it"

If it's something that's clearly not ever seeing sunlight, if it won't be at least instantly useful for me, if no one is buying/using it(in the future)... why do it? even a painting has to have a meaning and a purpose

Aren't the best actors the ones that are picky about work they'll take? Aren't the top industry guys who get teased by Facebook, Google and most promissing startups picky? I'm pretty sure they are... I'd guess it's because they know they are basically unable of half-assing stuff, so they need to choose well what they'll set out to do.

I'd say that's the positive version of people with this personality trait, the negative version of people with the same personality trait are the anxious, lost, starting-a-new-thing-everyday-and-never-finishing-anything guys... so IMO it's something for each person to work out