
Ask HN: I have a mental block designing software. Time to quit? - porker
This is embarrassing to admit after 15 years building web applications, but every project I hit a brick wall decomposing software.<p>I know the data inputs; I know the outputs. My prototype works. But my mind freezes when I try to rewrite the prototype into something maintainable that will run for 5+ years.<p>High level planning is fine, it&#x27;s getting from what needs to happen to how to decompose into classes&#x2F;methods&#x2F;libraries, dependencies (logging everywhere) and organise message passing.<p>My current project is choreographing 3 enterprise systems to work together. Once past the lack of documentation it&#x27;s conceptually simple; my code however is tangled.<p>Related, I have a mental block with drawing flow-charts. I can&#x27;t isolate tasks, decisions, actions. The chart becomes spaghetti, or too detailed&#x2F;too sparse to be useful. This is difficult when people want the system described on paper.<p>Either:<p>1) I need to learn something
2) It is an innate mental block, and I need to change fields<p>If I&#x27;m going to continue in software this needs to change, as I don&#x27;t feel I&#x27;m delivering quality. The more senior I get the less I can stick with prototyping or &quot;Just Works&quot; as an outcome.
======
eigenrick
What you're describing to me is you've built a very solid structure for
reasoning about programs, but it is too rigid, and too focused.

I would _heartily_ recommend learning a new language. You need to shake things
up a bit.

Two reasons:

1\.
[https://www.wikiwand.com/en/Linguistic_relativity](https://www.wikiwand.com/en/Linguistic_relativity)
\- Our language shapes our thought. This is true for spoken and programming
languages.

2\. You cannot properly understand a language unless you learn another. This
is partially based on point 1, but it needs to be said otherwise.

Since you're in web, I would recommend learning Elm for the front-end, and
Haskell in the back-end. Why? They will force you to design your systems
differently? Is it better or worse? That's subjective. The important thing is
that it is different.

For data design, have a look at things like datalog. Datomic is a good
playground for that.

~~~
porker
> I would heartily recommend learning a new language.

I've installed a nice selection of languages tonight. Can anyone suggest
sample projects to do as learning exercises? My ideas a) don't need the
language, b) are too complex or c) involve CRUD, and I do far too much of that
every day :)

TIA!

~~~
estefan
Read "Functional Programming in Scala" or any other good book on functional
programming to see things differently if you've just stuck with imperative so
far. There's also [http://iloveponies.github.io/120-hour-epic-sax-
marathon/](http://iloveponies.github.io/120-hour-epic-sax-marathon/)

Read books on concurrency ("Learning Concurrent Programming in Scala" if you
carry on with scala) and then you'll find different ways to write performant
concurrent code - for CRUD sites or other stuff.

I'm not pushing scala rather exposure to other ways of doing things you might
not have come across - functional programming, futures, akka, etc.

But yeah, also design patterns. And have you read
[http://highscalability.com/](http://highscalability.com/) for architecture
ideas?

~~~
porker
Thanks for your comment!

> But yeah, also design patterns. And have you read
> [http://highscalability.com/](http://highscalability.com/) for architecture
> ideas?

Lots. I would love to have clients in that league (I greatly enjoy scaling,
optimization, performance tuning)

------
blt
I'm going to take a wild guess and say you don't really care that much about
the purpose of your software. You're not a game developer who's been an avid
gamer since age 4, or a silicon valley entrepreneur who truly believes their
web app is going to change the world. You've identified a niche that's liable
to make some money, and are diligently plugging away at an implementation.

Your lack of inherent motivation from the product domain means you don't get
any satisfaction from just getting the requirements met, so you hunger for the
satisfaction of making a beautiful architecture. But all architectures have
flaws. As a competent engineer, you notice the flaws in whatever you design,
so your only source of satisfaction is compromised.

I would try to find work on a problem that excites you. When you get that
giddy rush just from seeing your program give a correct answer. I think most
software developers have felt this way about some project during their
learning process. Look back and try to remember where it happened for you.

You're in trouble if software craftsmanship is your _only_ source of
motivation. It's a great motivator, but it is not enough by itself to carry
you through a big project.

Now that you're in this project, it may be impossible to quit. I would find a
side project to work on that provides more satisfaction. That should make it
easier to plug-n-chug away at your current project without becoming depressed
at the inevitable small failures in elegance and design.

I've never understood how developers can be happy writing CRUD apps or wiring
together a bunch of pieces made by someone else. You gotta search for meaning.
Even if you take a pay cut, it's worth it to find work that you care about.

~~~
porker
> I'm going to take a wild guess and say you don't really care that much about
> the purpose of your software. You're not a game developer who's been an avid
> games since age 4, or a silicon valley entrepreneur who truly believes their
> web app is going to change the world. You've identified a niche that's
> liable to make some money, and are diligently plugging away at an
> implementation.

Spot on. I walked into this career by accident, the money was good and I
stayed. No, I don't know what I actually want to do - something I've never
known :)

> You're in trouble if software craftsmanship is your only source of
> motivation.

Spot on again. Pride in what I do is my only motivation at this point. I don't
get to see happy users, find out how I've improved lives, get user feedback
and _make things better_ for someone.

> Now that you're in this project, it may be impossible to quit.

It was impossible to turn down, as they're a good client and provide somewhat
interesting work (always unusual tasks). Otherwise I wouldn't have touched it.

> I've never understood how developers can be happy writing CRUD apps or
> pushing ads. You gotta search for meaning. Even if you take a pay cut, it's
> worth it to find work that you care about.

CRUD is the most soul-destroying code to write as doing it with good
craftsmanship is impossible (there I said it). Concerns overlap between the
front and backend. I do share your sentiment, but pay cuts are hard (I suspect
I undercharge at the moment, making them harder still).

Thanks for commenting!

~~~
richardboegli
Try your hand at Game programming, its great to mix things up and the
communities are great:

Unreal Engine 4: [https://www.unrealengine.com](https://www.unrealengine.com)

Maybe a mod for Unreal Tournament or Ark Survival:
[https://www.unrealtournament.com/blog/](https://www.unrealtournament.com/blog/)

[http://www.polygon.com/2015/8/14/9153379/ark-survival-
evolve...](http://www.polygon.com/2015/8/14/9153379/ark-survival-evolved-
hunger-games-mod-contest-survival-of-the-fittest)

Unity: [http://unity3d.com/](http://unity3d.com/)

Construct2: [https://www.scirra.com/](https://www.scirra.com/)

------
manyxcxi
Learning something is always helpful in some ways but it may not be exactly
what you need. It sounds like you may be at a point in your career where you
may be suffering some imposter syndrome when it comes to making the decisions
to move the PoC code to something more final. I hit a stage as well, years ago
where I was paralyzed by the 40 different ways to architect a Java web app and
all the arcane crap that came along with it. In general patterns only serve to
better understand and maintain your code. That should be your primary goal
during this time, is this readable and maintainable? After that, think about
ways you can make it more slick. Eventually, you get over the hump and those
patterns become part of your writing style and you don't even think about it
anymore, until you hear about some new cool pattern (MVC became THE THING
during my professional time) and you figure out how to incorporate it. After
this similar block I had in my career I never really had another, until...

I had a different type of block earlier this year. For the first time in my
career (I'm 32) I left my employer to work 100% for myself. I spent the first
six weeks so twisted up I'd go days without a line of code written- luckily
there was enough in the way of infrastructure and Ops tasks to give me
something to do. I think the stress of it all just kept me from being able to
focus like I need to in order to develop.

I started taking breakfast with the wife and kids, going to the gym every
single morning, walking around for at least 15 minutes at lunch, being home by
4:30 (instead of 6:30/7). I basically focused on everything else and after a
week or two it just started coming back and I feel like I'm faster and less
stressed than I've ever been. Even if I'm not I'm happier and my family is
significantly better than before, so I'd do it again in a heartbeat.

~~~
porker
> It sounds like you may be at a point in your career where you may be
> suffering some imposter syndrome when it comes to making the decisions to
> move the PoC code to something more final.

Definitely! I'm the imposter that imposter syndrome was defined around ;)

> That should be your primary goal during this time, is this readable and
> maintainable?

Oh, that's an interesting twist. Move the focus away from craftmanship of the
present-day to future-proofing. It may be more ugly but if I can fix/extend it
in 12 months without missing a beat then my job is better done.

> I spent the first six weeks so twisted up I'd go days without a line of code
> written- luckily there was enough in the way of infrastructure and Ops tasks
> to give me something to do. I think the stress of it all just kept me from
> being able to focus like I need to in order to develop.

I'm 31 and that's how I've spent a lot of time recently. Especially when not
working flat-out on a project to meet a deadline. Give me one of those and I'm
happy: head down, pounding it out.

------
sebastianconcpt
Sounds like a challenge on synthesis and perhaps design problems. I suggest
you to not do anything drastic but use a pair of fresh eyes on your project.
Manage to pair with someone and observe what this new guy finds challenging.
You have to be open to refactorings or code style improvements. Also the guys
must have to be good otherwise it could misguide you. What about using 3
instead of one? if they don't know each other you know the biases are under
some control. After getting that quality feedback, you can ask yourself those
questions again. You'll know better

~~~
porker
Good advice & I'd like to, my challenge is I work freelance/direct and don't
have other programmers around to ask.

I think paying for code reviews and some mentoring is the only way I can get
this feedback.

~~~
moberemk
Why pay for review though? If you're working on open-source stuff you can try
[http://codereview.stackexchange.com/](http://codereview.stackexchange.com/)
for help, or reach out to some trustworthy people in your network as a fresh
pair of eyes.

~~~
yeukhon
> My current project is choreographing 3 enterprise systems to work together

Well, don't think that's going to be open source.

------
jaredhalpert
Time to take a break. Decrease the amount of time you spend programming, and
increase the amount of time you spend with friends, family, being outside,
etc. You need to remove yourself; I recommend balance.

------
cableshaft
It sounds to me like you've dug yourself into too much of a rut of how
software _should_ be made that you can't just dive into it anymore, and get
bogged down in doing everything 'perfect' that you don't get anything done at
all.

I tend to just dive in, make a bit of a mess, and refactor/clean things up
before I check things in. I think it helps that I have tackled learning quite
a few different languages over the years and have done some working for
startups (which has no time to get things right, it's just got to get done
before the whole ship sinks).

Just pick a little piece of what you want to work on, get that working, then
add another bit, and another bit, until you have a full feature but still
runs, then clean it up, check it in, and move on to the next tiny piece. Don't
worry about documenting every little thing.

Learning a new language, especially a scripting language that can get things
done in a few lines of code and people don't obsess over diagrams and graphs
of the system could help. Python filled that niche for me.

Good luck!

------
weavie
> it's getting from what needs to happen to how to decompose into
> classes/methods/libraries

Perhaps the problem doesn't fit into an object oriented solution. You could
try learning a functional language and see if you find it fitting better. I
struggled for years with OO and was never happy with my solutions. Learning
functional was my road to Damascus experience.

If you have to go OO, there have been plenty of people who have thought long
and hard about how to make OO a suitable solution. Some classic books you may
want to consider :

\- Design patterns : elements of reusable object-oriented software (Gang of
four) - there are people who swear by this and turn it into their hammer for
which everything becomes a nail.

\- Patterns of Enterprise Application Architecture (Martin Fowler)

\- Refactoring: Improving the Design of Existing Code (Martin Fowler and Kent
Beck)

\- Growing Object-Oriented Software, Guided by Tests (Steve Freeman)

~~~
haxeny1
I would add

\- Object Oriented Software Construction (2nd. Bertrand Meyer)

I would recommend Eiffel if you want to elevate OO design. The issue with the
GOF book that I have is that there is a slant towards composition as opposed
to inheritance. When done right inheritance does result in compact code.
Multiple inheritance cannot be avoided (Scala proves this by trying to bring
in traits). Repeated inheritance actually helps reduce the code size as one
could inherit the same method twice to implement a doubly linked list from a
single linked list.

Note: In functional paradigm (Haskell) composition over inheritance makes
sense, though in object oriented languages choosing an approach that has a
compositional bias can impose some constraints that could be avoided
otherwise.

~~~
porker
> Object Oriented Software Construction (2nd. Bertrand Meyer)

That was the first programming book I ever bought! Sad to say it's sat unread
on the shelf for 13 years...

------
joeld42
You're thinking about it wrong. Don't think of it as "prototyping" or "just
works". If it just works, you're done, there's no need to rewrite it to make
it enterprisey.

What about it needs fixing? If it works for 100 items but not a million then
fix that. If it works but is difficult for others to change the code,
rearrange things until it's clearer. If it works by using some weird 3rd party
dependency that wouldn't be easy to deploy and update, just remove that
dependency, etc. If it works but you don't have an easy way to build or deploy
it, then work on the build system. If it's buggy, fix the bugs.

Don't make things more abstract just for the sake of abstraction. Don't add
message passing or enterprise logging just because they're enterprisey. Ask
what problem you are trying to solve by adding these.

------
zaphar
jaredhalpert says take a break and he's right. eigenrick says shake things up
a bit and learn a new language he's also right. Sebastienconcpt says get a
fresh pair of eyes, he is also right.

You just need a jolt. If it's feasible take vacation and build something fun
in a brand new language that is really different from anything else you've
ever used. Set goals like I'll work on this no longer than X hours a day where
X is small. Don't set deadlines for completion. The goal here isn't a finished
product it's to have fun tinkering with something that will break you out of
your block. And pick a language with a welcoming but passionate community that
can give your feedback on your code.

I do this all the time to keep myself fresh working on open source projects or
playing with new things. It expands my toolbox and gives my brain a break from
the constant legacy enterprise spaghetti-ware we so often work on.

And when it comes to your money earning work don't discount the Just Works
outcome. When you are glueing together legacy systems sometimes there is no
"beautiful and elegant" solution. Sometimes due to time and legacy constraints
your only option is duct-tape and bailing wire. Having an outlet to write more
elegant stuff in your spare time can be a valuable sanity saver.

Side Note: I wonder if there are any good places online to have virtual design
reviews done? Somewhere where the high level goals and constraints are listed
and people comment on the best way to achieve them? Language and platform
agnostic preferrably although some suggestions will inevitably work better in
some languages than others.

~~~
porker
> And pick a language with a welcoming but passionate community that can give
> your feedback on your code.

Any recommendations? The PHP community (where my commercial work is done) is
not that. And far too fad driven.

> And when it comes to your money earning work don't discount the Just Works
> outcome.

You're not the first person to say that to me. I should start listening!

> Having an outlet to write more elegant stuff in your spare time can be a
> valuable sanity saver.

Agree - if I despise what I write every day, it wears me down. I haven't
worked on anything fun (or open source) for years, focusing on making money.

Thanks mate!

~~~
zaphar
It's hard to say without knowing what's already in your current toolbox. But a
partial list might be:

* Erlang - Novel approaches to fault tolerant system design. Fail fast and recover somewhere else. Functional without the Type wizardry so you learn about immutability, pattern matching, and the typical functional language tools like folds, maps and so on.

* Clojure - A lisp that runs on the JVM. Learn about Code as data and how that empowers meta-programming. Also a functional language with interesting takes on how to share data in an immutable by default world. Lazy collections of a sort so you can learn about laziness.

* Haskell - Don't let the Category theory stuff scare you. Very interesting type system that can open your eyes to new ways to abstract things. Speaking of abstraction Haskell takes abstraction to a whole other level. Sometimes you can't see the program for all the abstraction going on. Militantly functional.

* Forth - Stack based programming, very small and compact. Radically different than anything else.

* Rust - Maybe you want to go a little bit closer to the metal for a change? Forces you to think about exactly _how_ you share data in a multithreaded world. Targets the same world as C++ but perhaps with an easier to enter community.

* Python - Cleaner more elegant language in the same family as PHP with a very welcoming community. Batteries included so it's easier to get started than any of the others.

* Go - Concurrency done right. Lots of tooling builtin to the distribution toolchain.

~~~
porker
Nice list, and overlaps a lot with my "Languages to try one day" Post-it note
(though I have Elixir instead of Erlang). And I'd never heard of Forth -
amazing what I learn!

My current toolbox is:

Professional level: PHP, databases, frontend development (but haven't picked a
JS framework due to choice paralysis)

Experience of: C, Python, Matlab, Fortran77, Prolog (my favourite, most mind-
bending experience yet. Magic!)

I tried learning about monads recently.
[https://www.youtube.com/watch?v=UvD1VjRvGIk](https://www.youtube.com/watch?v=UvD1VjRvGIk)
made sense (I was nodding my way through) but I kinda missed what a monad was
- this just seemed common sense that my languages didn't allow :)

Thanks, you've inspired me to play.

------
nostrademons
Don't decompose up-front. Instead, take your prototype and and start adding
the polish necessary to turn it into a finished product. When a subsystem
starts getting so complicated that can't understand it when you're changing
it, look at that subsystem in isolation and figure out how you might refactor
it to eliminate code duplication, create simple interfaces, etc. Eventually
you end up with clean system boundaries separated by message passing, but each
individual step doesn't require that you hold the whole system in your head.

------
ArekDymalski
How exactly does the "mental block" look like? Does it mean that you see
several possible approaches and struggle to decide which one to use? Do you
feel the pressure of the "into something maintainable that will run for 5+
years" requirement?

If the answer to 2nd and/or 3rd question is "yes", your description sounds
like a specific phase in the process of your professional development.

A lot of people grow with their competences (be it knowledge or skills) to the
moment when they start to realize the limitations of their competences. The
moment when they see a bigger picture of their work, the broader consequences
of a possible error, they feel bigger responsibility etc. It's a moment when
doubts start to creep in and self-esteem plummets. The paradox of this phase
is that objectively such people are very competent and capable to solve the
problems they face. But they become paralyzed by the pressure and self-doubt.

If this reminds you your situation (at least partially) then the solution is
called coaching - a method created to help competent people who lost their
faith in own competences, overcome the difficulties and grow further.

Other (often proposed, even here) solutions like switching a job, going to
some seminar, being more controlled by your boss or taking a vacation do not
solve this specific problem, and quite often lead to bigger decrease in self-
esteem.

Hope it was helpful and coherent - it's quite complicated subject to present
in a short comment.

~~~
porker
> How exactly does the "mental block" look like? Does it mean that you see
> several possible approaches and struggle to decide which one to use?

Sometimes yes (paralysis by choice), but in this case no. In this case defeat
and "I can't do it" are my initial response, and I shy away from programming.

> A lot of people grow with their competences (be it knowledge or skills) to
> the moment when they start to realize the limitations of their competences.
> The moment when they see a bigger picture of their work, the broader
> consequences of a possible error, they feel bigger responsibility etc. It's
> a moment when doubts start to creep in and self-esteem plummets. The paradox
> of this phase is that objectively such people are very competent and capable
> to solve the problems they face. But they become paralyzed by the pressure
> and self-doubt.

That fits. I know I know nothing. Then I wonder what it means to know. What is
meaning? You get the picture - it's a destructive deconstructive spiral
brought around by doubt. I inhabit "Imposter Syndrome", still trying to find a
way out.

> If this reminds you your situation (at least partially) then the solution is
> called coaching - a method created to help competent people who lost their
> faith in own competences, overcome the difficulties and grow further.

Do you have any pointers for finding a competent coach please, vs someone
who's overconfident in their own abilities?

Thanks for taking the time to comment, you've been really helpful.

~~~
ArekDymalski
>defeat and "I can't do it" are my initial response, and I shy away from
programming.

Was it always the case? Also: how do you finish your projects then?

>a competent coach please, vs someone who's overconfident in their own
abilities?

I'd say that anyone demonstrating a strong confidence either in the results
(ie. guaranteeing to help you) or own skills (ie. talking a lot about self)
has a giant red flag over his/her head. Good coaches have an interesting
mixture of modesty (no bragging), genuine focus on other people (more
questions than statements) and assertiveness - healthy interpersonal strength
when it comes to the coaching process, tools etc. Ideal coach will:

1\. Be very clear upfront about the distribution of responsibility between
him/her and you - it should be ~50-50. That means that being a coaching client
is a quite a challenge and it isn't always an easy experience. Anyone painting
a sunny, broad staircase up for you is a red flag :)

2\. Recognize you as an individual - without jumping to conclusions too early.

3\. Describe you the idea of coaching and how it is different from advice,
counseling, therapy and other forms of help which are sometimes confused with
coaching. Anything what smells new-ageish or like proven receipt for success =
mega red flag :)

4\. Describe you the process you will go through. Usually it's something like:
analysing your needs->setting development goal(s)->performing some
action->discussing the action+feedback->setting next goal and this repeats
until your main goal is reached (or the agreed upfront number of sessions is
finished). It's very important that coaching is like a good project: it has
clear, measured goal and is meant to end at some point. If the end is lost in
a distant fog - yes, it deserves red flag.

5\. Be very precise and metrics-oriented - there's no place for unclear mumbo-
jumbo in coaching. Unless someone likes red flags of course :)

So, these are the essential characteristics. Also it is very important that
you should feel comfortably in contact with that person - imagine openly
discussing your struggles, doubts as well as receiving feedback form him/her.
Regarding formal requirements: respected certificate (ICC/ICF/IIC/IAC) is a
proof that someone was trained enough to know the process and toolset. But
certificate isn't necessary because the process&tools are actually quite easy
to learn - everything depends on mindset/attitude and that's something what
could be shaped during the training but as usual - no guaranties.

Feel free to ask if something requires more info.

~~~
porker
> Was it always the case? Also: how do you finish your projects then?

No. Until I was 12-13 I had a real 'can do' attitude; I executed and others
looked on in amazement. And up until 5-7 years ago I loved programming and
making the computer do things.

Thanks for your excellent pointers to assessing a good coach - those are going
to be really helpful. I only have a question about 3) as I don't know what the
difference is myself :)

> 3\. Describe you the idea of coaching and how it is different from advice,
> counseling, therapy and other forms of help which are sometimes confused
> with coaching.

------
eternalban
Your #2 is drastic. Your #1, reasonable, and Martin Fowler is your friend
[1][2].

[1]:
[http://www.martinfowler.com/articles/enterprisePatterns.html](http://www.martinfowler.com/articles/enterprisePatterns.html)

[2]: [http://www.amazon.co.uk/Enterprise-Integration-
Patterns/dp/8...](http://www.amazon.co.uk/Enterprise-Integration-
Patterns/dp/8131725081?tag=duc08-21)

[edit: beefed up]

------
nerdy
If the code is tested then refactoring the ideas contained within is the way
to go. Do the smallest refactorings possible. Even well-respected programmers
who've authored tons of books and given talks do not sit down and 1-shot a
program. Elegant programs are the result of refining and designing.

The earlier in the development process you get to the refactoring and
isolating minimal testable components, the less of a tangle you have between
components to refactor; it's a lot like the wire tangles we're all so familiar
with). This not only means less refactoring, but it also means that the
existing code is easier to understand.

The tests are code. Treat the tests just like the rest of the code. I prefer
test-first but if you want to pull your cart in front of your horse then go
for it.

If the code is not tested I'd recommend testing it. There are really only 3
options to go from untested code base to tested code base: \- Try to put tests
under existing code (often inefficient) \- Refactor the system incrementally
as time goes on and features are added and bugs are fixed (slow process but
over time test coverage grows) \- Throw it out and start over (ouch but...
greenfield; if you do this you should _NOT_ add _any_ new features to the
live/production system or you'll end up in a rat race between the two forever)

Also if you're able to take any time away (preferably measured in days) the
fresh perspective might help too. Sometimes that little bit of total
separation is enough to come back and read a method name and wonder why the
hell it's even in that particular class. This scenario could be an early
warning for burnout too, take care of yourself!

~~~
porker
Testing. I have never done TDD; I put it off in the beginning as an extra cost
to clients (or time lost for me), and then when it became trendy I (as ever)
disliked the trend :) Ian Cooper sums up my thoughts better than anyone else:
[https://vimeo.com/68375232](https://vimeo.com/68375232)

I'm doing a rewrite of a small project and that seems ideal to give it a go.
To me functional testing is much more appealing, testing a simulated user's
experience through the website, yet TDD seems more popular - as easier?

> This scenario could be an early warning for burnout too, take care of
> yourself!

I've been in a burnout cycle for the last few years - a holiday in July was
the first time I switched off in 15 years, then a health scare last week and a
friend in palliative care have pushed me to reconsider. The lure of money
isn't as strong as it once was.

~~~
jerf
I think TDD is a great exercise to do a couple of times, and a great way for a
new developer to grow some good habits very quickly. I'm much, much less
convinced of its utility for people with experience.

However, you really do need some automated testing. One of the reasons I love
automated testing is that it permits me to make (nearly) monotonic forward
progress on my tasks. If part of your burnout here is that you are just
[expletive] tired of making changes in one bit of the code just for it to
create 6 more bugs in the rest of it, automated testing is the core solution
to that. I used to get that type of burnout... I don't anymore, except when
I'm stuck crawling around somebody else's code base where I can't add testing
for some reason.

Also, rather than posting another comment, rather than a new "field" you may
just need a new _job_. It isn't all CRUD coding. Mind you, there isn't really
much out there that is fascinating cutting-edge research every day, either;
even that involves a lot of grunt work. But it isn't _all_ CRUD work out
there, either.

~~~
porker
> However, you really do need some automated testing.

How would you distinguish automated testing from TDD?

> If part of your burnout here is that you are just [expletive] tired of
> making changes in one bit of the code just for it to create 6 more bugs in
> the rest of it, automated testing is the core solution to that.

I do so much legacy maintenance/development, and yes this would make my life a
lot easier. Not for my edits creating bugs (ok, sometimes ;)) but for platform
runtimes changing and ensuring nothing breaks (PHP 5.3 => PHP 5.6 for
example).

A new job... yes it's easy to get pigeonholed into a field. The work I've
loved the last few years has been tasks like "develop a custom plotting
algorithm so overlapping points aren't hidden (and labels don't overlap)".
This has been the work that's fallen in my lap; I haven't found a source for
such interesting projects (the ones that push me into areas I don't know).

Though that exciting plotting algorithm also required an entire CRUD admin
interface which nearly killed me (from lack of motivation). I need to
outsource those bits...

Alternatively there is the "productized consulting" route, but until I have a
product idea to productize I'll stick with what I do well: problem solving :)

~~~
jerf
"How would you distinguish automated testing from TDD?"

I would consider the distinguishing characteristic of TDD is that you write
the tests first. There's varying levels of strictness there from little more
than that restriction, to varying levels of dogmatism on writing code that
_only_ makes the test pass.

I consider this a great way for a new programmer to learn how to really
_design_ code. In fact it's one of the only ways I know to practically teach
design. However, once you have experience, the really dogmatic TDD involves
writing simple tests, then writing code to pass the tests that you, in your
experience, _know_ will be driving yourself into a local optima that will not
be able to carry the design. But when you lack that experience and knowledge,
well, it's a great way to get it while learning a good dose of YAGNI. (Which I
do not consider an absolute, but is something everybody needs to exercise.)

In contrast to "TDD" which I consider optional, automated testing has provided
so much value to me in so many diverse parts of the stack that I consider it a
no-brainer; it is a positive value to any project. The exceptions are things
where the code is _so_ not-amenable to testing that it is essentially
untestable. Note this _only_ applies to code written by _someone else_ ; if
you wrote it, fix it to be testable. In particular, GUIs are a _royal_ pain
here. It is my belief this is not a fundamental attribute of GUIs, but a GUI
not written to be testable is nigh untestable.

(And if you find it isn't bringing you value, you're probably doing it wrong.
There are wrong ways to do it. Test code is real code; apply all the arts of
refactoring, DRY, and learning from experience as you do to "real" code.)

~~~
porker
Thank you, that makes sense and _why_ TDD is so worthwhile for me to start
with.

For external services (I rarely write code that doesn't rely on at least one
SOAP endpoint) is there any alternative to mocking the service and the
response data? And then should your tests also verify that the remote service
works according to its contract?

As it would affect data in the live system, I cannot think of a way to safely
do it, but 3rd party systems have 'changed' their behaviours before now, and
that's a horrible bug to track down.

~~~
jerf
"For external services (I rarely write code that doesn't rely on at least one
SOAP endpoint) is there any alternative to mocking the service and the
response data?"

In practice, no. Though I prefer some variant to dependency injection to
"mocking". I consider the testability a first-class concern and code
accordingly. Exactly how this manifests varies suprisingly widely between
various languages. You don't mention your language, so I don't have specifics.

"And then should your tests also verify that the remote service works
according to its contract?"

Yes, but rather than thinking of that as "automated testing", you're better
off thinking of that as "service monitoring" that happens to look like
automated testing. Consider something like:
[http://n8v.enteuxis.org/2011/06/integrating-nagios-with-
test...](http://n8v.enteuxis.org/2011/06/integrating-nagios-with-test-driven-
development/)

------
lukebuehler
I recently hit a wall like this as well after designing many apps and
enterprise systems.

It was a problem that seemed simple at first, but I just kept banging my head
against it. I did exactly what some people described here, switched to a
functional language, but that didn't help. I started doubting myself.

I'm though it now, and I like the solution, although it could be better. Here
are some points that I've picked up along the way:

\- Remind yourself that you are not only generating value when you actually
write code. When you take a walk and think about the problem, draw stuff on
paper and so on, this is still work. You are still doing exactly your job.

\- This gets hard when coming up with a design takes longer than usual. It's
likely that you have run into a complex problem without knowing it. Endurance
& patience is key here.

\- Attack from different angles, don't consider the time wasted if you try
something and then end up throwing it away, you will have learned something,
even if it just eliminating one possible design option.

After you are through it, you might end up cherishing the experience.

------
KuhlMensch
[https://www.youtube.com/watch?v=GAFZcYlO5S0](https://www.youtube.com/watch?v=GAFZcYlO5S0)

[https://www.youtube.com/watch?v=iukBMY4apvI](https://www.youtube.com/watch?v=iukBMY4apvI)
(javascript module workflow that helped me (frontend dev) to transition to
modules)

help?

~~~
porker
Watching Simon Brown now & will look at his book too, thanks.

To take the current project, I have a CMS, Salesforce and ExactTarget. I'm
writing the glue to automate taking new content from the CMS, finding SF users
with the right permissions to see it, and emailing it to them via ExactTarget.

All good & working until... do I model it as 3 objects? Which object is
responsible for creating the DataTable at ExactTarget (populated with SF
data)? How is data going to be passed around cleanly between my
representations of all 3 services? Or should I represent the system more
abstractly as EmailToSend, EmailContent and use all three services within
these classes?

~~~
mistermann
Code it each way and post it online and someone will tell you _with absolute
certainty_ that you are Doing It Wrong because <some fashionable pattern>.
Take this person's recommended implementation and post it online and someone
else will tell you, again with _absolute certainty_ , that you are Doing It
Wrong because <some other fashionable pattern>.

All the while never mind that in 9/10 cases you could have written it either
way and it would be of sufficient quality and performance, run more or less
without issues for years, be conducive enough to periodic refactoring, etc.

I have a similar feeling as you about software today, although I'm not sure if
for the same reasons. I look around me and everything is just so damn
complicated. Writing code that is small and understandable is ruthlessly
mocked, while a gigantic, complex marvel of engineering (that does the exact
same thing) is looked upon as The Right Way. Functionality that would take a
week or two to implement 15 years ago now takes a month, or three.

------
themullet
Design Patterns! As you said your problem is "how to decompose into
classes/methods/libraries"

Book wise, I learnt them from "Head first design patterns" which is useful and
fun or for a quicker overview -
[http://www.java2s.com/Tutorials/Java/Java_Design_Patterns/in...](http://www.java2s.com/Tutorials/Java/Java_Design_Patterns/index.htm)
.

Also remember if you have other people around you delegate / meet them /
bounce ideas together.

~~~
porker
I bought my first "Head First Design Patterns" 12 years ago, worked through
it, and it didn't make sense. I've dipped into PoEAA and Gang of Four since,
with the same problem. The way messy life is meant to fit patterns... it
didn't gel.

Any thoughts?

And thanks for your great reminder about meeting up to bounce ideas around.
I'm going looking for them.

~~~
themullet
Difficult problem, I do know what you mean about fitting messy life into
patterns, that said some can still be invaluable at times.

Something like a singleton would be handy to stop multiple creation of an
object, for web design mvc just makes life a lot easier by decoupling
different parts and in theory for your project something like an observer
pattern could be created to notify other systems to a state change.
[http://www.java2s.com/Tutorials/Java/Java_Design_Patterns/01...](http://www.java2s.com/Tutorials/Java/Java_Design_Patterns/0170__Java_Observer_Pattern.htm)

In the end there's many ways and approaches, maybe even brushing up some class
design stuff in another language might help.

Good luck on your journey, sounds like you'll be fine.

------
stdbrouw
Relax. Anyone who writes software will code themselves into a mess sometimes.
Learn to love refactoring.

~~~
jordz
+1

Edit: This. One of the things I see a lot of people running into is this mis-
conception of "prototype" -> "production". The prototype is your proof of
concept, you take this application and you refactor it and optimize it as you
move into your production life cycle. You take the bits out and put new bits
in that are either un-optimized or don't work for the entire scope of that
specific problem.

Working code is better than nice code that doesn't work. Refactor small bits
as you progress.

------
cturner
About 2004, I reached a stall state. I did some serious wilderness time and it
took about six years to settle down into a new zone.

This was the trigger for me: (1) I had a development career where I tried to
build every application centred around databases; (2) I now had enough
experience under my belt to join the dots on where this ended up, and to see
that this was futile. But I wasn't able to see past my everything-is-a-hammer.

There is a technique I took a lot of value from in moving on, and I still use
it most days. It is this: think hard, ten times, about a problem that is
tickling you. I went through a notation phase, but now I just type out my
stream of conscious into a text document.

In the past, I would have spent a year building the first thing that came into
my house. Now I can consider a large number of ideas, and have context for
judging them to be good or bad ideas. The first thing I invest development
energy in is significantly better than it would otherwise have been.

It's cool too in that you don't need to be at your desk to do it. You can be
in a restaurant, or sitting on the beach, or on the bus, or on an exercise
bike. My favourite spot at the moment is lying on a couch. The good thing
about doing it away from your desk is it takes away the pressure to rush
through the uncertainty.

"choreographing 3 enterprise systems to work together"

How do you think about message-passing between them? Why do you do it that
way? Think ten times about different approaches you could use. You could use
the exercise to force you to question your passive assumptions. You may find
that you are scared by uncertainty. Your subsconscious could choose to fix on
something familiar, than to admit that it does not know. All the more reason
to do this away from your desk. Take a walk.

Feel free to ping me via my account email if you want to discuss your
situation in more detail.

------
MalcolmDiggs
Is it possible that you're suffering from "Analysis Paralysis"?
([https://en.wikipedia.org/wiki/Analysis_paralysis](https://en.wikipedia.org/wiki/Analysis_paralysis))

If you have 15 years of experience, I'm fairly confident that you know how to
solve these problems. In fact you probably know 10 different ways to solve
these problems. Maybe the overabundance of choice is becoming the bottleneck?

If so, perhaps there's a way you can simply force yourself to commit to a
methodology/problem-solving-method before you start tackling the actual
problems. Or, you could try breaking the development cycle into smaller, more
digestable, iterative chunks (so that you only have to think about the next
small step instead of the whole path forward).

~~~
porker
> If so, perhaps there's a way you can simply force yourself to commit to a
> methodology/problem-solving-method before you start tackling the actual
> problems.

I really like that idea. So for a given project where there are 5 ways I can
say "I'm going to solve it using approach 4. It may turn out to be better or
worse than the others but that's not what matters; using this method is what
I'm doing" and have that as my goal, not _finding the optimum method amongst
the 5 possible ways_.

~~~
MalcolmDiggs
Exactly. It's almost like recreating the pseudo-efficiency that newer
developers seem to possess (I only know one way to solve this, so I'm going to
do it that way, and do it fast). In your case, the loss of choice is
artificial, but it reaches the same result (getting it done).

------
abathur
When you explain your problem, I feel like we might share a trait, which I
could pithily describe as difficulty
separating/ranking/comparing/organizing/categorizing things. So, I find it
very taxing to make the kinds of decisions you describe. I usually get there,
but it's heavy lifting for me.

I don't think there's a silver bullet for you; you probably aren't incapable,
but you may need to suffer through sub-optimal organization to hone this
skill. What I will say is that, if you haven't worked on an existing project
with highly complex organization (ideally one that has _problems_ ), doing so
might help you. While I envision working directly on the project when I say
this, there are cases where working with complex dependencies you need to get
custom results out of can also be useful.

The point is that you want to be working with complex software (not simple
interfaces) that requires a strong understanding of its internals to
accomplish what you need. You'll know you're working on something on roughly
the right magnitude of complexity when you regularly realize that your
understanding of how the project or dependencies work is wrong, and your
solution doesn't account for some hitch in how they work. The meta-point is
that you're working with sub-optimal decisions other people made, and you
either just have to deal with them and simmer to yourself about how _you_
would reorganize this mess if it was within scope, or you have to start
refactoring their code (or building your own interface with it) that solves
the problems with their design. The meta-meta point is that we're better at
seeing problems in work other people have done than our own, but with effort
we can train our eye on other people's work and (eventually) learn to focus it
on our own.

At it's core, I think this is really about the cognitive overhead associated
with working on and understanding a given implementation, so working on things
that are cognitively overwhelming provides you an opportunity to learn about
how to manage cognitive overhead. The result is hopefully that you learn to
wrangle that complexity into an understanding that helps you do what you need
without doing surgery, you build an interface that reduces the cognitive
overhead enough that you can do what you need, or you refactor to accomplish
the same.

------
eyeinthepyramid
For me, a lot of this stuff only comes through trial and error. It's not
ideal, but I think over time, both the trials and errors improve in quality.

Basically as I work in the code, I see little things that could be easier or
make the code clearer. Then sometimes I get ideas for big changes. Sometimes
the changes make things worse, but that's why we have revision control right?

I think the same goes for drawing. Just get something on paper and look for
small changes. As you make changes, eventually you might see a larger change.

------
rbrogan
Whatever it is, it is only so different from your past work. It may help if
you sit down and try to put together an argument either that you can or you
cannot do the work. I would be skeptical and rigorous about it and try to
force it into defining exactly what it is that you cannot do. Chances are it
is a matter of framing up the work, and in trying to prove the work is
impossible, you will probably frame it up and make it be possible.

------
quantum_state
It seems to me it may help, if you have not done so, to refresh yourself with
some basic abstract math .. starting with set theory, topology, abstract
algebra such as group theory and group representation .. it would illustrate
how to walk up and down on the abstraction ladder to get to the essence of the
problems at hand .. once you have got the essence .. you would be liberated
and can spin it any ways you like ..

------
omginternets
If at all possible, take a vacation where you do anything _but_ code. Pull the
plug, if only for a week.

~~~
porker
I hate to say, I'm just back from a week's vacation. I struggled to switch off
& went down with shingles (brought on I am convinced by the work stress)

~~~
coda_
Give yourself some time to get back to healthy... any kind of health issue can
cloud your judgement. I always try to refrain from making big decisions when
I'm not feeling good. Shingles sucks, and the pain can linger... hope you're
doing better now!

~~~
porker
Thanks! The antivirals are doing magnificent work. The pain... is slowly
improving. Not bad for 1.5 weeks from first appearing.

------
palidanx
Do you have any training in UML? I feel that learning basic things like domain
diagrams, sequence diagrams, and state diagrams help decompose complex systems
into maintainable chunks.

------
Raphmedia
You could stay in your field, a field that you obliviously love, but try at
another company where you would have better ressources / coworkers at your
disposition.

Take days off too.

------
chipsy
Everyone's style is going to be differ on the high-level stuff - and you are
surely well beyond the point where any random blog post blabbing about
software architecture can guide you.

Keep walking back to the user and business requirements when you're in doubt.
In senior technical positions, you aren't focused on immediate issues in the
code, so much as on developing the right processes to attack those problems.
Software isn't expected to last very long, and when it does it's both
miraculous and ugly, as the tools and techniques become more and more arcane
over time. If the solution you develop can satisfy all parties for an extended
period, it is as correct as you can hope for - and part of that involves
choosing a technical process that can be kept running in an environment of
some ineptitude.

When you know the requirements, model the data first. Data comes first because
data lasts longer than code. Flexible data lets you do more with less code.
And if the data is very well defined you can usually use fewer language
features, too - which is a bonus to maintenance later. Think of features in
data terms first and lifecycles later.

The best-practices cruft you've learned over time - that's going to have to be
sacrificed when you identify ways that your designs can be simpler. You may
identify a pattern that is useful but not "with the grain" of the languages
you're using. That doesn't mean it's wrong, just that you're breaking past
some known boundaries of programming.

As you know, most projects get dirty in order to ship. Not doing that means
doing basic research to find the ideal technique, whereas there's always room
to tolerate a few more hacks, global flags, mystery-meat callbacks, etc. Your
best defense is for the data to be so good that all the code can remain
disposable and the system will still turn out more-or-less the same way if
rewritten from scratch, with a different-but-equally-awful set of hacks and
kludges.

Under the pressure of strong data, code that sticks around after several
cycles of disposal becomes the library code by default - and internal APIs
will become strong enough to prevent a "lava layer" from taking root, because
they'll consistently solve the immediate problem faster than any wrapper
layer. The maintainers will feel like geniuses because they'll keep finding
clever ways to use the existing data better, instead of mashing out another
travesty of reflection and classes-on-top-of-classes.

If data isn't sufficient, you may also have to define a protocol. Protocols,
if they work correctly, compact the state management into a small part of the
app, and leave you with more data.

~~~
porker
> When you know the requirements, model the data first. Data comes first
> because data lasts longer than code. Flexible data lets you do more with
> less code. And if the data is very well defined you can usually use fewer
> language features, too - which is a bonus to maintenance later. Think of
> features in data terms first and lifecycles later.

Boy am I SO GLAD to see someone say that! I think of my approach as "Data-
Driven Development", rooted in my days of scientific programming. The data is
what matters. On my largest project to date (8 months, entire business system)
I spent 25% of the time just on the data store schema. Needless to say it
worked & evolved without issue over 5 years.

Yet (at least in the PHP community) the fad is currently Domain-Driven
Development, BDD, and architectures ("another travesty of reflection and
classes-on-top-of-classes").

> The maintainers will feel like geniuses because they'll keep finding clever
> ways to use the existing data better

Oh yes, that's how my oldest systems are. Spaghetti code in parts, but the
data keeps on giving.

Thank you for taking the time to write this, you've ignited a small flame of
enthusiasm in me and reminded me why I enjoyed doing what I do!

------
jacooqs
5+ years? That's crazy for web applications. I cant imagine any code lasting
more than two years nowadays

------
ihsw
Quick question -- do you use unit tests?

~~~
porker
No. I missed the boat, then I felt functional testing was more important. Ian
Cooper sums up my thoughts better than anyone else:
[https://vimeo.com/68375232](https://vimeo.com/68375232)

------
mikekchar
It's hard to tell without looking at your code, but I suspect that you are
stumbling in to a wall that many do. What's in your favour is that you
recognise it, so you will probably be able to do something about it.

Systems grow over time. At the beginning, they are simple, which means that
they are forgiving. If you make a mistake, it is easy to see and correct. As
the software grows, it gets progressively more complex. Problems hide for a
long time and when they present themselves it is often very difficult to see
the actual problem.

What I'd like to do is challenge you to look at it a bit differently. Think
about the number of lines of code in the OS, system libraries, language,
libraries, frameworks, etc that actually go into your code. Even if you only
write 2 lines of code, there are probably hundreds of thousands of lines that
your 2 lines are sitting on top of. Why are your 2 lines easy to understand?

Obviously because there is a great separation between the complexity of the
underlying systems and your code. What you need to be able to do is to provide
that same separation in your own code.

As you mentioned that you don't work with others, I will recommend some other
approaches. However, if they don't work, I recommend that you change to an
office job where you can get the mentorship you require before you leave the
field altogether. 15 years is a lot of learning to give up.

#1: Read good code. One of the biggest problems that programmers have is that
they rely on their own code for examples of "good code". You are at a huge
advantage because you recognise that you need to improve your code. What
should you be reading? Read library code from whatever libraries you are
using. As you are doing "web development", I would advise you to shy away from
reading the source code of frameworks like Rails, but rather read the
underlying class libraries for the languages you use. If you are not using
languages that are open source, then it will make your life much more
difficult, but try your best to find open source and free software on the
internet to study.

#2. TDD. Every time. All the time. Not BDD (although BDD is an excellent
practice in its own right). You want to do TDD. This will force you to think
about your interfaces and to decompose. There are many books about TDD. Some
of them are good and some of them are bad. Read Test Driven Development: By
Example by Kent Beck. It is probably not the best book on TDD, but it will
give you the basics without confusing you.

If you are working with legacy systems and don't know how to TDD with those
legacy systems, read: Working Effectively With Legacy Code by Michael
Feathers. It is a very, very difficult book. Learn it all.

You may hate TDD. Many people do, but using it will make you a better
programmer. Personally, I don't think it's bad if you abandon it eventually,
but master it before you do. This will simplify the process for you and break
it down so that you can consider things in more manageable pieces. It will
move you away from "Just Works" because it is god awfully slow (or at least it
feels that way). Just pay attention to the effect it has on your code.

From where you seem to be now, to where you want to be, I estimate you've got
2-5 years of effort. Don't get discouraged if it doesn't come right away. This
is a discipline where you can get orders of magnitude better over time. Try to
be as patient as possible.

I hope this helps.

~~~
insanebits
I agree with a TDD, but it really depends on the framework/cms you're working,
some of them are incredibly hard to write tests agaist. One of the examples
would be writing unit tests for the WordPress plugin, if you're sticking to
the WordPress way of coding unit testing is practically impossible because
there is lot's of database access and events which do mess up.

You can write plugin folowing OO principles and abstract things out to be able
to mock them as nessesery. It really depends on the coding standarts you're
folowing.

As for unit testing. In my personal experience it enforces you to write
cleaner code, because when I started I had to do heavy refactoring to be able
to test my prototype, but as I wrote more testable code I started separate
unrelated parts which reduced amount of spagetti code.

Another way to achieve cleaner code in my opinion would be design patters
which lets you separate code logically(for example MVC), and make some things
cleaner(I'm big fan of laravel's Facades for example).

------
davelnewton
I'm confused why you list only two options.

