
Ask HN: Percent of time spent developing internal tools? - asher_
An issue I am facing at work at the moment is determining how much of my time I should spend developing internal tools (automation etc) that allow me to be more efficient in my job versus developing client-facing software.<p>How do you make a determination of how much investment you should make in this area? I need to keep value flowing to the customers in the short term but obviously internal tools allow me to do this better in the long term.
======
6ren
Tools are automation.

Rule of thumb: do it manually three times before automating it. That way,
you'll have a fair idea of what it is supposed to do (which is the hardest
part); you know whether it's predictable enough to be automated; and an how
often it will come up.

 _A programmer is someone who takes 5 hours to automate something that takes 5
minutes_

~~~
tinco
There are three cases for a task that takes a regular human 5 minutes and a
computer programmer 5 hours to automate:

1 The task is very complex, this means that there's a lot of room for human
error. A well-tested and specified program will reduce this fragility and
improve predictability.

2 The programmer is inexperienced, the programmer either is not a very good
programmer (yet) or is using tools that he is not familiar with. The 5 hours
mostly go into the learning process of the programmer, not in the task itself.

3 The task is simply something computers are not very good at, or hasn't been
fully explored. For example speech recognition. The 5 hours go into exploring
the domain, creating something reusable that the world could profit from.

In my opinion, it's always worth it, isn't that nice?

~~~
btilly
You miss the most common issue:

4 The task requires interacting with a system that was not designed with
automation in mind.

For instance you have a third party GUI which you don't have source to that
you have to push a button on based on what it is showing you. So you wind up
having to research this, work around it, etc. Often you can use a GUI test kit
to solve it temporarily, but that is going to be fragile - on the next upgrade
it will likely break.

In this case automation may not be worthwhile. What you're learning is not
connected with what you do. The automated solution is more fragile than the
human one. The time saved is not that important.

~~~
rhizome
The solution to that is to reimplement the functionality outside of the GUI.
Write the GUI out of your life.

~~~
btilly
Great if you implemented the thing with the GUI that you have to interact
with. Not so great if it comes from a third party.

Even if your people did implement it, depending, getting rid of the GUI might
be easier said than done. I remember an Access app that I had to automate
stuff around. In the end the best solution to all of the different popups that
were buried in their app was to make the machine it was on remotely
accessible, and have a watching process page someone if it took too long. A
human would then look at the screen, and press the appropriate button.

~~~
rhizome
Yeah, I worked with something like that about 10 years ago. We had an ETL
process that involved like 20 Access macros manually triggered in sequence
(click, wait, click, wait...over several days per cycle). I converted this
entire process to a Perl script that would be pointed at the raw data
directories, clean and process the data and throw it into the DB. Access was
taken completely out of the equation. It took a few months, but I was an IT
guy who had never programmed before and had to learn Perl at the same time.

------
JangoSteve
Of course it all depends on your actual business, income (i.e. how much time
can you _afford_ to spend on internal tools), needs, etc. Since you're talking
about client-facing software, I'm going to assume you're in a software
development company, which narrows it down a little.

For our company, we don't develop internal tools for any given problem until
it's a real pain. You'll know when it's an actual pain (as opposed to just an
idea), because you'll keep seeing it crop up in your day-to-day work, and if
you talk about it with others on your team, they'll say, "Yeah! I know exactly
what you mean, I was thinking about that the other day."

We wait until it's a real pain for a couple reasons:

1) We want to make sure we're not wasting our time; we're mostly engineers,
and we all get real, unadulterated enjoyment from building things. It's only
too easy for our enjoyment and enthusiasm for building a solution to fool us
into thinking there's a problem. But usually, if everyone on the team is
instantly excited about the prospect of solving some problem, that's usually a
good indicator that it's a real problem.

2) The longer we can put off something, the more information we'll have when
we go to solve it. In other words, we give ourselves time to figure out first
how we _want_ to work, and then find or build tools around that, rather than
trying to fit our system to some tool.

The second reason is really important too, because when we sit down to really
think about every facet of a problem, how it affects our work and how we have
been putting up with it and working around it, we'll often realize the best
solution isn't some new piece of software; it's merely leveraging some tool we
already have slightly differently.

For one example, we started losing feedback and updates in email, because some
person on the team would get left off the email (e.g. someone hit "reply"
instead of "reply-all"), and we wouldn't realize it until some task went un-
resolved for several days. We could have built or implemented some sort of
support desk system and made it the clients' responsibility to make sure
tickets are submitted and then integrated it with email. Or, we could just
create a unique email alias for each client, which copies everyone involved.
The latter took 2 minutes and improved communication threefold.

If you keep growing, you'll eventually reach a point where anything less than
a full-blown software solution just isn't going to cut it. But by that time,
you've seen how the team deals with the problem over time, and all the
pitfalls of other smaller or less-custom solutions, and you'll be able to
build the most awesome solution possible.

~~~
asher_
Thanks! I'm in agreement with you, I don't like spending a lot of time on
tooling if it isn't needed. I like only very simple tools, as I find complex
ones inevitably add friction in the workflow and thus end up unused.

~~~
dmgrow
I would suggest that it depends a lot on the stage of the company. Early on,
only build enough that you can legitimately track the key metrics. Other than
that, focus on building the product.

As the product and business matures, it's likely that you will need to spend
more time to ensure that the business can keep up with the product.

As a PM, I make sure that we've had to do the same thing manually a number of
times before we spend any engineering time on automating it, or setting up the
tools so that a non-engineer can do it.

------
meaty
I have actually automated myself out of a job before so be careful!
Fortunately the company saw the value of this and contracted me on an adhoc
basis.

I wad the unfortunate administrator for a large virtual mail system in the uk
which was run off exim and flat files across 50 machines (one per client).
This took 3 guys full time to manage as the machines were all over the uk and
needed regular maintenance. I centralized everything and moved it to two
virtual postfix machines backed with postgresql and wrote a web interface and
set up scripts for backup and administrative operations, including integrated
dns management.

Now, 12 years later, these guys are one of the larger hosting companies in the
UK...

Automation almost always pays off, but not how you anticipate in some cases.

~~~
rhizome
Why they didn't promote you for re-architecting their systems like that is a
failure on their part. I hope your contract fee made up for that!

~~~
meaty
a) they were assholes, b) yes - daily rate was previously my weekly rate :)

------
taliesinb
My answer at this very moment in time is around 1/3. Which seems a lot. Though
these projects have been less about automating drudgery and more about
unlocking new workflows and ways of understanding things.

When I first started working at my job I sneakily built a suite of tools for
visually working with NL grammars (instead of just buckling down and writing
the grammars). Although I don't maintain it anymore, it's now used by an
entire team. It was great that my then-managers saw that I was on to something
and let me obsess about it for a few weeks.

I recently spent a few weekends writing static code analysis tools to analyze
our codebase. There is some pretty exciting stuff that can be done there. It
hasn't paid off quite yet, though I know it will.

Another project I've been juggling with my 'official job' is a nice new
logging system with all kinds of cool features. There are so many challenges
about making a piece of infrastructure work technically _and_ culturally
across a large team, however. It's much easier to estimate how long the
technical part will take.

I've also written a bunch of tools that just failed to find any real use. In
retrospect, these were mostly vanity projects of some kind, whereas the
projects that have worked have always been about scratching an itch that I can
see other people have (whether or not they articulate it).

------
andrewf
I can think of many things that could qualify at "internal tools." The
question is broad and the answers depend on what your business is, who works
there, and what they do (yourself included!)

Possible "internal tools" that come to mind are: (1) Analytics (2) CRM-y stuff
for the sales and support people (3) Deployment and testing stuff for
technical people (4) CMS-y stuff to enable non-technical people to experiment
with, or even implement, things that are client-visible. And for technical
people to do so more efficiently. (5) Backup, redundancy, disaster recovery
etc

If none of these seem more important than others, I guess you need to ask:
where is the business now? Where will it need to be in <x> months? How are we
getting there? What are the pain points? How can the above improve this
(versus implementing new features)?

One cherry-picked example: should technical staff time be spent on improving
sales? If sales are your company's limiting factor and you don't have enough
salespeople, making them more efficient is really important. If you're still
in a soft launch-ish place, then it's not. If you have enough salespeople but
not enough leads, maybe mining your own data, or public data, could produce
some.

~~~
rhizome
You're generalizing too much. 1 and 2 are actually internal _products_ and
should be planned and budgeted out by higher-ups as projects. I have no idea
what would constitute #4 the way you describe it, but it sounds like a
business requirement that should also be dealt with as a project or internal
feature. Lastly, #5 is business continuity and a primary function of IT in
general.

This leaves #3 as a sector where the manual/automation see-saw can be improved
by tooling. The others are more aspects of business that _do not happen
otherwise_. Tooling is more concerned with easing the jobs of people who
implement the thing being streamlined with the tool. In a sales context this
would be something like switching to a shared address book or whiteboarding an
in/out scheduling chart, both of which they can do for themselves.
Interdepartmental deliverables are not tooling.

~~~
nowarninglabel
I think he was actually fair spot on, I work on an "Internal Tools" team, and
we do #1 - #4, in addition to maintaining the bug tracker, finance and audit
tools, and a translation/review product. #5 is definitely DevOps though.

~~~
rhizome
You both may be right in terms of the semantics and divisions of labor in your
respective companies/experiences, but they are more general than the
historical sense of tooling in a systems context.

------
PommeDeTerre
How is this different from any other business decision that you'd make?
Estimate the costs, estimate the benefits, estimate the rate of return, and
proceed if the rate of return exceeds that of the alternatives.

~~~
markbnine
I agree. Do the math. If you don't have the man power and the math makes
sense, contract it out.

------
chewxy
My team and I spent a lot of time developing internal tools (and/or
integrating them with other tools like CampaignMonitor and stuff).

Throughout our massive amounts of failures, we've learned some things (and
some things the hard way): there are basics, and there are nice-to-haves. Get
the basics right.

Our basics include an entirely zeroMQ based service-oriented architecture
(after 8-9 different iterations over the past 4 years). We spent a lot of time
writing a ZMQ based DHT, and service wrappers (kinda like zeroRPC[0]). Other
basics include a standardized front-end A/B testing based off genetify[1],
email service (with AWS' SES), paypal service, our fair matching services and
many others.

The one thing to note is that these basics didn't come overnight though. It
came through many many iterations of different products that failed. We merely
picked up the components that worked best.

There were things that were nice to have like automated testing and deployment
options that we considered. Even using tools like Puppet and/or Chef, we
determined, were not quite a good use of our time, considering our release
schedule isn't all that crazy, and deployments can be managed by one human
being.

But our situation does not apply to you. You need to think about what fits
your situation, and your styles. Maybe you are the kind that pushes a release
to production servers every 5 minutes. Then yes, having an automated test and
build system would be extremely useful.

Start with simple stuff like crons and tiny scripts. Let your internal tools
evolve on their own. Don't jump in to write massive automation for your
systems. Instead, do it in small parts, just enough to help you.

Hope this helps.

[0] <http://zerorpc.dotcloud.com/>

[1] <http://genetify.com/demo/>

------
twanlass
Very little. A lot of people automate waaaay to early (it's the engineer in
you). Of course this is very dependent on your situation (startup vs
agency/shop).

I always suggest a digital assistant first. Pay $4-10 an hour and let someone
else do it. Wait until that cost becomes a thorn in your side.

------
jakejake
First off, such a great question.

I think there is a danger in trying to make your own life too easy at the
expense of making your customers lives easier - especially during a startup
period.

I think for tools you have to just do the math. If it takes you 10 minutes
once a week and the tool you build takes three hours to develop - then the
effort will start paying off in 18 weeks. If it takes you 8 hours to build the
tool the you don't break even for 48 weeks (almost a year).

There could be other reasons to build a tool, though. For example the process
is error-prone and a tool will ensure it is done properly. Or it enables a
non-tech staff to take over something that allows them to help customers
better.

Lastly one task is always worth automating - that is data backups.

------
gbog
I'd leave the short term to short sighted people. In chess, in wars, in
business, in software, the world belong to those who see far away.

As Napoleon said: l'intendance suivra, which means that short term and
ancillary problems will be solved anyway.

Another way to say this is that if you keep the general long term direction,
it will be ok to use dirty hacks to fix in fee minutes the short term
problems, you will know how to do it in a way that we'll not hinder the long
term path.

But make sure your tools and automation are really in line with the long term
direction.

For example, building a chrome plugin to speed manual testing of a web app
would be wrong, architecting it to be fully unit testable would be right.

------
wrath
I think before you start automating you business and processes you first need
to know if your product will get any traction. Spending the time on making
sure you have the best possible MVP is the most important thing, IMO. You may
be launching a lot of tasks manually and modifying the database directly from
the console at first but so be it. The fact is that creating internal tools is
very expensive and only indirectly provides benefits to your product.

You'll know when you'll need to build better automation to your product; there
won't be enough time to get everything done in a day.

That said, there's a downside to following this approach. When you will need
to build automation tools you'll be conflicted with spending your time on
automation or on improving your product. Improving your product directly
provides benefits, tools indirectly provides benefits. It can be sometimes
hard to convince yourself or others that you need to spend x% of your time on
internal tools.

Where I work we're at a point where automation is very important. As the CTO
of the company I constantly have to challenge the CEO and others that spending
money on tools will give us some benefit in long term. The rule (or high level
design pattern) that we are trying to implement is for every module of our
product, there is a tool to understand the data better for debugging purposes
(e.g. a UI to see what step in the workflow the process is in and/or what
errors happened, etc), and if data is generated, there's an API/Service to get
the data out easily.

------
twelve45
I think Yishan Wong's thoughts on this really hits the mark: [http://algeri-
wong.com/yishan/engineering-management-tools-a...](http://algeri-
wong.com/yishan/engineering-management-tools-are-top-priority.html)

It is incredibly important to get your tools right so you can operate as
efficiently as possible. At my company we didn't get around to this till much
later, partly due to the dilemma you mentioned abt focusing on customer-facing
software vs. internal tools, and partly due to the fact that the people who
would benefit the most from automation (e.g. finance folks, inventory
managers, etc) don't have the same obsession with automating away mundane
tasks like us engineers do.

This lack of tools hurt us quite a bit once we started scaling, in ways that
impacted some customer relationships (because occasionally we didn't invoice
or ship correctly, etc). We have since invested significantly in tools, but we
should've done this to begin with.

To make the customer vs. internal tools compromise easier, you may even want
to get some basic tools in place via contract work/offshoring and then revisit
these later once you have the time/resources. The hard part really is
determining what to automate, and how much, and those are decisions that _you_
should make. The development itself can be done by someone else.

------
asher_
I see I need to add more specifics about my situation.

I work at a university developing research software (primarily - some is also
external customer facing). My customers are internal customers, thus
profitability and other business measures are not directly my concern.

The biggest issue I face with this is that my customers are non-technical
people, so if I were to tell them that our infrastructure is now software
defined and I have implemented analytics that will allow me to optimise the
caching layer, they will most likely look at me with a puzzled expression and
perhaps wonder what I'm actually doing since those things don't provide direct
value to them.

The types of internal tools I am thinking of are things like; analytics,
automated deployments, CI server etc.

One comment suggested I do a simple calculation like any other business
decision. This is something I cannot for the life of me work out how to do.
Internal tools will increase my efficiency in the future, so I guess at best I
could estimate a payback time for some of these tools (like automation). Even
so, how short a payback makes it worthwhile? For other tools, like analytics,
how can this type of calculation be made?

Additionally, there are some things that are the "right way" to do things,
such as software defined infrastructure vs managing it manually (which I
currently do). Investing time in this won't have any immediate payback, but is
nice to have in place in the case of a large failure when I need to recover
quickly. These types of value calculations for me are not easy to make.

sourthyme, I like the idea of spending one day a week working on this stuff.
Perhaps I need to have a conversation with my bosses/customers about this and
try to explain to them the value that internal tooling will bring.

Thanks for the answers so far, please keep them coming.

~~~
tinco
In my opinion, you should not tell people who do not understand how analytics
will improve your caching optimization that you are working on analytics. You
should tell them you are working on improving the caching layer. If you sound
confident that implementing analytics is required for this job to be done
properly they will have to trust you in this, _you_ are the programmer.

You are correct in your assertion that when people ask you what you are doing,
they want to know what value you are providing for them. So don't answer with
technical details, you already know they're not interested in that :) Whenever
a business stakeholder asks you what you are doing, trace back how what you
are doing provides value for them and inform them of that.

Note that value does not have to be very direct. Caching makes your
application faster, its business value is that the customers are happier when
using the product. It is for the marketing department to decide wether that is
worth it.

I would not bother too much with figuring out the exact numbers, they are not
important. You should for yourself decide wether something is useful or not,
if you are an experienced programmer I think this intuition will be right.
Almost always, tooling will reduce your technical debt.

Tooling is not only automation, it is also process documentation.

~~~
asher_
I appreciate this reply a lot. This type of situation is a new one for me, I
guess I should just bring up the subject as a whole and see how it goes. I
hope I don't run into too much resistance since it will result in a short term
decrease in deliverables.

------
markokocic
Everyone on this thread focus on business decisions should something be
automated based on perceived monetary value.

However, there is another important aspect in developing internal tools. In a
lot of setups, developers rarely have a chance to investigate new technology,
new development approach, different architecture, even different programming
language on a day to day basis. Work is mostly done using predefined set of
technologies.

For tools, the technology requirements are almost always much more relaxed, so
the developers themselves gets to choose and try different approaches. While
not every tool might end up being success in strictly measured time/money
saved sense, in the long run it will help developers stay up to date with the
technology, and even apply lessons learned in the real/product development.

As a bonus, employee retention might improve as people won't get bored so soon
if they have a chance to try doing something new from time to time.

------
drewcrawford
As a client-oriented business, I am a proud disciple of the Gospel of Tools.

1\. Tools are one of the few institutional advantages that are in the domain
of the programmer (rather than the MBA or the sales force). There are many
shops out there that suffer from a lack of technical leadership and compete on
the basis of (absolutely incredible!) strength of the sales force. Software
developers don't win at that game.

They can, however, win at having the best tools. You have to have some unfair
advantage, and in most developer-run studios, tools are easier to cultivate
than sales methodologies.

2\. Customer-facing tools provide customer-visible value. If you're writing
error-reporting software that helps debug issues on customer systems, that's
providing visible value. If you have client dashboards that show the status of
their project and predict ship dates, that's visible value. If your network
protocol is faster than REST for your vertical, that means faster software,
and that's visible value.

It's all about giving your customers something to hang their hat on to select
your shop. If they are getting "good vibes" from the other sales guys, but
your software automatically builds them a Powerpoint deck of the project
status to present at the weekly update meeting, then there's some class of
customers who will prefer working with you, and you are better insulated from
competition.

3\. If you and your competitors don't do tool investment, then all the boats
will rise at the same speed. But if you are doing tool investment, your boat
will rise faster. At some point you can do the same job and provide better
value than anyone else. When people figure that out, your competitors are
forced to first, cut prices to remain competitive, and second, raise expenses
to try and achieve tool parity. But you've already got a long head start, so
they will either run out of money, license the tools from you, or just focus
on the more sales/MBA-receptive customer base, and shrink to a niche offering.

4\. As I've hinted at, tools can be valuable and can be licensed to other
shops. The only way to really get out of the client "rat race" is to become a
product company. Tools are an easy way to make that transition; you have a
built-in customer base and an intimate knowledge of the problem domain.

------
mvkel
Easy! If this is your company, dedicate as many resources as possible to get
you to profitability the fastest.

Would adding better analytics allow you to convert more prospects? Sounds
pretty important for getting business, despite it being an internal tool.

If you're pre-revenue, or pre-sustainability, all your efforts need to depend
on becoming sustainable. Otherwise, you're making a great product for nobody.

------
sigsergv
When you really need tools, couldn't survive without them and couldn't buy
some 3rd-party so then you have to create some internal piece of software. And
it's not a matter of time or resources, just accurately calculate will it help
or not and decide.

Of course internal software MUST be developed using the same quality standards
as your frontend program (VCS/tests/code review/etc).

------
pbiggar
As a datapoint: when we (<https://circleci.com>) did our market research, we
estimated that a development team spent about 12% of its time on the parts of
testing/deployment that we would replace (mostly setting up a CI server w/
continuous deployment).

------
sourthyme
For new projects or existing projects that desperately need tools, %80-%90
percent is good. The rest can be used for small changes that have visually
large results to keep the uppers happy. After you have a basic tools stated
%20 or one day of the week to work on tools is good.

------
stcredzero
This is the wrong question to ask. The questions to ask are about ROI and
opportunity cost. If you spend 6 months on a library that lets you do a 6 hour
job in 1, then that's a good investment, if you can afford the up front cost.

~~~
dhimes
Depending on how often you have to do that 6 hour job, of course.

------
mokash
Well, I'd say that if you consider a task to be long, repetitive, time-
consuming and something you have to do often, work on something that wil make
that process a lot shorter and help you save time in the future.

------
malkia
For a game studio, that's significant portion of your time.

~~~
pyrotechnick
Particularly the larger studios.

In those, the answer to the OPs question, "Percent of time spent developing
internal tools?", at least on certain teams, is in excess of 90%.

------
typhonic
A part of the consideration should be the cost of buying the tools. If you
have to build all your own tools, there may be another element to the
analysis.

------
w_t_payne
Initially, about 80%, dropping to much less later on.

------
bosky101
imho, internal tools form a big part of your competitive advantage

you could spend all your time building these, and still just have scratched
the surface.

