
Automate Everything - the hacker way - tomblomfield
http://tomblomfield.com/post/17092502705/automate-everything
======
simonw
"I decided to roll my own in this case because I was interested in learning
about email handling after watching this great railscast from Ryan Bates. And
because paying $9 per month per user for something I could probably write
myself in a couple of hours seemed silly."

I find myself having to fight against this instinct almost every day. There
are a couple of problems with hacking for a couple of hours to save a few
bucks a month.

Firstly, you now have an additional piece of software to maintain - you're
committing yourself to an unknown quantity of future work.

Secondly, your software won't get any better without you actively improving
it. The nice thing about software you pay someone else for is that it gets
better over time.

It's a tough instinct to fight though. Building things is Fun. It's just that
there are probably other things you should be building that are more important
to your company.

~~~
tomblomfield
I know what you mean, and I'm torn. This project was built in about 4-5 hours
on a Saturday, mainly as a chance to learn about mail handling. It's ugly as
hell and there are no tests (!), so I don't know how well it will hold up as a
long-term solution - I guess we'll see.

I just find the $9/month per employee a bit repugnant - if we grow to 10
people, we'll be paying over $1000 a year for a piece of software that parses
subjects out of emails and stores attachments. Doesn't seem like a good deal.

~~~
6ren
As a rough guide, you might factor in another 8-10 hours, because, according
to Fred Brooks, a proper programming "product" takes about x3 as long as a
"program".

Taking a wider view, there's an appeal to "cloud components", where all
_components_ are rented - including ones as simple/trivial as this. For this
to work, it has to be cheap enough to be a no-brainer (esp for projects _made_
of such components). Say, 1/100 - pennies - e.g. 9 cents /m/e. In the long-
term, you'll see free open source alternatives, and maybe even free hosting
(cloud providers supply it free, to support their hosting of your actual
app)... and maintained.

------
mmahemoff
This kind of thinking is as useful to non-tech founders as it is dangerous to
technical founders.

For the non-techs, it's important for them to get their heads around the
possibility that they may be wasting large quantities of worker-days doing
what could be automated with a bash script.

For the techs, just because you can doesn't mean you should. Deciding to
automate something when there's a cheap SAAS offering is often a destructive
thing, mainly because even developers are prone to underestimate complexity
and maintenance costs, and to goldplate a solution that's already "good
enough".

I'm not saying Don't Automate. I'm just suggesting to be very cautious about
undertaking any automation initiative. The things to automate are the baggage
which you will take on your journey, and as such, they should be directly
related up your core product/service. Something like a mailing list handler is
probably not, and typical of the kind of underestimation that can take place
(what happens when you discover your homebrew mailing list is blocked by every
big mail provider's spam filter?).

~~~
tomblomfield
I agree with you - I think most programmers have this mentality automatically,
whereas the non-tech guys often don't. We're hiring for non-technical
operations/customer-service/biz-dev roles a the moment and we find ourselves
leaning towards people with a technical background.

I just don't know how much this "instinct to automate" can be taught.

------
djtriptych
I'll also add that this is the real value behind learning at least one
language REALLY well. For me it's Python, for this guy it's Ruby. That means
the language, it's ecosystem of platforms, and the standard lib.

Might be important to mention in these days where a new significant language
seems to break onto the scene every 8 months or so.

I submit that any working programmer ought to know (or be about the business
of knowing) at least one language well enough that they can go from idea to
working prototype without spending a ton of time reading documentation and
wondering about the canonical way to do something. One language where you can
automate stuff and solve small problems.

~~~
ams6110
For me it's shell scripts. Yeah they aren't ultimately as flexible as
something like python, but piping together grep, sed, awk, and other command-
line utilities, and some simple control-flow capabilities (if/then, loops,
case statements) you can do a lot quickly.

And you can always write a little piece in Python, if any of the standard
utilities are quite what you need.

~~~
sjs
What about Ruby and Perl and their ability to shell out as easily as you do in
a shell? (with backticks)

You have a real language available with (arguably) fewer weird things and
warts to be aware of or remember, better string manipulations and lots of
other advantages, but if you really just want to send something through `awk`
or `bc` you can do that just as easily as you do in bash and capture the
result in a Ruby or Perl variable.

Seems like a win-win situation. (Unless you have an exceptional circumstance
with requirements about running anything but shell scripts or something like
that.)

~~~
ReidZB
I write quite a few Ruby utilities (for myself) and this is what I do: use
backticks to harness the power of shell utilities, but have Ruby glue to do
more complex tasks with those tools.

For example, I recently wrote a quick Ruby script to test various parts of my
internet connection (ping default gateway, test DNS resolution, etc) because I
was bored and my internet connection was down. I used backticks to do things
like grab a nameserver from /etc/resolv.conf and then ping the server, but I
had the surrounding Ruby code doing things like parsing the output using
regular expressions and doing some logic on the results.

Sure, I could have written the thing in pure bash, but Ruby has better text
manipulation capabilities when it comes to things like parsing (in my
opinion), and with the backticks, I still was able to harness the power of the
shell anyways.

I'm not sure how portable such solutions are, but for a home-grown testing
program written on a whim out of boredom, it was certainly enjoyable to have
both the power of Ruby and the shell available.

------
jakejake
This is a perfect example of the old saying that a good programmer is a lazy
programmer. Lazy programmers hate doing the same monotonous things over and
over and will go to great lengths to avoid it!

This is a close cousin to what my grandfather used to call "working to get out
of work" in other words spending a lot of mental effort to avoid manual labor.
This usually involves fashioning a tool or device to move something that could
have been moved more quickly with simple, but back-breaking manual labor.

I tend to do both of these things a lot!

~~~
itmag
Lazysmart people are good people.

People who are eager to get up in the morning and toil in the sweat of their
brows are deeply suspect in my book.

I don't see much good coming out of that kind of attitude - when will they
stop and ponder the inescapable nuances of life?

The world needs more lazy inventor types laying around daydreaming on
lazyboys, not extra widget crankers milling around mindlessly in labyrinthine
office hives.

IMNSHO, of course :)

------
chrisdroukas
This is how and, more importantly, why I learned to program. My office uses a
horribly unintuitive centralized database from which employees 'pull reports'
to prove to supervisors the work they've done.

One of the guys in the office was maintaining a monthly report that involved
him burning 3 hours a week pulling and compiling reports into a needlessly
complex Excel spreadsheet. I offered to help out, and wrote an Excel macro
that did the job for him.

From there, I figured 'hell, why should _everyone_ have to spend time doing
this?' So I downloaded a copy of VB and learned enough of it to turn the Excel
macro into a simple deployable application that turned an absolute waste of
manpower into an easy-to-use desktop app.

In hindsight, a native application wasn't the best way to do it. Our office
has a strictly controlled internet/intranet policy, and the people maintaining
the database are likely too busy doing just that — maintenance — to build a
more robust data reporting system. That might be the beauty of learning on
your own, you quickly gain insight into how things should and could work.

It wasn't his point, but simonw is right: building things IS fun. I got
started and I haven't looked back.

~~~
mseebach
Perfect is the enemy of good.

If innovation had to sit around and wait for the "best solution" or be overly
concerned with how things "should" work, you'd be cooking your dinner by
pounding it with a rock and we'd all be in awe of enterprise software
architecture.

Innovation is about solving problems with the tools and resources that are
available, and kudos to you for doing just that. If more people would be
bothered (and, indeed, allowed) to learn a little programming, even in
something like VB, the productivity of an average office worker would explode.

------
tfb
I posted this on the blog but I'll copy it here as well. Please forgive the
shameless plug.

I think the OP is right in that wanting to automate repetitive tasks is
inherent to humans. And the problems discussed in this post are exactly the
kind my startup <http://loggur.com> aims to address. The goal is to let non-
programmers automate and also to let programmers automate certain tasks
(mainly data manipulation, analysis, triggers, and reports/notifications) in a
fraction of the time it would take to otherwise hand-code it.

I should also mention that I am fully aware that what Loggur attempts to do is
far from new, but I have yet to see an implementation that isn't a pain to
work with. And it's designed with the current (and predicted future) state of
things in mind and to be flexible enough to adapt to whatever the needs of the
future may be.

------
JohnLBevan
One thing that always surprises me is how many times companies reinvent the
wheel and the need for it. In your post you mention an expenses system;
something pretty much every company will have to implement. Since HMRC require
this information you'd think they'd provide a tool that everyone could "plug
into", giving them full visibility of the data they require in a standard
format with little effort, and allowing businesses to also benefit by just
downloading the relevant information from this service into their own systems
for any internal needs. If all companies were using the same service this
would also be one less thing for new employees to learn, since they'd be
familiar with the expenses process from previous jobs.

On a persona note, I'm definitely in the automater category; on the few
occasions when I've left a task unautomated because it was only supposed to be
a one off it's come back to bite me repeatedly when month after month that
same one off task pops up (admittedly after 2-3 repeats I then ignore the
advise that it's a one off and automate anyway). These days if I can automate
a task in slightly over the time it would take to perform that task, I
automate by default. If there's a significant difference I weigh up the
likelihood of repetition (by myself and/or others) against the extra time
required to automate it.

~~~
dhimes
With tasks I only perform occasionally, automation becomes my record of _how
to do_ those tasks. I find this extremely valuable for complex tasks
infrequently performed. Taking the time to write a script ensures that (1) I
pay attention to what I'm doing, not just run into a snag->google->get the
fix->forget (or never investigate!) what the hell the problem was; and (2)
allows me to quickly check that my procedure is repeatable; and (3) gives me a
step-by-step record of exactly what needs to be done, with no steps skipped
(like when I take notes) because they're "obvious." What is obvious when
you've been working a couple of hours to solve a problem is very different
than what is obvious when you haven't done something in a couple of months.

------
noonespecial
It's OT, but I've found that expense report submission requirements are often
(either willfully, our just willfully not improved) difficult in order to
reduce the rate at which they are submitted.

~~~
kragen
That's interesting. You'd think that would be irrational for the company as a
whole, no? If your employees are paying for their own housing, meals,
transportation, massages, etc., instead of the company taking those on as
business expenses, then for every US$1000 the company saves on expenses, it
has to pay its employees US$1300 or so in higher salaries to make up for it,
with the other US$300 going to income tax, payroll tax, etc. What's wrong with
my thinking, or is this really true?

~~~
noonespecial
Its just one of the dozens of sneaky ways that less than moral companies say
they're paying employees competitive rates and then... don't.

------
dsr_
...polls our pop3 server once per minute... ...attachments are saved to Amazon
S3... ...details are stored in Postgres...

This is the hacker way? Was it designed? It is not elegant. I think it was
grown. It uses technologies which are cool and hip but also dependent on third
party services.

POP3 is not a reliable delivery protocol for unique messages.

I'll leave it at that.

~~~
tomblomfield
The "hacker way" was a small nod to the recent Facebook filing - I don't it
necessary implies any kind of elegant design.

Zuckerberg explains "hacking just means building something quickly... Hackers
believe that something can always be better, and that nothing is ever
complete. They just have to go fix it — often in the face of people who say
it’s impossible or are content with the status quo...Hackers try to build the
best services over the long term by quickly releasing and learning from
smaller iterations rather than trying to get everything right all at once."

------
dpritchett
The informality of the solution seems at odds with the BigCo expense reporting
story told in the problem description: " _Presumably, hundreds of millions of
people have to do this every month, costing millions hours of lost
productivity. And in most companies, it’s because a well-meaning HR or Finance
person has said that It Must Be Done._ "

This email-based expense system hinged on specific keywords would be great for
a small business, but if you're a multinational with lots of legal
oversight...

~~~
cbs
>This email-based expense system hinged on specific keywords would be great
for a small business, but if you're a multinational with lots of legal
oversight

Its still not necessarily bad, such a system could be easily expanded to be an
quick input method to a system that makes the beancounters and legal happy
too. Say I snap a picture of a till receipt with my cellphone and send it off
to the expense account. The backend could also hook into our timesheet webapp.
I already have to choose the account I'm billing my hours to and some accounts
require I put in a note. My expense could show up right in the list of hours
I've clocked, then when I enter my hours at the end of the day I double check
all the expenses are there, are for the right amount, choose the account to
charge and pop in a description.

------
Tim-Boss
Does anyone know if (in the UK at least) HMRC require companies to submit the
actual receipts (physical bits of paper) or do scanned images suffice?

~~~
pbhjpbhj
<http://blog.receipt-bank.com/?p=428> quote HMRC as saying

"Records may be preserved on optical imaging systems, and the originals
discarded, provided that what is retained in digital form represents a
complete and unaltered image of the underlying paper document."

giving the reference as
[http://webarchive.nationalarchives.gov.uk/20110720224136/htt...](http://webarchive.nationalarchives.gov.uk/20110720224136/http://www.hmrc.gov.uk/bulletins/tb37.pdf)

------
josephcooney
To my mind the meta-hack here would be to do away with the need for expense
reporting entirely by giving everyone a corporate credit card like 37signals
did. <http://37signals.com/svn/posts/893-workplace-experiments>

Your start-up might not be in a position to do that, but removing the task
entirely always seems preferable to me.

------
jakeonthemove
I'm not a programmer, but I do like to automate and my favorite way at the
moment is using macros, bash scripts and WinAutomation - I can't believe how
much can be automated this way. Email replies, online searches, software
updates, website testing, testing content for uniqueness, proof reading and
renaming files, taking screenshots, adding tasks to the calendar - everything
can be automated using existing software and something to connect it all
together (in my case, it's WinAutomation).

Obviously, it's not scalable, but anything that saves you an hour of sorting
500 files by category is a win in my book...

~~~
nathan_f77
Sir, you are definitely a programmer.

------
bsoist
I write shell scripts to do just about everything. Good point about teaching
these skills to non-programmers.

------
kunashe
I think a balance between automation & manual gives the best efficiences.
Particularly in the rapid building pre production stage of a solution.

My context is data processing matlab (automation) & excel (manual).

------
polynomial
Interestingly, the Tibetan word for meditation (Gompa) means, literally, a
repetitive mental task.

Thus far, we have machines that think for us in countless ways, none yet that
meditate for us.

------
katsuyan
Tom, you might be interested in <http://wappwolf.com/> :)

------
seigel
The code would be fun to see if you have a moment.

