
Ask HN: Is freelance statistics/machine-learning consulting viable? - HN-regular
I'd love to hear from anyone who's making a living as a freelance statistics/machine-learning consultant, as I'm thinking of going this way myself and wondering whether it makes sense.  I prefer to do Bayesian data analysis, but I'll turn to whatever tool seems to fit the job at hand.  I have a lot of experience in scientific programming, not so much with industrial-scale software development (though I've read <i>Code Complete</i> and try to follow its main principles to the extent that they make sense for my relatively simple programs.)  I have a comfortable job in academia (not tenured) but for a variety of reasons I see hard times just over the horizon for the field of research I specialize in, and I'm looking around for what's going to come next.  I'm finding it surprisingly easy to interview for statistics/machine-learning type jobs at technology startups and I was thinking, these guys really want my skills, so why hitch my wagon to just one company when I could sell my services directly?  One thing I love about my current job is the autonomy and independence, and it would be great to preserve that.  And I don't really want to start a startup at this stage, because I don't want to have to do all the other work associated with that.  A consultancy seems like a nice balance, at least for now.  I know it will involve marketing, networking and collections, but that seems manageable.<p>So I am looking for evidence that this is a feasible approach.  Are companies interested in bringing in outside consultants to help solve statistics or machine-learning problems?  It's a complete guess, but the main problem I forsee is that ideal solutions usually come from a fairly close examination of the patterns in the target data, and if that data is sensitive there might be barriers to sharing it with outsiders.  I'm also curious about terms of the consulting gigs: how do you decide on the price and the criteria by which you'll be judged to have successfully completed the project?  Also, it would be interesting to know the extent to which customers accept remote collaboration, as I don't live in a major metropolitan area, and kind of like it that way, though I would be happy to travel in order to set up the deals.
======
patio11
If you have a track record of making competitors or peer organizations
staggering amounts of money, what you actually _do_ is almost irrelevant. So
yes, I vote "viable.". Heck, for certain (very simple) definitions of
statistics, I already do it.

Pricing: charge more. Too much work? Raise rates. Repeat until satisfied.

Criteria for success: you'll likely have more understanding than customers of
the likely outcomes of engagements, so communicate as best as possible, but
ultimately the engagement is a success if they are happy with the outcome and
is a failure if they are not. This counsels listening _very carefully_ when
they say what worries them, and taking an active role in picking any success
metrics.

------
_dps
I've been doing this successfully for a few years now and come from a very
similar background (scientific computing, had one foot in academia, initially
had mixed feelings about a startup). It's not that hard to find the work if
you know where to look (I work primarily with seed stage or series A stage
startups, but have worked with a few fortune 500s). It's a high-variance sort
of situation, but I do go through phases where I have to turn down work (a
good sign to raise your price on subsequent engagements!). I'd be happy to
chat about my experiences in detail (email in my profile).

For the sake of the HN discussion, I'll say that it's a lot easier to use
ML/stats consulting as a "sales lubricant" on top of custom software
development. The kinds of details you can tease out from most business data
aren't usually actionable without some kind of software that "closes the loop"
(e.g. a behavioral targeting system for customizing a web experience).

------
gexla
"And I don't really want to start a startup at this stage, because I don't
want to have to do all the other work associated with that."

Contracting yourself to others is a startup. It's a business and you do have
to do all the other work associated with that. The first step in being
successful as a contractor is taking it seriously and recognizing it as a real
business.

As for finding the work, I don't know how to help you there. You could take a
look through a site like Elance to see how much work there is available there.

------
mechanical_fish
I've worked in academic laboratories, and I've been a software consultant.
Nonetheless, the following is all speculation...

And my guess is that you'll find this to be a hard sell. I certainly wouldn't
try it unless you can find other folks in this line of business who can tell
you the ropes, or at least provide the existence proof that it works. If you
can't find anyone else, that is a very bad sign.

The first problem is the packaging. It's easier to sell a PHP consultant than
a generic "I'm a smart person who can learn your language of choice in a
weekend and then hack for you" consultant. It's easier to sell a Drupal
consultant than a generic PHP consultant. And it's easier to sell a Drupal
consultant who is the world expert on, say, Drupal ecommerce or migrating data
into Drupal than it is to sell a generic Drupal consultant.

So starting a consultancy is (surprise, surprise) a lot like starting any
business: You need to find product-market fit. The good news, though, is that
unlike the startup world you're not _racing_ to find product-market fit: Your
product by definition will not scale (if the customer could buy the solution
in a $89 box from Google, they obviously wouldn't hire consultants) so you can
just find other consultants and clone their product. If the market is at all
healthy, those consultants will probably _help_ you clone their product, and
thank you for it: A bigger ocean floats all boats, and the others need a
steady supply of new blood to help them pay the hotel bills for their industry
conferences.

So if there's a market for, e.g., SAS consultants, learn SAS and call yourself
a SAS consultant. Is there a well-known standard tool, or up-and-coming-
standard tool, in your industry that everyone wants to use but nobody knows
how to set up? That's a good candidate for a marketing hook.

I think you've identified your other, larger problems: A) Customers want in-
house employees because learning the ins and outs of the customers' data and
techniques is a long-term process, and B) everyone is super-secretive, because
data is incredibly expensive stuff to acquire and leaking the data at the
wrong time is like handing your competitors bundles of thousand-dollar bills.
To that I would add problem C): Your target market is either (i) academics,
who have relatively little money, relatively long timelines ("I'll request
some grant money to pay your consulting fee; I'll let you know in a year if we
get accepted"), and access to grad-student labor at below-market rates; or
(ii) big companies, which tend to be able to afford in-house staffs, and will
do so, because their secrecy concerns are even larger than usual.

There might be a category (iii): small-to-mid-size companies that can't afford
an in-house statistician. Maybe you can find and target that market. But it
might not be fun: By definition your market doesn't have much money, so
they're going to try to hire you at below-market rates; they're going to try
to get you to accept grand promises of future wealth in lieu of cash; they're
going to nickel-and-dime you every step of the way, risking the quality of the
work in the process; and of course there's the exciting possibility that they
won't pay you at all. Make friends among the client's clerical staff, and have
them keep an ear open for signs of potential bankruptcy. ;)

My suggestion is that you figure out what service the in-house statisticians
inside well-funded companies would pay for, and then offer that. Don't try to
_be_ the rent-to-own in-house statistician for a company that has no
statistician; Instead, offer a service that an in-house statistician would
love to have but can't find the time or skill to do on her own. Of course, how
does one best learn what in-house statisticians really want? By being one for
a while. Take one of those jobs that are throwing themselves at your feet,
hold it for a couple of years if you can, and tell yourself that it's market
research for your future consulting firm.

~~~
HN-regular
Thanks, Fish. I appreciate the advice.

~~~
mechanical_fish
Happy to get the party started. Just keep in mind that I am totally talking
out of... well, no actual experience in your field. ;)

------
earl
One guy who (apparently) successfully does it is Joseph Turian [1]. He is on
HN [2] and quora.

The reason I think you may have a hard time finding work is that, in my
experience, and I believe this to be a widely help opinion, you get far more
out of careful feature selection, cleaning, and filtering; and careful hand
tooling of algorithms to your domain than out of your raw ML tech. This has
been my experience at two companies (no names since there are tools on HN who
like to make posts out to be representative of the company, but if you want
I'm happy to discuss over email.) The problem then, for you, is that this
augurs poorly for hiring external consultants because the hard part is domain
specific knowledge and you don't want that to walk out the door. To the extent
that people want help setting up common toolkits like R, lucene, elastic
search, weka, mahout, vowpal rabbit, etc, there could be lucrative work.

I'm actually pretty curious about this myself. In fact, if you want to discuss
offline, drop me an email.

Good luck.

[1] <http://metaoptimize.com/blog/>

[2] <http://news.ycombinator.com/user?id=bravura>

