Ask HN: Is freelance statistics/machine-learning consulting viable?

[HN-regular]

I'd love to hear from anyone who's making a living as a freelance statistics/machine-learning consultant, as I'm thinking of going this way myself and wondering whether it makes sense. I prefer to do Bayesian data analysis, but I'll turn to whatever tool seems to fit the job at hand. I have a lot of experience in scientific programming, not so much with industrial-scale software development (though I've read Code Complete and try to follow its main principles to the extent that they make sense for my relatively simple programs.) I have a comfortable job in academia (not tenured) but for a variety of reasons I see hard times just over the horizon for the field of research I specialize in, and I'm looking around for what's going to come next. I'm finding it surprisingly easy to interview for statistics/machine-learning type jobs at technology startups and I was thinking, these guys really want my skills, so why hitch my wagon to just one company when I could sell my services directly? One thing I love about my current job is the autonomy and independence, and it would be great to preserve that. And I don't really want to start a startup at this stage, because I don't want to have to do all the other work associated with that. A consultancy seems like a nice balance, at least for now. I know it will involve marketing, networking and collections, but that seems manageable.

So I am looking for evidence that this is a feasible approach. Are companies interested in bringing in outside consultants to help solve statistics or machine-learning problems? It's a complete guess, but the main problem I forsee is that ideal solutions usually come from a fairly close examination of the patterns in the target data, and if that data is sensitive there might be barriers to sharing it with outsiders. I'm also curious about terms of the consulting gigs: how do you decide on the price and the criteria by which you'll be judged to have successfully completed the project? Also, it would be interesting to know the extent to which customers accept remote collaboration, as I don't live in a major metropolitan area, and kind of like it that way, though I would be happy to travel in order to set up the deals.

[patio11]

If you have a track record of making competitors or peer organizations staggering amounts of money, what you actually do is almost irrelevant. So yes, I vote "viable.". Heck, for certain (very simple) definitions of statistics, I already do it.

Pricing: charge more. Too much work? Raise rates. Repeat until satisfied.

Criteria for success: you'll likely have more understanding than customers of the likely outcomes of engagements, so communicate as best as possible, but ultimately the engagement is a success if they are happy with the outcome and is a failure if they are not. This counsels listening very carefully when they say what worries them, and taking an active role in picking any success metrics.

[_dps]

I've been doing this successfully for a few years now and come from a very similar background (scientific computing, had one foot in academia, initially had mixed feelings about a startup). It's not that hard to find the work if you know where to look (I work primarily with seed stage or series A stage startups, but have worked with a few fortune 500s). It's a high-variance sort of situation, but I do go through phases where I have to turn down work (a good sign to raise your price on subsequent engagements!). I'd be happy to chat about my experiences in detail (email in my profile).

For the sake of the HN discussion, I'll say that it's a lot easier to use ML/stats consulting as a "sales lubricant" on top of custom software development. The kinds of details you can tease out from most business data aren't usually actionable without some kind of software that "closes the loop" (e.g. a behavioral targeting system for customizing a web experience).

[gexla]

"And I don't really want to start a startup at this stage, because I don't want to have to do all the other work associated with that."

Contracting yourself to others is a startup. It's a business and you do have to do all the other work associated with that. The first step in being successful as a contractor is taking it seriously and recognizing it as a real business.

As for finding the work, I don't know how to help you there. You could take a look through a site like Elance to see how much work there is available there.

[mechanical_fish]

I've worked in academic laboratories, and I've been a software consultant. Nonetheless, the following is all speculation...

And my guess is that you'll find this to be a hard sell. I certainly wouldn't try it unless you can find other folks in this line of business who can tell you the ropes, or at least provide the existence proof that it works. If you can't find anyone else, that is a very bad sign.

The first problem is the packaging. It's easier to sell a PHP consultant than a generic "I'm a smart person who can learn your language of choice in a weekend and then hack for you" consultant. It's easier to sell a Drupal consultant than a generic PHP consultant. And it's easier to sell a Drupal consultant who is the world expert on, say, Drupal ecommerce or migrating data into Drupal than it is to sell a generic Drupal consultant.

So starting a consultancy is (surprise, surprise) a lot like starting any business: You need to find product-market fit. The good news, though, is that unlike the startup world you're not racing to find product-market fit: Your product by definition will not scale (if the customer could buy the solution in a $89 box from Google, they obviously wouldn't hire consultants) so you can just find other consultants and clone their product. If the market is at all healthy, those consultants will probably help you clone their product, and thank you for it: A bigger ocean floats all boats, and the others need a steady supply of new blood to help them pay the hotel bills for their industry conferences.

So if there's a market for, e.g., SAS consultants, learn SAS and call yourself a SAS consultant. Is there a well-known standard tool, or up-and-coming-standard tool, in your industry that everyone wants to use but nobody knows how to set up? That's a good candidate for a marketing hook.

I think you've identified your other, larger problems: A) Customers want in-house employees because learning the ins and outs of the customers' data and techniques is a long-term process, and B) everyone is super-secretive, because data is incredibly expensive stuff to acquire and leaking the data at the wrong time is like handing your competitors bundles of thousand-dollar bills. To that I would add problem C): Your target market is either (i) academics, who have relatively little money, relatively long timelines ("I'll request some grant money to pay your consulting fee; I'll let you know in a year if we get accepted"), and access to grad-student labor at below-market rates; or (ii) big companies, which tend to be able to afford in-house staffs, and will do so, because their secrecy concerns are even larger than usual.

There might be a category (iii): small-to-mid-size companies that can't afford an in-house statistician. Maybe you can find and target that market. But it might not be fun: By definition your market doesn't have much money, so they're going to try to hire you at below-market rates; they're going to try to get you to accept grand promises of future wealth in lieu of cash; they're going to nickel-and-dime you every step of the way, risking the quality of the work in the process; and of course there's the exciting possibility that they won't pay you at all. Make friends among the client's clerical staff, and have them keep an ear open for signs of potential bankruptcy. ;)

My suggestion is that you figure out what service the in-house statisticians inside well-funded companies would pay for, and then offer that. Don't try to be the rent-to-own in-house statistician for a company that has no statistician; Instead, offer a service that an in-house statistician would love to have but can't find the time or skill to do on her own. Of course, how does one best learn what in-house statisticians really want? By being one for a while. Take one of those jobs that are throwing themselves at your feet, hold it for a couple of years if you can, and tell yourself that it's market research for your future consulting firm.

[patio11 → mechanical_fish]

I have a different opinion with regards to midsized companies: if you have 20+ employees, you can afford to drop $X0k on a consulting engagement without it being more than a percent of payroll. Often, there are two or three people in the company who could a) individually decide to pull that trigger and b) reasonably speak about the matter to an outside consultant.

YMMV, but midsized companies have been my sweet spot for consulting engagements. They've got the agility of one-man shops with the attitude towards spending money of businesses as opposed to people.

[_dps → mechanical_fish]

I have a slightly different view on small/mid-size companies: it's not exactly that they can't afford an in-house math/stats person, but rather that there isn't enough work to occupy one full time (i.e. it's not a budget problem but an ROI problem). So if they confuse themselves into thinking they should hire one full time they necessarily need to offer below-market rates to get positive ROI.

It's much more common for small to mid-size companies to have 1-2 projects per year that could benefit from some mathematical analysis, and the ones I've worked with will happily pay for project work where the ROI is clear. And in those cases I think you can fairly negotiate an attractive rate.

[Yesh → mechanical_fish]

You maybe right about category (iii). I have need for stats/ML person to solve problems for features that can truly act as a competitive advantage. But haven't found anyone in my peer circles for various reasons (primarily lack of vocabulary (on my part) when talking to stats people.)

Maybe item(iii) is approachable by using some of the principles used by the SEO or web design companies in approaching clients.

[HN-regular → mechanical_fish]

Thanks, Fish. I appreciate the advice.

[mechanical_fish → HN-regular]

Happy to get the party started. Just keep in mind that I am totally talking out of... well, no actual experience in your field. ;)

[earl]

One guy who (apparently) successfully does it is Joseph Turian [1]. He is on HN [2] and quora.

The reason I think you may have a hard time finding work is that, in my experience, and I believe this to be a widely help opinion, you get far more out of careful feature selection, cleaning, and filtering; and careful hand tooling of algorithms to your domain than out of your raw ML tech. This has been my experience at two companies (no names since there are tools on HN who like to make posts out to be representative of the company, but if you want I'm happy to discuss over email.) The problem then, for you, is that this augurs poorly for hiring external consultants because the hard part is domain specific knowledge and you don't want that to walk out the door. To the extent that people want help setting up common toolkits like R, lucene, elastic search, weka, mahout, vowpal rabbit, etc, there could be lucrative work.

I'm actually pretty curious about this myself. In fact, if you want to discuss offline, drop me an email.

Good luck.

[1] http://metaoptimize.com/blog/

[2] http://news.ycombinator.com/user?id=bravura