Ask HN: How did you choose your specialization? - yamrzou
======
mlthoughts2018
This is my attempt to avoid far-mode abstract thinking when talking about
this, and reflect on the mistakes inherent into how I chose my specialization.

I work in machine learning, especially for “creative” leaning technology
companies. I have significant experience in quant finance and education
technology as well.

I chose my specialization because mathematics and statistics were my strongest
skill areas in my undergrad education. I won some math competitions and had
fun in team oriented math modeling competitions.

I debated going to grad school or working right out of college, ultimately
choosing to work at a defense research lab. I didn’t like it very much so I
went for my PhD a little leas than 2 years later.

I bounced around several jobs in the ubiquitous 2 year stints for a while,
finding startups to be deplorable with low pay, bad equity deals and poor
work/life benefits. I enjoyed quant finance a little but ultimately it was too
boring.

I finally started working in ML in ecommerce companies, and have about 12
years experience in 2 large ones, moving from engineer to architect to
manager.

I really like product development in this area. There are so many ways machine
learning adds tangible value for customers especially when the underlying
products involve creative aspects, like merchants selling and marketing
crafts, or people designing copy text or imagery. Meanwhile there are also
lots of internal use cases like demand forecasting, improvements in help
center search experiences, detecting abusive content or fraud, etc.

Nonetheless, it’s a real challenge to work on ML professionally. It receives
very undue skepticism and fights a huge uphill battle on every project because
other stakeholders don’t invest a lot of time to understand it. Somehow a
string of objective successes never sticks in their mind and every time a new
company leader is brought in, it’s a whole new game of avoiding their biases
and chipping away at unfounded skepticism.

On the engineering side, most of the work is unglamorous data cleaning &
pipeline maintenance, constant turf wars with operations and infra teams that
don’t want the responsibility of dealing with deployment constraints of ML
systems that break cookie cutter patterns for other classes of systems.

It’s also exceedingly political. I chose statistics and ML in part because at
some level it should supersede subjective opinions about any possible domain,
bordering on being a root subject of all philosophy and epistemology, and so
surely it supersedes politics when real money is on the line in businesses,
right?

Alas, not true. Machine learning engineers who display a naive sense of
intellectually honest curiosity applied to business decisions will get a rude
wake up slap by all sorts of executives and managers who don’t give a shit and
have ulterior motives.

See [0] for a good take on it that I really wish I understood before choosing
ML as my specialty.

I’ve hired and lost so much great ML engineering talent over the years, and it
takes a toll seeing these brilliant people not have their specialty or their
comparative advantage respected, to the point that someone who is a world
expert on computer vision is off trying to debug some inane kubernetes error,
and management can only see “fairness” arguments in favor of this instead of
realizing they’re flushing this person’s salary down the toilet by asking them
to do something many other people can do at a direct opportunity cost of them
using their specialization for things nobody else can do.

My advice for choosing a specialty is to interview lots and lots of people who
have jobs all across the career ladder for that specialty - interns, new
grads, mid-level employees, senior employees, managers and directors. Try to
get a holistic sense of what they like and dislike and what negative realities
you’re going to need to live with if you choose it.

[0]: [https://www.cato-unbound.org/2011/07/13/robin-hanson/who-
car...](https://www.cato-unbound.org/2011/07/13/robin-hanson/who-cares-about-
forecast-accuracy)

