
Four pitfalls of hill climbing - charlieegan3
http://chris-said.io/2016/02/28/four-pitfalls-of-hill-climbing/
======
ThrustVectoring
There's another one I've seen: climbing the wrong hill. Consider a news
website doing A/B testing on headlines, and optimizing for getting more clicks
on an individual headline compared to other candidates in the slot. The
problem is conflating "people want to click on a headline after visiting our
site" with "people want to visit our site", and optimizing for the former at
the expense of the latter.

The end result is clickbait headlines that alienate visitors while they fight
to grab attention.

~~~
abraae
A "wrong hill" example in our industry (recruitment software) is a focus on
getting as many as possible candidates to find and apply for jobs.

If I sell widgets, then yes, I want as many as possible people to come and buy
them, and I really only care about the colour of their money.

However when as a widget company I am advertising jobs, I really want only
_qualified candidates_ to apply. Otherwise I am drowning in people I just need
to sort through, and reject - and when I reject them, they may stop buying my
widgets.

In this situation, bounce rates (e.g. people leaving the site after viewing a
job) and a lot of other common metrics are meaningless without far more
context.

~~~
danieltillett
The equivalent in widget selling is time-wasting leads. When you are
advertising a job you are looking for leads and unqualified people are just
bad leads.

Having said this there is little you can do about this problem since any well
advertised position will attract the unqualified since the cost is so low to
just mass-blast your cv.

I have always wanted to have a fee (say $500) to apply for a position and
refunded $1000 to the candidates that actually have the skills they claim and
which match the job requirements. Real candidates would be rewarded for
applying (easy $500) paid for by all the time wasters and dunning-kruger
types. While this appeals to me I doubt this idea would be viable in the real
world :)

~~~
ThrustVectoring
I think what you're looking for is something like a preliminary interview
round that is cheap for you to administer, combined with a financial reward
for those you select to go to the next level of the process.

For incentive reasons, the budget for this _must_ come out of the hiring
company's recruiting budget. Otherwise, it's a great way for unethical folks
to extract money out of the applicant pool.

An alternative way is to set up some sort of competition with prizes, and then
recruit from those who do well at it.

~~~
danieltillett
Yes I agree. It might be possible to get around the unethical company problem
by outsourcing the cash distribution to an outside organisation.

------
cialowicz
This is a great little read, and really resonated with me. Like the author
mentioned, there is some value in A/B testing and an incremental approach to
product development. But too often organizations can become obsessed with this
path forward, and continuing with the same approach becomes like trying to
squeeze blood from a stone -- there are just no more significant gains to be
had.

Unfortunately the company I'm at right now feels like this. The product team
and CEO are obsessed with A/B testing, and don't have a more holistic approach
to product development. This means that our competitors are out-innovating us
in certain areas, and it's disappointing to watch.

~~~
avn2109
>> "...don't have a more holistic approach to product development..."

Interestingly, the article's author works at Twitter.

------
astazangasta
Haha, if we are going to use terrible reward function analogies to measure
success, why stop at A/B testing? Introduce your features as mutations to your
product! Start them off with one or two customers. Those features that make
your users quit in a huff will be eliminated. Those that make them recommend
your app to a friend will propagate. Survival of the fittest feature!

~~~
natosaichek
I love this terrible idea. I can totally imagine some terrified product
manager turning up the mutation rate to try to find some good set of features
while people leave because they don't want a mutating product.

~~~
techdragon
We sometimes call this "pivoting your business" ;-)

------
claar
I really liked the animations in this article. I found them very useful in
visualizing the authors points about the potential pitfalls of various product
development strategies, even if they exaggerate the effects a bit. Also neat
that they're SVGs instead of the usual GIFs.

------
llogiq
Note that this has little to do with hill-climbing algorithms and much with
A/B testing. Though there is an overlap in the problems both face.

I wonder if algorithmic solutions like randome restart or simulated annealing
could improve the outcome of A/B testing, too.

------
namenotrequired
> Bold changes require lots of resources. Maybe it’s mostly the success-bound
> companies who have enough resources to afford the bold changes.

And even when they do get the causality right, maybe big changes in production
are likely to cause big changes in outcome - providing both greater potential
reward and greater risk of losing everything.

Any sufficiently lucky contrarian is indistinguishable from genius.

Edit for clarity

------
cshimmin
Those javascript animations had my CPU running at 100% while I was reading the
article. I'm not sure I like this new web... can't we just go back to GIFs for
simple stuff like that?

~~~
restalis
The only reason GIFs are a better solution for you (i.e. not taking 100% of
the CPU resource) is because it's about a quality GIF render. What you really
need is quality of the software that powers the alternative solutions.

------
sopooneo
I often wonder if an agile development approach that only plans in the range
of very short iterations could run into the same problem.

------
xlayn
TL;DR: Incremental approach could prevent disruption benefits.

~~~
btowngar
There's three to four minutes of reading in this article, possibly less if
you're familiar with the concepts.

It would be great if users were required to click on the actual article and
wait a certain length of time before commenting.

~~~
xlayn
I did read it. Probably wording didn't help. I see it as the "Newton's method"
for finding roots that is know to have the problem of local maximum; you can
see different initial values for finding root as R&D.

