Ask HN: How does your company account for things that are hard to measure?

[hn_throwaway_99]

There were a bunch of good comments in the recent article about Amazon stuffing search results with ads around how ad revenue is easy to measure, while long term potentially negative impact to your brand is hard to measure.

I'm curious if other people have experiences with how to account for some of these hard to measure, but important factors for business success, especially in a world where metric analysis and constant A/B testing is the norm. Do you actually try to measure it? Use a fudge factor to account for it?

[starbugs]

If in doubt, consider trusting and using your gut feeling. In teams full of technical people, I have repeatedly made the experience that this skill — using your intuition – has either been unlearned, suppressed, or is actively or passively being discouraged by (technical) leadership.

But trusting your gut has some considerable advantages:

1. In the case of long term negative or positive effects you have something to base your decision on. Do you feel good with that decision? Or are you hesitant that it will contribute to your long term goals? What is your first thought without trying to dissect the matter? I have made the experience that people generally tend to have a really good intuition when it comes to these kinds of decisions.

2. In a rapidly changing environment you can make a decision now — you don't have to wait for a complicated measuring process to be finished. You can ask people how they feel about a certain change as well. Chances are you might get a gut feeling back.

3. You can be truly agile. Test something by putting it out and see what happens. You need an ongoing process for learning from what you experience and applying the necessary changes though.

4. A lot of pseudo-scientific methods for measuring outcomes beforehand are based on the assumption that their result will apply to the real world. This is not always the case. In the worst case you have wasted time by applying a complex technical process just to get an outcome that will waste you even more time, because it doesn't match what you predicted.

Don't overcomplicate stuff. Individuals and interaction over processes and tools.

[jackgolding]

At a local meetup we had a speaker come in and talk about "The Process" (en.wikipedia.org/wiki/Philadelphia_76ers#2013%E2%80%93present:_The_Process_era) which he deduced is design a strategy for long term success and see what happens. My thought was most tech employees and executives have very short tenure (3 years) so making short term decisions is incentivised (there is a lot of writing on this.) You can imagine as a product manager/GM/VP you'd push very hard on something like Amazon showing ads because its "new revenue" and regardless of if its a good idea in the long term for the business, its runs on the board in the short term which will provide yourself as an individual with the best results.

I don't know if people will be shocked with that type of thinking, but I've been thinking about it more and more this year.

[dasmoth → jackgolding]

It’s interesting that they’re calling this “The Process”. In general, I tend to associate that word with approaches that try to make decision-making more organisational (which tends to mean metrics above all). In this case, it does seem more a case of hire someone strong and give them substantial authority.