
If you want developers to adopt your software, make it less risky (2017) - mooreds
https://www.reifyworks.com/writing/2017-02-21-if-you-want-developers-to-adopt-your-software-make-it-less-risky
======
pjc50
Underlying paper: [http://se-naist.jp/pman3/pman3.cgi?DOWNLOAD=579](http://se-
naist.jp/pman3/pman3.cgi?DOWNLOAD=579)

Survey N = 20, selected by cold-emailing open source developers.

~~~
mooreds
Wow, that's not very many.

~~~
SamuelAdams
Not at all. To be statistically significant, you typically need 1000 or more
test cases / individuals.

~~~
jefftk
It depends on what you measured. For example, if you ask 20 people a yes/no
question and they all say yes, the chance that they would all agree by chance
is 1 in 2^19.

~~~
neogodless
_If they are selected randomly_

(On the other hand, if you go to a board game convention and ask 20 people if
they like board games...)

~~~
UncleMeat
Sure, but that has little to do with sample size.

------
perspective1
Doesn't pass the sniff test. By developer use, npm, for all of its security
issues and orphan packages is the most successful package ecosystem of all
time.

~~~
pjc50
The _time_ risk of NPM (which is the factor considered by this paper) is small
though. Security risk is way down most people's considerations.

~~~
perspective1
The paper extrapolates from its survey questions about _developer time_ to
_loss profitability_ so I think it's fair to discuss risk as more than just
_developer time._ Even so, the _developer time_ risk of poorly maintained
packages is enormous (and not just security-related). Consider the _time_ loss
from a security issue in a package you have to replace, fix yourself or wait
for someone else to fix. Not to mention the non-time losses. The npm ecosystem
has thrived despite this and other risks-- if you want developers to adopt
your software, make it interesting, easy to talk-about, frictionless to get
started with and shiny. _Time_ is a logical consideration that takes a
backseat.

------
rocqua
I think the answers of these questions should probably be read through 'glossy
folder skepticism'. If someone asks me about

> A tool that always reduces working time by 2 hours

VS

> A tool which saves 6 hours with 90% chance and costs 4 hours with 10% chance

I pick the first because there is a lot less space to fudge the first claim.
Certainly, when such probabilities are quoted I am careful not to assume those
probabilities will be accurate in my usage of the product.

------
ken
Why would I want developers to adopt my software? Like "Doctors make the worst
patients", my experience is that developers are the worst users. Doubly so, if
you're looking for them to ever pay you.

~~~
holtalanm
I've had the opposite experience.

Usually, developers have a greater understanding of the effort that goes into
making changes to software, and so are usually more understanding if a bug
isn't fixed right away.

~~~
hermitdev
As a developer, when I hit a potential bug in an OSS lib I'm using, first
thing I do is assume I'm doing something wrong (mileage varies by quality of
the lib, of course). Then, back into a minimal reproducible example, if at all
possible. By the time I actually submit a bug, I've already debugged for hours
or days, sometimes even over weeks (once took me about 6 weeks of off again/on
again debugging to find a bug in a Python DB lib I was using; the bug was in
the C lib and in the certain circumstance I was hitting, uninitialized memory
was being read).

Another case, I had a user raise an issue with mangling of datetimes in a C++
lib. Short of it is we were hitting an integer overflow issue. Under the hood,
our datetime lib was using Boost to do the heavy lifting; our lib was mostly a
facade over Boost. Boost being what it is, there was massive template
instantiations. Templates all the way down... Anyways, somewhere along the
instantiation chain, a 64-bit int got truncated to a 32-bit int and then
promoted back to a 64-bit int. I think the bug was reported, and I don't
there's a resolution some 10 years later (I haven't been doing C++ for a
while, so haven't kept up). Definitely an issue trying to compute schedules
for some 30 years in the future (Unix 32-bit Y2K type issue is in 2037 when
32-bit time_t overflows).

Also, I assume failure. I mean, software is written by fallible human beings.
There will be bugs, crashes, etc. Sometimes the bug is in the implementation,
sometimes it can even be a bug in the requirements. Users asked for X, but
they really needed Y. If you want a favorable resolution to an issue, you need
to be ready to work with the developer, not against them. This may be
providing logs, inputs, steps to reproduce, etc.

I've had times where, as a developer, I've had users provide me fairly
detailed steps to reproduce, but, well, I couldn't reproduce it. In those
cases, ended up sending something along the lines of "let's cut the email;
I'll come to your desk and watch exactly what you do." In those cases, they
usually omitted a step they thought was inconsequential, but really wasn't
(not that they'd know that).

I had one very expensive (for the client) incident at a former employer where
we eventually sold the source code to the client to divest ourselves of a
joint venture. The client then turned around and hired a 3rd party developer
to continue development. As part of the hand off, we documented in pain
staking detail the environment, versions of dependencies used, etc. It was a
Python app and only have about 2 non-stdlib dependencies. However, those 2
dependencies each had bugs that had not been fixed upstream (patches had been
submitted). We left very explicit instructions about download this specific
version, then apply supplied patch, then install. We started getting emails
along the lines of "Nothing works!!", and of course, we were like: "Did you
follow the instructions?" and the response was: "Yes, of course we did.". We
were unable to reproduce on our side, as we actually followed the instructions
to the letter. After several weeks of back and forth and not being able to
reproduce, I offered to visit the 3rd party developer to sit with them while
they attempted to setup their environment. We started from scratch, and
immediately it was apparent that they completely disregarded our instructions
and downloaded the latest version of everything and didn't apply the patches
(which wouldn't have worked because of the differences between the versions).
3 weeks of my time, billed at $500/hr 10 years ago. Yeah, I billed $60,000
over 3 weeks because a 3rd party didn't follow very explicit and detailed
instructions. Not that I saw a dime of that...

------
afarrell
I find it strange that searching in the text of the paper and the commentary
on it for "documentation", "tutorial", "errors", "help", "faq", and "doc"
turns up nothing. Anecdotally, salespeople who try to persuade companies to
adopt a payments API have found that it is very useful to have both tutorial
and reference documentation. I think if you're asking questions about software
adoption, you'd want to study the impact of documentation availability and
clarity.

~~~
chessturk
You are 100% right. I made a rubric for my company's sales department to grade
third party APIs against.

Documentation, tutorial, working example on github, and sdk in
<teams_favorite_lang> were all worth points.

------
neilv
One question is what motivations the researchers realize.

In qualitative interviews, perhaps some subjects will say they frequently
choose new and less-proven or less-personally-known frameworks because they're
looking for greater advantage (which I think is often true).

But how many subjects are motivated to add frameworks simply because they are
(or might) become good things to have on their resume, regardless of risk to
their current employer (also often true, at least in the US, I think)?

Would researchers' focus on perception of risk (to the project?) miss this
motivation, and be trying to fit the wrong psych quiz questions to the wrong
models?

(Alternatively, the researchers appear to all be based in Japan, which might
not have US dotcom prevalence of short-term job-hopping, so perhaps their
results are more applicable there than in the US?)

------
rhacker
I don't know if software developers are all that risk adverse. I mean we're
typically changing development environments and tools a lot, usually at a very
high expense to change, with little evidence of time savings. Sometimes
however those changes may feel like time savings or is just a cooler way to
code. For example consider someone coding standard REST services vs adopting
GraphQL. The adoption itself is a huge risk, probably exponential cost of
introduction, probably a little more convenient code once that initial risk is
taken on. Yet companies are adopting it like gangbusters. Similarly with React
and a lot of other (Facebook) tech.

Ton's of risk, arguably lots of reward, but doesn't seem to be perfectly
correlated with risk alone.

~~~
maxerickson
You'll frequently see people that make $1,000 a day complaining that software
that saves them lots of time costs $100.

~~~
ska
I don't see that much.

But I do see people making $1,000 a day complaining that everyone wants to
upsell you into $20/mo SAAS offering with a bunch of crap you don't need,
rather than sell you the thing you need.

For something you rely on, easy decision. For occasional use though? Often not
worth it.

------
fnord77
I don't understand what they're getting at.

working out the premise, assuming these probabilities are uniform over 10 uses
of the tool:

1 a. reduces time by 2 * 10 = 20 hours

1 b. reduces time by 5 * 5 + 5 * 0 = 25 hours

2 b. reduces time by 4 * 7 + (-1 * 3) = 25 hours

3 b. reduces time by 6 * 9 + (-4 * 1) = 50 hours

------
ZainRiz
Related note:

This is a variation of a winning sales technique taught by Jay Abraham [1]:
Reverse the risk to get a competitive edge

People are unwilling to take on risk. Do what you can to reduce that risk
(even if it means taking more of the risk upon yourself, e.g. offering free
returns) and those potential customers will be more willing to buy what you're
selling

[1] [https://booksummaryclub.com/getting-everything-you-can-
out-o...](https://booksummaryclub.com/getting-everything-you-can-out-of-all-
youve-got-summary/)

------
roland35
As others have pointed out the study itself is too small to draw significant
conclusions, but I agree with the hypothesis at least that developers are
risk-adverse with their time and data.

I think developers especially are worried about getting locked into an
ecosystem, so as a new product it may be useful to make it easy and obvious in
how to both IMPORT and EXPORT data from competitors.

~~~
mooreds
They (the OPs) have an article for that as well:

[https://www.reifyworks.com/writing/2019-02-25-the-first-
ques...](https://www.reifyworks.com/writing/2019-02-25-the-first-question-to-
ask-when-evaluating-a-platform)

------
0wis
If we assume this is right, it could be interesting to search why.

Is it the same for small teams and big businesses ? Are indie developers less
biased towards risk avoidance ?

It feels to me that when you are in a large organisation, you have incentives
to be risk averse, moreover in your expertise domain where you have to justify
every delay, problem, risk...etc

