
How to talk to an open source project as a large scale or interesting user - whocanfly
http://blog.powerdns.com/2014/05/05/how-to-talk-to-an-open-source-software-project-as-a-large-scale-or-otherwise-interesting-user/
======
baudehlo
I have definitely had this issue with Haraka (a highly scalable SMTP server).
On the one hand we have Craigslist as a user, who have been incredibly open
and contributed heavily to the project, and on the other we have had people
post cryptic bug reports that it didn't work for them, but clearly they were
trying to use it at an interesting scale, and we would love to have helped.

We pride ourselves on being one of the friendliest MTA communities out there,
so to find people who give up and quit using it is always sad. Many times
we've contributed custom code to companies for free to help them out, but we
can't do that if you are silent. So speak up, companies, even if it's only in
private messages.

~~~
Mikushi
On paper I agree, from the other side of the fence, here is what the process
looks like:

This looks good for what we want to do, let's test it out. Cryptic error.
Google for 24h for a solution, can't find one. Ask on official mailing list /
forum / bug tracker. Getting ignored as usual, or dismissed rudely. Never use
that product ever again.

I'm a CTO and I do that process more often that I would want to, sometimes I
don't get that far because Install instruction sucks, and usually that's
enough to deter me.

A concrete example: Since The Guardian opened Scribe (their CMS WYSIWYG
editor) I had it in the back of my mind for weeks because it looked really
interesting and close to what I had been looking for. We were doing CMS
upgrades recently, so I wanted to take the occasion to plug in Scribe.

It took only 2h for me to put Scribe on my "never use ever again" list. It all
look pretty neat at glance, quick install, nice plugin system, yadda yadda,...
However, their one line install simply doesn't work, their is no list of
dependencies anywhere, and simply no documentation. Logged an issue notifying
them of the poor quality of their setup instructions. Got a reply 7 days
later, which starts a discussion. But now it's too late, still have personal
interest for Scribe, however my company has already moved on.

I'm all for OpenSource, have some myself (though, more of a "here is what I
did recently" type of guy), however if you want to pretend to be a serious
library/framework/piece of software, that includes decent documentation with
clear install instruction for most common distributions.

Plus, dialogue is good, but most of the time I need it right now, or in a very
short time frame, so more often that not I will simply drop back to another
less interesting piece of library (or nasty one) that simply works, because
that is what matters, it works when I need it. (Again with the Scribe example,
no matter how much I dislike TinyMCE, I ended up using it, because one line of
JavaScript is all it took for it to work).

~~~
baudehlo
I absolutely agree and this is a problem that a lot of open source projects
face (Haraka is not alone in this). In part it comes down to the fact that
most open source isn't worked on full-time, and so documentation often takes a
back seat to getting your feature-du-jour working.

But absolutely we could all use some improvement in these areas.

~~~
Mikushi
Yeah, just to be clear I'm not bitching at maintainers, just giving the
perspective of a potential company wanting to use a library. Though usually I
do try to follow up if I think the software/lib is really nice, but it's often
on a personal level, not for my company.

Documentation is pretty boring to write,and not as fun as feature-du-jour, but
may be there could be some guidelines about what a basic documentation should
have, specially with GitHub it could be straight forward to have a Markdown
template for basic doc (Installation, FAQ, Requirements, Contact points, ...),
I know I use one.

------
roberthahn
What I'm seeing here is a customer support problem. And IMHO, PowerDNS isn't
taking the correct steps to solve it.

It's funny, because the answer is right in the article. If your customer:

* is large and/or interesting * has a legal/security/marketing/ team that's worried about sharing information publicly * has an IT dept that doesn't know the whole company is depending on a single open source product * feels they have to use gmail accounts and made-up names to talk to you

Then maybe a public forum really _isn 't_ the place to expect your customers
to ask for help.

I would suggest creating a partnership program for businesses like Cloudflare
- swap tech support for publicity (which will then help drive adoption)

~~~
sremani
I think they should have a VIP program akin like WordPress. I think every
serious Open Source project should have some kind of separated channel for
Corporate/BigInstalls.

I would not entirely place the blame on PowerDNS here, it also points out that
how our Corporate citizens can do better for the eco-system by being active
partners with the Open Source projects they rely on, instead of hide and snipe
mailing-listers.

~~~
jcurbo
I agree completely. They have this paragraph:

 _Another reason why employees at big corporations are loath to contact open
source projects is that they assume they won’t get help, and need to pay for
support. And, while it is true that someone needs to pay the bills here, we
realise that for many users, a support agreement is just not going to happen.
Corporate IT might not even know the whole company is relying on an open
source project. Questions might be asked. For such organisations, we are fine
with providing free help on the public mailing lists. But to give proper
weight and context to your issue, it helps if we know who you are._

I think corporate owners are in the _best_ position to offer money for
services. They are usually looking specifically for some sort of support
contract they can pay for! (If Corporate IT doesn't know, that company has
bigger problems anyway.) Something like a VIP program would be a great way to
get these types of users in the door. Feed all that money back into your open
source project - if you're big enough you probably have project support costs
anyway (hosting, etc).

I agree with your second point as well, and I think it reinforces my point,
because you can take those internal people that you are dedicating to the task
and make them the active partners to the open source project.

------
jackgavigan
It's interesting how the title "How to talk to an open source project as a
large scale or interesting user" implies that the onus is on the _user_ to
approach the open source project in the correct manner.

It might be more productive to think about "How an open source project should
talk to a large scale or interesting user".

I was deploying open source software in large corporations more than a decade
ago[1] and I formed the impression that open source projects could do a LOT
more to engage with and support their users. It sometimes felt like the
developers just didn't care about users. Documentation was often sorely
lacking[2] and "support" typically came in the form of mailing lists that were
full of users trying to figure out how the software worked, and swapping
advice and tips - often a case of the blind leading the blind.

The end result was that corporate users would sometimes choose commercial
software, despite the cost, because it came with decent support and/or had
functionality that the open source alternative lacked.

These days, I think that a product manager is an essential part of any open
source project team.

1: e.g. [http://mail-archives.apache.org/mod_mbox/tomcat-
users/200009...](http://mail-archives.apache.org/mod_mbox/tomcat-
users/200009.mbox/%3COFF2F061DD.C2B7EB05-ON00256960.004C94C4%40srv.uk.deuba.com%3E)

2: e.g. [https://www.mail-archive.com/linux-
raid@vger.rutgers.edu/msg...](https://www.mail-archive.com/linux-
raid@vger.rutgers.edu/msg00947.html)

~~~
x0x0
your comments are, bluntly, massively entitled. One of the open source
tradeoffs is that they often come with slightly more install/configuration
work; in trade for usage freedom, the ability to modify the source, and of
course, zero required cash outlay. I share ml code on cran because I wrote it
and I'm happy to have other people use it; that doesn't come with a
requirement for me to spend more than zero time helping them do so and when I
do, it's because I happen to feel like it. Telling me you're doing something
interesting (to me!) will often help interest me in helping you. The massive
wave of demanding users that want hand holding typically don't even get a
reply from me.

~~~
Karunamon
>your comments are, bluntly, massively entitled.

Really? I see no entitlement, merely a very common annoyance that FOSS
projects are traditionally very bad at support (and your frankly, shitty,
attitude is part of that, and part of the reason many enterprises consider
community open source projects to be absolute last resort)

If I had a penny for every time I had a question ignored in IRC or mailing
list ghost towns, I'd be running my own Y Combinator by this point. Those that
will actually take the time to help users out, or projects with sufficient and
good enough documentation that the "hand holding" as you put it, isn't
necessary, are like refreshing oases in a vast desert.

~~~
DerpDerpDerp
I mean, would you be okay with support that cost money?

That seems to be the model that Red Hat went with, for instance.

~~~
Karunamon
I would be okay with no support (explicitly mentioned) vs no support where
that's not implied to be the case.

    
    
        <rant>
    

There's nothing that sucks quite so bad as a link on a project page which says
"Go here for help" \- you do so, then find out you are in an IRC channel with
500 people, none of which ever speak, any questions are simply lost in the
ether among the hundreds of join/quit messages per hour.

In other words, don't say you provide support if you don't intend on giving
it. At least a message along the lines of:

    
    
        This is provided as is, where is, if it breaks you get to keep both parts
    

..forewarns me to either not waste my time, or be prepared to waste quite a
bit of time.

As it stands, now I've wasted time, and now I hate your project for being an
unsupported piece of shit, and now I hate the developers for fucking lying
about it.

This is a literal experience I'm fighting with _right now_.

    
    
        </rant>
    

And worse, every time a user encounters this situation, it tars all of FOSS
with the same brush. It's what leads to people joining companies and
remembering their experience and telling people not to use X Y and Z due to
that bad experience.

~~~
lacksconfidence
turn off the joins/parts/quits in your irc client(we use irc at work, and its
the first thing i suggest to new employees)

~~~
Karunamon
It's not that I can't see the message anymore, it's that whoever might be able
to help likely won't be able to see it.

------
sambeau
OFF-TOPIC FOR ENGLISH-AS-SECOND-LANGUAGE:

For HN'ers who are still learning the finer points of English this title shows
one of the many weird inconsistencies of the language.

Normally, when trying to say 'a thing' where the thing starts with a vowel we
change the 'a' into 'an' so 'a axe' becomes 'an axe', 'a umbrella' becomes 'an
umbrella'.

However this is a notable exception: when pronouncing 'user' there is an
unwritten 'y' e.g. it is said 'yoozer', thus we wouldn't normally add another
consonant here so it should be 'a user' not 'an user'. If you say it out loud
you can hear it (and hopefully feel the extra tongue work that, traditionally,
we try to avoid).

Another common exception is 'Hotel'. It takes 'an' rather than the expected
'a' as for some reason we drop the 'H'. For me it rolls better off the tongue
with 'an' but this may purely be due to conditioning.

'Herb' is a difficult one as in the US it's pronounced 'erb' while in the UK
it has a hard 'H'. Thus in the UK we use 'A herb' — I assume the US says 'an
Herb' — correct me if I'm wrong.

It's nonsense like this that makes me pity you brave fellows who attempt to
master our pidgin of a language!

Edit: change 'pigeon' to 'pidgin' :)

~~~
dspillett
I (northern UK based) find "an user" feels wrong as to me the "y" implied by
the way "u" is pronounced is a consonant sound so it should be "a user" \-
though I pronounce the "a" in a softish manner (more towards "ey"), it might
be different if you pronounce "a" harder or pronounce the "u" as more towards
"oo" (reducing the y-like mouth movements).

Regarding herb: in written word it is "a herb" for correctness, though few in
the north pronounce it exactly that way as the "h" is dropped to produce "an
'erb" or sometimes (though this makes you sound uneducated due to certain
stereotypes) more like "a nerb" (this time with the harder "a" as in "an".

    
    
      It's nonsense like this that makes me pity you brave fellows who attempt to master our pidgin of a language!
    

The fairly arbitrary set of incompatible rules and standards that make up the
English language mean it is difficult to learn perfectly (especially as there
are arguments as to that "perfectly" means in this context) but it makes the
language easy to learn to the point where you can survive using it in the
languages native land(s) easier than many others. We are so used to
subconsciously processing all the different accents we have in the relatively
small area that is the UK and the extra nuances that creep in from elsewhere
(poor education, influences from other cultures mainly via TV/film & music,
etc.), that unless we are feeling pedantic we also naturally screen out the
mistakes non-native speakers make.

It is common to respond to people apologising for their bad English with
something along the lines of "don't worry, I deal with worse from people who
have been here for generations". We are not being patronising here (well, at
least some of us aren't!): sometimes we _genuinely_ do have to internally
correct/translate more when talking to someone from 300 miles away than we do
when talking to a Pole/Spaniard/what-ever who is not (yet) particularly fluent
in English.

~~~
vidarh
For "hotel" and some others, I believe the difference is whether or not the
"h" is aspirated. That is, whether or not you take a short break at the word
boundary. If you say it "in one breath", you get "an-(h)otel" with a very soft
h and no or very little aspiration.

If you make the word boundary more pronounced, you get "a hotel" with a
clearly noticeable air-burst, and a clearly distinct "h".

Same process as with "herb" etc. and some other words that can be pronounced
both with and without aspiration in English.

This is much the same as in French, where if the preceding word end in a vowel
(with certain exceptions) and the current word starts with a vowel sound, you
can usually elide the last vowel in the preceding words _except_ when the "h"
is aspirated. My French teacher liked to say that the French "thinks" they are
pronouncing the "h" in those few cases (aspiration usually refers to cases
where there's a strong burst of air in pronouncing the sound, but in French
"aspirated h" does not actually involve fully pronouncing an "h", though there
may be some aspiration)

(I may confuse this - this is based on recollection of my French classes 20
years ago...)

------
ibmthrowaway218
This is one of the pitfalls of a public only support mailing list. I know I
post using a gmail psuedonym to get support for various open source apps that
I use for side projects.

I'd be looking to provide an extra private/internal support email address made
available to the largest customers.

~~~
ciupicri
Except that even the basic support [1] includes that:

> _Support via private (email) ticketing system_ , with guaranteed 8 hour
> response time, during European office hours (9AM – 6PM, or 3AM – noon EST).
> Includes resolution of 5 incidents. Does not include DNSSEC support.

although CloudFlare should have probably bought the carrier grade support.

[1]:
[https://www.powerdns.com/support.html](https://www.powerdns.com/support.html)

~~~
smackfu
And no prices on that page at all, so you need to contact them, and then you
are dealing with a salesperson who sees CloudFare as a whale who needs to buy
the $$$ carrier-grade support and get them a big commission.

~~~
danielweber
If the customer would rather build their own system than risk talking to
someone on the phone where money might be discussed, I'm not sure that the
open-source vendor is the problem.

~~~
smackfu
The contention was that Cloudfare could have just signed up for the cheap
support plan if they wanted private problem reporting, and I don't think that
would be as easy as was suggested.

------
chris_wot
Curious to understand what the issue was.

~~~
jgrahamc
I work on our DNS server, but I don't have a copy of the email that Matthew
sent to the PowerDNS guys, but DNS is fundamental to what we do and so the
ability to modify the DNS server was pretty important.

I think we got to the point where we just said "there are N features that we
want" and decided to build our own. The result is that we've been able to do
all sorts of stuff that's CloudFlare-specific: bear in mind that we handle two
different sorts of DNS queries: external ones (that might or might not serve a
CloudFlare IP that is proxying for a customer site) and internal ones (that
need access to public, non-CloudFlare IPs). We also recently introduced a
CNAME-flattening feature: [http://blog.cloudflare.com/introducing-cname-
flattening-rfc-...](http://blog.cloudflare.com/introducing-cname-flattening-
rfc-compliant-cnames-at-a-domains-root)

There are other things we want to do to be fast in two ways: fast to respond
([http://www.solvedns.com/dns-comparison/2014/04](http://www.solvedns.com/dns-
comparison/2014/04)) and fast to update (which means integration with our
replicated, global settings databases).

We also have all sorts of special things that we do to deal with attacks on
our DNS infrastructure that include the DNS server talking directly to NICs
for filtering and packet handling. We're also generating bpf bytecode directly
for specific filtering activities.

Bottom line was that DNS is a core technology for CloudFlare; it made sense to
totally own it. I don't think this says anything bad about PowerDNS. Note:
HTTP serving is a core technology. If nginx weren't so configurable
(especially through OpenResty) we could easily have been having this
discussion about a web server. But equally agentzh (who wrote OpenResty) works
for us.

------
pilif
tangentially related: Thanks for once again pointing to PowerDNS. I've been
using bind forever and I never really looked out for other alternatives ("it
works - why should I bother learning something new?").

I had no idea what I was missing as from now on, I'll never again have to deal
with forgotten serials or manually adding slave zones to the secondary DNS
server.

Everybody still running with bind, thinking it's "good enough", ask yourself
how many times you were inconvenienced by either of the two issues and decide
whether it might be worth looking at alternatives.

The fact that PowerDNS apparently also is really performant doesn't matter for
my mostly minimal use case, but it's good to know that if something great
happens, I won't have to migrate again (not that migration is hard - powerdns
comes with a helper tool to read your bind config and produce SQL for your
zone files)

~~~
dmourati
May I recommend the excellent "Alternative DNS Servers" by JP Mens:
[http://www.amazon.com/Alternative-DNS-Servers-Deployment-
Bac...](http://www.amazon.com/Alternative-DNS-Servers-Deployment-Back-
Ends/dp/0954452992/ref=sr_1_1?ie=UTF8&qid=1399390889&sr=8-1&keywords=dns+book+mens)

------
dmourati
It is impossible to comment on this specific example without seeing the actual
email exchange. What I can say, is that a user _chooses_ how to disclose their
identity and the identity of their company making the best analysis of an
imperfect situation. Having worked for largely smaller companies (not in the
business of protecting the global internet), I've been relatively lax in
posting who I am and where I work. This transparency has been helpful on more
than a few occasions.

If you are a "large and/or interesting" company, it would behoove you to
disclose your identity in a safe and responsible way to projects (open source
or otherwise) who represent any important part of your business. The trick is
to do so in such a manner so as to derive the many benefits (increased
responsiveness) without the many liabilities (spam, sales calls, etc).

By remaining mum about who you are, the signal to noise ratio of your requests
do not stand out from the crowd. This is unfortunate for both sides. It does,
however, reinforce the notion that companies and projects should aspire to
provide excellent support to all users. You never know when you have a "secret
shopper" in your midst.

------
doxcf434
Often when a company want's to "help" it means make a sale regardless of how
poor the technical implementation is, waste your time with SE level support,
and power point presentations. When instead they should be focusing on the
product first. If your user is reporting a scaling issue and your product is
all about scale, that should be your first priority.

Vendors also need to remember, your users change companies, a user could be at
some small unknown company on day, and the next be in control of a large
budget at a high profile company. Your reputation follows that user, and what
they want to see is you're making a good product. If the smallest company is
reporting an issue that would affect your customers, I'd want to see that
you're addressing the issue with that company so I don't have to at my
large/interesting company.

------
alexdowad
No sympathy. If your support for users is poor, then it is simply poor, and no
excuses please. "We didn't know how big they were" doesn't cut the mustard.

------
gilrain
To complain that the only reason you gave someone bad support was because you
didn't know who they were is pretty tone deaf.

Would you want to dine somewhere that got panned by a restaurant critic and
then wrote an essay saying, "Well, we'd have been a lot more careful with his
food if we'd known he was an important critic!"?

~~~
octo_t
Your analogy is flawed. This is like a restaurant critic saying the portions
were far too small, but failing to reveal when they ordered that it wasn't
just for one person, but actually a party of 12 people.

~~~
markbnj
Hmmm, I'd agree if the issues were all performance-related, or if the author
had said "make sure to let us know if you are a large deployment." What the
author actually said was "large or impressive" and that point was reinforced
several times. I don't know how you measure "impressive" but it sounds a lot
like "special" to me.

~~~
octo_t
I imagine if you were using their product in an HFT setup (somehow) with only
a few servers, but with sub millisecond requirements they might also be
interested.

------
philjackson
I read the first part of that quote as "White PowerDNS" and wondered if I was
dreaming.

------
EGreg
How are DNS servers different from nameservers?

