
Put a Fork in Caddy; It's Done - neflabs
https://neflabs.com/blog/caddy-server/
======
mholt
Hey everyone -- Matt here.

Not quite sure why this is news; we've been discussing telemetry for a long[1]
time[2].

I haven't actually watched the video because I've been really busy finishing
my last semester of graduate school and switching research labs to start my
masters thesis this week.

Several of us in the research community have agreed that telemetry can be a
net good for the Web. Similar to how Firefox, the privacy-focused browser, has
telemetry on by default to help improve the Web.

A huge thanks to everyone who contributes to Caddy and makes it -- in my
opinion -- one of the best, easiest-to-use web servers around. We have over
200 contributors and it is amazing what the open source community has pulled
together, despite the growing pains we faced last year [3].

I hope you love using Caddy, and if you don't, you don't have to use it.

[1]: [https://caddy.community/t/caddy-0-11-will-have-telemetry-
dis...](https://caddy.community/t/caddy-0-11-will-have-telemetry-
discuss/3610?u=matt)

[2]: [https://caddy.community/t/the-caddy-telemetry-
project/3224?u...](https://caddy.community/t/the-caddy-telemetry-
project/3224?u=matt)

[3]: [https://caddy.community/t/the-realities-of-being-a-foss-
main...](https://caddy.community/t/the-realities-of-being-a-foss-
maintainer/2728?u=matt)

~~~
neflabs
Matt, your response here is a very nice advertisement for Caddy, and a glimpse
into your academic life, but you aren't addressing the very real privacy
concerns of your users.

~~~
onewhonknocks
One can easily opt-out. What's the issue here?

~~~
slenk
I just went to the Caddy website and GitHUb - if I was a new user I don't know
that telemetry is being collected nor how to turn it off.

~~~
detaro
Telemetry is being discussed, it hasn't been added yet. (And people have
rightly pointed out in the discussion that it has to be well-documented to be
acceptable)

------
conradk
It's amazing the lack of gratefulness some people have for open source devs.
Matt Holt, which I don't know nor have ever met, has spend a tremendous amount
of time developing Caddy and making it available for free. It's creating a lot
of value, regardless of what you think of recent telemetry announcements.

And still, apparently, some people will use the slightest disagreement to say
that the whole project is garbage, just like this article is doing. I think
we're lucky that open source developers don't get deterred by these kinds of
article, because they sure could be. What does religion have to do with the
quality of a free software project ? Does your server run better if coded by
an atheist ?

Instead of being disdainful, a more constructive thing to do would be to
openly talk about forking, on Caddy's forum, and see the response. If nothing
changes, fork and convince people that your fork is better.

~~~
pmlnr
Being able to fork in case of disagreements is a feature of open source, not a
bug.

Some of these forks get re-merged a few years later (see ffmpeg), others
don't, but this is not bad at all.

The ideal solution would be to build Caddy with flags that disable telemetry,
just like Firefox is built for Debian.

~~~
detaro
There's a difference between announcing a fork with "we disagree with the
following upstream decisions and thus are making this fork that will remove
them" and broadcasting widely something along the lines of "I hate this, it's
totally unacceptable, it's done, could someone fork it for us?" though,
smearing all kind of barely related things into it.

------
AndrewStephens
Allow me to spend a few words shooting the messenger: This blog post comes
across as petulant whining with a side order of personal attack. The author
needs someone to buy them a drink and explain that Caddy is just not that into
them and there are plenty of fish in the sea.

Having said that, I have no idea why the makers of Caddy think that telemetry
is a good idea. None of the examples given on the Caddy site make any sense to
me except for maybe reporting crashes. Who cares about the depth of
certificate chains? What value does it bring?

~~~
elcore
> Who cares about the depth of certificate chains? What value does it bring?

The data will be used to analyze the Internet from the server perspective,
similar to e.g Mozilla collecting data from the clients perspective. It
can/will be used by e.g researchers to improve the Internet (security, speed
etc.)

~~~
AndrewStephens
Who are these researchers? Does Caddy have the kind of market share that would
make that information useful?

I could sort of understand it is this was being pitched as a way to improve
Caddy by reporting back crashes or misconfigurations. But what I've heard
makes no sense to me.

------
ezekg
> This is no longer just a conversation on privacy; this is a hostage
> situation.

Give me a break. I'm tired of this type of drama and FUD coming up through
sensationalized posts like this. It's _open source_ \-- there's no hostage
situation.

I stopped reading at that whiney BS.

Matt is looking to collect anonymized data so that he knows how his _product_
is being used and how it can be improved -- which must be a tough situation,
considering most SaaS companies can throw whatever trackers they want up into
their apps and be done with it -- on-prem software is a little different when
it comes to usage statistics, etc., and I think this is acceptable.

You can grep this to see all of the data they're collecting,

    
    
        go telemetry.
    

You can _easily_ opt-out.

~~~
slenk
The only issue I have is that I don't see any mention about the telemtry
collection or how to opt out on their homepage or github readme...unless these
features aren't actually out yet and I misunderstood everything

~~~
detaro
It isn't out yet, mholt announced the plans for the next version and asked for
feedback about it.

------
carlhjerpe
My thoughts seem controversial, but i don't see the problem with things like
this when you're able to opt out. As long as there's some kind of notice
somewhere saying that you'll be part of data collection by default.

People should already know this here but it seems they often ignore it: Data
collection can be useful for developers so they can see what features are used
the most and which are used the least. If there's a nice feature that people
aren't using then maybe it should be "promoted" better in documentation for
people to find it, optimise functions people are using amongst other things.

~~~
oblio
I don't have any solid sources backing what I'm saying right now, but based on
what I've read and what I know about sales and marketing, the difference
between opt-in and opt-out is huge (much bigger than the 3 letter difference
:) ).

With opt-in you only reach something like 10% of your users, if you're like,
while opt-out is the exact opposite, you reach 80-90%.

This creates massive financial incentives to be sneaky and push opt-out.

And every time this happens for products where there's decent competition, the
people doing this lose a chunk of their users...

------
AWebOfBrown
> How can any server administrator trust Matt Holt or his software again?

I sincerely hope much of the community views this kind of personal attack on
an open source developer for what it is: _disgusting_.

If this kind of behavior is encouraged, imagine the message it sends to people
building open source software. Not only does your open source work not
directly reward you financially, but it will be used as ammunition to tarnish
your reputation, all because someone doesn't like the direction you're taking
your project.

Honestly, this is shameful.

------
firebacon
What a loveley sidecar ad-hominem. I'm sure his religious beliefs had a major
impact on the telemetry issue.

~~~
TheCowboy
It seems weird to be linking to the developer's (social media? church?)
profile in an attempt to illustrate a conflict in values or imply hypocrisy.
That is potentially out of line to me as the Caddy site itself doesn't link to
this profile or (that I could find) reference it in any form.

I don't think it is attacking his religion, but the criticism of the project
direction could be made without that.

------
tptacek
This introduction to this piece, with the utterly pointless link to Holt's
profile on a religious social network, is startlingly inappropriate. It says
something far more memorable and disturbing about "Nefarious Labs" than the
piece does about Caddy.

------
Vendan
This is not the first contentious decision for caddy, and it does already have
a fork:
[https://github.com/WedgeServer/wedge](https://github.com/WedgeServer/wedge)

On top of that, "fast, automatic TLS HTTP2 capable web server" is not some
complex feat in Go. HTTP2 is already baked into the stdlib, and you can add
automatic TLS via LE in a few lines of code:
[https://godoc.org/golang.org/x/crypto/acme/autocert](https://godoc.org/golang.org/x/crypto/acme/autocert).

A few alternatives:

[https://github.com/labstack/armor](https://github.com/labstack/armor)

[https://github.com/containous/traefik](https://github.com/containous/traefik)

------
Gigablah
This looks like a simple grab for attention by a company that touts "server
hardening" and "device hardening" as part of their services, one of their
projects being "c0llude", a "self-hosted, flat-file collaboration tool for
small teams and activists". It's supposed to prevent tracking by "government
lawyers and spies", so let's take a look at the source code:

[https://github.com/neflabs/c0llude/blob/master/api/caldel.ph...](https://github.com/neflabs/c0llude/blob/master/api/caldel.php)

Wow, that's just embarrassing.

------
flarco
Seems the OP lacks objectivity, especially with the religious reference. With
that said, I agree that telemetry should not be on by default, but prompted.

~~~
neflabs
We think his public statement, "Earning your trust is my most important
interpersonal goal" is both relevant and good. We're not attacking his
religion - we want Matt to stick to his publicly stated principles.

------
AdmiralAsshat
Linking to the Caddy dev's Mormon profile in the third sentence seems like a
really sleazy low-blow.

~~~
neflabs
We think his public statement, "Earning your trust is my most important
interpersonal goal" is both relevant and good. We're not attacking his
religion - we want Matt to stick to his publicly stated principles.

~~~
AdmiralAsshat
Then I would suggest changing the wording to just "profile". Whether you
realize it or not, the Mormons are a less-than-revered religious minority in
many parts of the US, and dropping that fact so early in the article comes
across as poisoning the well against him.

There's plenty of reason to be upset with Mr. Holt. His own reply elsewhere in
this thread reads more like a PR response than a real reply. But keep the
contention on-topic and less like a personal hit.

~~~
mholt
> His own reply elsewhere in this thread reads more like a PR response than a
> real reply.

What would you like me to say?

~~~
AdmiralAsshat
Hello, Matt!

To be clear, I am not a Caddy user and have no horse in this race. I tend to
sympathize with the privacy-conscious, however, having been a user who _turned
off_ telemetry in Firefox after the Mr. Robot scandal. Let me see if I can
explain why your response comes across as tone-deaf:

1) Your first response is "I haven't actually watched the video," which
immediately suggests that you're not going to actually engage with the claims
so much as tackle a strawman version of the claim. Now perhaps the author is
repeating an accusation that he has made in the past, and so you actually
_are_ familiar with it already, but that's not how this comes across.

2) Your next response--"Several of us in the research community have agreed
that telemetry can be a net good for the Web."\--is not really doing anything
to assuage the privacy concerns. It's not a technical refutation, and it's not
a particularly fleshed-out emotional appeal, either. It's basically, "We
disagree."

Put another way, let's imagine for a sec that you were a Tobacco CEO and the
following exchange was recorded:

Reporter: Sir, we have a multitude of evidence that smoking is conclusively,
irreversibly detrimental to human health.

CEO: Actually, a number of scientists and health officials have agreed that
smoking is good.

Do you realize how tone-deaf that non-answer comes across?

3) Your final response is the most "PR" part, as it first advertises the
product, then pivots away from the contention at hand in favor of praising how
wonderful it is that it's open source and has a vast number of contributors.
\---

I've already done the transposition analogy once, so I'm hesitant to do it
again lest it look like I'm demonizing you, but I want you to read the below
and see how you would perceive this response if it came from the CEO of J.Crew
about accusations of child labor in its clothing factories:

    
    
      "Hey everyone. James here.
    
      I haven't actually reviewed the accusations yet because I've been at a conference.
    
      We believe that allowing underage employees to fill a limited number of positions at 
      are factories allows impoverished families to bring in badly needed revenue, and 
      ultimately serves as a net positive for these needy communities.
    
      A huge thanks to everyone for shopping at J.Crew and making it the World's Best 
      Clothing Line™ five years and counting!"
    

\---

Hopefully that makes sense. It may not have been your intent, but perception
is critical when you're the public face of the company. You can gain or lose a
ton of goodwill among your users depending on whether you attempt to receive
their criticisms with an open ear and work towards a solution, or dismiss them
and dodge around the question. And even if you're doing the former, the mere
perception of the latter can be damaging.

Good luck.

~~~
yuhong
This completely ignores that the definition of open source code is that it can
be audited.

------
dsissitka
Mirror:
[https://web.archive.org/web/20180430141629/https://neflabs.c...](https://web.archive.org/web/20180430141629/https://neflabs.com/blog/caddy-
server/)

------
cjcampbell
It’s important to debate privacy, but it’s also important to understand that
privacy is much bigger than tracking and telemetry. From my position, Neflabs
compromises their own authority when they resort to bullying by mocking Matt
for his faith.

~~~
neflabs
We think his public statement, "Earning your trust is my most important
interpersonal goal" is both relevant and good. We're not attacking his
religion - we want Matt to stick to his publicly stated principles.

------
Touche
I sort of agree with this post but not for the reasons given. That developer
doesn't want a negative video posted to his forum is not a big problem to me.

The reason I left Caddy was that it just didn't feel that stable to me. By
that I don't mean that there are bugs, just that he keeps changing things. A
web server is not something I want to have breaking changes all of the time. I
want to deploy it and leave it alone; only upgrade for security patches. So
I've decided to switch to AWS.

~~~
mholt
When we get to 1.0, breaking changes won't happen on the same major version,
so upgrading will be more reliable. :) Thanks for testing it out before 1.0 so
we can get it right!

------
stevekemp
I can't reach the site to read the post, but it seems that Caddy has been
plagued with controvosy/drama as they tried to monetise their project - as
past discussion here shows:

[https://news.ycombinator.com/item?id=15237923](https://news.ycombinator.com/item?id=15237923)

It's a nice-looking project, and the integrated Let's Encrypt is great for
non-sophisticated users, but the user-base must be suffering.

~~~
pbreit
I’m surprised it survived the commercial rollout. AFAICT, the commercial
rollout required $300/year per instance for any and all commercial usage. It
made it sound like you weren’t even allowed to use the open source version for
commercial use (is that even possible?).

~~~
tobya
The commercial licence for the prebuilt binaries is $50 pm for 2 instances.
You can build yourself from source with no restrictions.

~~~
pbreit
OK, I see the requirement to pay is only for the binaries. It does seem like
grabbing binaries from gocaddy.com is the preferred installation method vs.
apt-get, etc for nginx.

$300 is the annual rate for 1 instance.

------
jamespo
Article timed out for me, seems neflabs needs new webserver software.

~~~
tobya
:) [https://caddyserver.com](https://caddyserver.com)

------
eklavya
From the caddy blog post, I can’t seem to find any objectionable behaviour.
What are you the objections if any?

------
rrdharan
The original announcement is here:
[https://caddyserver.com/blog/caddy-0_11-telemetry](https://caddyserver.com/blog/caddy-0_11-telemetry)

And forum discussion here: [https://caddy.community/t/caddy-0-11-will-have-
telemetry-dis...](https://caddy.community/t/caddy-0-11-will-have-telemetry-
discuss/3610/5)

I thought some of the ideas in the "middle ground" section discussing opt-in
versus opt-out being the default depending on how you obtained the software
were sort of interesting, hadn't heard that compromise suggested before.

~~~
rrdharan
Skimming through that discussion, it seems like the developer is also somewhat
naively optimistic and possibly underinformed regarding how much of his own
and his customers/users' effort will be required to comply with the GDPR while
gathering this data.

~~~
josteink
A server-installation data is not data about a particular _user_. It’s a
information about a piece of running software.

GDPR does not regulate information you can store about software components. It
merely ensures that companies can only store information about _people_ which
the person has given explicit and implicit consent for, and that they can
account for this consent.

Log-data from a running service disconnected from any identifiable personal
data is in no way covered by GDPR.

~~~
TimothyBJacobs
It sounds like it is collecting User Agent strings which depending on who you
ask is personal data.

~~~
josteink
That identifies _browser version_ and _operating system_ combinations in a way
which is aggregated and 100% decoupled in a irreversible way from the actual
browsing session as conducted by the user(s), given by the browser,
automatically, to everyone by default on every request.

You won’t find a single lawyer anywhere who considers this to be privacy
sensitive and definitely not covered by the GDPR.

~~~
rrdharan
I'm not convinced.

[https://www.iubenda.com/blog/device-fingerprinting-and-
cooki...](https://www.iubenda.com/blog/device-fingerprinting-and-cookie-law/)

My understanding is that anything that enables fingerprinting is potentially
covered.

[EDIT] So, here's a better link that specifically discusses fingerprinting and
user agents in a post-GDPR world:

[https://www.connectedpath.com/all-posts/2018/3/3/gdpr-and-
fi...](https://www.connectedpath.com/all-posts/2018/3/3/gdpr-and-
fingerprinting-how-are-you-being-tracked)

My assumption was that the GDPR was attempting to be sufficiently broad such
as to cover these kind of fingerprinting techniques but I guess not?

At least the second link makes it sound like at least some portion of people
are likely to turn more towards device fingerprinting techniques specifically
because they _are_ GDPR-safe.

~~~
josteink
I think trying to frame something you give away to everyone, always, without
anyone asking for it can legally be framed as privacy sensitive information.
That would simply be absurd.

The GDPR regulations largely represents common sense and decensy and this
über-paranoid consideration about what “may” be covered or not is not really
productive use of time.

Example: if you explicitly email someone, according to the GDPR the recipient
has been given an implicit right to store your email and email-address.
Because there’s no way for them not to. Because that’s just how email and
computers works.

I can’t imagine a fucking _user-agent_ string shared by billion of other users
enjoys higher protection.

The GDPR is not insane. Chill.

~~~
rouzh
> I think trying to frame something you give away to everyone, always, without
> anyone asking for it can legally be framed as privacy sensitive information.
> That would simply be absurd…The GDPR is not insane. Chill.

Isn't it? Just one particularly absurd example: logging IP addresses in your
httpd's access logs can be considered a violation of GDPR. [1][2][3]

[1]: [https://www.whitecase.com/publications/alert/court-
confirms-...](https://www.whitecase.com/publications/alert/court-confirms-ip-
addresses-are-personal-data-some-cases)

[2]: [https://www.gdpr360.com/gdpr-ip-addresses-and-
classification...](https://www.gdpr360.com/gdpr-ip-addresses-and-
classification-theory-and-practice)

[3]: [https://www.smashingmagazine.com/2018/02/gdpr-for-web-
develo...](https://www.smashingmagazine.com/2018/02/gdpr-for-web-
developers/#what-is-personal-data)

------
stephenr
Not surprised at all.

And for all the jibes people like to make when comparing things like apache to
Caddy etc: guess which sole http/2 server passes all the spec tests?

Hint: it’s the one “that looks like a dinosaur” from 1995.

~~~
elcore
The Go community is working on it :) #upstream

------
moron4hire
It's concerning to me that people are drawing parallels to Cambridge Analytica
over this. It shows the deep, deep misunderstanding among not just lay people,
but people who should really know better, about what Cambridge Analytica and
Facebook actually did.

------
atomical
Isn't this easy to disable with /etc/hosts?

