Hacker News new | comments | ask | show | jobs | submit login
[flagged] Put a Fork in Caddy; It's Done (neflabs.com)
68 points by neflabs 9 months ago | hide | past | web | favorite | 84 comments

Hey everyone -- Matt here.

Not quite sure why this is news; we've been discussing telemetry for a long[1] time[2].

I haven't actually watched the video because I've been really busy finishing my last semester of graduate school and switching research labs to start my masters thesis this week.

Several of us in the research community have agreed that telemetry can be a net good for the Web. Similar to how Firefox, the privacy-focused browser, has telemetry on by default to help improve the Web.

A huge thanks to everyone who contributes to Caddy and makes it -- in my opinion -- one of the best, easiest-to-use web servers around. We have over 200 contributors and it is amazing what the open source community has pulled together, despite the growing pains we faced last year [3].

I hope you love using Caddy, and if you don't, you don't have to use it.

[1]: https://caddy.community/t/caddy-0-11-will-have-telemetry-dis...

[2]: https://caddy.community/t/the-caddy-telemetry-project/3224?u...

[3]: https://caddy.community/t/the-realities-of-being-a-foss-main...

I haven’t used Caddy myself and am not prepared to weigh in on the telemetry discussion, but I still think you deserve a thank you for such a sizeable contribution to the open source community!

Matt, your response here is a very nice advertisement for Caddy, and a glimpse into your academic life, but you aren't addressing the very real privacy concerns of your users.

Looking through the discussions linked, he's done more for a reasonable discussion of this than you have. And I say that as someone who's not a fan of many things around Caddy (and would very much prefer opt-out if I were in any danger of using stock Caddy).

Next time, maybe spend more time on elaborating on the "very real privacy concerns" and less on personalities.

How is deleting comments from his own community "done more for a reasonable discussion of this than you have"? Your argument has no basis, you're simply shooting the messenger.

that's not the only thing he has done in that discussion, and I'm basing my comment on that.

Your video contains little argument that hasn't been already in the discussion, goes in no detail what specifically you object to, personally insults him, and being a video isn't a very good contribution in a text-forum anyways: in sum more then enough reasons for a moderator to remove it, and adding very little to the overall discussion.

I'm confused. If telemetry is an option and you can opt out, why not just do that? Also, because everything is open source, you're welcome to inspect the source to verify that it's behaving as expected and not reporting telemetry. You mentioned that you're not a Go programmer and/or you don't have time to do this...okay, so pay someone else who is a Go programmer to do this for you. Or is any software that costs money too much?

Matt has a right, as the creator of that work, to ask for some form of compensation for the portion of his life invested into the project. It's a completely fair and reasonable request. If you're set on avoiding any form of compensation (financial or otherwise), then you don't have any rights to demand anything from him. Just go use nginx and be done with it.

You're conflating two issues. If Matt wants money, we're happy to pay. That's a separate issue from telemetry.

We don't want a) a sole developer making decisions which the community has rejected, and b) software which sends telemetry by default.

> a) a sole developer making decisions which the community has rejected

Have you run a poll to be sure that >50% of the community rejects the change? Anyway, online polls are not reliable, it's difficult to be avoid sock puppets, people that don't use the software, ensure that people that disagree with your proposal care to vote, ...

Just make a fork and if most of the community agree with you, your fork will be most popular and shadow the original.

If Matt feels that having telemetry on by default is the form of compensation he would like for his time/investment, I don't see that as a problem.

I can see that different people would take exception to that which is why there's an off switch for telemetry.

If you're having a difficult time accepting that telemetry exists at all, then I'm almost certain Matt would be happy to create a non-telemetry version for paid subscribers.

One can easily opt-out. What's the issue here?

I just went to the Caddy website and GitHUb - if I was a new user I don't know that telemetry is being collected nor how to turn it off.

Telemetry is being discussed, it hasn't been added yet. (And people have rightly pointed out in the discussion that it has to be well-documented to be acceptable)

It's amazing the lack of gratefulness some people have for open source devs. Matt Holt, which I don't know nor have ever met, has spend a tremendous amount of time developing Caddy and making it available for free. It's creating a lot of value, regardless of what you think of recent telemetry announcements.

And still, apparently, some people will use the slightest disagreement to say that the whole project is garbage, just like this article is doing. I think we're lucky that open source developers don't get deterred by these kinds of article, because they sure could be. What does religion have to do with the quality of a free software project ? Does your server run better if coded by an atheist ?

Instead of being disdainful, a more constructive thing to do would be to openly talk about forking, on Caddy's forum, and see the response. If nothing changes, fork and convince people that your fork is better.

Being able to fork in case of disagreements is a feature of open source, not a bug.

Some of these forks get re-merged a few years later (see ffmpeg), others don't, but this is not bad at all.

The ideal solution would be to build Caddy with flags that disable telemetry, just like Firefox is built for Debian.

There's a difference between announcing a fork with "we disagree with the following upstream decisions and thus are making this fork that will remove them" and broadcasting widely something along the lines of "I hate this, it's totally unacceptable, it's done, could someone fork it for us?" though, smearing all kind of barely related things into it.

Totally! Forks can be great.

Saying "Caddy was supposed to be great" or "This is no longer just a conversation on privacy; this is a hostage situation" is not right. Caddy is great, but Neflabs doesn't agree with the direction it's taking. The solution is not describing the free software's developer as a hostage taker.

The solution, as you said, is forking.

Apache httpd is itself a (patchy) fork of NCSA httpd.

It's hard to imagine a productive forking discussion taking place on Caddy's forums directly. Certainly not in that announcement.

At any rate having read this article I didn't see anything saying that "the whole project was garbage".

The video (which Matt Holt removed) is also actually worth watching IMO, although I can see why he doesn't want it posted on the forum given its tone.

The video has a personality calling Matt an 'asshole'. That was unnecessary. Once that statement was made, I do not blame Matt for removing it. Name calling is not the way to go about addressing this important issue. It is indeed unfortunate that it has tainted what was otherwise a useful point of view.

Agree. They also call him a 'dick', repeatedly, and insinuate that he possibly has plans to monetize the gathered data. It escalates quickly and sadly I think all of that speculation and name-calling merely detracts from the strength and importance of the argument.

Matt said in his statement above, "I haven't actually watched the video" so the truth is, he removed it simply because he asked the community for feedback and decided to censor any opinion which didn't agree with his goals.

Allow me to spend a few words shooting the messenger: This blog post comes across as petulant whining with a side order of personal attack. The author needs someone to buy them a drink and explain that Caddy is just not that into them and there are plenty of fish in the sea.

Having said that, I have no idea why the makers of Caddy think that telemetry is a good idea. None of the examples given on the Caddy site make any sense to me except for maybe reporting crashes. Who cares about the depth of certificate chains? What value does it bring?

> Who cares about the depth of certificate chains? What value does it bring?

The data will be used to analyze the Internet from the server perspective, similar to e.g Mozilla collecting data from the clients perspective. It can/will be used by e.g researchers to improve the Internet (security, speed etc.)

Who are these researchers? Does Caddy have the kind of market share that would make that information useful?

I could sort of understand it is this was being pitched as a way to improve Caddy by reporting back crashes or misconfigurations. But what I've heard makes no sense to me.

> This is no longer just a conversation on privacy; this is a hostage situation.

Give me a break. I'm tired of this type of drama and FUD coming up through sensationalized posts like this. It's open source -- there's no hostage situation.

I stopped reading at that whiney BS.

Matt is looking to collect anonymized data so that he knows how his product is being used and how it can be improved -- which must be a tough situation, considering most SaaS companies can throw whatever trackers they want up into their apps and be done with it -- on-prem software is a little different when it comes to usage statistics, etc., and I think this is acceptable.

You can grep this to see all of the data they're collecting,

    go telemetry.
You can easily opt-out.

The only issue I have is that I don't see any mention about the telemtry collection or how to opt out on their homepage or github readme...unless these features aren't actually out yet and I misunderstood everything

It isn't out yet, mholt announced the plans for the next version and asked for feedback about it.

My thoughts seem controversial, but i don't see the problem with things like this when you're able to opt out. As long as there's some kind of notice somewhere saying that you'll be part of data collection by default.

People should already know this here but it seems they often ignore it: Data collection can be useful for developers so they can see what features are used the most and which are used the least. If there's a nice feature that people aren't using then maybe it should be "promoted" better in documentation for people to find it, optimise functions people are using amongst other things.

I don't have any solid sources backing what I'm saying right now, but based on what I've read and what I know about sales and marketing, the difference between opt-in and opt-out is huge (much bigger than the 3 letter difference :) ).

With opt-in you only reach something like 10% of your users, if you're like, while opt-out is the exact opposite, you reach 80-90%.

This creates massive financial incentives to be sneaky and push opt-out.

And every time this happens for products where there's decent competition, the people doing this lose a chunk of their users...

> How can any server administrator trust Matt Holt or his software again?

I sincerely hope much of the community views this kind of personal attack on an open source developer for what it is: disgusting.

If this kind of behavior is encouraged, imagine the message it sends to people building open source software. Not only does your open source work not directly reward you financially, but it will be used as ammunition to tarnish your reputation, all because someone doesn't like the direction you're taking your project.

Honestly, this is shameful.

What a loveley sidecar ad-hominem. I'm sure his religious beliefs had a major impact on the telemetry issue.

It seems weird to be linking to the developer's (social media? church?) profile in an attempt to illustrate a conflict in values or imply hypocrisy. That is potentially out of line to me as the Caddy site itself doesn't link to this profile or (that I could find) reference it in any form.

I don't think it is attacking his religion, but the criticism of the project direction could be made without that.


Religious flamewar will get you banned here. Please don't post like this again.


This introduction to this piece, with the utterly pointless link to Holt's profile on a religious social network, is startlingly inappropriate. It says something far more memorable and disturbing about "Nefarious Labs" than the piece does about Caddy.

This is not the first contentious decision for caddy, and it does already have a fork: https://github.com/WedgeServer/wedge

On top of that, "fast, automatic TLS HTTP2 capable web server" is not some complex feat in Go. HTTP2 is already baked into the stdlib, and you can add automatic TLS via LE in a few lines of code: https://godoc.org/golang.org/x/crypto/acme/autocert.

A few alternatives:



This looks like a simple grab for attention by a company that touts "server hardening" and "device hardening" as part of their services, one of their projects being "c0llude", a "self-hosted, flat-file collaboration tool for small teams and activists". It's supposed to prevent tracking by "government lawyers and spies", so let's take a look at the source code:


Wow, that's just embarrassing.

Seems the OP lacks objectivity, especially with the religious reference. With that said, I agree that telemetry should not be on by default, but prompted.

We think his public statement, "Earning your trust is my most important interpersonal goal" is both relevant and good. We're not attacking his religion - we want Matt to stick to his publicly stated principles.

Linking to the Caddy dev's Mormon profile in the third sentence seems like a really sleazy low-blow.

We think his public statement, "Earning your trust is my most important interpersonal goal" is both relevant and good. We're not attacking his religion - we want Matt to stick to his publicly stated principles.

If it's just about the statement "Earning your trust is my most important interpersonal goal", why include the remainder of the comment? If it's not part of your commentary, why reference it all? You've already seen fit to elide most of the rest of the profile - your thinly-veiled attempts to publically shame him for his Christianity are as shameful as your verbal diarrhea while pontificating that the only IMAGINABLE reasons for collecting telemetry are because he's going to sell the data.

Collecting telemetry is a decades-old method of getting real data about the behaviour of a program in real-world environments. This is why web browsers, smartphones, and OSes all have telemetry collection routines.

Then I would suggest changing the wording to just "profile". Whether you realize it or not, the Mormons are a less-than-revered religious minority in many parts of the US, and dropping that fact so early in the article comes across as poisoning the well against him.

There's plenty of reason to be upset with Mr. Holt. His own reply elsewhere in this thread reads more like a PR response than a real reply. But keep the contention on-topic and less like a personal hit.

> His own reply elsewhere in this thread reads more like a PR response than a real reply.

What would you like me to say?

Hello, Matt!

To be clear, I am not a Caddy user and have no horse in this race. I tend to sympathize with the privacy-conscious, however, having been a user who turned off telemetry in Firefox after the Mr. Robot scandal. Let me see if I can explain why your response comes across as tone-deaf:

1) Your first response is "I haven't actually watched the video," which immediately suggests that you're not going to actually engage with the claims so much as tackle a strawman version of the claim. Now perhaps the author is repeating an accusation that he has made in the past, and so you actually are familiar with it already, but that's not how this comes across.

2) Your next response--"Several of us in the research community have agreed that telemetry can be a net good for the Web."--is not really doing anything to assuage the privacy concerns. It's not a technical refutation, and it's not a particularly fleshed-out emotional appeal, either. It's basically, "We disagree."

Put another way, let's imagine for a sec that you were a Tobacco CEO and the following exchange was recorded:

Reporter: Sir, we have a multitude of evidence that smoking is conclusively, irreversibly detrimental to human health.

CEO: Actually, a number of scientists and health officials have agreed that smoking is good.

Do you realize how tone-deaf that non-answer comes across?

3) Your final response is the most "PR" part, as it first advertises the product, then pivots away from the contention at hand in favor of praising how wonderful it is that it's open source and has a vast number of contributors. ---

I've already done the transposition analogy once, so I'm hesitant to do it again lest it look like I'm demonizing you, but I want you to read the below and see how you would perceive this response if it came from the CEO of J.Crew about accusations of child labor in its clothing factories:

  "Hey everyone. James here.

  I haven't actually reviewed the accusations yet because I've been at a conference.

  We believe that allowing underage employees to fill a limited number of positions at 
  are factories allows impoverished families to bring in badly needed revenue, and 
  ultimately serves as a net positive for these needy communities.

  A huge thanks to everyone for shopping at J.Crew and making it the World's Best 
  Clothing Line™ five years and counting!"

Hopefully that makes sense. It may not have been your intent, but perception is critical when you're the public face of the company. You can gain or lose a ton of goodwill among your users depending on whether you attempt to receive their criticisms with an open ear and work towards a solution, or dismiss them and dodge around the question. And even if you're doing the former, the mere perception of the latter can be damaging.

Good luck.

This completely ignores that the definition of open source code is that it can be audited.

For example, you could say, "Clearly the community feels strongly about this issue, and while I still believe Caddy should include telemetry, it will be opt-in, not the default. Sorry for censoring posts on the Caddy forum, it won't happen again."

Fair point. Done.

It’s important to debate privacy, but it’s also important to understand that privacy is much bigger than tracking and telemetry. From my position, Neflabs compromises their own authority when they resort to bullying by mocking Matt for his faith.

We think his public statement, "Earning your trust is my most important interpersonal goal" is both relevant and good. We're not attacking his religion - we want Matt to stick to his publicly stated principles.

I sort of agree with this post but not for the reasons given. That developer doesn't want a negative video posted to his forum is not a big problem to me.

The reason I left Caddy was that it just didn't feel that stable to me. By that I don't mean that there are bugs, just that he keeps changing things. A web server is not something I want to have breaking changes all of the time. I want to deploy it and leave it alone; only upgrade for security patches. So I've decided to switch to AWS.

When we get to 1.0, breaking changes won't happen on the same major version, so upgrading will be more reliable. :) Thanks for testing it out before 1.0 so we can get it right!

Article timed out for me, seems neflabs needs new webserver software.

Agreed, we're using Caddy right now. Time to switch!

From the caddy blog post, I can’t seem to find any objectionable behaviour. What are you the objections if any?

The original announcement is here: https://caddyserver.com/blog/caddy-0_11-telemetry

And forum discussion here: https://caddy.community/t/caddy-0-11-will-have-telemetry-dis...

I thought some of the ideas in the "middle ground" section discussing opt-in versus opt-out being the default depending on how you obtained the software were sort of interesting, hadn't heard that compromise suggested before.

Reading that it really feels like he only posed the question for discussion so that he can say that he got input. It doesn't seem like the feedback is going to sway him an inch.

I'm really not a fan of default telemetry (and other aspects of Caddy), but some of the feedback in that thread is really bad and I can totally understand mholts reactions to it. Which is a shame, since it risks to drown out more detailed comments.

Skimming through that discussion, it seems like the developer is also somewhat naively optimistic and possibly underinformed regarding how much of his own and his customers/users' effort will be required to comply with the GDPR while gathering this data.

A server-installation data is not data about a particular user. It’s a information about a piece of running software.

GDPR does not regulate information you can store about software components. It merely ensures that companies can only store information about people which the person has given explicit and implicit consent for, and that they can account for this consent.

Log-data from a running service disconnected from any identifiable personal data is in no way covered by GDPR.

It sounds like it is collecting User Agent strings which depending on who you ask is personal data.

That identifies browser version and operating system combinations in a way which is aggregated and 100% decoupled in a irreversible way from the actual browsing session as conducted by the user(s), given by the browser, automatically, to everyone by default on every request.

You won’t find a single lawyer anywhere who considers this to be privacy sensitive and definitely not covered by the GDPR.

I'm not convinced.


My understanding is that anything that enables fingerprinting is potentially covered.

[EDIT] So, here's a better link that specifically discusses fingerprinting and user agents in a post-GDPR world:


My assumption was that the GDPR was attempting to be sufficiently broad such as to cover these kind of fingerprinting techniques but I guess not?

At least the second link makes it sound like at least some portion of people are likely to turn more towards device fingerprinting techniques specifically because they are GDPR-safe.

I think trying to frame something you give away to everyone, always, without anyone asking for it can legally be framed as privacy sensitive information. That would simply be absurd.

The GDPR regulations largely represents common sense and decensy and this über-paranoid consideration about what “may” be covered or not is not really productive use of time.

Example: if you explicitly email someone, according to the GDPR the recipient has been given an implicit right to store your email and email-address. Because there’s no way for them not to. Because that’s just how email and computers works.

I can’t imagine a fucking user-agent string shared by billion of other users enjoys higher protection.

The GDPR is not insane. Chill.

> I think trying to frame something you give away to everyone, always, without anyone asking for it can legally be framed as privacy sensitive information. That would simply be absurd…The GDPR is not insane. Chill.

Isn't it? Just one particularly absurd example: logging IP addresses in your httpd's access logs can be considered a violation of GDPR. [1][2][3]

[1]: https://www.whitecase.com/publications/alert/court-confirms-...

[2]: https://www.gdpr360.com/gdpr-ip-addresses-and-classification...

[3]: https://www.smashingmagazine.com/2018/02/gdpr-for-web-develo...

IANAL, but I could set my own completely custom user agent - I don't even need much technical expertise, a simple browser add-on would suffice - and by logging that string, I could be (depending on how unique I made my own user agent) uniquely identified.

Out of all the metrics Caddy plans to collect, it's the only one I think has some merit to its complainants. It might be simpler to only keep user agents that conform to common browser standards. But this has all been discussed in the Caddy forum thread itself, and we'd welcome your input there!

Which of "this data" do you see as relevant under GDPR?

Per the sibling thread - User Agent strings.

I can't reach the site to read the post, but it seems that Caddy has been plagued with controvosy/drama as they tried to monetise their project - as past discussion here shows:


It's a nice-looking project, and the integrated Let's Encrypt is great for non-sophisticated users, but the user-base must be suffering.

I'm personally happy Matt is charging for his work. More open-source devs should charge for commercial use a la Sidekiq, ngrok, and Caddy.

Free OSS is great, but for these types of tools, so is paid OSS.

There's a large pool of small, useful dev tools that could be making money instead of begging for it. And I'm glad Matt is a part of the much smaller group that makes money.

I have no qualms about commercial services either, it's just a very different model here compared to nginx (which is about the closest comparison project I can think of).

Huh? Nginx also has a commercial license, Nginx Plus.


I was suggesting the drama caused previously was due to the confusing way it was handled - pay to avoid a header, or to use "official" binaries.

I'm well aware that nginx has a commercial side, which is why I mentioned it as a comparison.

I’m surprised it survived the commercial rollout. AFAICT, the commercial rollout required $300/year per instance for any and all commercial usage. It made it sound like you weren’t even allowed to use the open source version for commercial use (is that even possible?).

You're free to do anything with the source code that's permitted by the Apache license, including commercial usage. The commercial licensing only affects the binaries offered on the site.

I'd agree that it wasn't very well-handled and being the biggest hassle for the wrong kind of people, but I believe it always clearly talked about their binaries only.

The commercial licence for the prebuilt binaries is $50 pm for 2 instances. You can build yourself from source with no restrictions.

OK, I see the requirement to pay is only for the binaries. It does seem like grabbing binaries from gocaddy.com is the preferred installation method vs. apt-get, etc for nginx.

$300 is the annual rate for 1 instance.

Not surprised at all.

And for all the jibes people like to make when comparing things like apache to Caddy etc: guess which sole http/2 server passes all the spec tests?

Hint: it’s the one “that looks like a dinosaur” from 1995.

The Go community is working on it :) #upstream

It's concerning to me that people are drawing parallels to Cambridge Analytica over this. It shows the deep, deep misunderstanding among not just lay people, but people who should really know better, about what Cambridge Analytica and Facebook actually did.

Isn't this easy to disable with /etc/hosts?

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact