
Why kernel development still uses email - dankohn1
https://lwn.net/SubscriberLink/702177/e2712c9c41c0c683/
======
nxc18
Email is an amazing tool. I don't understand the rush to get rid of it.

It is a free standard with many implementations on both the client and server.
It is exceptionally easy to manage email. Many servers have great systems for
categorizing emails and storing them long-term. Email is fast. Everyone has
one.

There is this weird myth that Email magically kills productivity while Slack
and GitHub don't. I never understood this - slack and GitHub have a lot more
distraction with their social media components.

I see the day coming soon when in order to work with other people I'll need to
find and install one of the x tens of IM clients. The day when I can't find
anything because it is in a million places and its real hard to search 10
years of conversation when messages don't have basic metadata like a subject.
:(

~~~
deckard1
> I'll need to find and install one of the x tens of IM clients.

Only one? :)

It got so bad at my company, that at one time we were having conversations on
3 platforms. And, of course, no one emails anything! Because it's all done on
Slack or whatever now. So everyone is 100% out of the loop at all times.

I'm up to about 20 Slack channels. I'm constant switching between things to
see if I need the information. We have Github integration (and 10 other
things) into everything, so it's a constant stream of garbage. And stupid gif
animation integration. God forbid you take time off work. Good luck catching
up on all that mess.

People like to shit on IRC today. But at least I got to choose the client I
wanted, and log to an actual file that I could actually grep on. And timestamp
on. And _organize_. And block all the stupid nonsense. First thing on the
cutting block would be ignore all Github.

~~~
usgroup
It harkens to a yesteryear of decentralised everything and a spirit that
wanted to keep it as such: an island for every man...

Now it's more of a huge resort super island filled with lots of other people
where everyone perpetually rents everything.

Too cynical?

~~~
uola
> Too cynical?

No, too romantic. E-mail was never good enough to be decentralized, much less
at the "every man" level.

------
DSMan195276
I think it's worth adding, in the age of git sending patches by email is
easier then ever - it definitely shows that git was developed by Linus with
the kernel's development in mind. There is literally a `git format-patch`
command which spits out a full formatted email (or emails) of your various
commits, which can then be sent to a mail client to be edited and sent out. Or
even easier, just use git to send the patches as email direct via `git send-
email`.

The harder part is touched on in the article - email clients and email
servers. All the patches sent to the mailing-list are plain text, and lots of
clients or servers either mess with the plain text before sending or after
receiving. And with that they also take some configuring to avoid sending
attachments as base64 - which will be rejected by the kernel. Gmail thankfully
doesn't mess anything up on the sever side, but the web client wraps your
email to 80 characters per line making it unusable. Because of the above
issues, using a separate mail client is essentially required, and setting them
up can be annoying and there aren't really tons of them to choose from. I
personally use `mutt`, but I can definitely understand why some wouldn't want
to bother.

~~~
bluedino
Outlook loves messing with messages. "Detecting and removing" extra line
breaks is one of its features

~~~
mrweasel
Outlook it probably the worst email client currently on the market and has
been for the all of it's existence.

It is my belief of the issue at lot of people have with email stems from
Outlook. One example is the weird instance of top posting has made it hard to
read many conversations and fostered a mentality of firing of quick response,
reducing Outlook/email to a poor mans IM client.

There's a ton of improvements that could be made to Outlook to make it better.

~~~
molteanu
Nope, Lotus Notes takes that prize.

~~~
mrweasel
I forgot about Notes. Yes, Notes is worse.

------
Daviey
The article totally missed the mark on Gerrit. I think the heavy user is now
OpenStack and not Android...

The article claims that Gerrit is just a small part, but if used fully it
really isn't. It can be used to track the entire patch lifecycle. Which LKML
doens't do via email.. because it isn't very good at it... it uses
'patchwork'.

OpenStack uses Gerrit to manage 537 git repositories. In the last year, 86,027
changesets were merged in the last year out of 111,292 commits. [0]

The real values are the smart searching capability (with email subscription to
that search),integration with CI, inline code reviews.. simple rebasing, acls
and easy digestion who reviewed.

Once setup, the workflow is super simple:

    
    
      $ git commit
      $ git review
      $ # celebrate success
    

Further, why all the hate on project managers?

[0]
[http://activity.openstack.org/dash/browser/](http://activity.openstack.org/dash/browser/)

------
na85
I find the idea that a technology should be replaced simply because it is old
to be very irksome. Those developers out there blindly contributing to the
trend of replacing "tried and true" with "new and shiny" really need to take a
step back and gain some perspective.

Email works great for the vast, vast majority of users.

~~~
Klathmon
While I agree, the opposite is also an issue. Developers which either look
down upon anything new as "pointless" or in some cases actively try to stop
improvements.

Looking at the kernel dev system, it works great, but it has its share of
problems. I've never even thought about contributing to the kernel because of
the work involved to just submit a patch, that process can be improved.

Making it easier for newcomers, and updating some of the workflow to be more
aligned with current tools could really help.

~~~
userbinator
I think it's a deliberate barrier to entry: they're basically saying "if you
can't be bothered to learn how to do it, we don't want your contributions."

Linus occasionally rants about some of the crap that gets in, and I have a
feeling he certainly doesn't want more, which would happen if the barrier was
lowered. Here's a somewhat recent example:

[http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01331.html](http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01331.html)

~~~
qb45
Probably you are right, though in the case you linked the barrier was clearly
insufficient. And OTOH, I have seen many cases where actually sane one-time
contributors had to spend unreasonable time changing trivial patches to match
the guidelines.

    
    
      [RFC PATCH] cmdline: Hide "debug" from /proc/cmdline
    
      It has come to our attention that a system running a specific user
      space init program will not boot if you add "debug" to the kernel
      command line.
    

Lol. I sometimes get disgusted by the kind of crap people merge, but it's nice
that at least this one got the ridicule it deserved.

------
ludamad
"Some systems, like Outlook, will uniformly corrupt patches; as a result,
companies doing kernel development tend to keep a Linux machine that they can
use to send patches in a corner somewhere." Isn't a Linux desktop also
convenient for the development itself?

~~~
technion
I developed a series of patches for the RPi kernel fork. I use the Pi
headless, and generally access it from a Windows machine.

I never tried to send those patches upstream - and the Pi fork uses Github for
such things - but I'm entirely confident that if I tried to paste those
patches into an email client somewhere along the line, I'd get an upstream
maintainer pointing out that magical line breaks blew something up somewhere.

~~~
aisofteng
What I read is "I am entirely confident about what would happen if I tried to
do something I've never tried to do", which is useless speculation.

~~~
ludamad
What I read is that he develops kernel patches entirely on a Windows desktop,
which is a relevant anecdote.

------
drewg123
When I was at Google, gerrit was indeed used internally. Things may have
changed in the last 18months, but moving off the old tool to Gerrit was such a
huge project that I'd be surprised if they'd moved on to something else.

~~~
xjia
+1. AFAIK Gerrit is still being used for kernel-related projects.

------
blaisio
So I get and agree with what he's saying about how GitHub duplicated email,
but as a millennial who uses Vim, I find GitHub much more intuitive and useful
than sending patches via email. In general, I disagree with the issues he has
with GitHub - most of them are wishy washy disagreements that could easily be
solved by adding things to GitHub or using a chrome extension.

IMO the only real, logical reason they need to keep using email is "Email
works just as well as GitHub, and we've already been using email, so we might
as well continue using it." Which is a good reason on it's own - there's no
reason to try and come up with problems with GitHub. (Again, not saying GitHub
is perfect, but I don't think the reasons he gave were particularly valid.)

~~~
michaelmrose
I don't really feel that you being a millennial or using vim is relevant.

I have no idea how you would go about getting github to implement whatever
functionality is desired and even so would come with the following
disadvantages

\-- Requiring a chrome extension limiting people to using chrome + a singular
website vs any email client under the sun

\-- wouldn't work offline

\-- Presumably it would be more challenging for sight impaired individuals to
use a js heavy website vs their email. They stated they had existing
contributors who were so impaired.

\-- Since people still presumably do a lot of discussion via email it would
mean using at minimum github AND email some of which would refer back and
forth to other discussions.

\-- Relying on github would require tying free software infrastructure to a
single non free software tool. Recalling bitkeeper this didn't go well last
time.

\-- Hosted github costs additional money.

\-- The primary and perhaps only benefit would be making it easier for people
to come in off the street and drop patches with minimal effort but this would
actually cost effort to manage and the kind of individual who can't or likely
wont figure out how to use existing infrastructure may not be an optimal
contributor in any case.

Basically your solution involves not only effort to create tools that don't
exist, the cooperation of a third party, but the end result would almost
certainly be objectively worse in a number of ways.

I feel like you didn't really address any of the things the actual expert on
the matter said and just hand waved his entire argument away.

~~~
RaleyField
> Requiring a chrome extension limiting people to using chrome

And people who use extensions. Beyond ad blocker on some machines I don't use
extensions on principle because of security concerns.

~~~
aisofteng
Catehorically not using extensions, regardless of their permissions system,
sounds less like "security concerns" and more like "fear rooted in ignorance."
Continuing to use some of them (ad blockers), regardless of said "security
concerns", sounds like cognitive dissonance.

~~~
michaelmrose
If any single extension author is compromised you may be yourself but ads are
also a malware vector thus the position is consistent.

------
ncouture
I like to think that Linux kernel developers master most of the tools they
use, especially maintainers who have power to accept patches.

From my own perspective and in different contexts, I can tell that email can
be as efficient as it can be inefficient and that depends mostly on its users.

------
taeric
The number of systems we have created to try and move collaboration off of
email is hilariously large.

Between email and lisp, is there any other system that you can find in any
sufficiently large system?

~~~
brandmeyer
> Between email and lisp, is there any other system that you can find in any
> sufficiently large system?

Make. How many systems try really really hard to just update the things that
need to be changed in an output store from a different input store using a
serial number? Nah, just use the date. That's almost like a serial number,
isn't it? Except for those handful of things for which we couldn't be bothered
to set up the dependencies correctly - those get updated every time. It won't
be _that_ expensive to refresh just a handful every run...

------
sambe
Email is great and I can see why they continue to use it, but there is a
danger of being blinkered. Saying that GitHub reinvented email because they
have comment threads is... almost trolling. Especially when also saying email
integrates well with patchwork (presumably written due to the inability of
email to handle issue/PR tracking).

Both have strengths and weaknesses.

------
u801e
One thing I've always wondered is why email based patch submission and review
was never done via a newsgroup over NNTP.

One advantage a newsgroup has over an email list is that you don't have to set
up filters in your email client, and that every message posted doesn't have to
be specifically CC'd to a certain group of people. Also, it's much easier to
browse past posts.

Currently this can be done via the gmane NNTP gateway for the kernel and git
mailing lists, but I can't see a good reason why a newsgroup wouldn't be
preferred over a mailing list.

------
csl
Is Gerrit really that terrible? I do agree that clicking through each file is
a waste, but I love the ability to work on several different reviews and keep
track of their different build statuses.

Also, where I use it we have different levels of requirements for different
branches (more people usually need to approve reviews on soon-to-be-released
point branches). We use it for a large project in MLOCs but relatively few
developers (less than a hundred). I can't see it would work for 4000 devs at
all, though.

~~~
adrianratnapala
Perhaps I don't understand the Gerrit workflow, but last time I read about
Gerrit it looked like it had a philosophy of "one commit, one review".

That sounds like an anethema for the kernel, where they want to review a
series of tidied up commits, one per email.

~~~
kiallmacinnes
Gerrit does have "one commit, one review", but - that's not a bad thing. If a
change is well written and structured, merging the first in the chain without
the second+ is both harmless and is progress towards the end goal.

There's also patch chains in Gerrit, so you can see the X commits that make up
the full change, and the second+ cannot be merged before the first.

------
cm3
For continued decentralized operations something like
[https://github.com/google/git-appraise](https://github.com/google/git-
appraise) looks good. It's like Fossil-SCM's distributed tickets but for code
review. It can also mirror to Gihtub and Phabricator and has a web interface
too. Most importantly it doesn't break the decentralized aspect of git.

------
icc97
> "On the other hand, GitHub does not scale to larger projects. He pointed at
> the Kubernetes project"

It seems according to this, that Github is fine for a project until it scales
to such a size as you need an alternative.

So even this is agreeing that small projects should start with Github.

It would be useful to know the lower bound of users/developers at which point
you need to switch to a more flexible system like email.

~~~
stephenr
s/should/could/

------
exDM69
tl;dr: my wish is a code review convention for git-notes and a friendly UI on
top.

If only Git would allow storing reviews and issues in the repo proper.
Something like Git notes with conventions to refer to files/lines in a
file/different revisions etc. Something similar to Fossil.

This would solve half of the problem. The other half would be the UI layer.

Now the reviews live in a database, outside of the repo, and can't be
downloaded locally for offline work.

This would allow doing code reviews and managing issues e.g. while commuting
in a train. And everything would be pretty fast compared to waiting for a web
page to load for every file you want to review and connecting to a server for
every comment you write.

~~~
ktt
Something like this [https://github.com/google/git-
appraise](https://github.com/google/git-appraise) ?

~~~
exDM69
Yes, something like that. I've seen it before and it sparked my interest.

Not sure about using json, though.

Now this project is just the first step, now it needs a code review tool (UI)
on top that's good enough to match Gerrit, GitHub, etc.

------
known
Email solves
[https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_prob...](https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem)
AND
[http://www.beej.us/guide/bgipc/output/html/singlepage/bgipc....](http://www.beej.us/guide/bgipc/output/html/singlepage/bgipc.html#fifopc)

------
piotrkaminski
Amusingly, the article references Kubernetes and its growing pains a couple
times, but fails to mention that it adopted Reviewable (on top of GitHub) for
doing more complicated code reviews. I don't know if Reviewable would scale to
kernel development -- and it certainly doesn't support offline work, etc. --
but at least Kubernetes seems to be doing all right with it.

(Disclosure: I'm the founder of Reviewable.)

------
crististm
I don't have anything against e-mail but I don't get what's wrong with Gerrit
either [1]. We used it on several projects and it was pretty transparent to
the work flow.

[1]except for some Java gc issues that required the process to be restarted
every now and then - pretty annoying though

------
lucaspottersky
Simple: humans don't like change.

We say we do like it, but the truth is that we don't.

------
rpedela
For Github organizations, can't you forward notifications to a mailing list?
Then everyone on the mailing list can see everyone's comments and respond
through email but you still get to keep Github and all of its advantages.

~~~
Manishearth
Or people can just subscribe to a repo. Github provides pretty decent tools
for filtering these via mail filters, too (sadly the on-site github
notifications aren't as fine grained). GKHs assertion that you can't follow
threads via email is false.

------
begriffs
Are there any good references online about how to switch from Github-based
collaboration to one based on email and patches? I'd like to give it a try and
see how it feels.

~~~
gkya
Put mailman/majordomo and cgit/gitweb on a server, and a little front page for
the project if you like. Use spamassassin with mailman, restrict allowed mail
mime types to plain text and patches. Maybe use one of those tools that pick
patches up from your mbox. Quilt is a good patch manager. I'd use emacs org
mode for issues and todos.

------
notacoward
As always, it's worth pointing out that this is just about the Linux kernel.
Different scale, different developer community, different code structure,
different testing/CI methodology (i.e. practically none) than just about every
other project out there. Email works for them - great. Other things work for
other people - also great. Theirs is one interesting data point, but that's
all it is.

~~~
catern
>As always, it's worth pointing out that this is just about the Linux kernel.
Different scale, different developer community, different code structure,
different testing/CI methodology (i.e. practically none) than just about every
other project out there.

Eh? Patches on mailing lists is the "traditional" way for open source projects
to operate. Many projects "still" do things this way - indeed I wish more
projects did, for the reasons outlined in this article!

~~~
notacoward
You're trying to refute a much stronger claim than I actually made. I was
pointing out these differences to explain why what's best for the Linux kernel
_might_ not be best for another project. I wasn't saying email is bad, that it
couldn't be the best for any other project, etc. The space of projects for
which email is a good choice might be larger than just the Linux kernel, but
it's much _much_ smaller than the whole space of open-source projects as well.
Given this community's tendency to cargo-cult whatever the highest-profile
projects do without thinking about how different circumstances might affect
applicability, I thought it was worth highlighting.

------
dschiptsov
Because it ain't broken.)

------
wildchild
What to use instead? Blockchain?

