
Why kernel development still uses email (2016) - pmoriarty
https://lwn.net/Articles/702177/
======
jasode
_> , kernel developers still use email because it is faster than any of the
alternatives. Over the course of the last year, the project accepted about
eight changes per hour — every hour — from over 4,000 developers sponsored by
over 400 companies_

Compare with Google's ~1875-per-hour from 2015[1]:

 _> , and Google’s 25,000 engineers make about 45,000 commits (changes) to the
repository each day._

Obviously, we can't extrapolate that Google should switch to an internal SMTP
project management system layered on top of Perforce because _" email is
faster than any of the alternatives"_. Same for other companies with high
commit rates to their source repos like Microsoft and Facebook.

Since Linus & Linux community never did an A-vs-B experiment of email vs open-
source-JIRA-clone as a project workflow, I think we can only safely say that
email works best for a disparate group of volunteers with different employers.
Email is the lowest common denominator that everyone could agree to use.

You couldn't get 4000 _opinionated_ developers to agree to use an open-source
JIRA-clone or MS-Team-Foundation-clone or some other ALM[2]-in-the-cloud.

Therefore, listing advantages of email clients such as "threaded tree views"
etc is actually _hiding_ the underlying reason email-as-workflow works for
Linux. Email isn't about "throughput"; it's "acceptance".

[1] [https://www.wired.com/2015/09/google-2-billion-lines-
codeand...](https://www.wired.com/2015/09/google-2-billion-lines-codeand-one-
place/)

[2]
[https://en.wikipedia.org/wiki/Application_lifecycle_manageme...](https://en.wikipedia.org/wiki/Application_lifecycle_management)

~~~
abiox
> Compare with Google

isn't google's situation a monorepo composed of many distinct projects? there
doesn't seem much utility in the comparison.

> Email isn't about "throughput"; it's "acceptance".

this seems to be the similar, in the other direction. using (or not) an alm in
an organization isn't about "throughput", it's about "accepting" a company's
policy assertions.

~~~
jasode
_> using (or not) an alm in an organization isn't about "throughput", it's
about "accepting" a company's policy assertions._

I don't think that equating it from the reverse direction works. Here's why:

Using the article's count of ~4000 developers, let's consider _all software
companies_ that employ 4000+ developers as part of a "natural experiment"[1].
This includes companies and organizations like MS, Facebook, NASA, SAP,
Oracle, etc.

Each company "votes" in the _marketplace of ideas_ to coordinate software
development for a complex project. One company might use JIRA. Another uses MS
Team Foundation. Another uses FogCreek. Maybe another employer might use SMTP
emails similar to Linux kernel development.

Since companies have every incentive to develop software with cost-efficiency
and velocity, they will want to use the best workflow possible.

Since no single software company I know of uses SMTP emails as their ALM
workflow, it either means that (1) companies are wasting money & losing
iteration velocity by avoiding email -- or -- (2) the Linux kernel community
has special circumstances that makes using a true unified ALM unrealistic. I
believe that special circumstance is: 4000 programmers _work for different
employers_ instead of a single employer.

[1]
[https://en.wikipedia.org/wiki/Natural_experiment](https://en.wikipedia.org/wiki/Natural_experiment)

~~~
abiox
> Since companies have every incentive to develop software with cost-
> efficiency and velocity, they will want to use the best workflow possible.

in the abstract these are generally viewed as or presumed to be positive, but
the pressures and objectives of a business can be rather complex and other
things will often compete for finite resources.

having worked in a number of large organizations, the waste and inefficiencies
one can find is absolutely staggering.

outside the sales pitches, it's also not clear how an alm interacts with
notions of 'cost-efficienty' and 'velocity'. introducing process and
bureaucracy isn't always a win.

perhaps the presumed predilection of large organizations for alms can be found
in alms addressing (or seeming to address) other needs, pressures or
requirements.

~~~
jasode
_> having worked in a number of large organizations, the waste and
inefficiencies one can find is absolutely staggering._

Sure, I agree this is true in a general sense (Dilbert cartoons, Peter
Principle, etc). However, for things like _choosing technology_ to help with
market superiority, companies (especially _software engineering_ companies)
will eventually gravitate to something better.

We could argue that companies are "inefficient" for not embracing remote work-
from-home instead of open offices, etc. (general inefficiencies). But, even
badly run companies don't insist on using inferior floppy drives instead of
USB drives or wireless network file transfers. Yes, a law firm might limp
along with an 15-year old version of Microsoft Word 2000, but a _software
engineering_ company will not. (technology inefficiencies)

 _> introducing process and bureaucracy isn't always a win._

The Linux dev community _also_ has process & bureaucracy. See my previous link
for the _strict and precise_ formatting rules to submit kernel patches in
emails. _That 's process._ Just because it's not in a web form with fields
doesn't mean it's not "process". They're just using emails as the workflow for
it. (Linus has famously chastised contributors for not following "the rules".)
Process & bureaucracy is not bad thing -- its effort just has to match the
scope of work being delivered. (E.g. using JIRA for a "hello world" toy
program is overkill.) The Linux kernel community seems to have converged on
the level of bureaucracy that works for them.

It's interesting that even young YCombinator companies with much less than
4000 employees also don't stay with SMTP email-as-ALM as they get bigger. If
email-as-ALM was a better software dev philosophy, there would be a huge
arbitrage opportunity to outcompete everybody. Instead, the email-as-ALM fits
Linux kernel devs' specific circumstances.

------
TazeTSchnitzel
One of the best things about email is that, with the right client, you can
view discussions as hierarchical trees. That means you can have multiple
simultaneous discussions about a topic asynchronously and not lose track of
anything.

GitHub and the like only work that well when there's maybe one or two things
being discussed at once, because there's just a single linear stream of
comments.

~~~
pmoriarty
This is one of the things that web-based forums have yet to get right. Email
(and NNTP news) clients from 20 or 30 years ago are far superior in this
respect, because they can intelligently deal with threading and folding. These
features alone makes large conversations much easier to deal with than on web-
forums.

To add to that, email (and NNTP news) clients even from 20 or 30 years ago
have other powerful features that web forums have yet to catch up on:

\- kill files[1] (which you can use to filter out unwanted articles/mails
based on content or metadata such as subject, user, etc)

\- scoring

\- user-configurable anti-spam filtering or other "intelligent" filtering
(such as bayesian filtering not just for spam/ham, but for
interesting/unintersting content)

\- tagging not just on a site-wide level but at the client level so each user
can tag messages/articles the way they make sense to them

\- other advanced filtering and scripting based on any of the above

Web-based forums are just incredibly primitive compared to this many-decade-
old technology.

[1] -
[https://wikivisually.com/wiki/Kill_file](https://wikivisually.com/wiki/Kill_file)

~~~
nextos
Exactly. Gnus is a marvelous client because it has really good NNTP and
threading support (silencing and automatic scoring) that allows you to cope
with gigantic volumes of email.

It's a bit sad the code is messy and it's a tad slow compared to other
options.

Notmuch for Emacs has so much potential to replace Gnus. The interface is
incredibly fast and friendly (a classic tree view and a threaded view that
IMHO surpasses Gmail). Plus super quick search.

I think it could become a killer application for Emacs (the other two now are
org-mode and Magit IMHO).

However, there's no good tooling to keep Notmuch in two-way sync with maildirs
right now. Everything works based on tags, which is incredibly flexible, but
there should be some tooling to implement logic that allows moving mails
around based on their tags, so that the remote IMAP store gets tidy.

People usually rely on scripts for simple use cases, which is fine albeit a
bit hacky. Some DSL that resembles Sieve, but implemented in ELisp would be
cool.

~~~
pmoriarty
Notmuch has potential, but also a bunch of limitations and issues compared to
Gnus.

Notmuch doesn't have scoring, it doesn't have BBDB integration, it has no way
to get arbitrary message headers from elisp, it only indexes a handful of
headers.

I've heard it is possible to use notmuch as a backend for Gnus, to get the
best of both worlds.

As for two-way sync, some people have come up ways of doing that, using
something like Sieve.

For me two-way sync is not super critical, as I've come to view the webmail
interface of my mail provider as just a backup to use in case my laptop dies
or something. In all other cases, I'll just use notmuch, for which local tags
are enough.

~~~
nextos
I ran Gnus with notmuch backend (nnir-notmuch) for many months. It's not very
fast as Gnus is not very quick on Maildirs.

Scoring could be quite easy to implement with auto-tagging. Tags are a
unifying concept in email. You can implement any custom user logic with tags.
What notmuch is missing is a bit of tooling to do this. It'd be cool if this
tooling had access to any arbitrary header, as you have pointed out.

I feel I don't need BBDB integration when I have autocompletion based on the
email database. That said, there's bbdb-notmuch.el, and I assume other options
as message-mode is really a Gnus thing.

I think basic two-way sync is almost always needed. Otherwise you end up with
a huge inbox and even if you don't care at all about the remote IMAP store,
mbsync calls will start slowing down.

~~~
pmoriarty
Autocompletion from the notmuch database is fine when you don't have many
addresses in there. But when you are subscribed to a bunch of large mailing
lists and you try to search your notmuch database for "John" you're going to
get a thousand completion candidates.

If instead you search just your BBDB database, which only has the contacts you
actually correspond with, you're going to get just the useful and relevant
ones... and you'll always have the notmuch database to search on as a backup,
in case the contact you want to to correspond with is not in BBDB yet.

~~~
nextos
Sure, all these points are true. Notmuch is a bit unfinished I think. What
email setup do you have?

~~~
pmoriarty
I'm actually using notmuch in emacs myself, but I'm just running in to a bunch
of its limitations and looking around for an alternative.

Very sorry to hear that Gnus with a notmuch backend is slow, as that's what I
was considering trying next.

~~~
nextos
Maybe give Gnus a try anyway. It might not be that slow on good hardware. Or
just run it coupled with a Dovecot server.

I really like Notmuch's interface. I think limitations are perhaps not that
hard to patch on Elisp unless you have a complicated setup.

I'm trying to implement a DSL to coordinate tag <-> maildir sync on top of a
few macros.

------
kashyapc
I split my time between Gerrit-based and email-based projects. While I
appreciate Gerrit for its benefits (like the excellent CI integration), I
instinctively tend to gravitate towards my email-based workflow that is
blazing fast, allows me to work completely offline (once OfflineIMAP fetches
email in the morning), and lets me process tons of email / patches.

The said offline workflow (setup with: Mutt, OfflineIMAP, Postfix for local
mail queueing, and Notmuch for fast indexing / searching / tagging) gives me
unalloyed joy compared to the clickety-clicky, RSI-inducing pain that is
Gerrit.

The problems listed in the below email, by Dan Berrangé, about Gerrit are
still a bloody pain in the neck for me:

[https://lists.nongnu.org/archive/html/qemu-
devel/2014-02/msg...](https://lists.nongnu.org/archive/html/qemu-
devel/2014-02/msg00232.html)

[I learnt 2 days ago (can't find the URL to it; typing from my phone) that
Gerrit now allows you to respond via email (strangely the announcement only
mentioned the dreaded HTML email). I'm a bit cautious to say "sounds
promising" without trying it yet.]

------
IncRnd
It doesn't make sense for large projects to move to other systems. Control is
lost of the history, which automatically disqualifies any platform. Using
services others than e-mail will also entail a revamp and move to a different
service every few years.

Email is a far lower cost and more reliable medium most other methods.

~~~
u801e
I've always wondered why newsgroups aren't used instead of email. They pretty
much have the same advantages of email and they're far better in terms of
searching archives compared to email. Web interfaces for searching email list
archives are painful to use. If it wasn't for services like gmane (and later
public-inbox), it would be very difficult to view archives of those lists.

~~~
IncRnd
Newsgroups could be an exception to what I wrote. That's a good point. But,
newsgroups don't use the special protocols that I meant, which may not be
long-lived enough.

------
hdhzy
I wonder if they considered git appraise [0] as it allows offline work and
synchronization due to underlying transport (git).

[0]: [https://github.com/google/git-appraise](https://github.com/google/git-
appraise)

------
blauditore
(Unrelated to the e-mail topic)

> A poor craftsman famously complains about his tools, Greg said, but a good
> craftsman knows how to choose excellent tools.

Unless their company forces tools on them. :(

~~~
ivan_gammel
Actually good craftsman knows how to use the tools available to him to do the
job in the most effective way. Choice is a luxury that is not always present.

~~~
482794793792894
Just because you say that doesn't make it true, though. Sometimes a tool just
is worse, even if you become skilled at it.

I can try to work around Microsoft Word's UI by learning the keyboard
shortcuts, but I can't do anything against Internet Explorer taking its sweet
time loading webpages. No skill in the world will make that faster.

~~~
ivan_gammel
Well, that's the point. If you can get your job done with IE (I can only guess
what kind of job is that), you just do it without pointless clicking, pressing
F1 button or anything else that does not belong to the path of success. Talks
about "it's worse", "omg it's so slow", "why not firefox/chrome/whatever" are
meaningless when you are in strict corporate environment with only IE allowed
or even possible. True professionals do not pay much attention to the tools.
They pick the best when they have choice or find the way to use what's
available, because craftsmanship is not about the tools.

------
Iv
I think a crucial point is missed: email does not have a single point of
failure.

If you go through a website, a server downtime halts the development. If the
mailing list crashes, patches can still be sent and some discussions kind of
continue until it is resolved.

Individual email servers may fail, but it is easy to work around it. Many
people have several email and other channels through which to be reached.

Also, a very important point that Linus explained was at the core of some
design decisions for git: it works offline. You have your email archive
offline so if you take a plan, are at a conference in a place with no internet
(does still exist, sadly), are on the road, you can still go through your
archives and the patches you have to examine are still available.

------
thg
Discussion from a year ago:
[https://news.ycombinator.com/item?id=12620468](https://news.ycombinator.com/item?id=12620468)

------
Sir_Cmpwn
I've been working on a side project to bring the best of both worlds. It aims
to provide git repository hosting, issue tracking, continuous integration,
_and_ mailing lists to organize efforts. But I'll also be providing web tools
for submitting and reviewing patches that provides a workflow familiar to
GitHub users, but backed by mailing lists.

If anyone is interested in getting involved in the closed alpha for this
software, my email's in my profile. Hit me up!

------
unlmtd1
Easy: because using a webapp to track issues sucks donkey balls, and matrix
still has no threading and good terminal client.

------
CoreXtreme
How about using Discourse? I think it scales well. Would love to hear your
experience with this.

~~~
usernam
I find Discourse absolutely horrendous. Discourse mailing-list mode is a tack-
on at best, and the formatting you get in mail notifications basically require
you to be online to visit the link. And the gamification is also a huge put-
off.

Web forums made us forgot how far we came with NNTP years ago. Even the
shittiest NNTP could do subthread splitting, hiding and scoring. Handling
hundreds of messages per day was _easy_. The convenience of doing-it-all under
one powerful client was massive.

You can still do that with mailing lists, as long as it's not as badly managed
as forums-with-a-wanna-be-mailing-list-mode addon as Discourse is.

------
uiri
Ironically enough, Google actually does use Gerrit for their own kernel
development.

~~~
jwilk
What's ironic about it?

------
shmerl
I can understand it for the kernel. What about Debian bugs?

------
zozo123
Because you can read email from the command line.

------
na85
For 99% of folks outside the HN echo chamber, email works fine. Spam is a
problem, but I've never heard or read of anyone who thinks email needs to go
the way of the dodo except on this site.

~~~
erikbye
Email is the pinnacle of human communication.

Without notifications, which I, of course, do not use, email is the least
intrusive and least disrupting way someone can contact me. I simply check my
email when I feel like it, which is about two times daily, morning and
sometime late evening.

Then, even when I look at my email, I don't have to respond. I don't have to
do anything. And if there's an email I want to reply to, I won't necessarily
do so right now. Maybe tomorrow. Maybe never.

The best part is the record of all communications and the fact that you have a
searchable archive. Man, I wish I could search all my other conversations with
people. In fact, I think I should just start recording everything, every day,
all the time, and convert the audio to text.

~~~
keithpeter
_" In fact, I think I should just start recording everything, every day, all
the time, and convert the audio to text."_

About a decade or perhaps 15 years ago in Birmingham UK, the Ikon gallery had
a multi-artist exhibition of a conceptual nature. one of the artists had used
a micro-cassette recorder to record every word he uttered over a week and then
had the tapes transcribed by audio-typists. The results where exhibited as an
impressively high stack of A4 sheets. Unfortunately, they were, apart from a
few sample sheets on the wall, piled in a _glass case_ thus preventing viewers
from sampling the text. I was taking bets on the sheets that were hidden from
view being blank. The artist also exhibited the micro-cassette recorder and
tapes used.

Google has nothing to say about the exhibit, but it has to be said my memory
is a bit short on key words.

A sibling post mentioned Ted Nelson.

------
m-j-fox
What I want to know is: which of Tower, GitKraken, GitHub Desktop or Stash do
the kernel developers prefer?

~~~
pritambaral
Doubt they'd prefer any of those. None of those are open source, every single
one is a GUI app (which has a history of limiting access to the plumbing that
can be provided), and some of those don't even run on Linux.

