
Load-Bearing Internet People - ingve
http://esr.ibiblio.org/?p=8383
======
tptacek
Or, you know, he could just get a job like everyone else does, including
virtually everyone who _actually does_ maintain critical internet
infrastructure.

Meanwhile, for those keeping score at home, a quick recap of the ways in which
Raymond, esteemed author of fetchmail and maintainer of gpsd, has referred to
himself:

* A "load-bearing Internet person"

* "one of the senior technical cadre that makes the Internet work"

* "one of the half-dozen or so most influential people" in open source, "in fact, a lot of people would put me among the top three".

* Hisham ibn-Sindbad (the Black Wazir) in Arabian Nights

* Someone you could reasonably infer is "the most famous programmer in the world" (but Knuth doesn't count because he's not famous outside of CS, nor is Bill Gates because he's not famous as a programmer, nor is Kevin Mitnick or Aaron Swartz).

* A member of a "small cadre of old hands" like Vint Cerf and Dave Taht with "the specialized technical knowledge required for Internet disruption on a massive scale".

* A person "creating the computer code that makes your digital world work"; "every time you use a Web browser, locate yourself on Google Maps, draw money from an ATM, or play on a game console, you rely on computer code I wrote and gave away."

~~~
pvg
There's a succinct summary of his position in the classical literature:

[https://www.youtube.com/watch?v=f9aM_dT5VMI&t=182](https://www.youtube.com/watch?v=f9aM_dT5VMI&t=182)

------
QuadrupleA
In spite of all its amazing aspects, one obvious drawback of the open source
movement is that it devalues a lot of stuff that you could once build a
business around and support a lot of people making a living. Category after
category of software businesses have declined in the face of great free
alternatives - OS's, languages and compilers, server software, databases.

Tears for Larry Ellison aside, I do feel social systems / economies work best
when there's a mutual exchange of value, a win-win rather than a win-lose, and
the classic simple payment in exchange for software has a certain fairness and
sanity to it. E.g. the iOS app store was a lot nicer and high quality before
ad-supported / in-app purchases became the norm. Anyway something like ESR
proposes here seems promising.

I'm not an open source maintainer myself (except for my magnum opus
[https://github.com/QuadrupleA/private-secure-sharing-
buttons](https://github.com/QuadrupleA/private-secure-sharing-buttons)) but
seems like a lot of important-project maintainers find themselves in a
demanding position with little reward besides ego-stroking or future promise
of job opportunities to make money elsewhere.

~~~
jandrewrogers
A key nuance is that open source has largely only devalued software that any
competent software engineer could write. In this sense, open source replaces
software that was essentially selling labor rather than expertise. A developer
always has the option to write this software themselves if they don't want to
pay for it. The price of a product of undifferentiated labor will converge on
the cost of that labor in a well-functioning market, which in the case of open
source is effectively zero.

On the other hand, software that requires rare and difficult to acquire
expertise -- software development where competence is insufficient -- faces
little threat from open source in practice. This creates two-tier markets that
differentiate on the replicability of _capability_. Products that are
replicable by any software developer eventually will be in open source. But
there are still large gaps in capabilities between closed and open source in
some markets because the average software developer has no obvious way to
replicate those capabilities on a purely technical level.

In essence, you can only make money if you are doing hardcore R&D. This
strongly incentivizes the creation of new capabilities but also
disincentivizes publication of CS research.

You see this in markets like databases, where open source has captured almost
the entire market for undifferentiated capabilities, and there is a lucrative
high-end market with unique product capabilities that don't exist in open
source or CS literature. The trend toward treating CS research as trade
secrets, originally started because algorithm patents were impractical to
enforce, turned out to be effective at maintaining profitability in high-end
software products if open source can't replicate capability.

~~~
Tomte
> A key nuance is that open source has largely only devalued software that any
> competent software engineer could write

I don't think so. Take GCC or Clang, for example. While I have taken a lot of
compiler courses at university, I couldn't build a production compiler for a
real language, without dedicating a decade of my life or so to it.

What Open Source projects really bring to the table is massive manpower over
decades. Those projects that become popular, at least. And then no smaller dev
shop (or single developer except Fabrice Bellard and a handful others) can
possibly compete.

~~~
jandrewrogers
There are legitimate exceptions (and as I was writing that LLVM was the one
that popped into my mind) but they are quite rare in practice if you look at
open source as a whole. And Intel's compiler is still significantly better at
optimization, as of a couple years ago at least.

Open source _sometimes_ has a manpower advantage, usually when companies are
paying for the development, but manpower per se doesn't address the
significant expertise advantage of closed source in many areas. Quantity is
not a good substitute for quality.

~~~
0815test
> significant expertise advantage of closed source in many areas

Citation _very much_ needed here! The average closed source product has
_zilch_ expertise advantage compared to FLOSS. And the expertise that _is_
embedded in FLOSS is actually verifiable (as well as, crucially, being
_resilient over time_ \- sometimes enduring for decades in a "load-bearing"
role with no long-term maintenance issues whatsoever), in a way that closed
source could never be.

~~~
jandrewrogers
Most closed source applications contain no differentiating technical
expertise, this is true. By "expertise", I am referring to deep expertise in
computer science domains where the state-of-the-art is rarely published. The
areas where you see large gaps in capability are domains that are
intrinsically computer science limited e.g. most things involving high-
scale/high-performance data infrastructure, real-time sensor processing,
database kernels, etc. There is a surprising amount of highly evolved
algorithms and designs that are not in literature but nonetheless show up in a
multiple closed source systems.

Customers pay for the capabilities and performance that expertise affords.
Most of it just manifests as "speeds and feeds" i.e. orders of magnitude more
throughput, scalability, etc. In much rarer cases, there are manifest
capabilities that don't have an obvious solution in published computer
science, never mind open source (e.g. large scale semantic models of physical
reality).

~~~
0815test
This is better described as "niche" expertise than anything _significantly_
'deep' or 'high-end'. Basically, stuff that (1) is _not_ truly business-
critical, at least compared to the closest FLOSS equivalent, and (2) is so
inherently niche that it would _not_ derive any benefit from the typical
network effects that drive people to standardize on open source solutions. ESR
himself discusses this case in his book _The Magic Cauldron_ ; one example he
provides of something where FLOSS is probably not called for, is software for
calculating cutting patterns for sawmills. A different one, is the business-
management "secret sauce" that's embedded in many ERP packages. Note that the
"not truly business-critical" proviso is quite relevant; for something that
_is_ critical to your business, however "niche" it might be, not having access
to the source code would be an unacceptable risk!

------
gravypod
I'm also pretty sure that if these incentives were in place we would see a lot
of engineers switch to these really interesting tasks. No one wants to shovel
http requests back and forth, they want to build things. I think a lot of
internet, and software, infrastructure is more compelling work than what most
high level engineers work on.

~~~
thrwayxyz
It is but the number of people who can understand it's value is inversely
proportional to the interest of the work. I had to setup a TAI server for a
trading company I worked in because no other time mechanism could record the
trades happening reliably without duplicates. I would have been interested in
making it a universal service. It would take me an hour long presentation to
explain to other programmers with a physics background why this was absolutely
essential. I can't imagine ever convincing a suit as to why they should pay
for it.

~~~
nitrogen
_I can 't imagine ever convincing a suit as to why they should pay for it._

 _...no other time mechanism could record the trades happening reliably
without duplicates._

Start with the core value added, describe the world with and without in
cost/benefit terms, and work from there.

~~~
thrwayxyz
My opportunity cost is a mid six figure salary vs the hustle of trying to sell
something that can be copied very easily by others. Not worth my time or
effort any more.

------
wbl
Some of the examples are people who are actively harmful. Harlan Stern is the
reason NTP WG is so dysfunctional as he insists that his implementation and
his private plans are more important then what everyone else wants, and the WG
exists to ratify his plans.

~~~
toyg
Well, it _is_ ESR speaking...

~~~
dfrage
Who just happens to be the tech lead to a partial answer to this problem:
[http://ntpsec.org/](http://ntpsec.org/) Which implicitly criticizes the
reference implementation.

~~~
tptacek
As I understand it, NTPsec, a hostile fork of ntpd, is not a well-regarded
project. Look at the "project accomplishments" page and see what they _don 't_
claim to have accomplished: the elimination, _prior to publication_ , of any
vulnerabilities in a msinstream/default ntpd configuration. They reorganized a
bunch of code, swapped strcpy's (and strncpy's) with strlcpy, moved the
project out of Bitkeeper (something that has nothing to do with security but
is the _first_ listed achievement on the site), and generally removed stuff
nobody enables in ntpd.

Before it lost funding, Raymond was openly discussing rewriting the whole
thing in Go, which sort of gives the lie to the idea that the project was
operating in good faith.

~~~
dfrage
Accusing ESR and the rest of the NTPsec project of fraud is a very serious
claim. Could you explain in more detail why contemplating rewriting of most or
all of the project in Go as he was learning the language is such a definitive
tell?

~~~
tptacek
I haven't accused anyone of the crime of fraud; fraud requires an active
intent to acquire something of value through misrepresentation, and I'm happy
to concede that forces other than intentional misrepresentation are at work
here.

The premise of the ntpsec project was that ntpd was an unloved and mismanaged
codebase that suffered, as a result, from security flaws. Raymond and his team
would take over the code, in something similar to the manner the openssh
project took over SSH, and eliminate security vulnerabilities. The project
needed funding because ordinary developers wouldn't take on such a thankless
task --- maintenance programming on a giant C codebase --- without
compensation.

A reimplementation of NTP in a different language is not at all the same
project --- as you can see from all the NTP projects that already exist in Go
and Rust, for which nobody appears to be begging contributions. Not to mention
the obvious fact that people don't run new implementations of NTP in Go or
Rust _because they can 't_, and so abandoning the ntpd codebase eliminates
almost all of the purported value of the project to the Internet.

~~~
wbl
Most sites can switch from ntpd to something else. See for instance systemd
timescynd which really doesn't have a reason to exist. And changing to chronyd
was a very quick switch.

I think it is really inertia. Time synchronization goes unloved at a lot of
places.

~~~
tptacek
I don't disagree! In particular, a ground-up Rust replacement for the 20% of
ntpd that everyone relies on would do a lot of good and be deployable
virtually everywhere ntpd is today (Raymond proposed a Go rewrite --- I
strongly prefer Go to Rust, but Go has a garbage-collected runtime).

But that's besides the point. Pushing a hostile fork of a popular project,
raising money for it, and then _abandoning the codebase entirely_ for a
rewrite takes a "special" kind of chutzpah.

------
praptak
What software for a critical internet service or a library does ESR maintain?
I checked his Wikipedia entry and have not found anything.

~~~
plorkyeran
ESR's whole life has been spent pointing at people who actually do important
things and saying "I'm with them", and he's done it so much that people assume
that it must be true.

------
icebraining
(Without getting into the discussion of who is an LBIP or not)

> Where there’s no profit stream, markets are not going to directly solve this
> problem.

The market is a process of matching suppliers with consumers, and contrary to
the previous statement, it _has_ solved the problem, by finding a very cheap
supplier: you.

LBIP are admirable selfless people, but I think this attitude is as misguided
as a parent doing their kid's homework for them. The rest of us won't care to
find a better solution until we start feeling the pressure, and we won't feel
it while the load is being born by them. And the author is essentially asking
regular individuals to throw a few tips to the LBIPs, so that the current
broken model can be maintained. I'm not sure we wouldn't be better off letting
it fail.

------
jauer
No mention of a org that's funding things that are actually "internet load
bearing":
[https://www.coreinfrastructure.org/grants/](https://www.coreinfrastructure.org/grants/)

Also, this might be better described as Load-Bearing Individual Participants.
If only to avoid distraction around what counts as internet-critical.

Beyond funding, what happens when they get burned out or take a vacation?
Walls have many bricks. What makes these people unique? I'd expect anyone in a
senior position to be working on growing people to help share the load, open
source or corporate. There's only so much you can do by your self.

------
harryh
The idea that ESR is a LBIP is laughable. Do not give this horrible person
your hard earned money.

~~~
_vertigo
I’m out of the loop, why do you say he’s a horrible person?

~~~
neilk
ESR has done, and apparently continues to do, many useful things for open
source. But his various personality disorders have reset that karma back to
zero, or possibly put him in negative territory.

It started early... [https://lists.debian.org/debian-
user/1999/04/msg00623.html](https://lists.debian.org/debian-
user/1999/04/msg00623.html)

And here is a generally well-sourced summary:
[https://rationalwiki.org/wiki/Eric_S._Raymond](https://rationalwiki.org/wiki/Eric_S._Raymond)

~~~
brandmeyer
Its kinda hard to accept a reference as rational when the opening summary
describes him as having "batshit insane wingnut tendencies." Those are not
appeals to reason, they are appeals to emotion and tribalism.

~~~
tptacek
Raymond is an avowed and weirdly obsessed racial supremacist (and, at times,
an advocate of political violence). That's not an appeal to tribalism but
rather a recorded fact; both conservatives and liberals alike would find his
stated beliefs horrifying if laid out in front of them rather than scattered
throughout his various discursive blog posts and comment threads.

There is a general and valid concern in our industry about a liberal orthodoxy
that makes it at least socially unsafe to express political thoughts. People
who share that concern should have an even bigger problem with Raymond, who is
an attention-seeking caricature of conservative or libertarian belief. People
who have couched their bigotry in far more careful and subtle language than
Raymond have found themselves ostracized from conservative circles, and for
good reason.

I think the better way to engage with Raymond is on his manifest deficiencies
as a professional engineer and technologist, and wouldn't want to start a
rebuttal to his request for donations with his politics. But I also wouldn't
want to let stand the idea that the opprobrium he attracts is rooted in
orthodoxy or tribalism.

~~~
mwcampbell
> his manifest deficiencies as a professional engineer and technologist

Do you have references to more information on that?

~~~
tptacek
I believe I could back that statement up with references, yes.

------
NickBusey
Personally I think the problem with a lot of these kind of Patreon setups is
they are generally pitched as "I once wrote some very useful software, and
sometimes I maintain it."

While that's great and all, for me at least it doesn't justify why I should
send them money every month.

I think the Open Source model needs to evolve, and if these people want
monthly income, they should be "showing their work" a lot more. Put out a
regular (weekly or more often) YouTube or blog series. Maybe stream your work
on Twitch (like I do :) ). Provide some active community involvement. Then I
would be far more inclined to donate to someone, rather than having to wonder
all the time if I'm paying for them to work on this stuff, or if I'm paying
them so they can go on an extra vacation next year for work they completed
years ago and have mostly ignored since then.

~~~
cs02rm0
I had similar thoughts about how they could use YouTube or something
equivalent to open access a little in return. Grow the cake and all that.

I then had quick Google and it seems a few years back some were seeing a
conflict between ESR's Internet Civil Engineering Institute and something
called the Core Infrastructure Initiative[1] which it seems is pulling in
millions.

Perhaps it just lost out.

[1] [https://www.coreinfrastructure.org](https://www.coreinfrastructure.org)

~~~
thrwayxyz
These people are not entertainers. Demand they post their work on YouTube
makes as much sense as demanding architects use smoke signals to communicate
their progress.

~~~
cs02rm0
They're not entertainers, there should be no demands.

But I'd think talking a little about their interesting work might serve to
better convey what they do to those who are aware of it and reach a wider
audience. What ESR has tried obviously hasn't worked, would a half hour chat
about their work once a year or a twitch session, or whatever else really make
_no_ sense under any circumstance?

