Or, you know, he could just get a job like everyone else does, including virtually everyone who actually does maintain critical internet infrastructure.
Meanwhile, for those keeping score at home, a quick recap of the ways in which Raymond, esteemed author of fetchmail and maintainer of gpsd, has referred to himself:
* A "load-bearing Internet person"
* "one of the senior technical cadre that makes the Internet work"
* "one of the half-dozen or so most influential people" in open source, "in fact, a lot of people would put me among the top three".
* Hisham ibn-Sindbad (the Black Wazir) in Arabian Nights
* Someone you could reasonably infer is "the most famous programmer in the world" (but Knuth doesn't count because he's not famous outside of CS, nor is Bill Gates because he's not famous as a programmer, nor is Kevin Mitnick or Aaron Swartz).
* A member of a "small cadre of old hands" like Vint Cerf and Dave Taht with "the specialized technical knowledge required for Internet disruption on a massive scale".
* A person "creating the computer code that makes your digital world work"; "every time you use a Web browser, locate yourself on Google Maps, draw money from an ATM, or play on a game console, you rely on computer code I wrote and gave away."
In spite of all its amazing aspects, one obvious drawback of the open source movement is that it devalues a lot of stuff that you could once build a business around and support a lot of people making a living. Category after category of software businesses have declined in the face of great free alternatives - OS's, languages and compilers, server software, databases.
Tears for Larry Ellison aside, I do feel social systems / economies work best when there's a mutual exchange of value, a win-win rather than a win-lose, and the classic simple payment in exchange for software has a certain fairness and sanity to it. E.g. the iOS app store was a lot nicer and high quality before ad-supported / in-app purchases became the norm. Anyway something like ESR proposes here seems promising.
I'm not an open source maintainer myself (except for my magnum opus https://github.com/QuadrupleA/private-secure-sharing-buttons) but seems like a lot of important-project maintainers find themselves in a demanding position with little reward besides ego-stroking or future promise of job opportunities to make money elsewhere.
A key nuance is that open source has largely only devalued software that any competent software engineer could write. In this sense, open source replaces software that was essentially selling labor rather than expertise. A developer always has the option to write this software themselves if they don't want to pay for it. The price of a product of undifferentiated labor will converge on the cost of that labor in a well-functioning market, which in the case of open source is effectively zero.
On the other hand, software that requires rare and difficult to acquire expertise -- software development where competence is insufficient -- faces little threat from open source in practice. This creates two-tier markets that differentiate on the replicability of capability. Products that are replicable by any software developer eventually will be in open source. But there are still large gaps in capabilities between closed and open source in some markets because the average software developer has no obvious way to replicate those capabilities on a purely technical level.
In essence, you can only make money if you are doing hardcore R&D. This strongly incentivizes the creation of new capabilities but also disincentivizes publication of CS research.
You see this in markets like databases, where open source has captured almost the entire market for undifferentiated capabilities, and there is a lucrative high-end market with unique product capabilities that don't exist in open source or CS literature. The trend toward treating CS research as trade secrets, originally started because algorithm patents were impractical to enforce, turned out to be effective at maintaining profitability in high-end software products if open source can't replicate capability.
> A key nuance is that open source has largely only devalued software that any competent software engineer could write
I don't think so. Take GCC or Clang, for example. While I have taken a lot of compiler courses at university, I couldn't build a production compiler for a real language, without dedicating a decade of my life or so to it.
What Open Source projects really bring to the table is massive manpower over decades. Those projects that become popular, at least. And then no smaller dev shop (or single developer except Fabrice Bellard and a handful others) can possibly compete.
There are legitimate exceptions (and as I was writing that LLVM was the one that popped into my mind) but they are quite rare in practice if you look at open source as a whole. And Intel's compiler is still significantly better at optimization, as of a couple years ago at least.
Open source sometimes has a manpower advantage, usually when companies are paying for the development, but manpower per se doesn't address the significant expertise advantage of closed source in many areas. Quantity is not a good substitute for quality.
> significant expertise advantage of closed source in many areas
Citation very much needed here! The average closed source product has zilch expertise advantage compared to FLOSS. And the expertise that is embedded in FLOSS is actually verifiable (as well as, crucially, being resilient over time - sometimes enduring for decades in a "load-bearing" role with no long-term maintenance issues whatsoever), in a way that closed source could never be.
Most closed source applications contain no differentiating technical expertise, this is true. By "expertise", I am referring to deep expertise in computer science domains where the state-of-the-art is rarely published. The areas where you see large gaps in capability are domains that are intrinsically computer science limited e.g. most things involving high-scale/high-performance data infrastructure, real-time sensor processing, database kernels, etc. There is a surprising amount of highly evolved algorithms and designs that are not in literature but nonetheless show up in a multiple closed source systems.
Customers pay for the capabilities and performance that expertise affords. Most of it just manifests as "speeds and feeds" i.e. orders of magnitude more throughput, scalability, etc. In much rarer cases, there are manifest capabilities that don't have an obvious solution in published computer science, never mind open source (e.g. large scale semantic models of physical reality).
This is better described as "niche" expertise than anything significantly 'deep' or 'high-end'. Basically, stuff that (1) is not truly business-critical, at least compared to the closest FLOSS equivalent, and (2) is so inherently niche that it would not derive any benefit from the typical network effects that drive people to standardize on open source solutions. ESR himself discusses this case in his book The Magic Cauldron; one example he provides of something where FLOSS is probably not called for, is software for calculating cutting patterns for sawmills. A different one, is the business-management "secret sauce" that's embedded in many ERP packages. Note that the "not truly business-critical" proviso is quite relevant; for something that is critical to your business, however "niche" it might be, not having access to the source code would be an unacceptable risk!
> I do feel social systems / economies work best when there's a mutual exchange of value
That’s too broad of a claim to possibly be true. Many things are required for smoothly running economies, and sometimes a ledger is the problem and sometimes it’s the solution.
I think it's more because there's a lot of software that's too cheap to charge for, but has too many edge cases for the average programmer to get right.
Logging is a perfect example, server-side logs are all handled by free frameworks (too cheap to charge for, but everyone needs it), but client-side is often handled by companies (just complex enough to make money on).
The former is often handled by LBIPs like Eric, where-as the latter is either written in house (usually badly) or you get something like Sentry.
If the useful open source software development work was not scarce these maintainers would certainly not be under immense pressure.
To boot, the entire idea of post scarcity is preposterous. Until the theorized heat death of the universe, there will always be uneven distribution of resources and some things that are better than others.
I'm also pretty sure that if these incentives were in place we would see a lot of engineers switch to these really interesting tasks. No one wants to shovel http requests back and forth, they want to build things. I think a lot of internet, and software, infrastructure is more compelling work than what most high level engineers work on.
It is but the number of people who can understand it's value is inversely proportional to the interest of the work. I had to setup a TAI server for a trading company I worked in because no other time mechanism could record the trades happening reliably without duplicates. I would have been interested in making it a universal service. It would take me an hour long presentation to explain to other programmers with a physics background why this was absolutely essential. I can't imagine ever convincing a suit as to why they should pay for it.
My opportunity cost is a mid six figure salary vs the hustle of trying to sell something that can be copied very easily by others. Not worth my time or effort any more.
Some of the examples are people who are actively harmful. Harlan Stern is the reason NTP WG is so dysfunctional as he insists that his implementation and his private plans are more important then what everyone else wants, and the WG exists to ratify his plans.
As I understand it, NTPsec, a hostile fork of ntpd, is not a well-regarded project. Look at the "project accomplishments" page and see what they don't claim to have accomplished: the elimination, prior to publication, of any vulnerabilities in a msinstream/default ntpd configuration. They reorganized a bunch of code, swapped strcpy's (and strncpy's) with strlcpy, moved the project out of Bitkeeper (something that has nothing to do with security but is the first listed achievement on the site), and generally removed stuff nobody enables in ntpd.
Before it lost funding, Raymond was openly discussing rewriting the whole thing in Go, which sort of gives the lie to the idea that the project was operating in good faith.
Accusing ESR and the rest of the NTPsec project of fraud is a very serious claim. Could you explain in more detail why contemplating rewriting of most or all of the project in Go as he was learning the language is such a definitive tell?
I haven't accused anyone of the crime of fraud; fraud requires an active intent to acquire something of value through misrepresentation, and I'm happy to concede that forces other than intentional misrepresentation are at work here.
The premise of the ntpsec project was that ntpd was an unloved and mismanaged codebase that suffered, as a result, from security flaws. Raymond and his team would take over the code, in something similar to the manner the openssh project took over SSH, and eliminate security vulnerabilities. The project needed funding because ordinary developers wouldn't take on such a thankless task --- maintenance programming on a giant C codebase --- without compensation.
A reimplementation of NTP in a different language is not at all the same project --- as you can see from all the NTP projects that already exist in Go and Rust, for which nobody appears to be begging contributions. Not to mention the obvious fact that people don't run new implementations of NTP in Go or Rust because they can't, and so abandoning the ntpd codebase eliminates almost all of the purported value of the project to the Internet.
Most sites can switch from ntpd to something else. See for instance systemd timescynd which really doesn't have a reason to exist. And changing to chronyd was a very quick switch.
I think it is really inertia. Time synchronization goes unloved at a lot of places.
I don't disagree! In particular, a ground-up Rust replacement for the 20% of ntpd that everyone relies on would do a lot of good and be deployable virtually everywhere ntpd is today (Raymond proposed a Go rewrite --- I strongly prefer Go to Rust, but Go has a garbage-collected runtime).
But that's besides the point. Pushing a hostile fork of a popular project, raising money for it, and then abandoning the codebase entirely for a rewrite takes a "special" kind of chutzpah.
ESR's whole life has been spent pointing at people who actually do important things and saying "I'm with them", and he's done it so much that people assume that it must be true.
(Without getting into the discussion of who is an LBIP or not)
> Where there’s no profit stream, markets are not going to directly solve this problem.
The market is a process of matching suppliers with consumers, and contrary to the previous statement, it has solved the problem, by finding a very cheap supplier: you.
LBIP are admirable selfless people, but I think this attitude is as misguided as a parent doing their kid's homework for them. The rest of us won't care to find a better solution until we start feeling the pressure, and we won't feel it while the load is being born by them. And the author is essentially asking regular individuals to throw a few tips to the LBIPs, so that the current broken model can be maintained. I'm not sure we wouldn't be better off letting it fail.
Also, this might be better described as Load-Bearing Individual Participants. If only to avoid distraction around what counts as internet-critical.
Beyond funding, what happens when they get burned out or take a vacation? Walls have many bricks. What makes these people unique?
I'd expect anyone in a senior position to be working on growing people to help share the load, open source or corporate. There's only so much you can do by your self.
It's sort of out of scope so let's not dwell on his slow descent into predictably cantankerous obscurity. You can descend into his blogs, sword and torch in hand, ready to do ideological combat with the horrors from beyond the stars that lurk there on your own time.
The key idea: that he is an ILBP (or has been in 10 years) is absurd. He is not, and he claims he's not hurting for money.
> The key idea: that he is an ILBP (or has been in 10 years) is absurd. He is not....
This attitude denies support to projects like NTPsec, for which he's the technical lead, your take on this concept only applies to current maintainers of existing projects.
Even then, he's converting GCC to git, the latter indirectly bears a great deal of "Internet Load".
I deny support for NTPsec specifically because I think it's an idea who's time has passed and now soldiers on because of inertia rather than good sense. It's sort of a meme that any project ending in "sec" is vestigal.
So no: they don't get my support. Why would they? Same with DNSsec. Useless project, please desist.
Can I just use this spot to remind everyone that when one of his commenters found an integer handling bug in the ntpsec codebase, Raymond said "I will neither confirm nor deny that I left it in there deliberately to see who would be sharp enough to spot it".
You can find it in the thread on his blog post titled (I am not making this up) "Thinking like a master programmer, redux".
Another fun fact: Cure53 audited ntpd and ntpsec concurrently, and found an instance where ntpsec rewrote a function and managed to regress out a patch for a security vulnerability, reintroducing it into their codebase. (By the way: overwhelmingly, with I think just one exception --- not counting the regression above --- the significant findings in that report applied uniformly to both ntpsec and ntpd).
Additional fun: until 2017, the ntpsec project apparently didn't even enable system/runtime mitigations like ASLR (according to the "Fix/Validation log" in the Mozilla SOS project).
Conclusion of that report: "While the NTPsec project emphasizes cleaning up its ancestors’ flaws, the difference regarding quality between the original code and the current implementation was not as great as anticipated."
Yes, this is unconstructive legalism. Is anyone out there checking whether he's written something about Iranian agents attaching strumpet mines to his car which he then successfully demagnetized with his Aikido?
Apropos of nothing at all, if someone were to gather his greatest hits into some sort of collection, it should probably be named Bearin' Load
ESR has done, and apparently continues to do, many useful things for open source. But his various personality disorders have reset that karma back to zero, or possibly put him in negative territory.
Its kinda hard to accept a reference as rational when the opening summary describes him as having "batshit insane wingnut tendencies." Those are not appeals to reason, they are appeals to emotion and tribalism.
Raymond is an avowed and weirdly obsessed racial supremacist (and, at times, an advocate of political violence). That's not an appeal to tribalism but rather a recorded fact; both conservatives and liberals alike would find his stated beliefs horrifying if laid out in front of them rather than scattered throughout his various discursive blog posts and comment threads.
There is a general and valid concern in our industry about a liberal orthodoxy that makes it at least socially unsafe to express political thoughts. People who share that concern should have an even bigger problem with Raymond, who is an attention-seeking caricature of conservative or libertarian belief. People who have couched their bigotry in far more careful and subtle language than Raymond have found themselves ostracized from conservative circles, and for good reason.
I think the better way to engage with Raymond is on his manifest deficiencies as a professional engineer and technologist, and wouldn't want to start a rebuttal to his request for donations with his politics. But I also wouldn't want to let stand the idea that the opprobrium he attracts is rooted in orthodoxy or tribalism.
Lovely. To inject some psychological precision, only his narcissism can be considered a personality disorder. Being old, white, or male... no. Not yet.
Some distance down the list is reposurgeon, which has been used to convert Gnu Emacs and many smaller/younger repositories to git, and is in the process of converting GCC: http://esr.ibiblio.org/?tag=reposurgeon
> gpsd is a service daemon that monitors a GPS attached to a serial or USB port, decodes the position/velocity/time information it sends, and republishes in a simple uniform format on an IANA-designated TCP/IP port. This enables multiple applications to read from a GPS without contention. The distribution also provides C and Python libraries to encapsulate the client side of talking with gpsd.
> Eric S. Raymond has been the technical lead of GPSD, a close peer project of NTP and one of its principal time sources, since 2004. GPSD has billions of deployments in Android smartphones world wide and is a mission-critical component in most of the world’s drones and driverless cars and robot submarines.
The top item, where he's also the technical lead:
> NTPsec
> A stripped down-security-hardened and generally improved version of the NTP reference code. Features code bulk reduced by a factor of 4, better monitoring and diagnostic tools, and Network Time Security.
Aspires to become one, but it's early in the process to see if it'll succeed.
Those are projects I've vaguely followed over the years. Reading down the list, this claims to be one, and the claim is partially falsifiable:
> giflib
> The ubiquitous service library for rendering GIFs. I handed off the project 1994 to avoid problems with the U.S. patent system, but accepted back the lead in 2012. This code had the odd effect of making me virtually omnipresent; it seems nobody has ever bothered to write a replacement, and it's now ubiquitous in web browsers, cellphones and gaming consoles. In a nicely ironic touch, it earned me an appearance in the credits of the Microsoft XBox.
GPSD was historically an important project, but I wouldn't call it "load bearing" in the same sense as core networking or service contributions.
It's also increasingly less important as the changes dropped in 4.19 are picked up by downstream software authors. Most software installations that care about gps are deployed in SBC configurations. A lot of other folks (e.g., hobbyists with external microcontrollers or arm SBCs) are parsing directly.
Folks most interested in linux attached hardware are either older school hardware hackers (who rely on this project) or folks using new LoRa radios (in which case that stuff is in the card and annoyingly locked down because it's part of some LoRa monetization schemes).
Sooooo yes. Not a bullshit project. But no, not a ILBS project.
I'll repeat the claim from the NTPsec project page:
> GPSD has billions of deployments in Android smartphones world wide and is a mission-critical component in most of the world’s drones and driverless cars and robot submarines.
And tools, to support for example the development of tools like Emacs and GCC, indirectly support "core networking or services".
You can narrowly define "core networking or service contributions" to exclude everyone by Linus Torvalds and Vint Cerf, but that's boring.
Having hand built and written software for a lot of drones, gpsd is not mission critical in most drones and I dunno where they get that claim.
You only use gpsd for embedded hardware when you have no drivers OR you're in 2017.
As for Android... again that capability is not important to the internet. It's important to Google cheaply getting a feature launched. This seems to me to be specifically ignoring the anti-corporate-centric intent of ESR's post to elevate his importance.
This essay is important without these sub-discussions. I fully agree with ESR that capitalism fails to sustain the internet and ruthlessly rides the backs of maybe four dozen skilled individuals in the world who, when they're gone, will be sorely missed and the world will suddenly become more expensive if others don't take up the call.
I just don't think ESR is in that critical group. He might be in a group of people writing widely used software. And that's great and important. But it really doesn't seem like what he himself is discussing.
> You can narrowly define "core networking or service contributions" to exclude everyone by Linus Torvalds and Vint Cerf, but that's boring.
Both of whom are well compensated for their work, no? Seems to me like we should look more critically at who is not being served by capitalism and help them since their work has value.
Raymond didn't write gpsd, did he? He took it over as the "maintainer". But as 'geofft observed awhile ago, if you actually git-log the project, you'll see that most of the substantive changes to gpsd aren't Raymond's.
Is it possible that what Raymond really has is a (waning) talent for getting his name attached to other people's work?
This is merely a list of the "critical Internet services or libraries" that he claims in one way or another, and ensuring the maintenance of such software is part of the whole thesis.
To scare quote that, and denigrate it over "substantive changes" ... well, I must thank for your solid support of his thesis.
I don't understand your objection. Either you do the work or you don't. There's no "critical Internet infrastructure" value in simply attaching your name to things.
Here you're blatantly accusing ESR of fraud, which per you "requires an active intent to acquire something of value through misrepresentation", to wit, soliciting money for "attaching his name to things".
We have no basis for a discussion of the work you claim he's not doing.
I don't know where you're going with this, you haven't given me anything to reply to here, and so I suppose I'll simply stand by what I wrote. I am certainly not accusing anyone of committing a crime.
But none of these are within the implicit scope of "Internet Load Bearing Software" though, are they? We're aware of his prior involvement with ILBS, but he hasn't been sharp in that game for a long time now.
But I think ESR's point about recognizing how load bearing individuals and small groups are being crushed in the wave of monetizable internet growth is a good point. We should folks on people currently in that position.
The signature line is a Google+ link. I was making fun of it because Google+ is over now and that link goes to a shutdown page. (Hence "when was this written?")
Personally I think the problem with a lot of these kind of Patreon setups is they are generally pitched as "I once wrote some very useful software, and sometimes I maintain it."
While that's great and all, for me at least it doesn't justify why I should send them money every month.
I think the Open Source model needs to evolve, and if these people want monthly income, they should be "showing their work" a lot more. Put out a regular (weekly or more often) YouTube or blog series. Maybe stream your work on Twitch (like I do :) ). Provide some active community involvement. Then I would be far more inclined to donate to someone, rather than having to wonder all the time if I'm paying for them to work on this stuff, or if I'm paying them so they can go on an extra vacation next year for work they completed years ago and have mostly ignored since then.
I had similar thoughts about how they could use YouTube or something equivalent to open access a little in return. Grow the cake and all that.
I then had quick Google and it seems a few years back some were seeing a conflict between ESR's Internet Civil Engineering Institute and something called the Core Infrastructure Initiative[1] which it seems is pulling in millions.
These people are not entertainers. Demand they post their work on YouTube makes as much sense as demanding architects use smoke signals to communicate their progress.
They're not entertainers, there should be no demands.
But I'd think talking a little about their interesting work might serve to better convey what they do to those who are aware of it and reach a wider audience. What ESR has tried obviously hasn't worked, would a half hour chat about their work once a year or a twitch session, or whatever else really make no sense under any circumstance?
This is morally equivalent to the pointy headed boss who demands people work from the office because work that is not done in his field of view does not exist.
Not really. The boss is being picky while getting paid, whereas OP is offering a donation while being picky. Pretty different from a moral standpoint.
I would say it’s morally equivalent to not buying a product because they haven’t updated their blog in a long time and you can’t tell if the company is still active.
A person should confirm the maintenance status of most of these programs by looking at the public interface of whatever source control system is being used.
> I think the Open Source model needs to evolve, and if these people want monthly income, they should be "showing their work" a lot more.
I think there should be a "like" button on Facebook so that users can communicate more positive vibes on the internet. Maybe even make it easy to "like" things by embedding some code on web pages so that the positive vibes spread to the web as a whole.
In both cases it seems we're ignoring important and obvious consequences of the concept between the quote marks.
There are some wonderful developers on Patreon. I tend to support those who explain their work, because it showcases the work and provides extra value. Granted, that extra value also means extra work. But I think it's more than worth it because more people will realize the value of one's work.
Meanwhile, for those keeping score at home, a quick recap of the ways in which Raymond, esteemed author of fetchmail and maintainer of gpsd, has referred to himself:
* A "load-bearing Internet person"
* "one of the senior technical cadre that makes the Internet work"
* "one of the half-dozen or so most influential people" in open source, "in fact, a lot of people would put me among the top three".
* Hisham ibn-Sindbad (the Black Wazir) in Arabian Nights
* Someone you could reasonably infer is "the most famous programmer in the world" (but Knuth doesn't count because he's not famous outside of CS, nor is Bill Gates because he's not famous as a programmer, nor is Kevin Mitnick or Aaron Swartz).
* A member of a "small cadre of old hands" like Vint Cerf and Dave Taht with "the specialized technical knowledge required for Internet disruption on a massive scale".
* A person "creating the computer code that makes your digital world work"; "every time you use a Web browser, locate yourself on Google Maps, draw money from an ATM, or play on a game console, you rely on computer code I wrote and gave away."