I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems. Unless their specific personal/professional projects are at risk they have no responsibility to hurry and fix a thing.
You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems. They should have been buried in useful pull requests.
For that matter, they should have really picked them all up in private jets and flown them to neutral working space with those engineers for a one or two week hackathon/code sprint to clean up the outstanding issues and set the project on a sustainable path. To get those maintainers there they should offer a six figure consulting fee and negotiate with their current employers to secure their temporary help.
I can't believe these folks just get abandoned like this while CEOs/CTOs from rich companies wring their hands wailing about the problems and not offering solutions.
> I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems. Unless their specific personal/professional projects are at risk they have no responsibility to hurry and fix a thing.
Sorry, but what's the hard part to understand? Open source maintainers end up in this position because they are nice, helpful people who like using computers to solve problems for others. People who spend years on a project and then see a bigger problem arise don't suddenly turn that off. With the bigger problem, they'll want to work harder, not just hoist a middle finger and go binge Netflix without a care in the world.
But I totally agree with you on the CTOs, etc. I don't expect random programmers who like working on logging to also be good at solving complicated sociotechological problems around paying for global infrastructure. But it boggles my mind that none of these richly rewarded, supposedly brilliant experts at organizing engineers has gotten out in front of this. If not out of community spirit or social responsibility, then out of pure self interest.
> none of these richly rewarded, supposedly brilliant experts at organizing engineers has gotten out in front of this
Indeed. Each of them has had to spend the last few days madly trying to fix this problem to avoid exposing exposing their infrastructure. Each has been, in some way, replicating the wheel to do so. I'm curious how many will actually submit their findings to the original OSS so others can learn from their experience?
There's always resources to put a fire but rarely enough to install a sprinkler system.
For sure. And this was a richly predictable fire. Not this specific problem, of course. But there have been enough fires like this in the past that it makes no sense for companies to act like it won't happen again and again.
Oh? One that's comfortable and easily available to the kind of person who has spent years on an open-source project? One that won't increase the crap they're getting from project users and random strangers? Do tell.
Yes. It's quite simple. There's absolutely no need from maintainers to "rush" to do anything, just do it in a comfortable timeline. And no, this is not equivalent to giving people the finger. Only dishonest people would say so.
Ah, the magic "just". I don't think you've really grasped the kind of people who spend years maintaining open-source projects. This is like asking sysadmins to ignore alarms and downtime. The kinds of people who can be comfortable ignoring downtime rarely become sysadmins and don't last if they do.
Feel free to prove me wrong by pointing to the major open-source project you've led for years and how good you have been at ignoring problems with it.
No, the magic word is "healthy". I said "healthy, acceptable, middle ground". This means learning to say "no" or at least "later, when I have the time".
I have maintained open-source projects myself for about 15 years, and it is completely different from sysadmin alarms. I can do it on my time, and even if with vulnerabilities I do it on the time I have previously allotted for doing it, since my family and my job comes first. And no, I won't dox myself. There, I "just" said "no" to you. That's a healthy way of establishing boundaries.
By the way, feel free to prove that I'm wrong in saying that by pointing to any law that says I have to do otherwise.
And no, just because people have unhealthy behaviours doesn't make it a rule or a law.
Oh? Which projects? I'm skeptical that they are of the scale and importance we're talking about here. And as somebody who has done open-source stuff and also done sysadmin work, I say that there are commonalities, so you can't just handwave it away.
I agree that unhealthy behaviors don't make it a rule. But deep-set behaviors don't change overnight. And it's not clear to me that people who are perfectly healthy would ever put themselves in the situation of maintaining important infrastructure for free. As mentioned elsewhere, I've closed down an open-source project of mine because it grew too big and became more of a drain than a pleasure. But if everybody did that, we'd have a lot fewer open-source projects. So I think your breezy "just" only works for the kind of people who would never have ended up with this problem in the first place.
The answer is not to close open source projects, but establish healthy boundaries between your audience and yourself. And no, you don't have to be healthy to establish those boundaries, but you do need these boundaries if you want to be healthy.
Rather than giving up and closing a project, my answer to abuse and unreasonable demands is to be liberal with ignoring and blocking. It's not my fault that someone chose to berate me, but it's my choice to keep giving them a way to do it. But most of the time, a simple "I will do it when I have time" is more than enough. I believe in carefully curating the online places I manage in order to maintain a healthy atmosphere. I choose health and well-being above "popularity at all costs".
About the projects, I'm not gonna dox myself, and I don't think the cross-examination is warranted or even in the spirit of this site. I believe my post stands by itself. You can doubt all you want, but that only solidifies my belief I'm doing the right thing.
I'm not saying you aren't doing the right thing. Good for you. Stay healthy.
I am saying you are delusional to talk as if what's easy for you is what's easy for others. And that you're failing to consider that people who are in XKCD's "random person in Nebraska" bucket [1] are much less likely to have good habits and healthy boundaries in the first place, because people like that end up bailing much earlier.
Bullshit. I utterly and completely disagree that the boundaries I mention are hard to implement. I will also say that boundaries and limits are not only healthy, they are absolutely necessary for long-term, large-scale, open source projects, and this is why those projects survive. People on FOSS rarely get to be long-term maintainers without establishing clear boundaries, so most active long-term maintainers obviously do have the boundaries I mention. This is a lot of people. I'm clearly not taking hobby projects, I'm talking about packages with thousands to millions of weekly downloads, for example.
The myth of the hero maintainer must die.
Binging Netflix is completely acceptable if you're in your free time. EVEN if the package has a RCE vulnerability. And most maintainers of popular projects will agree with that.
Easy for you does not mean easy for others. If you think boundaries are easy for everybody, go read Reddit's AITA, where you will find a flood of people who struggle with it, generally because important figures in their lives train them not to have boundaries: https://www.reddit.com/r/AmItheAsshole/
I'm glad it's easy for you. But therapists can spend years working with people on establishing and maintaining boundaries in important relationships. You can't just handwave the difficulty away. Especially when so many tech jobs discourage having a good sense of boundaries in the first place. E.g., all the startups where people are expected to be bought in to a vision of changing the world, working crazy hours. All the bosses that talk about their teams being like families. All the places that talk up commitment and loyalty and going the extra mile. Which again you can see a flood of if you care to look: https://www.reddit.com/r/antiwork/
> And most maintainers of popular projects will agree with that.
You're reaching and you're constantly misrepresenting and replying to exaggerated interpretations of my views. Unpaid OSS maintainer work is nothing like startup work or a similar to the issues in antiwork. The boundaries are much easier and much more obviously necessary than in any other relationship because direct contact is not mandatory, and productive work is impossible without them. Also, even if impossible for one person, other co-maintainers can even help with those boundaries, and there are tools available for that. Of course there will be people unable to work healthily with OSS, but those people will burn out very quickly and they won't be able to be long-term maintainers, period. Also, keep in mind that a large part of OSS maintainers are paid by companies to work with OSS and they do it on company time.
I'd likewise ask for a citation of some maintainer agreeing that watching Netflix on their free time is akin to "giving the middle finger" to users. That was one of the most toxic phrases related to OSS maintainer work I've ever seen on this website, and those views should not be spread as if they were acceptable. I stand by my opinion that the "work harder" vs "watch Netflix" is a completely fabricated dichotomy, and there is a very viable middle ground available (and necessary) for everyone.
> You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would (...) mitigate the problems.
FAANG engineer here, and one who had to work extra hours to redeploy services with the log4j vulnerability fix. I'm not sure you understand the scope and constraints of this sort of problem. Log4j's maintainers have a far more difficult and challenging job than FANGs or any other consumer of a FLOSS package, who only need to consider their own personal internal constraints, and if push comes to shove can even force backwards-incompatible changes. The priority of any company, FANG or not, is to plug their own security holes ASAP. Until that's addressed the thought of diverting resources to fix someone else's security issues doesn't even register on the radar. I mean, are you willing to spend your weekend working around the clock to fix my problems? Why do you expect others like me to do that, then? Instead I'm spending a relaxing weekend with my family with the confort of knowing my service is safe. Why wouldn't I?
I'm not saying you, as an engineer for those companies, should be the one to donate your time and energy toward the problem. We all have competing priorities, as do the maintainers of those FLOSS packages.
I'm saying that your company's CTO, especially one with a very large companies, could likely identify two or three engineers who they pull into a meeting and say "reach out to these guys and get them whatever they need. Here's my cell, call me the moment you need the plane or additional resources."
Seriously, if a CTO has a budget of a few hundred million dollars and thousands of dedicated employees, how hard is it to throw a few crumbs to the open source community to change this situation from being one of a burden on a volunteer effort to, instead, one where they feel like they're in the middle of an international event where their knowledge and services are vital to keeping the internet alive?
Again, I'm exaggerating, but you see where I'm going with this. It's a missed opportunity for some seriously great PR out of a seriously bad situation.
Some time ago, I saw this suggestion from a disaster relief specialist directed towards those who want to help with disaster relief: the best thing you can do after a disaster is stay away. Taking yourself to the disaster zone very often at best consumes scarce resources from those present to manage the disaster trying to bring you up to speed and at worst creates new problems that need solving.
It's not hard to extend this to the kind of software security flaw here. If I'm a developer on a package with a critical security vulnerability that needs to be fixed now, sending me extra developers who know absolutely nothing about the code I'm working on isn't going to be helpful--it's just going to waste my time trying to bring them up to speed (or more often, telling them to just go away). If I actually need help, I'll ask the people who I know can help me; trying to sift through unsolicited help to figure out who actually has the skills to do so would take too much time.
So think hard about what help Google et al could actually be providing to help log4j here. If you have to resort to clear exaggeration to find examples... maybe that's a sign that there actually isn't all that much that they could be doing that would actually be helpful.
> So think hard about what help Google et al could actually be providing to help log4j here.
I think you make a perfectly valid point and one that shouldn't be overlooked. How about this:
"Here's a $100K and an isolated penthouse suite down the road rented for the month where you can focus on fixing the problem and not be interrupted by screaming children. Here's a phone number if you need to delegate any specific tasks to additional teams."
Incentive to help. No added pressure. Just one practical example.
I don’t quite understand why you keep coming back to luxury apartments and private jets.
If children and family were viewed as too much of a distraction, I’m pretty sure the CTO (in this scenario) would simply choose a developer who lacks those distractions.
Let’s say the engineers chosen do have family. Why wouldn’t the company just comp a room in a local hotel?
I'm so confused. I thought we were talking about a single volunteer open source developer responsible for a vital tool, and it was too onerous to give them additional staff.
If you want to help someone, give them cash. A blank check. Not "here's what I think would be helpful and now you should arrange to use it". Not a week at a penthouse, not a butler, not a private jet. Enough cash to pay for those things if they want them.
It isn't quite simple; when negotiating it is better to give cash. When donating it is better to give goods. Particularly if there is more than one person involved on the receiving side.
You didn’t explain why it’s a perfect valid point
It doesn’t seem reasonable for Johnson & Johnston at a valuation of 1/2 trillion to free load
You are kind of talking about greater good, perhaps those charitable donations should go to medical research or homeless shelters rather than reducing the burden on for profit companies
The valid point is that too many cooks can spoil the soup. Mythical man month, if you will. Adding people who don't have the institutional knowledge to a software project even if they are rock stars at their own companies could do more harm inadvertently when trying to fix something time critical. So the additional proposal made here acknowledges that, and instead tries to remove as many non-work distractions and discomforts as possible for the people who CAN reliably fix this fast.
For sure, but what could be done is eliminating any non-superflous task so they can focus on resolving that specific problem.
Have a team handle all github issues and media inquiries. Another team focus on initially evaluating all incoming pull requests to check for egregious errors or applicability.
Only after making it through the gauntlet would the original maintainers need to read and/or respond to them.
Especially when such overwhelming public attention and pressure overtakes a relatively small team like this one.
There is still a risk that the kind of time required for the maintainers to have to get those teams up to speed on the project and how it works and what needs to be done could be just as much of a distraction. Adding more triage teams might be good in the future, but for now, adding more outsiders without proper context might just add stress.
As with openssl, what needs to be done is that these volunteers need to be given cash so this is more than just a volunteer project. If a particular corporate entity doesn’t want to sponsor some of the maintainers to work on it full time, then the project needs full-time sponsorship by the Linux Foundation, ASF or under the OpenJDK.
This explains how such a “solution” Benefits log4j and log4j users. The part I question from the start is why “google should”
Vs “google should” pump the equivalent money into medical research vs “google should” make what is does better
When there's a disaster, there will be emergency responders from other jurisdictions lined up on the state lines as the commanders call to ask if assistance is needed.
This situation is fairly urgent, but I think you might not realize just how many people a CTO at one of these companies manages. There are going to be "OSS fires" more or less constantly so "some major OSS project has a bad vuln" is not the sort of thing that gets a CTO at a company like Google or Facebook out of bed. I've only seen this happen a very few times and they were for problems that were way more serious and complex.
But that is not to say that nothing is being done. At Google, at least, there are organized efforts staffed with plenty of people that are trying to solve the much much much bigger problem of "secure all of our open source dependencies and all future dependencies" rather than the individual problem of "secure this one dependency."
And PR? Google has been running projects like OSSFuzz for years and I haven't really seen it materialize as a large amount of positive PR, even in the tech community.
> And PR? Google has been running projects like OSSFuzz for years and I haven't really seen it materialize as a large amount of positive PR, even in the tech community.
Google's Project Zero is both very helpful and gets them A LOT of PR, both tech and mainstream.
GPZ isn't oncall for urgent bugfixes and, while a truly excellent project filled with great people, isn't the core team responsible for safe imported code.
> There are going to be "OSS fires" more or less constantly so "some major OSS project has a bad vuln" is not the sort of thing that gets a CTO at a company like Google or Facebook out of bed.
If all your IT projects have an RCE vulnerability that’s relatively easy to exploit, that should keep you up at night.
The RCE existed prior to this disclosure. If I can't sleep today, why should I have been able to sleep a week ago? The dirty secret is that an absolutely enormous amount of code is vulnerable and that the solution to software security is not "fix RCEs as they are discovered as fast as possible." If having RCEs keeps you up at night, then I don't believe that there is a single engineer at almost any company in the world that interfaces with the internet that should be able to sleep.
The actual solutions here are at a more abstract layer than individual vulns.
> The RCE existed prior to this disclosure. If I can't sleep today, why should I have been able to sleep a week ago?
A week ago, this vulnerability might have been known at most to a few three-letter agencies. Today, every two-bit script kiddie will be trying to exploit it. It's not hard to see how the situation has changed.
No, the fact of having these vulnerabilities is not a problem (I mean, obviously, but at the level you describe). The problem is having them be known to the world. Especially with a level of publicity like this.
As with earlier comments this seems to oversimplify the problem of throwing people at a problem. Adding people to a project puts more pressure on the current maintainers, to authenticate, validate, train and support newcomers.
I am guessing FANG engineers (and most engineers in general) would unanimously suggest "delete this ridiculous , ill-conceived JNDI integration ASAP. If people want JNDI integration, use a custom opt-in log4j appender. Don't let this shit be enabled by default". Yet that may not sit well with the log4j folks.
> I mean, are you willing to spend your weekend working around the clock to fix my problems?
Surely the difference is you are getting paid, and if your boss says, help these guys out, you can do it? As opposed to some guys with jobs who have a project on the side. The big guys could even do something like offer to pay the maintainers and maybe they can take leave or something.
I agree with both sentiments. The big guys are under no obligation to fix an issue in some library they happen to use. But the log4j guys are under even less obligation when they do it in their spare time.
> Surely the difference is you are getting paid, and if your boss says, help these guys out, you can do it?
No, I'm not getting paid. What leads you to believe in that? My targets are defined yearly and are very well defined, and patching random FLOSS projects is not one of them. And what leads you to believe that others, such as my boss, don't have their own milestones to meet, and instead take random FLOSS requests from random people on the internet?
A FANG is not a magical entity where any engineer can drop everything they're doing at the drop of a hat to work on external projects, let alone one whose only possible outcome is at best total indifference and at worse we get the company to own a problem affecting everyone for no reason whatsoever.
I'm not under the impression that GP means to suggest you personally have any obligation to donate time to OSS by virtue of being an employee at a large company.
Something I believe we agree on is that it is in the interest of large tech companies to spend time fixing critical security bugs in their own programs, regardless of who originally wrote the malfunctioning code and for whom said code was written.
One way to fix those bugs would be to create a patch for the external OSS library in instances where such a library is the origin of the vulnerability. This is especially practical when that library is used heavily as a basic piece of the company's common software development framework.
GP appears to be arguing that these patches should be upstreamed instead of simply being maintained internally until the bug is patched by someone else in the OSS community.
I think that what throwaway is saying, perhaps without trying to do so, is that you can't expect people in a FANG to care about the best interest of their employer, not if there are metrics set up that don't reflect the interest in question. You can't pay six figures salaries and expect to find people without razor sharp focus on personal gain.
I really don't understand why you are defining this as a random, external project. Your software is dependent on this project! It's right in the term "dependency"!
I’m not suggesting you donate time. I’m suggesting that if a large company depends on open source projects, it may be in their best interests to either use some engineering resources to help out those projects, i.e their engineers would do it as part of their job, or to spend their money on the maintainers of those projects.
If the big guys don’t want to do that, fair enough. But the open source maintainers are not under any obligation to work to anyones time lines either.
> You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would (...) mitigate the problems.
Your "(...)" elides the word "help," which completely changes the meaning of the quote, and your reply is constructed uncharitably as if that word wasn't in the original statement.
Somehow, I find what you are saying here to be totally unplausible.
> Log4j's maintainers have a far more difficult and challenging job than FANGs
You are saying that the companies that built advanced ML-based Chess/Go engines like Alpha Zero/Go can't solve a simple logging bug involving string substitution?
If your company ends up using the product in all your teams/project and products wouldn't it be in the company's interest to keep the product safe?
How do we know you're not a CTO/C--/manager in your 'faang' just taking this opportunity to bitch about how bad and unreliable open source is? You do have a track record when it comes to this.
> I mean, are you willing to spend your weekend working around the clock to fix my problems?
Speaking as an individual, of course you want to sit by the pool this weekend.
But as a professional representative of your org. surely you'll recognized the unsustainability of the situation and that it's far from ideal even in the pure self-interest of the company in question.
>> I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems.
Agreed, while reading it I also disagreed at this point:
>> the maintainers of log4j would have loved to remove this bad feature long ago, but could not because of the backwards compatibility promises they are held to.
Nobody is holding them to anything. If they want to remove an old feature, go right ahead. If those using it think it's that important they can fork the project and maintain it themselves. Oh right, that would take effort or money.
I don't get this argument. Part of sharing your work is making sure what you put out is actually helpful to people. If they remove features people really like, then the library won't be as helpful - so it's perfectly fine for the OG devs to maintain this feature. The same thing with "scrambling" to fix - that could be because a sword is hanging over your head, or because you care about the people who use your work. Thinking this way, I can perfectly see them working hard to fixing this bug.
It is still a choice you make, do you want to be nice and do everything anyone asks of you ? Or stick to your principles and build only what you think is right?
Developers are social creatures like anyone else and like validation and recognition from peers, that is understandable but not an excuse.
It is no different from being able to say no when your boss ask you to build something you don't believe in or is against your morals, it is tough but our choice nonetheless.
Their choice then to support this feature, and now to patch it. They could have no either time, no point in complaining about it after making the choice.
>> Part of sharing your work is making sure what you put out is actually helpful to people.
No, people get to decide for themselves what is helpful to them. Assuming developers want to make the best tool they can, they still have to do so within the resource constraints they have. Dropping a feature or ignoring requests is part of that.
I understand it perfectly. Log4j is used in many Enterprise systems. Java is a fairly conservative language. Combine both together and you get much hesitancy to break backwards compatibility ingrained in the Java world.
Are data breaches actually treated as all that seriously? For all the talk about cyber security, there seems to generally be little investment. It appears to be viewed as more of a reputational concern than an operational one.
A past organization of mine had a data breach (the kind that ended up making the news everywhere). A few people left (probably making it worse with all the turnover there), but I would be surprised if anything really changed in that organization.
If the company is in healthcare or finance, yes. Otherwise the typical answer is no. Most companies just load up on cyber insurance and call it a day. That said, reputational concern, is a big thing for companies. Take Dropbox for example. Early on they suffered several security breaches, and had a bad reputation around security. They've since built out a fairly large security program, in part because bad security can block deals, especially in the enterprise space.
I'll note that there's been more investment in security the last 4-5 years. Most B2B companies do a SOC2, and early on, so there tends to be a baseline of competence.
A data breach isn't the primary concern here. This exploit allows full pwnage of a system and could take down entire networks for as long as it takes to rebuild them.
This is not really about data breaches. The first widely spread automated attacks seem to drop cryptominers, however, we should expect that (if it's not already happened) within a week or so this will get used as the entry point for ransomware attacks, since it gives attackers a solid way of getting of code execution into company servers for anyone who has not solved this issue.
> I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems.
If the RCE had been responsibly disclosed instead of via tweets and PR comments, maybe there wouldn't have had to be so much scrambling. And indeed maybe ASF could have found corporate OSPOs to help with remediation.
There are lots pixels being spilled on how the users of open source software should be paying for it (?), but I haven't seen much criticism of the vulnerability not being responsibly disclosed.
to the best of my knowledge it was discovered via a minecraft exploit and I don't think minecraft players are generally the "responsible disclosure" kinda people.
Simply said, this was a feature that was intended to make it easier for one company to use their structure with ldap lookups for where/how to log. The author of the change did what many people encourage others to do when working with open source "here's some code that I wrote, I'm contributing it back upstream."
If this was part of something "bigger", it sat quiet for the better part of a decade.
Yes... but... realize that log4j2 was in beta releases at the time, being maintained by one developer as part of a "I want to redesign how it works".
As an open source developer working on a project that hadn't even been formally released, I'd be quite pleased to have someone else contributing the features that they found useful back upstream in an effort to make it a better project.
Yes, this is what jndi is supposed to do. Was it done as best as it could be? Probably not. But it isn't something that's only in log4j2
But I'm not going to fault a solo developer of some beta software in the world of 2013 for not rejecting a patch because every angle wasn't thought out.
Considering that Minecraft players have built functional computers in-game using redstone circuits, it's not surprising to me at all. Minecraft tends to attract folks who have an eye for detail, and a passion for figuring out how things work (a.k.a. hacking).
Or they didn't know what they had. People spend a surprising large amount of time trying to cheat at video games. They could have stumbled onto it in this context without realizing the much larger picture.
yeah it's nuts. I wonder what the upper limit on a responsible disclosure bug bounty would have been(of course who would pay that is the question because it's an opensource project with a few maintainers) vs. the nation or underground price. This has been described as a once in a decade RCE.
There’s no hiding something this easily exploitable. This isn’t rowhammer or spectre where you need a degree to understand it. Copy and paste this in and that’s it. It would have never survived “responsible disclosure”
From my reading of it, it was (12 days ago) "here's a pull request to remove a feature." And for a bit over a week it went through the normal process and got merged into the code 7 days ago.
Then three days ago someone asked "Is it a security vulnerability" and then everything happened.
This wasn't any great "there's a security bug that needs to be patched yesterday" and then a fix and release but rather an after the fact, when looking at it, was it realized the scope of the issue.
I'm not sure about Amazon but Google project's zero and openfuzz teams seem to be doing a lot of good work when it comes to open-source security -- more would be nice always
Personally I'd like something like a security health card/metric on opensource libaries that we could tie into CI systems/pull requests or something
in the past there were so few libarries it wasn't as daunting
I'd be able reason about stuff like libpng, libttf ..etc and think about them or even support them but now some projects are massive hodgepodges of thousands upon thousnads of packages
I admit to a certain level of exaggeration but, at the same time, we are talking literal peanuts to a large company. They could spend a million dollars and it'd be a rounding error on their balance sheet.
In all seriousness, taking actions like I identified above would cost the companies virtually nothing but result in huge long-term benefits by signaling to the rest of the open source world that "we love your work and will be right beside you helping if the chips are down."
This is, of course, not a suitable compensation model for popular open source projects. Thats a separate conversation.
The compensation model is the problem. It doesn't mesh well with the way corporations function. If you don't charge for your product or services, corporations have no standard mechanism for addressing that some other way.
Open source contracts essentially state that companies can use the product with no relevant obligations, so they do. The "huge long term benefit" you claim can't be reliably translated onto a balance sheet, so it isn't.
And when companies do get involved, it's often explicitly for their direct commercial benefit, like Amazon's ElasticSearch distribution.
There's also a race to the bottom aspect to it. If an open source package e.g. charges for commercial use, something more free is likely to replace it.
> The compensation model is the problem. It doesn't mesh well with the way corporations function. If you don't charge for your product or services, corporations have no standard mechanism for addressing that some other way.
FSF: Pardon me, here's a little process that encourages sharing so that everyone can...
Corporations: gnawing on a giant proprietary turkey leg Doesn't Mesh Well nom nom nom No Standard Mechanism nom nom nom
**
Corporations: yelling through a mouthful of turkey What in sam hell is Linux why does IBM have a billboard about Linux why don't we have Linux?
Underling: Sir that's open source, our standard licensing mechanism wouldn't...
Corporations: belching through a mouthful of turkey change the goddamned mechanism I want linux and pass me that mayo
> Pardon me, here's a little process that encourages sharing
But they're following the process. What the comment I replied to was saying was that they should go beyond it. I'm pointing out that the lack of standard processes for doing that is what prevents it from happening.
That's just the reality of the situation. Yelling at clouds isn't going to change that.
> I admit to a certain level of exaggeration but, at the same time, we are talking literal peanuts to a large company.
I'm not sure you understand what you are asking, and I'm kind of dumbfounded by the sense of entitlement of your request. You are expecting others like me to be forced to work weekends on a problem that doesn't concern me (because my service is already patched) for absolutely nothing in return, and instead risking owning a problem and the blame of not coming up with a one-size-fits-all magic bullet.
All downsides and absolutely no upside at all, for my employer and let alone for myself.
Let me ask you this: how much of your personal time did you invested in coming up with a fix for this vulnerability? And yet you feel entitled to demanding this from others?
You do realize that this issue wasn't discovered this morning, and indeed part of the point is that if it wasn't people in their spare time it could've been patched days ago, and you could've fixed all your services during the week? So you prefer working weekends to fix a delayed bug instead of devs being paid to help fix it for everybody during the working week?
> Usually (as per ASF rules) the team should wait 72 hours after creating a release candidate before publishing the release to give the community enough time to review and cast their votes. We are building consensus to shorten that window for this particular release, given its urgency.
if this was a difficult task to fix, maybe support to ensure the devs can properly focus on the task would be a valid thing. however, this sounded like something not very difficult to fix. it will take longer for all of the end users to deploy fixes in their envs that it took for the patch to become available.
This commentary is exactly the problematic commentary the authors were referring to in the quote that David included near the top of his article. You may not be being so directly brash about their state, but you still imply with added dramatic extension that the project is in some dire state. That's likely a significant overreaction. There's a feature in the project that leads to a security concern, making it fixable likely took a small number of hours, communicating and dealing with the drama is what has and would continue to take the time, as well as pushing back on a plethora of demands for "full review" or "you must travel outside of your home country so I feel comfortable again".
I poked some fun at the issue, because this is in many ways an amusing issue - a feature that would rarely be added in more recent times, but at the time it was introduced we as an industry considered the whole space differently. I'm sure the maintainers are having a tough time, and theres no need to point fingers at them, hell there's no need to point fingers at the implementor of the feature. It's a mistake in retrospect but that doesn't make anyone unworthy of respect.
The actions you identified make a lot of assumptions, some are exclusionary, some are potentially offensive:
- not everyone lives in the same country
- not everyone can travel on a whim
- the developers don't need to be close to you or their customers unless they want to be.
- the project is not in some dire state in need of "saving"
As an Apache project there is a foundation that can help organize funding, and so if there's a funding problem with the project the discussion should start there. Yes, open source at large is underfunded, but this isn't a standalone project on a personal git host. Apache has some (probably not enough) funding, but most importantly it has the industry contacts and relationships to do better here.
There's a problem with "doing better" though, which I rarely see come up in these conversations. There are lots of libraries, such as log4j, that don't necessarily need full time staff or full time funding. They need spurts of funding at key times, to handle the trickle of regular but infrequent patches, to roll releases periodically, and occasionally such as this to dedicate some significant time to handling an exceptionally rare event. What this requires, in my opinion, more than arbitrary dollars, is a slush fund of professional time sponsorship, for employers of key contributors to be ready to make space available during work hours for this work, without adding any more pressure to the situation. Depending on the situation this may or may not require additional funding, but for a healthy ecosystem finding ways to arrange for this, and helping employers be comfortable with it is the step necessary to address the wide scale problem of small to mid size projects burning people's personal time in unplanned ways.
If I misspoke or offended anyone it was certainly not intentional. The bulk of my comment was based on the linked tweet:
> "Log4j maintainers have been working sleeplessly on mitigation measures; fixes, docs, CVE, replies to inquiries, etc. Yet nothing is stopping people to bash us, for work we aren't paid for..."
I guess that sounded to me like a very small group of isolated volunteers struggling to handle a lot of press and attention along with demands from numerous loud users. I thought perhaps they could have benefited from having some additional support.
I don't think that throwing a bunch of new people in a project will necessarily make dealing with the problem (code, doc, communication issues) less stressful for the maintainers involved.
Quite the opposite.
The only way you can make this be less stressful for the maintainers is for them to delegate it all to some corporation to handle and go shopping, but this effectively means they are no longer the maintainers of that project.
Surely there is the possibility that these million dollars are just donated to the maintainers as compensation for a prompt resolution, but I don't think that is so easy to pull off. CTO may have some staffing but paying large sums to external contractors on a short notice is not something easy to pull off in any large organization.
No. I think this whole line of reasoning doesn't resonate much with reality.
For argument’s sake, at least, I don’t consider anything suggested here as definitively “over-the-top”. It may seem (or be) unrealistic in practice (for reasons I don’t know), but the suggestion is far from unconscionable— it may, in fact, be the lowest cost solution to what could cost mega-corps billions in current (and potential future) fines/liabilities. To the extent it sounds like an exaggeration, I think that embodies the point of the comment— there are some (almost unreconcilable) concerns that impact the interplay of corporations and open source development.
As you said, the solution isn't the hard part. The reason that large companies aren't deploying their own solutions for this issue isn't that their engineers engineers that are incapable of developing their own solutions, but because then they would have to carry that patch forever, and if a problem was found with their particular solution they would be on the hook for it.
And yes, I do think this, "but everyone else is doing the same thing so it isn't really our fault" attitude is a problem.
> You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems.
Google doesn't even use log4j. What are you talking about? The spirit of open source does not dictate that the richest companies automatically shoulder the burden of maintenance of projects they do not even use. Google already has initiatives like Summer of Code that help open source projects it does not use, and I think it's perfectly fine to draw the line there.
> divert the best engineers they had
So the lessons from the mythical man-month are forgotten here. At this point I don't think adding more manpower helps.
What? It was already fixed. You just need to update. There's no need for the world's top fintech programmers to hack it out on a mountaintop somewhere.
Also, the reason the maintainers are rushing to fix it is: they're worried about losing "market share". Having been in open-source circles for a long time, maintainers care GREATLY about how many users they have. They just like watching their download stats go up every year. Even if it beings them no financial rewards. It's a sort of addiction.
> maintainers care GREATLY about how many users they have.
They do. Until they don't.
That inevitable day when they get yelled at in a github issue thread by a user who didn't bother reading the documentation, while staring at their kid in the living room playing video games and start wondering to themselves "why am I doing this hobby in my spare time again?"
Mild dopamine hits to affirmation-addicted programmers is not the sturdiest foundation upon which to build enterprise-grade software libraries.
An influx of pull requests is also equally difficult for open source projects.
Anything sufficiently at scale needs a set of maintainers that the commercial tech companies would then collaborate with to get the PRs going.
Otherwise if everyone's just panicking and rushing to submit PRs, they'll inundate the maintainer. There's also no guarantee that even the best engineers at these companies are intimately familiar with the project, and might introduce regressions or other vulnerabilities in the process.
Anyway I do agree companies should be working with OSS devs, but it shouldn't be rushed or knee jerk. It should be collaborative and measured.
> You'd think...these multi-billion dollar companies...would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems.
For the general case, the problem is that a reporter might report the vulnerability to the open source project, then the project needs to keep it a secret while they make a fix. There isn't a great way to leverage these stakeholders. It's obviously different for something like Android that is open source, but clearly Google.
This is a problem in open source: everybody wants the fruits of labor without paying for it. The log4j vulnerability is what happens when you don't pay for it.
but being open source, you can pull out the crazy glue and stick them back together. if you feel generous, you can submit your crazy glue solution to the devs. if they like it, they can then make it part of the package.
You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems. They should have been buried in useful pull requests.
For that matter, they should have really picked them all up in private jets and flown them to neutral working space with those engineers for a one or two week hackathon/code sprint to clean up the outstanding issues and set the project on a sustainable path. To get those maintainers there they should offer a six figure consulting fee and negotiate with their current employers to secure their temporary help.
I can't believe these folks just get abandoned like this while CEOs/CTOs from rich companies wring their hands wailing about the problems and not offering solutions.