There are a lot of people gloating about this on Twitter and elsewhere, but to me it just seems sad; the ultimate morass of monoculture and an endgame with no competition. An ode to "the OS layer no longer matters, don't bother putting any effort into it someone else will". Bystander paradox.
I know Linux fans don't see it that way.. young me thought Linux was indomitable and it was foolish for any company and group of people to not invest all their time and efforts into it. Older me knows better, that's not how corporations and groups of people work best. It leads to stagnation and apathy. It kills the fertile ground for ideas needed to make non-linear progression.
I don't know the solution to all this, but systems software is really going the way electronics engineering and circuit design have gone. I am sure executive types without engineering background would love to "outsource" it all to the perceived interchangeable cogs in "low cost regions".. but the rate of technological progress is about to drop of a cliff if my generation doesn't pick up the torch real soon now.
That said, it is somewhat premature as another commentator points out IBM's CNK. CNK is inventor's thinking.. the gall to ask "do we even need a general purpose OS?" regardless of what the market or eventual answer may be.
>> There are a lot of people gloating about this on Twitter and elsewhere, but to me it just seems sad; the ultimate morass of monoculture and an endgame with no competition. An ode to "the OS layer no longer matters, don't bother putting any effort into it someone else will". Bystander paradox.
>>I know Linux fans don't see it that way..
I am not a Linux fan per se (as I type this from my Win10 desktop), but I legitimately don't understand what your comment is about. I've read it a few times and I'm still not exactly sure what you are complaining about.
How is Linux a monoculture, or this the endgame... or this stagnation?
For big metal rigs like supercomputers, there is almost no choice but Linux. Combined with Linux' dominance of the server market in general, there's a risk emerging to get stuck in a monoculture on the OS layer.
Most companies, including very big ones with a long history of OS development are abandoning their efforts in favor of just using Linux. The perception is often that there is no innovation that can be done on the OS layer that will earn back its investment.
This leads to a lack of experimentation, tunnel vision, and in the long run stagnation.
Companies experiment on Linux everyday. Unless they want to use a different programming language, they can implement all sorts of innovation in the Linux Kernel without incurring in massive overhead costs of rolling their own.
I don't think the situation is nearly as bad as portrait here.
They can always use *BSD as a second choice too.
Additionally, where's the stagnation we are so worried about here? Any real world examples?
> Additionally, where's the stagnation we are so worried about here? Any real world examples?
BeOS? Singularity? Plan 9? Hell, just read the last 20 years of OS research and realize that large swathes of it will never get used because anything that gets implemented has to be compatible with the way the Linux kernel does things. (By way of analogy, consider the x86 architecture, also a '70s era technology, and the massive contortions Intel had to go through to maintain compatibility over the years while adding new features and how all the newer alternative architectures died in the marketplace because x86 was too entrenched.)
Fuschia and, ironically, Windows are the last hopes for advancing OS architecture and that's really only possible because the semi-monopolists that created them have enough cash and engineering talent to push them until they take off.
We are talking about HPC computing and how Linux being the only viable choice leads to stagnation.
You are proposing those OSes advance the status quo (and I agree) but how is it that HPC computing is stagnated because Linux is the only viable option? If anything, HPC would be an awesome testbed for those experimental OSes and they still choose Linux.
Where's HPC stagnated because awesome ideas from other OSes can't be used?
Finally, are these other ideas so extremely awesome that they require a new OS from scratch (with all the uphill battle that it's to create it)? Can they be implemented on top of the Linux kernel? In which case, is Linux's success really such a bad thing?
Hey, Haiku's still around, and while we don't have time to experiment with such crazy ideas ourselves (keeping an operating system project afloat is no joke), we'll more than likely accept patches for such things ... provided they actually work and aren't just needless bloat, and especially if they fix or improve some area we've neglected :-p
I understand that there is experimentation and innovation inside and on top of the Linux kernel, but that still leads to some kind of local optimum, the best approach that fits within the design and implementation of the Linux kernel.
I'm not saying this is necessarily a problem, but it risks becoming a problem.
It's clear we're seeing reduced investment in Linux competitors in exchange for Linux flavours from large players the past decade or so.
The competition has moved inside Linux. The large companies all maintain their own patch sets. There are still ways they believe that they can make OSes better, they just don't believe they can compete with Linux in all the other ways, so they just write their own modifications for it instead.
Which is absolutely wonderful, because Linux is licensed under GPL, so the best improvements end up in the main tree and everyone benefits.
That's a great example of https://news.ycombinator.com/item?id=15772233 (currently also on the front page), where something appears superior on the surface, but there are good reasons why it didn't "win".
"grsecurity/PaX" didn't show up in the main tree but is alive in a different branch or as patch. Various distributions and users are free to enable and use it upon demand.
It kind of prove Linux kernel is not mono-culture and natural selection and evolution are happening.
I believe the AOSP's kernel patches are also not mainlined - also proof Linux kernel are evolving very healthily.
That's all baseless speculation and hand-wringing over straw-boogeyman.
A 9 year old can install Linux and experiment and "innovate" all she wants. Linux's open source nature and the community is what powers its ongoing evolution, and that is encoded in its essence. I fail to see how the vein of Linux innovation on big iron is any close to being spent.
A "winner" is all but guaranteed when it comes to slicing the smallest performance gains in any field. And this is a niche field of performance; as I said in another comment, Win10 is still superior to Linux in other compute-bound systems (HBM2 is incapable of being exploited to its fullest potential outside of Win10 due to AMD's poor compute/blockchain drivers for Linux, for example - if you need blazing fast memops for the price, nothing beats Vega architecture right now).
The linux kernel has a certain architecture which informs the design of every OS built on top of it. There is effectively no competition of core OS architecture ideas anymore because linux won. Now, linux won because it was better, but is it really the right OS architecture for the 3d xpoint / quantum-mechanical hardware stack of the future? The monoculture makes it harder for people to experiment, and slows down discontinuous innovation in OS design.
Or, just imagine microsoft won and you substitute windows for linux in this thought exercise. Wouldn’t that be a problem for innovation?
From an architectural perspective, it's really *nix that won. To the degree that we worry about software monocultures in that vein, it really doesn't matter much if it's only Linux or if it's Linux + AIX + various other Unixes. That Linux is open source and allows for customization and variants makes it less of a monoculture but it still embodies a specific architectural approach. (Although GPU computation is one big change that's taken place.)
The TOP500 list does tend to be oriented to highly parallel systems because of the benchmark used. However, the different architectures like vector systems have largely gone by the wayside anyway. As CMOS scaling runs its course I expect we'll see more experimentation. Whether some of that requires a new operating system model as well remains to be seen.
>>Or, just imagine microsoft won and you substitute windows for linux in this thought exercise. Wouldn’t that be a problem for innovation?
There is no "winner" as far as I can tell in the grand scheme of things. We're talking about a set niche of computing. Win10 is still superior for a lot of GPU-clustering specific work; HBM2 memops via Vega architecture comes to mind as the Linux compute drivers are simply inadequate and the Windows tooling is significantly better.
Why isn't the promise of https://en.wikipedia.org/wiki/Singularity_%28operating_syste... being delved into? Who is pushing the microkernel ideal? Alan Kay has talked about how back in his time one had to do the full stack: hardware to kernelspace to userspace. Why is hardware being developed running Linux instead of an OS tailored to use that hardware?
Maybe it has to do with the Linux kernel being malleable enough to meet these specialized needs. My comment is a bit open ended, there's probably good reasons like the ideas I referenced had underlying issues. Microkernel security isn't really necessary on a supercomputer where anything running on it is more or less trusted code
I'm afraid security is important on typical HPC systems (at least university ones). I'm horrified to be assured cliinical data on the filesystems is secure when there are trivial privilege escalations on the system. We really do need a microkernel-type capability operating system, though most of what I see is failure to do system management or sane configuration of the batch system.
I recommend the book "The Supermen" about Seymour Cray, National Labs, etc and that will give you the necessary context to understand. Bottom line, it didn't used to be this way and supercomputing degenerates to lowest common denominator without starting from square one every now and then.
Not gloating – expressing pride in an open collective effort of hundreds of corporations and thousands upon thousands of individuals.
> monoculture
Only a bad thing if proprietary – witness how many Linux distros there are, consider the difference between Android and whatever runs on a supercomputer.
> I don't know the solution to all this
There's no problem, hence no need for a solution. You want competition? Then join any of the other free/open OS efforts and contribute your time and energy.
Hear, hear!, from the perspective of someone who's worked on an HPC
resource manager, and fondly remembers data collection/analysis on
real-time micro-kernel-ish distributed systems in the '80s. I don't
want proprietary software, of course; we could hack those '80s systems
in interesting ways, even if it wasn't actually free software.
I've sighed while removing code for systems like Irix, and Super-UX
(which I thought was dead, but seems not to be). Their extended Unix
resource management features were specifically useful for HPC-ish
batch systems. The deficiencies of Linux are partially mitigated if
you restrict the system to one job per node at the likely cost of
wasted cores, but then -- like Bluegene -- you can ask what the kernel
is there for. An example of the issues is lack of a sane way to
manage and account memory for a job's process groups; please don't say
that the cgroups mess provides that. No, OOM isn't what's wanted when
you hit a cgroup limit, for instance.
Then we have no real solution for security in GNU/Linux HPC systems --
or other mainstream systems (even if they did have HPC hardware
support).
> the ultimate morass of monoculture and an endgame with no competition. An ode to "the OS layer no longer matters, don't bother putting any effort into it someone else will". Bystander paradox.
It is not a monoculture, it's a common framework. Most farms have soil, and sure, some farms can operate without soil, but there is a lot of flexibility with soil. No two soils are alike, soil developers at each supersoil centre^W^Wfarm are free to derive and innovate as much as they'd like, but soil is a good place to start.
If you want a microkernel, port the Linux drivers to it and you're off to the races. If you want a monolithic kernel, there's basically nothing you're missing out on with Linux.
This is one of the most facile comments I’ve seen on HN. You are backing up your claims with absolutely zero data. More work goes into general purpose operating systems than ever before. The commit rate on Linux, the BSDs is higher than ever. Some of the best kernel innovations have been mainlined in the last 12 months (extension of eBPF, layer 2 encryption, ShiftFS, etc). Where has all this work come from? The corporations you are maligning, mostly Facebook, Google, Microsoft, the Chip manufacturers, Apple.
I am in Systems and Kernel development at least take a look at some commit data before you spew your unwarranted pessimism next time.
Thanks, I win the prize :). The corporations you list have little to do with supercomputing. I recommend the book "The Supermen" about Seymour Cray, National Labs, etc and that will give you the necessary context to understand. Bottom line, it didn't used to be this way and supercomputing degenerates to lowest common denominator without starting from square one every now and then.
I fear the technological progress has already dropped off a cliff. The last big things were plastic and ICs, and those have effectively been tapped out.
We are hitting that s curve plateau, and unless we have something like another world war (and even that may not be enough, as i do not get the impression we have anything sitting in the wing that is just lacking in funds to go massive) to open wide the purse strings i suspect there will be only a plateau from here on out.
Yeah you're right the curve was entirely driven by the world wars and cold war. It's not just purse strings, but collective purpose, need, and oversight that drove the curve.
I feel you're still overlooking the still maintained Minix and all the BSD Unix variants. If we see all of those become discontinued. Then we have a major problem. On other fronts the kernel of macOS is open sourced and then there's ReactOS and even QubesOS to some degree though I'm not sure how much of Qubes is Linux in all honesty.
Linux is not a mono-culture single entity with no competition. It is infinitely modifiable, and all the modifications compete with each-other, and that is why it dominates.
I know Linux fans don't see it that way.. young me thought Linux was indomitable and it was foolish for any company and group of people to not invest all their time and efforts into it. Older me knows better, that's not how corporations and groups of people work best. It leads to stagnation and apathy. It kills the fertile ground for ideas needed to make non-linear progression.
I don't know the solution to all this, but systems software is really going the way electronics engineering and circuit design have gone. I am sure executive types without engineering background would love to "outsource" it all to the perceived interchangeable cogs in "low cost regions".. but the rate of technological progress is about to drop of a cliff if my generation doesn't pick up the torch real soon now.
That said, it is somewhat premature as another commentator points out IBM's CNK. CNK is inventor's thinking.. the gall to ask "do we even need a general purpose OS?" regardless of what the market or eventual answer may be.