> This is the most important infrastructure project that we’ve had in this country for decades. Everyone agrees — this is non-partisan. Everyone knows we have to do it.
Considering the current political climate and rampant government cuts to important services, I very much doubt “everyone agrees” and that this is the best time to be planning such an important transition.
Yeah, couldn't this easily split in a group supporting the FAA to implement a better system, versus a group trying to contract it out to the private sector? Before you know it, IBM* is printing money again. (* substitute with Evil Corp of your choosing)
"Everyone agrees - this is non-partisan" is itself a piece of rhetoric designed to create that reality in a situation where it's in doubt. If everyone actually agreed you wouldn't need to emphasize it.
I don't know why you're being downvoted, I fully agree personally, my only concern at least would be that once the transition project gets started, if the wrong "loudest" vessels in government need to make some noise about overspending to distract from other things, maybe this project might be used as a scape goat.
I can see it already actually : "The FAA was working fine and yet they want these X billions to have shiny new silicon valley machines, paid to big tech by the tax payers"
As horrible as that sounds, I don't think many people would say that it couldn't happen.
And then what? Those words mean nothing to the people with the most power and motivation (or lack of care) to derail the whole thing.
It’s about as effective as placing a monkey in a porcelain shop then walking away while commenting loudly “Now now, it is very important none of the porcelain breaks, everyone knows it must remain intact”. The monkey doesn’t give a shit.
Butchering a proverb: “The best time to reorganise your porcelain store was before you bought a monkey. The second best time is after you sell the monkey.”
I sincerely don’t even know what you’re talking about right now. But it definitely isn’t related to my original argument. My point was about the task and the right time to tackle it, while you seem to be hung up on the words of the Secretary. The words are immaterial to this practical matter, as is any vague general concept of “changing the world”. I’m talking about this specific case, not building a philosophical thesis on the subject of improving humanity.
Yea, I'd be more worried that they're going to hire a 19 year old who knows nothing about tech or aviation, but who happens to be the son of a Heritage Foundation big shot, to head up the FAA tech modernization project. The scariest part of this administration is how unnecessary knowledge and expertise is when it comes to hiring the leaders.
Tomshardware could do better reporting. There is no such thing as a computer that can’t fail, or a component that can’t be replaced. Does our reporter think the entire system was installed 25 years ago, and not one component has been replaced since? More likely it’s the ship of Theseus, and not one component is original.
I’ve replaced whole systems without interruption. You build in compatibility, then replace every computer one by one, and phase out use of the compatibility. It’s not rocket surgery.
Technical sites could be superior to the reporting in the general media on technical issues. It doesn’t have to be be stenography.
I think they are just trying to get across how critical this infrastructure is, it can't be powered down for several days and unscheduled maintenance is risky.
The other problem they are up against is that there are not many people around that still understand how it works or what the edge cases are.
Upgrading these large distributed systems can be painful. The NHS tried to upgrade their software, over £10 bn later and they abandoned it [1].
Minor nit: There are classes of computers which contain redundant CPUs, PSUs, memory, etc where the components are hot swappable. Very specialized and expensive hardware which is statistically unlikely to ever need a reboot!
Source: My cousin used to sell these systems back in the 90's and 00's.
Operating systems have gotten a whole lot more reliable since Windows 95. The way I remember it, Windows 98 would regularly corrupt itself and need to be manually reinstalled. I'd done it so many times that I could pretty much recite the license key from memory. Modern Linux is rock solid. Even Windows 10 is very stable. They might be 'bloated', but modern OS's are way, way more stable.
corrupt itself and need to be manually reinstalled
In my experience that's normally the fault of third-party software, and otherwise quite easy to determine and avoid/fix. Now OSes with more protections just hide those bugs, causing most software to regress to a barely-working state.
I ran 98SE as a daily driver from late 1999 until 2010, and it was reinstalled at most 3 times, not even coinciding with hardware upgrades.
Or of just a power outage or driver causing a loss of write back cache.
95 and 98 and ME crashed on a regular basis. I specifically remember upgrading from ME to XP and being so happy with the massively improved stability of the NT kernel over the 9x kernels.
If you think that's 9x was stable and reliable, you may be thinking very nostalgicly.
I am not so sure. I've ran 98 on bad hardware, and it crashed regularly. So much so, that I installed linux on it already in 1998, and that was much more stable. It only crashed now and then. No doubt in both cases the poor hardware was the cause of it.
Anyway, two years later I got a brand-new laptop with good hardware that was running 98se. As far as I remember, it didn't crash during normal usage. By then I was studying computer science, and would sometimes write or run programs that would make it crash, but that was on me. I did dual boot in Linux, and that didn't have any problems on that machine either.
Fun fact, I still have that laptop, it's over 25 years old now, but it still works and runs Windows 98se!
Or a modem driver reading the stream and writing shit - I still have some of those burping mp3s. But if you blame this solely on the OS then you may be thinking very nostalgically too.
Hell, it most of the time worked on some combo of the cheapest parts - modern systems wont even get to UEFI boot part on the parts of the same quality.
> If you think that's 9x was stable and reliable, you may be thinking very nostalgicly.
I agree. Remember Plug'n'Play? It was so bad that we used to call it Plug'n'Pray. It frequently caused PC crashes. Modern OSes are a miracle in how stable they are with drivers.
I've seen enough stories of power outages permanently damaging SSDs, that if you have bad power from your utilities provider and can't get them to fix it, then I recommend investing in a UPS.
Operating systems were always more reliable than Windows95 from the day it was introduced. Protected memory and process privilege were not exactly unknown when DEC was selling VMS. Or for that matter when Microsoft was selling Windows NT. That the FAA cheaped out then, choosing an inferior system with no technical merit, is prelude to the current problem.
I've noticed that operating systems can get very flaky when the disk space gets tight. It seems that too much code does not check for disk full write failures.
It was still very much like modern systems. If you didn't install, uninstall, or aggressively reconfigure things they were pretty stable, and controlled changes could be achieved. Some of the problem though was that the systems required a lot of that to do anything fun with them at home.
As the article points out: the hardware is at risk of physically failing and it’s getting harder to replace like for like. That’s the reason for looking at an upgrade. Hell, even turning the machines off to replace them is a challenge since some systems need to run 24/7!
Not necessarily. For example, if there is custom hardware used for communications with other systems, such as radar for example, there might be specific timing and latency requirements that could be difficult to meet under emulation.
The most recent Dolphin Emulator post referenced a bug they had where memory cards were written to too quickly under the emulator (and even on actual hardware if you had memory cards that were sufficiently fast) which caused some games problems because they did not expect save files to be written so quickly. Imagine things like that, but where the worst case isn't having Wind Waker hang while saving, but planes crashing.
A long time ago, ..., so add a huge IIRC for the details.
Anyway, a long time ago, in a physic lab class, we had a custom spectrometer device that was controlled using the printer port. (Probably a light, a detector, a diffraction grid and a steppers motor, and perhaps some weird card as an interface to the computer.) It was controlled by a custom Pascal program in the DOS command line.
It was connected to a Win 95 computer that died and we had to replace it with a Win 2000 one. The problem is that anyone can read/write the printer port in Win 95 but that's not possible in Win 2000. [1] We had to make some magic, probably rewrite the program and use some weird device driver to write to the printer port (IIRC again). And multitasking messed the timing. At the end it worked, but upgrading the computer was not easy.
[1] If that makes no sense, replace that with a Win 3.1 to Win 95 transition, but I think it was not that old.
I had similar many years ago in a custom paint shop. We had an expensive colorimeter that interfaced over serial with a program that ran in DOS.
When the pysical computer gave out, I replaced it with a reasonably new one but instead of using a modern OS I installed MS-DOS in order to get it up and running as reliably and quickly as possible.
If I were doing the same today, I'd likely get a new computer and install FreeDOS.
Back when I was designing electronic circuits, the rule was to design for minimum speed, but faster speed should not cause a failure. The rationale was that newer parts were usually faster, and the older parts disappeared.
Of course, nothing can prevent poorly designed code and hardware.
The speed thing was just an example that easily came to mind. I can imagine there are other kinds of analog vs digital interactions that might be occurring that may not be easily replicated under emulation. Especially with a system that grew somewhat organically over the last half-century.
Emulation is likely possible, probably for many of the systems involved, but this is not a field where bugs, especially ones introduced due to emulation, would be easily acceptable.
> Of course, nothing can prevent poorly designed code and hardware.
Agreed, but the reality is that here, trying to fix things and ending up breaking them can and probably will kill people.
> Being terrified of progress means you're likely to kill far more people.
I'm not really sure what you're arguing for, or against, here. Nobody denies that changes are needed. The hardware is failing and buying new isn't possible to do seamlessly because of the age of the software.
The discussion is whether emulation is a possibility and people are pointing out that this comes with its own risks, so the discussion should centre around whether the cost/risk of emulation is lower than the cost/risk of building completely new systems.
I would not be surprised if many of these systems rely on several ISA cards with proprietary protocols and drivers.
I am only aware of a single modern-ish motherboard with ISA, the MS-98A9, and it only supports Intel 3rd Gen Core series CPUs.
That said, if it was a large enough project, reverse engineering and re-implementing using modern components would likely be feasible. Turning each of these into network services handled by something more akin to a RaspberryPi could modernize the data sources while providing a sustainable and modular replacement strategy. The problem is that its not "sexy" enough to get a major government project, and it would not grease the correct palms that a multi-billion dollar next-gen complex proprietary replacement would.
I'm sorry, but this just sounds like quitting before you start.
For example, I've looked into emulating DOS, because I don't like the existing emulators. But I don't need to emulate floppy disk drives, or their drivers, or their hardware cards. I just write some software that can fake a disk drive and hook it to the I/O interrupts. That would be a simple project.
There's nothing sophisticated about DOS.
In hindsight, I'm baffled that it took many many years for people to develop clones of DOS.
For example, EDLIN. A trivial program. I'd write it in a high level language like D, get it to work, then hand-translate it to asm. The executable loader is absurdly simple. And so on.
If it uses any digital hardware directly accessed by software (something that Win9x still allowed, see DOS games w/ ISA sound cards), it makes virtualization impossible.
Why is that? Doesn't virtualization virtualize the hardware? I'm not sure why virtualization is impossible there, as QEMU can run plenty of old DOS games.
Yes, we know that floppy disks and drives will wear out, and they have few if any sources for new repair parts. So the fact that the system is still more or less working today doesn't mean it isn't doomed and needs to be replaced before experiencing a catastrophic unrecoverable failure.
2. There are floppy emulators that replicate the functionality of floppy drives with flash
3. The above two probably absorb all of the demand today, but even if they didn't, the volume is so low that fixed manufacturing costs per unit could likely push unit prices well beyond even $50. The tooling for factories often costs millions and unless you are selling in high volume, you will have quite a high fixed cost per unit.
Because it wouldn't be profitable? How many do you think they could sell to a dying market, and what would those manufacturing costs be? What experts could you tap who know this space? they are all gone
The point is it’s enough legroom to be reasonably cautious in the rollout rather than needing to get a big contractor to do a major and therefore expensive push.
I read some years ago - IIRC the letters pages of BYTE, which dates it - about a critical factory control system in a company somewhere running on an IBM XT. The MFM drive had started to show some errors, so they got in touch with IBM, who being IBM, did not have any drives in stock (they'd stopped making them 15 years previously), but could retool a manufacturing line and make some. They offered to do it for $250k/drive. The company paid up.
That was cheaper at that time, than modernising that system. But it's clearly not long-term scalable.
I've heard of S/360s in KTLO mode in basements keeping banks running. Teams of people slowly crafting COBOL to get new features in at a cost of thousand of dollars a day each, and it "still works". But from a risk point of view, this is also ridiculous.
Safety critical systems have different economics. Yes, you can keep the floppy systems going, but the cost of keeping them going is rising exponentially each year, and at some point a failure will cost one or more airliners full of civilians and the blame will be put on not having a reasonable upgrade policy.
Sometimes you have to fix things before they stop working, or the cost is not just eyewateringly expensive in terms of dollars, but of human lives too.
IBM mainframes can run software written in the 1960s without modification. There’s no reason anyone would keep using an obsolete mainframe, and IBM usually leased them anyway and would refuse to support obsoleted machines.
You clearly don’t know what has been happening in the World of S/360 (and similar), support contracts in recent years.
Costs are rising heavily. IBM sold off most of that business, to people who don’t really want it as the skill base to support it is retiring and it’s too expensive to easily replace. This has been going on for a couple of decades, but it’s now gaining more and more pace.
Let's be a little more reasonable. I don't think anyone is saying we need AI. There are numerous other technological advances between floppy drives and AI that our air traffic control system could benefit.
Does it work? Sure. You have to ask more questions. How much does it cost to keep it working? How much would it cost to upgrade? If we do nothing, along what sort of timeline can we expect it to stop working, or become cost prohibitive to maintain?
Well also, 20 years is less time then you think. For a system of this magnitude, deploying the replacement could easily take 5 years to get all the way through to full completion. So that's 1/4 of your runway gone right there.
Every year you delay is pushing that lower, and then there's whether the funding is available because you're in fairweather economic conditions or if crisis will happen concordantly with some other crisis (I.e. do you want to be stuck replacing air traffic control systems in a rush because some war has wiped out the floppy supply chain right as your air logistics is a critical issue?)
The article completed skipped over this. This video was released literally a week ago and is completely mocking the FAA. Floppy disks are a big joke in this video.
For retrofit purposes, it's probably attainable to use solid state (no moving parts) floppy disk emulators that use USB thumb drives or CF/SD cards instead of error-prone, real floppy disks. Every time a floppy drive moves over a sector to read or write, it wears that area mechanically. Magnetically, bits just seem to rot from floppy disks randomly with time more likely failure mode for previously good floppies.
Let me complain you about how error-prone and unreliable are real floppy disks. ):
> For retrofit purposes, it's probably attainable to use solid state (no moving parts) floppy disk emulators that use USB thumb drives or CF/SD cards instead of error-prone, real floppy disks.
Yes, but if it is just a PC running Windows 95, likely simpler to get the software working under newer Windows, or if worst comes to worst, keep Windows 95 and stick it in a VM. I doubt there is any specialised hardware on the Windows 95 machines, the specialised hardware is likely connected to something else.
The use case where physical floppy emulators really shine is with much more exotic legacy systems. Some years ago there was a furore that the US nuclear arsenal was still being managed using 8-inch floppy disks (used in IBM Series/1s, 16-bit minicomputers from the 1970s). USAF was proud to publicly announce they’d successfully transitioned the US nuclear arsenal to be floppy-free. I don’t know if they said publicly exactly how they did it, but I suspect they kept the Series/1 minicomputers and just replaced the 8-inch floppy drives with hardware emulators (which probably each cost an utter fortune when you add up the premiums anyone will charge for it being the military, being highly classified, and above all being related to glowing things that go boom)
> USAF was proud to publicly announce they’d successfully transitioned the US nuclear arsenal to be floppy-free. I don’t know if they said publicly exactly how they did it,
They only said a “highly-secure solid state digital storage solution”. At least that's all I could find.[0] The article indicates that things get repaired down to the "component level", but specialist civilians.
And this bit was interesting as well:
"While SACC’s hardware is decades old, its software is constantly refreshed by young Air Force programmers who learn software development skills at Offutt’s Rapid Agile Development Lab. Most work on the software and interfaces seen by end-users like intercontinental ballistic missile launch crews, rewriting legacy code to make it more modern and sustainable, said Master Sgt. Travis Menard, 595th SCS’s programming section chief."
> I doubt there is any specialised hardware on the Windows 95 machines, the specialised hardware is likely connected to something else.
Based on my experience with older government systems, this is likely an incorrect assumption. It was extremely popular in the 90s to create custom hardware that integrated directly to windows machines. I've had to reverse engineer so many drivers to upgrade old bespoke equipment to integrate with newer OSs
The reason why I doubt it for the FAA, is I know in the 1990s the core ATC system ran on IBM mainframes and the specialised equipment was directly or indirectly connected to the mainframes. Hence, I expect the Windows PCs were used to run terminal emulators, email/messaging, various auxiliary applications.
Aside from that, the upgrades to this critical infrastructure should be resistant to hacking and other vulnerabilities
They should realise that, unlike e.g. USB drives or SSDs or even HDDs[1], floppy disks are dumb raw media and cannot contain any "hidden" behaviour, and the failure modes are well-known.
As of the early 2000s, ATC was still using vacuum tubes. In fact, the FAA was the single biggest buyer of vacuum tubes in the world at the time, almost all of them sourced from former Soviet bloc countries. I think they've all been replaced by now, but I can't say that with 100% certainty.
Which is what baffles me about the current situation and gives me a lot of hope for this effort. We should've been updating this stuff in the 90s, but successive administrations of both parties have just passed the ball on this one.
I see this the opposite way: kudos to the FAA for sticking it out so long on legacy hardware and software as long as they have!
ATC is a safety-critical function that has what amounts to a 100% uptime requirement. Whatever system they're running currently either works or has known flaws that they know how to work around, and air traffic controllers have been trained on these systems for more than a generation now. Upgrading merely for the sake of being up to date would have been foolish no matter how much funding Congress would have given them.
If they're saying that they need the upgrade now, I'll trust them on that, but it was the right call to make it last.
> ATC is a safety-critical function that has what amounts to a 100% uptime requirement. Whatever system they're running currently either works or has known flaws that they know how to work around, and air traffic controllers have been trained on these systems for more than a generation now.
The problem is that Eurocontrol (for example) has modernized their systems without much fuss, and UK NATS even has remote tower ATC now (https://www.youtube.com/video/Ii_Gz1WbBGA). It seems that FAA is stuck in the past, not just using old systems because it's reliable.
> Upgrading merely for the sake of being up to date would have been foolish no matter how much funding Congress would have given them.
I would agree if the system is still fit and proper, but even in 2005 the ATC systems in the US is not really fit and proper that there has been multiple plans to rehaul the system. It is really miraculous that the only system failure happened in 2023 (NOTAM offline), but that's due to tireless dedication that's certainly burning unneded manpower.
Unlike in Europe where civil servants have the sway to just do it, it seems that the US is an expert in political bickering on things that aren't really political.
My understanding is that remote tower ATC is something that had to happen at that airport due to geographic constraints rather than some kind of next step for ATC in general. Given a choice between being able to physically look out the window and not, from what I understand being able to see out is always preferable.
The rest I don't know enough to comment on, so I'll assume you're correct.
> ATC is a safety-critical function that has what amounts to a 100% uptime requirement. Whatever system they're running currently either works or has known flaws that they know how to work around, and air traffic controllers have been trained on these systems for more than a generation now. Upgrading merely for the sake of being up to date would have been foolish no matter how much funding Congress would have given them.
I do not have enough knowledge to disagree on this. But I will say the FAA is still on floppy disks when the US Nuclear Arsenal moved off floppies back in 2019.
Yes, they have different requirements and yes, SACCS was using 8 inch IBM mainframe floppies from the 70s, but they are both 24/7 critical systems.
> If they're saying that they need the upgrade now, I'll trust them on that, but it was the right call to make it last.
The real answer is likely embarrassing incidents that came up during the start of this presidency. There is now political will to address it; instead of 'before' it becomes a problem. They are on Windows 95-it was budget issues.
> I do not have enough knowledge to disagree on this. But I will say the FAA is still on floppy disks when the US Nuclear Arsenal moved off floppies back in 2019.
Well this isn't very long in terms of overhauling safety-critical systems that have many decades worth of processes and infrastructure built up around them, is it?
> Well this isn't very long in terms of overhauling safety-critical systems that have many decades worth of processes and infrastructure built up around them, is it?
Do you actually know if 6 years is enough time? If so, provide info.
As for myself, I do not know. I do know once it was reported they were on floppy disks; they finished moving off it in 3 years. If wasn't for the media report I doubt it would happen. Once again, different systems, but likely public embarrassment motivated the move more than anything else.
Upgrading merely for the sake of being up to date is essential if you want to retain the capability to upgrade at all. You have to exercise your upgrade muscles, otherwise they become weak and flabby. At some point you're so far behind that upgrading becomes nearly impossible, and then everything just collapses.
What I don’t understand in such reports is why there is no mention at all how this is done in other countries. Do they all still use floppy disks? Did they do an upgrade? How did it go? Surely this would be valuable information.
A considerable number of countries use systems based on ex-Eurocat now TopSky, for their air traffic controllers, which is a distributed system.
Install and updates are via a registry based system, and it supports Windows, Linux, macOS - because its mostly written in Ada and R, as of 2012. (Most are running on top of Linux, as far as I'm aware).
Is this one of those things like phones or banking where the early adopters are stuck with old tech due to inertia while late adopters are using newer technology?
A lot of airports are privately or quasi privately (Canada has a non profit called NavCanada) run which allows for advance planning of CapEx instead of depending on Congress. The US system where the FAA oversees and runs ATC is, to put it nicely, unique.
I think truth is more so that each is using contemporary technology from time the project started. So those that started later have newer. But that does not mean modern. As in current day.
I get that FAA hardware/software is a time-tested, safety-critical system that has resisted many prior modernization efforts but...how do other countries run their systems? Surely they're not all using floppies. I doubt there are many (any?) countries with a flight volume like the US but overall, flight safety is pretty good world-wide (again, with exceptions).
Their governments fund the upgrades instead of running around claiming their flight agencies are full of corruption and inefficiency with no basis in reality.
The upgrades have been funded for decades. It is an execution issue, not a money issue. Many other parts of the Federal government are in the same condition: software upgrades that are infinite money sinks that never produce much after decades of effort.
I've worked around some of these programs. I've had visibility into some of them for 15 years over which there has been zero forward progress despite unreasonably large amounts of money being spent. It is no secret why those programs are permanently broken but no one wants to have that conversation.
I think most takes on this are overly reductive. The whole situation is sad really.
The root cause, to the extent that one exists, is that no one is accountable for successful execution in a very literal and systemic way. Some parts of the government I've worked in are worse than others, but it is endemic. This leads to a textbook case of Pournelle's Iron Law. There are no negative consequences for a handful of people aggressively maximizing their personal benefit and acquisition of power as their primary objective. This is how you get the fiefdom-building, feather-bedding, and the usual revolving-door corruption that these programs are notorious for.
Most people involved in these programs aren't like that but enough people are that it is impossible for people trying to do their jobs competently to get anything done. The people that defect are the people that end up controlling the programs because that is how the incentives work.
Inefficiency and corruption are a symptom, not the disease. The incentives virtually guarantee that these programs become playgrounds for sociopaths. Average workers on these programs are put in the demoralizing position of either having their good effort constantly undermined by leaders that don't care about the mission and are openly making decisions for personal benefit or to defect to the side of the sociopaths so they at least get some personal benefit out of it. Most of the best and most competent people I know eventually leave Federal service entirely.
A second-order consequence of this is that over time, no one competent wants to work on the programs that are run this way. Through churn these programs slowly fill up with mostly useless seat warmers who don't mind a job where no one expects productive outcomes. It is a kind of stealth UBI for government employees. Some people request assignment to these programs.
You never hear about the programs where the leadership is actually competent and cares about the objective because these actually function pretty well. But the incentives are such that this is the exception rather than the rule.
I'm not even sure how you would fix it, I suspect it is politically impossible. When companies become overtly like this they tend to slowly self-immolate into irrelevancy. Governments lack these negative feedback loops in any meaningful sense.
> The root cause, to the extent that one exists, is that no one is accountable for successful execution in a very literal and systemic way.
Not even the secretary of transportation? Wouldn't this have been a really great way for the previous one to show he can get things done? Or does the position lack the requisite authority?
Somewhat authority, you need Congress to sign off on the money and they will want to influence it to their preferred vendors. Also, when Secretary of Transportation wants to run for higher office, he does not want some boondoggle project that looks terrible hanging over his next office run.
I remember reading a drone startup saying they had an easier time operating in Kenya than in the US because Kenya's ATC system was fully modernized, with every aircraft tracked at all times.
What non-pilot techies fail to comprehend is that the entire ATC system is designed to operate in a no-communication failure mode. This includes features such as the paper strips, mandatory holding points, timed approaches, clearance void times, etc. It is all designed so if you have a complete communication failure in IFR conditions you can land without hitting anyone else.
Any fancy new system of, for example an in-cockpit text based clearance/routing display using an LTE network, will need to be backed up with a process that can be accomplished with a pencil, a compass, and silence.
For anyone considering doing development in this space, sign up for a 20 hour instrument flying ground school, preferably one taught by a retired old fart rather than a 25 year old “instructor” with no actual experience.
> For anyone considering doing development in this space, sign up for a 20 hour instrument flying ground school, preferably one taught by a retired old fart rather than a 25 year old “instructor” with no actual experience.
That's a bit overkill in the world of DOGE. They'll just use some AI to design a system by a bunch of 20-somethings that have no experience whatsoever. Then, they'll come here and brag about it.
The reason this story is in the news currently is because the current FAA administrator specifically turned the FAA's floppy disks into a story about why DOGE should be involved at the FAA.
> “Big news,” Duffy said in a 5 February post on Musk’s social media platform, X. “Talked to the DOGE team. They are going to plug in to help upgrade our aviation system.”
> Musk himself responded by saying that DOGE will make “rapid safety upgrades to the air traffic control system”.
There is definitely value in stability and the US system worked well for a long time, but it is not the only way for a safe ATC.
So OK, if you want to do development in this space, do learn about the US setup from a retired old fart who worked with paper strips and thinks the system should stay like this forever. But also learn about systems in Europe, Japan and other places; and realize that ATC can move away from the stone age. My 2c.
The irony is that the US routinely uses much more capable software for almost identical purposes in domains like battle space management. It isn’t like the US doesn’t have this software, more that the FAA doesn’t consider anything derivative of that tech as an option.
My original response was sharper than I intended; I am aware of both the air-defense and ATC systems for both the civilian and military use (so worked in each cell of that mini-2x2 table). The military is much more tolerant of risks and, even if implementing them would lead to an increase in safety and convenience using military systems for civilian ATC will likely cause all sorts of problems due to differences in training, planning, etc.
I was involved in writing ATC software in the 90s for a European country (the FATMI system for Finland), and they were definitely using paper strips at the time, and I believe the design did not change this. I wasn't involved in the flight strip printing though, I was working on SIDs and STARs, airspace sectorisation, that sort of thing.
It would be interesting to know how things have changed since then, as obviously nearly 30 years has passed since that system would have been commissioned!
A first step to mitigate some of the risk would be to move the
system to a virtualised system. This could be in each location
or more centralised which would make the maintenance of the
fleet of old computers easier.
Floppy can be copied to hard disks and will not have to worry
about failures of mechanical parts involved in reading floppy drives.
Developing a brand new system would take quit a lot of time.
As all systems du if they need extreme uptime.
Starting that effort now is ok but I would guess it would be take
at leas a couple of years. Significant work would have to understand
in detail what the current system does and does not do, and then
map out what a system should do.
We IT folks tend to quickly propose solutions to systems whose complexities we do not completely understand. That's fine when it is about serving ads or managing book orders. It's not ok when the stakes are high.
Virtualization just adds another layer of complexity to an already fragile system which literally thousands of human lives depend on every day. Adding more complexity is not a neutral act here, but neglectful manslaughter waiting to happen. Aviation is a low-tech, never-touch-a-running-system, risk-averse environment for a reason.
Floppies were useful because you could easily take them and take them to another, secondary, sometimes air gapped backup system. Replacing this functionality means replicating not just the data transfer, but also the safety architecture - which includes physical isolation and manual fallback paths. To recreate, the best chance would probably be something like storing the relevant info on thumb drives - but then you have whole new family of attack vectors by hostile forces (anyone still remember Stuxnet), which floppies did not have in that form?
And then there's the pesky aspect of international interoperability. One country alone cannot just storm forward. We are looking at decades of upgrades and alignments here. And that process already is underway. But proposing a radical change without acknowledging the full scope of what that entails - from certification cycles to human factors to geopolitical coordination - is not progress, it’s hubris.
It is true, it introduces new risks.
and would have to be tested well and attempt to be certified.
I am not saying its something you "just do overnight".
Still it removes more risks than it introdcues.
(IF it tests out ok)
Though they have a stack of replacement PCs ready to go and lots of
floppy drives and floppy disks to quikly replace whatever may break.
Writing new code from scratch, introduces a lot more risks.
but also offers a promise of something much better.
Eeeeexcept that floppies are horrifically unreliable. I remember feeding disk number 27 out of 33 only to get a "bad sector" error an hour into a software install. I'm still salty about that one.
"It's not broken" is the cry of the bad manager that hasn't done the proper analysis, hasn't actually looked at the pros and cons, but has simply become complacent and comfortable with the devil they know.
If they're still using physical floppies, then their process is broken now, so virtualising it will almost certainly un-break it.
A simple "clarifier" for this kind of thought process that I like to use is: If you were already using the new option (virtualised legacy hardware), would you think it a good idea to convert it to using open drives with convenient dust ingress, non-existent support and supply chain, glacially slow mechanical moving parts, and hilariously antiquated crunching noises for all data access? Would you? Really? Or would you recoil in horror at the very idea?
I use the same kind of logic on people who think staying on Windows Server 2012 in <current year> is a good idea. Would you downgrade Windows Server 2025 to 2012? Why not? You think it's a great platform, apparently!
PS: I worked on a large scale DOS-era software virtualisation project where we moved ~20K users onto a Windows + Citrix platform. We eliminated about 6000 floppy drives and about a million(!) tapes, and the resulting system was so much faster and reliable than the original that people were trying to bribe the project manager to be put at the front of the migration queue.
> I remember feeding disk number 27 out of 33 only to get a "bad sector" error an hour into a software install.
I love this fixation on floppy disks. The article likely brought it up because it is a recognizably obsolete technology, but didn't cite why (or even if) it was a problem. I'm sorry, but a nightmarish software installation scenario doesn't cut it. It is highly unlikely that they are doing in situ software installations from floppy diskettes.
The danger in such armchair quarterbacking is that it undermines the authority of the agencies that are in charge of making decisions. If there are legitimate reasons to question their authority, by all means do so. Yet, when doing so, understand their requirements and provide evidence as to why their authority should be questioned. Also be prepared to be unsatisfied by some of their answers due to differences in perspectives.
You make good points, but now Citrix and Microsoft have them over a barrel. Curious how such a migration looks in 2025 with Microsoft pushing everything to Azure, though and Citrix's acquisition by Vista Equity (2022).
That migration occurred in 2007, the whole thing was replaced by a web app in 2017. Over that decade they saved many millions of dollars. Large scale disturbed legacy hardware is much more expensive than Citrix licenses!
When we started in 2007, systems still using floppies were considered ludicrously legacy and people could hardly believe me when I told them it was still in production and used for a critical system affecting millions of people.
Anyone still using floppies in 2025 has no excuse, stop trying to justify incompetence and sloth.
> I remember feeding disk number 27 out of 33 only to get a "bad sector" error an hour into a software install. I'm still salty about that one.
That's why mission-critical systems have several sets of floppy disks, and disk-multiplication stations.
> Would you? Really? Or would you recoil in horror at the very idea?
Depends. If the old system is certified and has all error modes defined, while the other new system is a black box with exciting new ways to screw up, I'd go old system ten out of ten times. Which incidentally is why NASA uses ancient chips when they build new robotic drones.
> I worked on a large scale DOS-era software virtualisation project where we moved ~20K users onto a Windows + Citrix platform.
Respectfully: How many lives would you have extinguished had your new system failed? How many failure modes did you encounter during your virtualisation project? How many external systems - which also relied on a very specific way of doing things and would have murdered people if talked to wrongly did you interface with?
No need to answer. We have all had such projects. We know things break before, during, and after the switchover. Only in some environments, systems absolutely cannot break, ever. Aviation is not your average 'let's get us a new mail server' migration project.
I’ve worked on life & death Citrix modernisation projects several times: the local equivalent of the 911 emergency phone call centre and then computers on wheels used for during paediatric surgeries. A help line for suicidal children too.
People conflate the usecase with the technology, assuming that “important thing” must have some mystical properties that requires legacy or some other “special flavor” of IT architecture. They’re wrong.
The best example of this flawed thinking was some person arguing with me about the computer upgrade that F-22 fighters are receiving this year… to the same level of performance as a first-gen Apple Watch!
Of course, that costs an absurd amount of money and is already delayed.
“But it’s a stealth fighter!” people will argue until they’re blue in the face.
Sure. Yes. But that’s a property of the outside surface, not the computer inside.
Other modern fighters, including stealth fighters, have hilariously better computers for a fraction of the cost. The F-22 procurement process was corrupted and some vendor is doing the minimum, twenty years late, at ten times the price. That’s what happened. Everything else is a “story”. A fiction. A cover of the ass type.
Same thing here. There’s a contract for providing IT services to the FAA. It’s a bad contract. That’s what happened. That’s all. There is no mystical or magical capability provided by floppies that can’t be better served by, for example, USB thumb drives.
At the risk of replying to someone with "troll" in their username...
Yes, but the entire point is that other fighter planes have identical requirements but don't have comically out-dated avionics.
Don't guess. Don't make up stories. Don't carry water for incompetent people that are protecting their own backsides.
Compare. Look at what other, more competent people have done, and use that as your benchmark.
That's always the key with these things. You don't have to be an expert. You don't have to have secret knowledge. Other people do. Just look at what they've achieved (or haven't), and compare against that.
When people come up with excuses, you don't have to believe them, even if they're experts in an esoteric, specialist field such as "nuclear-war-resistant stealth fighter design". Even if they're some sort of "authority", so are other people that designed their own stealth fighters.
This is a very useful life skill. Use it!
As a random example, there's countless arguments from "experts" such as economists, politicians, and industry thought leaders about how unaffordable universal government-funded health-care would be in the United States. Meanwhile, dozens of similar countries have done it for decades! Just look at the success of other countries and then dismiss the excuses you hear back at home out of hand because now you know: they're just excuses, not reasons.
I wonder if anyone makes a virtual floppy drive that replicates the performance characteristics. I.e. to avoid a faster virtual drive uncovering dormant race conditions. Something like a developer assuming "I have enough time to do this processing before the disc makes another rotation" etc.
I think any of the "off the shelf" gotek emulators should suffice for this. They're made for people to keep playing games on old hardware. I would assume copy protection and other shenanigans would be the creme de la creme of abusing the hardware.
This is to get rid of the media only. You'll still be using the original compute hardware. But it would be an interesting step.
I feel that most of the desire to upgrade is cultural and not technical. People love to talk about the floppies being used while its just a small part of the equation. Cost and risk of creating a new system with the same reliability expectations is hard when the incumbent has decades of iteration. For systems that do not require more performance or energy efficiency the accounting on upgrading looks very different.
What are the chances they ditch floppies on the hardware end and switch to GNU/Linux running DOSBox to preserve the software stack?
Presumably (hopefully) the existing system is airgapped in some way or otherwise restricted to communication with other ATC systems, so DOSBox-X running Win95/98[1] could act as a drop-in replacement for the software side...
While you can still get high quality floppy disks, there is nothing wrong with them. You must handle them properly and keep them clean, but that's only a problem for children and idiots. When demand falls and they are no longer profitable to manufacture at high quality levels, low quality media dominate the supply and you have problems.
I watched this happen with floppy media. When floppy disks were common in the 80's you had great quality disks from top tier Japanese manufacturers at low cost. Media failure was rare and you could rely on a disk day after day for years. Then, as demand for floppies dropped, and these manufacturers fobbed off legacy products to low cost manufacturers, floppy media became terrible.
By the mid to late 90's, floppy media bought retail was very unreliable. For a brief time I was salvaging stacks of disks that came with commercial software because the software vendors were still able to secure good media, while the retail blanks you found in stores was just this side of e-waste. I used them with expensive instruments that had integral (high quality) floppy drives, long after PCs stopped using them.
Nothing if it still does the job. I buy new things when they can't do their job anymore or I have a new problem to solve. My desktop and car are over 12 years old.
Does that matter? Manufacturing them ain't hard and current supplies are big enough to sustain the small demand. What matters is that I can buy floppies and will be able to for a very long time if not indefinitely.
He also pointed at the root cause and possible solution there. Which is re-categorizing spending on this as essential instead of as something that's nice to have and becomes the victim of cuts almost immediately after anytime some budget is actually allocated.
An interesting point here is maybe that there's a whole world outside the US where planes fly and communicate. For example the EU has its own issues on this front but is modernizing what it does. Airspaces here are pretty dense and busy. It's not necessary to reinvent a lot of wheels here. The US could just look across its borders and learn from what is being done there.
As soon as there's a reasonable budget for this, there are all sorts of perfectly reasonable things that can be done. The core issue isn't technical.
> For example the EU has its own issues on this front but is modernizing what it does.
The EU has been almost single-handedly designing the international interoperability standards for the kinds of information that the US uses floppies to move around.
I'm not sure any country still remembers the lessons that would be useful for the US on this one situation.
The problem, once Congress gets wind of the amount of real money that will need to be spent, plus the time it will really take to develop and fully test, it is cancelled.
Of course I fully expect this to be TIP (Test in Production), thus for maybe 10 years, flying in the US could be very dangerous. Lets hope the pilots will be able to manually avoid other planes.
Would you like to trust your life win95 and floppies definitely no but paper strips is something really robust and in light of crowd-strike or the outage in Newark I think a truly independent backup ‚system‘ is a good idea. Particularly as the next system will come with some early bugs.
The problem is all of the big software consultancy services are optimized to maximize revenues / minimize their own risk when working with big / dumb government agencies.
The reporting on this has been an atrocious, lazy, embarrassment to journalism (aka: a normal day) so I tried my best to look into it.
As far as I can tell the only systems that use floppy disks are IDS-4 terminals, of which there are a couple hundred left in the US, the rest having been upgraded (to IDS-5 or similar systems) over the last 30 years.
I don't know if it is small regional airports with no money, large international airports with few moments of downtime time needed for the upgrade, a mix those two, bad luck with the bureaucratic wheel-of-priorities spin, or what.
But there's no context to any of these articles, only "FLOPPY DRIVES LOL" so I had to take the time to find out what systems were actually impacted.
I mean, it could have been an old HP Oscilloscope in a RF rack that used floppy drives to store images and log data, or it could have been the Master Control Program of the entire air traffic control network.
There's a slight difference in impact between those two.
It appears as though there are multiple competitors/replacements to IDS-4 so the solution is to cut a check and block off some time on the calendar.
edit: every single journalist who just grabs a couple of tweets, adds some commentary, and dusts off their hands muttering "job well done" should encased in a Lucite cube and displayed in the town square as an object of ridicule.
Still have yet to see any evidence of exactly how this tech is being used, just statements from an established liar with no qualifications and clear motives to sabotage the federal government so it can be taken over by private corporations.
It's possible that what is actually going is that the Windows 95 serves as "bootloader" for something like VxWorks - KUKA robots with KRC-1 controllers do something similar.
Setting a protocol to handle air traffic control and collision prevention in airspace around airports is a 100% automatable problem. You don't even need a centralized control system. This can be handled entirely with software running on each plane. Same way a flock of birds can fly and never collide with each other.
Unfortunately that's not how things work in practice https://en.wikipedia.org/wiki/Split-brain_(computing) If Jepsen fails every database coming from single source, imagine the chaos of synchronising a P2P of various clients of various versions over a very noisy link. We can't even achieve that with home automation meshes that send maybe 3 types of messages!
Also you need to handle planes without computers - you can land a personal plane at almost any airport. (With lots of caveats but still) Also you need to handle planes with failing automation. Also you really want to know the situation on the runways, so there's really no need to remove the single source of truth here.
We operate cars on the road with not only no centralized system, but also minimally defined and enforced protocol, and yet Waymo has achieved a near zero collision rate inside a swarm of cars that are not running equivalent software. And this is in a situation where cars are only a few feet from each other while operating at top speed. So you can come up with a million objections but they are all solvable. As for automation failure, the rate of that can be easily made lower than the rate of human failure, which currently is fatal to a plane.
Waymo is significantly more dangerous than air travel to a degree that the comparison is actively offensive to everyone currently alive on Earth.
We can't even get cars, working in more-or-less two dimensions, to go without constantly running into each other and being one of the major causes of death in human civilizations. Waymo "solved" that problem in about, oh let me see here... yeah 0.0001% of cases. So, we're almost there! That's, like a couple cities out of all cities on Earth.
Yeah that's bad. Really, really, really bad. Like so bad it's not even worth talking about and comparing to air travel.
That's a completely different system. In a car everything still resolves in seconds and needs minimal planning -corrections can happen in real time. That's completely different from "there's a queue of planes coming in and you have to manage the throughput available on the ground". You can't just hit emergency breaks on everything and slowly resolve the situation. These things are not comparable.
99.9% success rate is probably not good enough when you consider that the vehicle in question costs $200 million, and has 150 - 350 humans on board. It’s not a “whoopsie” like with cars where the damage is $50k at most (waymo vehicle) and maybe 2-3 people injured or dead. Also dead pilots who are already in short supply.
Also cars only care about what’s going on in a horizontal plane. 3D space is clearly more complex and probably requires more computational power i’m sure. Consider that boeing couldn’t even correctly write software to keep their 737 MAX planes from crashing into the ground (MCAS software issue). Something that simple was too hard for them. Speaks volumes doesn’t it?
Not saying it’s impossible. Just saying that this clearly isn’t a “just do it” problem. Waymo’s been working on their software for how many years now? and they still have minor crashes lol. Not an easy problem at all.
Also consider that airlines usually outsource their software development so they don’t have cutting edge expertise in house.
So what happens when a plane has a critical failure (related to this P2P communication), how does it land? How would other planes nearby magically know what the plane that is in distress is going to do? It’s basically an unpredictable peer in the network.
YOLO i guess? :)
Hypothetically the nearby planes can detect that unresponsive plane on radar or other sensors, and try to react together as an intelligent swarm, to avoid it and let that plane land manually. But it’s not so simple. Planes are not loaded with full fuel tanks, only a bit extra. Some planes may have already underwent a go-around if the airport is busy. So it’s not just “land without crashing”, it’s also a prioritization issue.
IMO we certainly need humans in the loop, in a centralized fashion, to “orchestrate” a manual emergency landing if there is some critical cascading failure or bug in the software. I agree that in the happy path (99% of cases) it’s possible to automate it all. In theory.
Things get more complicated when you consider that small planes (flown for hobby, flight school, etc) have waaaaay less tech. That can’t work in some peer to peer fashion without a major upgrade to all those planes too. And the owners of those planes are not corporations making billions.
What happens when an airplane's pilots have to radio ATC to request an emergency landing, and the planes' sensors have failed so it can't safety land itself?
If the plane is able to communicate with ATC it is able to broadcast to the other planes that it is on an emergency landing, so this is not an issue. Even if it has lost all communication this is still not an issue because all the sensors of the other planes can see it. So if the disabled plane immediately goes for an emergency landing, all other planes in the area are able to see its position and that it is not responding to pings, and therefore set safe courses that avoid it. This really isn't a very difficult problem.
Considering the current political climate and rampant government cuts to important services, I very much doubt “everyone agrees” and that this is the best time to be planning such an important transition.
reply