Hacker News new | past | comments | ask | show | jobs | submit login
Dieselgate, but for trains – some heavyweight hardware hacking (badcyber.com)
683 points by GruHe on Dec 8, 2023 | hide | past | favorite | 293 comments



Recent and related: Polish trains lock up when serviced in third-party workshops - https://news.ycombinator.com/item?id=38530885 - Dec 2023 (347 comments)


Its insane how brazen this is. Code that 'bricks' the train locomotive if its gps coordinates remain with bounds of a competing repair facility for more than ten days! This is way beyond putting information barriers to repair, like undocumented interfaces or even crypto-signed firmware. This is actively malicious destruction of property. I don't know anything about the legal system in Poland, but I can't imagine how this gets by.


It will be stuck in legal hell due to conflicts of interests. Trains already exist, and they need to work - but maintenance/repair companies cannot legally modify software of them due to copyrights. It's a catch22 situation.

I honestly hope that company will be fined to the oblivion, and for criminal charges for that, but i doubt it will happen.


Supposedly tho the maintenance company would want to sue to at least dispute their contractual penalties?

> A day of train downtime in the workshop costs over 1000 USD in contractual penalties, and there are several trains stuck, so the tension level in the SPS is rising.

Also LSR, because evidently they were interested in holding a tender before and so likely don't want to be forced by Newag into overpriced maintenance contracts?


Of course laws vary, and Polish copyright law might be completely crazy, but around here copyright only covers distribution of copies. It does not make it illegal to modify software that you own. It only limits distribution of copies of that software, modified or otherwise. If the owner of the train wants to modify the software then there is probably nothing stopping them.


I don't know anything about Polish law either, but in the US, copyright law (DMCA in particular) makes it illegal to modify the software in a device you own, if it requires circumventing protection code or devices. Which it probably would in this case.


According to the Polish copyright law, by default one can reverse engineer and modify licensed software without author's permission to ensure interoperability with other software and for fixing bugs. Such right can be explicitly denied by the copyright owner, though.


But could the authors be attacked for treason, destruction of property, or a simili-Patriot Act? Besides, have we learnt something about any public software being required to be delivered as open-source?


> If the owner of the train wants to modify the software then there is probably nothing stopping them.

This of assumes that the owner of the train company has the skills to do this. In reality they probably would need outside help and that company might fall foul of copyright issues. (when they are distributing the modified code back to the train company, for example)

But the real problem of course is that all of this code is very likely safety critical. Can they modify it? Probably. Is it a good idea? Not really.


Performing a task for someone, whether directly or as a third party contract, generally doesn't invoke copyright.


As a CEO of a railway company, not sure I'd want folks to patch the binaries on my trains. Pretty sure an insurance company wouldn't like that in case shit happens.

Rather, the companies with botched trains should demand from Newag to remove their locks, for free, and on the side sue them.


> Rather, the companies with botched trains should demand from Newag to remove their locks, for free, and on the side sue them.

This is the way.


> It will be stuck in legal hell due to conflicts of interests. Trains already exist, and they need to work - but maintenance/repair companies cannot legally modify software of them due to copyrights. It's a catch22 situation.

Which is why any government funding for new trains should come with the requirement for open source firmware.

Well better yet would be to abolish or reform copyright to be more aligned with the interests of society at large but international pressure makes that even less likely.


This is the kind of thing that will destroy a nation's manufacturing industry overnight.

Who in their right mind would buy kind of equipment from a Polish company knowing that this kind of nonsense is both widespread and that their legal system has no solution?

Hoestly, "Dieselgate" is not a fitting corollary for this travesty. This is considerably more sinister. Hopefully whatever happens from here will be an agent of change for the better.


Newag is actually trying to expand into Italy and a few years back they sold (and already delivered) 11 of their Impuls 2 trains (newer variant of the ones described in the article) to Ferrovie del Sud Est. I'm really wondering whether they got the same extortion software as the ones in Poland or did they maybe spare a new client on a new market.


Aren’t we all doing this when we buy software from the cloud?


No? When using the cloud we are renting resources and can (in theory) switch providers as we wish. Here they bought a machine that vendor purposefully broke after some time or with purpose to disable competition from doing their job.


The only practical solution is to re-nationalise this company.


How does one become an train repair company if there are no trains that you are allowed to repair?


We will see firmware wars were open source train software wind.


If an individual did this, they'd go to prison.


Article 254a of the Polish Penal Code addresses the obstruction of railway operations and other critical infrastructure. Violating this law can result in a prison sentence ranging from 6 months to 8 years.

It doesn't matter whether the act was committed as part of a company's operations or as an individual's private endeavour.

To all software engineers: please refrain from engaging in criminal activities. If you are instructed to do something illegal, it is important to report it to the relevant authorities.


I think jakozaur is correct, and don't know why they're being downvoted. Here is the legal statute they are referencing:

    Art. 254a. Disruption of a network; damage. Anyone who takes, destroys, damages or renders unfit for use an element of a water supply, sewage, heating, electricity, gas or telecommunications network, or a railway, tramway, trolley bus or metro line, thereby causing a disturbance in the operation of all or part of such network or line, is liable to imprisonment for six months to eight years.
Source: https://supertrans2014.files.wordpress.com/2014/06/the-crimi... page 32

I certainly think that this malware meets the criteria set forth in that law: "renders unfit for use an element of ... a railway ... , thereby causing a disturbance in the operation of all or part of such network or line".

Seems pretty cut & dry to me. I hope some people face real jail time for this. As another comment mentioned, it will probably be a "fall guy" (perhaps a middle manager) but that will still deter future managers from authorizing such fraud, even if the orders come from above. Future managers might reject such orders since it's not worth jail time.


As a person very familiar with Polish legal system and code I would be far from saying it's a cut & dry case.

What we have in this case is a company doing something that is wrong, not an individual. To sentence somebody with the statute you and the parent are referring to you need to have a case against an individual, not a company. Secondly, except for this specific statute you have to take into consideration the general rules of the penal code (Zasady odpowiedzialności karnej) and in this case you have to assign blame, find out that the intent, the knowledge of what they were doing etc.

In general, in best case this will be a breach of contract, a civil case, there probably won't be jail time. Don't get your hopes up.

Don't get me wrong, what they did is nefarious, but at the same time I don't think there should be Jail time, just huge fines and some scrutiny (maybe NIK, ABW, CBA etc. - other polish three later agencies) on Newag, maybe barring them from some future deals.

Source: I might be a lawyer...


Deter middle managers from what? Implementing shady business practices that skirt the edge of legality? That's day-to-day business, the only way to avoid that would be to quit. Sure, no one would commit this exact offense again, but (a) the practice will (would, if any conviction actually happens, big if) be changed just enough to make it legally ambiguous again, and (b) the law would probably be changed to make it legal.


There's a third option, which is: to not quit, but fight back against legal-but-immoral practices from within the corporation.

Have you seen The Incredibles (pixar film). This scene is exactly what I'm talking about:

https://www.youtube.com/watch?v=O_VMXa9k5KU


Yes, you might get the odd Schindler every now and then who tries to do just that, but most are probably in it for the money and not to fight some uphill battle.



> Dear software engineers, please do not commit a crime

Yes, developers shouldn't knowingly write code to commit crime, but developers don't tend to receive instructions that directly. Unsurprisingly, the company doesn't mention to every employee that they are knowingly breaking the law.

Instead, developers receive a request to build a feature, and it typically won't be at all obvious that the intended use of that feature is to commit a crime. There might even be a legitimate use of the feature, and then someone finds it can be abused to commit a crime.


Sometimes it may not be obvious but the feature still might seem super suspicious. For example, suppose that the malware discussed in this article was broken down into two sub-features assigned to different people: geofencing detection, and bricking the train. The person writing the "bricking the train" part should have realized that there is practically no legitimate reason for that code to be written, and if they ask their manager for a reason and are told "don't worry about why, just write the code", they should report this suspicious activity to law enforcement. There are many reasons that law enforcement would want to know, including that the engineer's manager might not even be acting in the company's own interests but might have taken a bribe from a hostile foreign power.


> The person writing the "bricking the train" part should have realized that there is practically no legitimate reason for that code to be written

Hey Janusz, can you build a safety feature that prevents the train from operating under certain conditions. We don’t know all the conditions yet, so leave it flexible.


Even so, wouldn't someone still have to write either an if-then statement, or a database entry, to connect the geofencing capability to the bricking capability? Even if that was only a single line of code or SQL, it seems like a smoking gun and whoever did it can't possibly plead ignorance. No one who can operate a keyboard is that dumb.


OK, let's try how it goes:

Hey Czesław, I cannot leave it flexible, because it's a train that can run over 100km/h with 500 passengers inside, so I need to know the details to perform the required safety analysis. All in all, my name will be in the commit log if someone runs... git blame.


Here's a fix for the git blame part:

GIT_AUTHOR_NAME=Czeslaw GIT_AUTHOR_EMAIL=czeslaw@januszex.com git commit

(obviously, just kidding)


For some reason I'm afraid to check if that website exists :)


Yes, a real website. I actually bothered to check.


I think I owe you an explanation, as I was referring to the fact that in Poland "Januszex" is a meme.

Back in the 90s, when market economy started here, many many businesses' names had the "-ex" suffix, as apparently it sounded western.

And Janusz is a popular first name, which became a meme for a specific type of a businessman, I suspect that because years ago when the meme started, many of those businessman actually had that name (i.e. it was very popular in that generation).

Now, when you open januszex.com, what's there is a meme of "Janusz Alfa" [1] (I think it should be self-explicable), and a related long-nosed monkey meme (I guess they could be thought of as "Janusz Beta" ot sth like that).

EDIT: [1] The real person from the photo is a Polish politician. I couldn't find if he somehow "triggered" the whole meme situation, so I guess it was just a coincidence--someone stumbled upon his photo, used it, and the rest is history.


Let's repeat this one for the parts of the peanut gallery harping on irrelevant issues such as whether object orientation was part of the design methodology or SEL4 part of the firmware runtime stack:

"To all software engineers; please refrain from engaging in criminal activities. If you are instructed to do something illegal, it is important to report it to the relevant authorities."


Companies are made up of individuals. I'm all for holding everybody who contributed to this malware accountable.


Unfortunately that is why fall guys were invented. I never liked the idea of punishing a company based on their revenue, but in this kind of case that is the only way to get the actual owners of the company to listen and punish the people actually responsible.


I don't believe the Polish judicial systems has experience in dealing with corporate crime, especially of the tech-related kind. I'm a bit afraid of disappointment here.


Wait, what? You don’t think a country with a population of 41M has experience with corporate crime?


The country does I'm sure but my guess would be that the judiciary probably doesn't because the judicial tools to deal with are quite likely absent or quite recently enacted and because there have been few prosecutions.


38M and no, not this kind of corporate crime. Plain financial fraud - sure. This case is much more complicated though.


Does it have experience in dealing with...sabotage? Specifically, a country that has a war on its eastern doorstep?

I mean - how is "let's mess with something on purpose so that trains won't run" NOT sabotage, since such time as railways exist?


No, we do not have. Sabotage is rare to nonexisting and cases in past were rare and of "teenager builds device to control switch on tram tracks, derails tram for fun" type.


It's funny how, in the western world, as a company, you can commit crimes and take a pat on the wrist, but, as an individual, you get to jail for the same crimes.


It’s simpler than that. A rich company or individual can often avoid jail. A poor company can just fold. It’s the poor individuals who suffer.


Sadly, the general populace didn’t hire lobbyists to represent them. Our representatives were supposed to be built into the system, but that unfortunately made them part of the game, rather than some of the players.


Can’t you create an NGO that will collectively represent and lobby on behalf of the group, hiring lobbyists from membership fees and other fundraisers? Holy hell, maybe create a political party?


Me personally? No, I don’t think I have the connections, patience, or talent for that. If I did I’d probably do it for a big company instead, they pay better than “we the people,” I think.


or once you have enough money to not care about it as much , you can focus on "we the people" but I doubt thats likely .


rules for thee but not for me is not a western invention


An individual did do this. Companies do not suddenly grow arms and brains and learn to code, at least not quite yet.


You could very seriously start a war by doing things like this.


No, they would not. It would be entirely a civil matter that would be resolved in litigation.


A contractor in the UK put a time-lock in the software he was contracted to write because he was concerned about non-payment. He didn't get paid and the software duly stopped working. He was successfully prosecuted under the Computer Misuse Act. He had some justification (unlike the Polish train manufacturer) but it didn't help him avoid prosecution. I've no idea what the law in Poland says.


That’s pretty ridiculous.


Why? Tolerating that kind of behavior is what is ridiculous.

If anything, us software developers should be held liable for more of the damages that we cause.


So it’s ok when a megacorp like Microsoft does it, but if an individual developer does it then it’s a crime? When you fail to pay for Windows, it turns off various features, does it not? When you stop paying for your MMO, it stops letting you play the game, right? So if I am contracted to develop some accounting software and they fail to pay me, why shouldn’t it stop working?

This is why it would only be a contract dispute; they will argue that the contract did not state up front that if they failed to pay the developer that the software wouldn’t work. The MMO certainly has pages and pages of legalese that everyone knows lets the MMO’s developer get away with anything they want. The independent contractor needs explicit language in their contract to specify exactly what will happen if they aren’t paid on time. That should include late fees, but probably also software that refuses to work until someone fills out a credit card payment form.


And if it is a big or even state company we need to save and ensure workplaces, or "hello dear lobbyist with that big suitcase!" :D


> This is actively malicious destruction of property.

Seems more like malicious denial of service, with the goal of enriching the malicious actor.

A motivated legal team would likely be able to find Serious Charges that could apply. Especially if these specific trains / locomotives happen to be "Critical Infrastructure" (not guaranteed).


It's one thing to implement a secret handshake and underdocument some procedures to make your competitors look incompetent, but actively breaking your product when it's in your competitor's shop - that reqires some chutzpah.



Exactly what I immediately thought of as well.


That wasn’t intentional though.


Google fucks up Firefox experience so often. For a company that large, both intentionally doing it and ignorance (eg. not testing on Firefox) is actually malice. Pretty much only company that has this problem.

Go check out GCP web UI on Firefox and tell me it's not intentional.


Ooooops! Somebody put this delay here tooooootally by accident and nobody noticed it when shipping to production! Silly devs!


Totally unintentional, I'm sure.


Exactly, like that time they slowed down only chrome and every other browser was still fast. Oh wait that never ever happened.


*provable, you mean


Title is a bit misleading, because this *gate is not about faking ecology, and trying to pass certification in artificial conditions, as dieselgate was, but simulating fake failures instead. The company hardcoded algorithms that would report failures of parts that work correctly (like a compressor), if it detected that train has been repaired by another company (based on location readings), and stop the train from running


It's an example of fraudulent / malicious behavior found by decompiling industrial logic controllers, with incontrovertible evidence of illegality. Obviously no two situations are ever going to be the exact same, but I think it's clear why the analogy was made.


Yes, but “Dieselgate” is not appropriate here because that term has “cheating” loaded onto it, which represents a different struggle for companies than vendor lock-in. What this company is doing is related to DRM and arguably closer to what John Deere does with its products.


The example here includes faking a compressor failure, which is a bit beyond ‘vendor lock-in’.


yeah - this is basically stealing


> The company hardcoded algorithms that would report failures of parts that work correctly (like a compressor), if it detected that train has been repaired by another company (based on location readings), and stop the train from running

This isn't correct by my understanding - there's actually two separate things here:

- The company made their trains stop functioning after spending 10 days at competing maintenance locations, based on GPS

- In one firmware, they hardcoded to pretend a compressor failure a few days after the next scheduled maintenance for the train


I'm writing from memory so I could be wrong, but based on (IIRC) two articles and some comments I read in Polish:

- The 10 days limit was their initial attempt, no GPS involved

- After it turned out the trains were immobilized in the "wrong" location, they added GPS geofencing

- The compressor failure was supposed to happen every day till the end of the year, starting on the scheduled maintenance date. (This might as well be bad coding.)

Again, I could get some details wrong, especially that the articles I read were kinda all over the place (subjectively, IMHO).


>it is hard to find an institution in Poland that has done anything beyond kindly expressing interest in the matter. We are not aware of any action taken either by the Office of Consumer and Competition Protection or by the Railway Transport Office,

That the worst part of all that.


The government anti-corruption office is formally investigating this now, which means almost certainly people will end up going to jail. The office of consumer protection doesn't have anywhere near the power these guys have.


When the companies see that this behavior is not punished, they'll basically need to implement their own versions of it to stay competitive!


It’s cute that you think they haven’t done that already…


See also the discussion/comments of the previous post of a related article: https://news.ycombinator.com/item?id=38530885


I wonder if the solution to all these screwy engine controls (tampering with emissions testing, preventing 3rd party repairs, etc.) is to standardize the interfaces to these systems so they can be replaced.

Standardizing the outputs of the sensors would let us swap in and out various components to ensure the system is not cheating the regulators.


F1 does this. All teams are required to run the same, approved, ECU[1]. They can change certain mapping tables and such but it's a sealed unit and they can't replace the firmware.

[1]: https://wheelsports.co/formula-1s-standardised-ecu-explained...


This is quite interesting because you can imagine that lobbyists would argue that standardization would “stymie innovation.” If F1 does it why can’t you?


Indeed. Standardizing certain components may reduce some potential innovation, however I've long thought that the public sector would be better off buying modular systems with well-defined interfaces rather than the behemoths do-it-all oh-so-often fail.

At work we're a small team, providing a B2B application to perform a small, but very important task for our customers. We integrate with tons of other systems, at our largest customer we talk to 30 other systems. We're highly specialized and we rely on being good at exchanging data with other systems that are good at what they do.

This allows us to innovate and provide great value for our niche, while the other systems can focus on getting better at what they do, rather than implementing a half-assed solution because it's not their core focus.


F1 is an ecosystem in which the competitors agree to a set of rules as a prerequisite of participating. Among the guiding principles of the rules are "limit spending on aspects that don't meaningfully affect the competition" and "when possible, make it easy to enforce the other rules". Using a single, common ECU (which is both complicated to manufacture and doesn't directly influence performance [0]) saves all the teams (bar McLaren) from having to go down a rabbit hole of doing semiconductor design and manufacturing and makes it easy for the governing body to enforce rules about how the ECU is configured.

[0] How the ECU is configured does, but the ECU itself doesn't


Most of F1's "innovation" is around finding ways to beat the rules, not necessarily coming up with new technologies.


One could easily argue that F1 hasn't innovated much in the last decade or so. The coolest stuff we get is clever aero and advantageous workarounds that get outlawed extremely fast.


You are extremely downplaying the "clever" aero. Remember, simulation time and costs are regulated in F1. The innovation is producing extremely effective aero with minimal brute force simulation.


I'm aware that the tech to do so is really cool. It doesn't make the racing better though. It's not more exciting, the cars aren't really faster because of it because they're limited in other ways, and it's not really more competitive.

What F1 needs is disruption, more options, like too many options that all of them won't be test-able, less standardization. Even the limited sim time is a problem, because there are certainly optimizations left on the table that can't really be found otherwise. What you wind up with is a A-team and a B-team of mostly the same designs. That's booring, and moreso because if the cars really were 100% standardized, at very least it'd be competitive in terms of driver skill.

It's why these days, I don't watch much F1, I much prefer the more 'indie' racing leagues where, on any given raceday, anyone can win. The days of F1 being that way are long gone, and, it's not likely to change at-all.


There are devices for automobiles that intercept sensor data and feed back fake data to the ECU to bypass emissions controls. It's a fairly simple to do.

I have a buddy with a WRX that absolutely should not pass smog, has no cats, big turbos, tune, etc, but it has no codes, passes every time without issue because the sensor data is synthetic that governs those things.


The government emmissions test doesn’t measure what comes out of the tail pipe?


Nope, relies on the ECU to communicate over the OBDII port the emissions status.

There are visual inspections, but, they're not terribly thorough, and they don't disassemble anything.

That's for normal cars mind, for diesel semis/lori's they also have a measure of how much smoke comes out of the stacks at a given load, and how long it lingers in the air, so even if the ECU reads clean, if it's running dirty like that it'll still fail.


Or even just requiring the manufacturer to provide all source code to the customer and the tools to update/replace the software. Would be nice to get rid of the black holes that is firmware and allow for auditing.


It's a bit more than standardizing, since you must also remove the barriers to changing the software. And you don't need full standardization, just publicity.

But yes, it's basically it.


Seems like deliberate sabotage via software to force the costumer to buy the manufacturer’s services instead of 3rd party (cheaper) ones.

Curious to see the court’s decision.


There’s no question that it’s sabotage. The only thing left to prove is the culprit, which is with 99% the manufacturer (motive, means, opportunity) but obviously need to be established in a court who is responsible and criminally culpable.

The fact that lawmakers, courts and the public are lost in the tech is a problem, but surely this crime can be fitted into existing criminal code against sabotage… although the methods are “new” the crime itself is classic.


"Lawlessness is the condition in which your adversary refers you to a law he made."


It's almost like you found a lossy encoding for discussing governance.

For example, both the NYC taxi medallion system and warlords controlling a city in a failed state would get input as "lawlessness" in your encoding.

But then if I ask what is the quality of life in each instance, I can't get that answer because there aren't bits in your encoding for that.


My impression is that the quality of train firmware is generally not very good, and I hope that this scandal will lead to greater scrutiny. 3 years ago, Deutsche Bahn publicly complained of "grotesque" software problems with newly delivered Bombardier trains. For example, when train drivers changed the direction of travel, the train software would crash. It then took 1 hour to boot the train up again [0]. Switzerland had similar problems in 2018 [1].

As a computer scientist, I find this embarrassing. Just compare these modern trains to the old trains built in East Germany [2] during the 80ies that were pulling old West German carriages [3] from the 50ies here until recently. Minimal or no usage of digital electronics. No "boot times". They just worked. And if they didn't, the train driver usually knew where to hit the engine with a hammer to fix it. You cannot expect a train driver to hack into the train firmware and fire up gdb to find out why it doesn't move.

[0] https://www.sueddeutsche.de/wirtschaft/deutsche-bahn-ic-1.47...

[1] https://bahnblogstelle.com/33872/twindexx-swiss-express-soft...

[2] https://de.wikipedia.org/wiki/DR-Baureihe_243

[3] https://de.wikipedia.org/wiki/N-Wagen


I think the compensation given to software developers by companies that view software as their product has drawn many of the skilled software developers away from jobs that would have once grabbed them because of the fun factor. Companies that make things that contain software are not in markets prepared to pay 2 and 3 times what they were for software. What you are left with is people who are willing to accept that fun factor as the difference in TC, and people who couldn't get jobs that paid more. These are the people we have making most of our safety critical systems. Go look at software developer compensation at X vs spaceX. That is the market at work. Fun does count as TC, but you also end up with people who aren't good developers pivoting to engineer new processes and tools in these domains. They latch on to whatever fad full stack is just getting over a case of, and try to apply it to train firmware. It wouldn't surprise me to find out they are all about scrumfall and have 10x more text in jira than git. And they have restful apis, or service oriented architecture in a safety critical embedded system.


I think you are correct. Because the pay is so miserable the talent pool is mostly vba developing engineers from inside the company. Because of that they can’t hire good technical leads that know or can enforce good practices or design good architecture. The result is a giant mess of software in trains planes and automobiles


Some of the problems here might have been logic problems by inept coders; however, the underlying theme of this scandal is corrupt management. Even the erroneous code was an explicit piece of fraud that almost certainly was done under order by someone in the management chain.


That is true. You can get few times more money as regular Spring Java developer making CRUD in some bodyshop than writing industrial software for local Polish company.


> Companies that make things that contain software are not in markets prepared to pay 2 and 3 times what they were for software

The quality of SW has nothing to do with the pay. Notice that FAANG SW developers do not deliver safety critical SW.

There are more things to SW development than writing code.


If I had a choice and had to choose between (1) high-pay, not-safety-critical at FAANG, vs. (2) low-pay, safety-critical, I can't imagine why I'd go for (2), outside of maybe hobby when I had too much money already.


Speaking from experience in vehicle firmware, the controller component belongs to a separate profession.


Facebook, Amazon, Netflix, Google, Twitter are not selling software but they are able to attract a lot of skilled developers


They all are selling software, with SaaS model.

You can compare this situation with Boeing. And issues they had with software of 737max.


Because this software is not made by software engineers, it's made by plc programmers, electric circuit designers and whoever did drift into the field.

Except for beckhoff to tc3 they haven't made it to object orientation yet, so the field is stuck as a whole in the blue screen mines of yore. Managing complexity with thin standard docs, no version control while the machines grow ever more complex sensor and actuator wise..

You can not treat modern machines like small embedded hobby devices - but the industry does.

Some outside-programmers make good money coming in and solving these yesterday's problems with proper software architecture and good c development practices. But the industries doesn't learn from this. Making software will forever not be a profession for them.


I'm not sure if you've ever used modern software. It's sometimes amazing just how unreliable it is. Web browsers crash every few weeks, windows is known for regularly needing a reboot, evince regularly crashes on me, you can't call 911 with some of cell phones, ... . This reminds me of https://danluu.com/everything-is-broken/ .

The clearest example of the difference of reliability is looking at public digital signage (on transit and elsewhere). If it's based on LED segments or something similarly basic (with old-school embedded software development) it will basically always work. New LCD Screens inside trains/busses and outside working with a modern software setup (using an OS, often with a pc architecture, quite often just displaying a website) are broken ~10%-20% of the time. Looking at (for example) busses, a large portion of the time the screen will either be blank, not display anything, old information or just wrong information. Going inside fast food restaurants with large LCDs for the menu, often something is broken, frozen or something else.

It is of course possible to make modern software more reliable. It's just much, much harder than making embedded software or PLC programming reliable. Software can be easily made more complex, but it's hard to make it non-complex or to wrap the complexity so it isn't an issue anymore. The ecosystem isn't set up for non-complexity.


I think to make software more reliable, you'd have to go back to the "waterfall" method of development.

If we went back to Dijkstra's notion of correctness by construction, then a specification for the program would be made, and then a programmer would prove their part of the code correct to the specification. They would write the precondition and postcondition of every effectful statement, document the invariant of every loop, and prove by induction that each loop does what it's supposed to do. Basically, annotate your program with Hiare triples. (There are books about how to do this). Then, extensive tests should be run for as much of rhe program as possible.

Nowadays, we have tools for this so that we don't actually have to write a proof by induction for every loop; instead, we have bounded model checkers. In theory, the manual proof writing could be isolated to the parts of the program whose properties a bounded model checker cannot verify.

However, it seems like this whole plan is infeasible unless regulations are written that enforce this onto the industry. It would make them a lot less productive, and therefore less profitable. The only benefit would be that software is more reliable. By necessity, it would have to become simpler, too. For instance, there's absolutely no way that web browsers like Chromium, with 38 million lines of code, will ever be verified, because they're too large and complex.


Such regulations exist for avionics and aerospace. They were written in blood.


> I'm not sure if you've ever used modern software. It's sometimes amazing just how unreliable it is. Web browsers crash every few weeks, windows is known for regularly needing a reboot

I'm not sure if you remember older software properly. The idea that an entire desktop computer, much less a web browser, could even run for weeks without a reboot would blow the mind of '90s me.

Netscape Navigator was the direct cause of at least 80% of the Mac OS 8/9 crashes I've experienced in my life. It was less common for the browser to entirely take down a Windows system in that era but I sure do remember seeing a lot of those application crash dialogs until the modern era of tabbed browsing made people care more about stability.

Prior to Windows 7 becoming mainstream I never had to check uptime when diagnosing weird behavior on end user computers because they just naturally got rebooted regularly. Now there are a lot of computers that only reboot when they receive updates that require one, and depending on the user might not get those updates very often. I see systems that haven't been rebooted in the better part of a year and some times longer every few weeks. Even more now with Windows 10+ "Fast Start" where "Shut Down" actually means hibernate so users who think they're shutting down every night actually haven't had a fresh boot in months.


Yep, I don't eat fast food nearly as much as I used to but whenever I go in to a place with self-service ordering "kiosks" one or more of the kiosks is often out of service or frozen up, sometimes with a Windows error screen, or just stuck in a reboot loop, or it randomly resets in the middle of entering an order.


There are trains (Polish ones, funnily enough) that will happily show you the "choose the location of this network" dialog from Windows7 on their passenger information screen.


> Because this software is not made by software engineers, it's made by plc programmers, electric circuit designers and whoever did drift into the field.

There are more "engineers" writing software for your car or a train than "engineers" at Microsoft, Google, Apple or Facebook.

I don't think that someone will be happy when driving with 100 km/h on a highway, the car will suddenly decide to restart itself. There are bugs everywhere where profits are put before engineering but calling those people names is not constructive. Especially when they use SW created by "engineers" which crash with no apparent reason when they are doing their work.


> ... they haven't made it to object orientation yet, ...

Not always a blessing and I've actually recently been thinking (e.g. in context of Lua) if object orientation is in most situations not better to avoid.


100% agree with this. IMO there are a few efforts to modernize PLC programming but I feel like they are still stuck in the 1990s software development. Take a look at Codesys, got Git support few years ago and in very bad shape. How do you test your code, in the field or buy another Codesys testing plugin....which is in rough shape.

The issue is as machines get way more complex this issue gets worse. Also there are generations of PLC devs that still want to stick with ladder logic. Huge fragmentation.


Ladder is a fine choice for visually animating Boolean logic

The fact is the unmodernized plc platforms deliver working machines and plants. The projects are built, commissioned, and unlike modern software they are finished and left to operate instead of constantly modified With no benefit to the end user


... sort of. Past a certain level of complexity you get into trouble with just PLC programmers alone.

Don't get me wrong. <PLC programmers> tend to also have electrical skills and some are pretty good at their jobs, and you can learn a lot from them!

But ... they're not programmer programmers. It's a different skill-set with only so much overlap.


The ladder paradigm has its place, as does function block diagrams, as does what the plc vendors call ‘structured text’.

controlling physical processes like oil refineries, power generation, or sawmills is different from most computer programs as are the consequences of bugs or errors so it makes sense that it takes different people with different interests and skills.


I can't do the nuances justice here for sure. But to make a long story short:

PLCs replace relay circuits for Industrial Electrical technicians (hence: Ladder). And make no mistake: relay circuits are important, and is something they are very adept at!

However, this doesn't imply that they are automatically able to write or debug regular Python or C code or what have you.

-----

PS. If you let a PLC programmer at any language, they'll find a way to do ladder logic. Illustrative example (combining some features I've seen in textbooks and IRL, formatting included):

  if Tank_Level_Reached == True and EF132121 == False and Ready_To_Start == True and Finished == False and Belt == Not_Ready and Stage == 5 and Position_Left_Arm == Up :
    move_left = True
    move_right = False

   if Tank_Level_Reached == True and EF132121 == False and Ready_To_Start == True and Finished == False and Belt == Not_Ready and Stage == 5 and Position_Left_Arm == Down:
   move_left = False
   move_right = True

   if Tank_Level_Reached == True and EF132121 == False and Ready_To_Start == False and Finished == False and Belt == Not_Ready and Stage == 5 and (Position_Left_Arm == Down or Position_Left_Arm == Up):
    move_left = False
    move_right = False
etc ...


Then why is nobody using assembler for modern software projects?


Sometimes low-tech is just better. Here in Finland we got Sr1 electric trains from the Soviet Union in the 70's, and after some renovations the model is likely to stay in use at least until 2030.


Simply of old designs is often a blessing as long as the drawing and documentation is readable and good. It can be hard to get replacement electronics for 1970s designs so sometimes you have to design new components but the functionality was relatively simple back then so it’s possible to build a 1:1 replacement


They've actually been modernized with newer power electronics and some of the soviet oversights have been addressed. They're still very reliable, and now somewhat more efficient.


Old electronics of that era could be drawn by a simple schematic and usually only performed one or 2 functions. That makes designing a drop in replacement very easy.


That is also my impression as well. The softwareization of trains has led to deep regressions in both basic reliability and interoperability/flexilibity. Many modern trains suffer from software issues for basic driving [0] and delays when getting the software approved [1]. But the loss of compatability is in my opinion the worst regression. Modern EMUs basically only work together with other EMUs of the same batch. Even the same model ordered by two different companies often don't work together and basically forget about trying to use EMUs of different companies or ordered over a decade apart together. Meanwhile pre-digital everything it was common to use e.g. trams of different generations together and rewire them to work with each other. Older train cars work together without issues, good luck trying to use an IC2 and a Railjet together (or a RailJet and ICE-L). Even certain locomotives and train cars would often only work with each other.

It is way harder for different computerized systems to work together due to the higher complexity and more obfuscation (a traditional logic circuitboard is often easily reverse engineered. Reverse engineering software is a very specialized task). This is also very noticeable in other sectors, where interoperability has become much worse due to moving to proprietary digital protocols.

This is in part due to the difficulty in getting software approved as compared to previous tech (due to software being so intransparent) but also because of truly lacking quality. One of the reasons Bombardier was so deep in trouble was bad software, even leading to a contract of over 40 ordered trains just being cancelled ([2]).

In my opinion building reliable (and understandable) software is way harder than building logic or even mechanical systems. I don't know what the solution is, but it's been a problem for a long time.

[0]: https://www.vrt.be/vrtnws/de/2013/02/12/belgische_bahn_storn... [1]: https://www.augsburger-allgemeine.de/augsburg/Neue-Zuege-auf... [2]: https://de.wikipedia.org/wiki/Bombardier_Talent_3#%C3%96BB


Except that this isn't really a story about poorly written software; it's a story about corrupt management. Further, if we look at Boeing's recent issues with the 737Max, it's the same thing. In both of these cases, the bad software was almost certainly ordered to be written by management acting fraudulently for profit. The one error that has been discussed in the article was a stupid mistake, quite possibly due to the logic conditions being made overly complicated in order to enable the fraud, but the recurrent theme of all of the real underlying issues found was intentional design malfeasance, not incompetence.


You could say that about most software where the fresher the framework the more glaring the holes - here's a recent post about it: "Software disenchantment" https://tonsky.me/blog/disenchantment/


> Minimal or no usage of digital electronics

Everyone in this thread seems to be forgetting that those might be useful and not just fancy toys. I prefer trains that have digital signage indicating their location, and connections at the next station. Higher level of automation in trains (e.g. Communications-based train control) also drastically increases efficiencies in speed and scheduling, allowing more trains on the same tracks, and minimises time wasted waiting or accelerating/decelerating needlessly.

The problem is poorly implemented software, not the existence of software.


Oh, the 243..

You know? Shortly after German reunification they appeared in NRW in front of, or behind (pushing them/'Wendetraktion') the Regional-Expresses, S-Bahn, doing the heavy lifting of mass-transportation there, on all or at least most important relations in the Rhine-Ruhr-Area. Not that much later we got DOSTOS there, but brand-new ones.

I remember wondering why that was at the times, and that they couldn't be bad at all, because otherwise they'd break down more often, leading to delays. Which I rarely experienced at the times, as almost daily user.

Die verdammte Scheissbahn war da noch in Ordnung!


I don't think fixing the software failure will improve DB's punctuality


What if it's caused by random reboots lasting up to 1h? :)

Of course I'm half joking, but I wouldn't be surprised if software problems contributed to the delays (which tend to chain, as trains need to wait for each other, etc).


Curiously, I was riding a TGV in France few weeks ago, and due to some track problems, we had to change the direction and go back to previous station (and then use different track to go forward again).

It took more than 1h to start going back, and I wonder how much of it was making sure the track is clear, and how much was "rebooting the train" or whatever else was necessary.


I'm guessing that these days anything might need to be rebooted, including track themselves (I mean: there are misc detectors there).. Sometimes I'm in awe how strong the overarching duct tape is :)


In the case of Bombardier, I suspect contracting also contributes to the problem. The same for financial institutions.


> No "boot times". They just worked

Haha wait until you find out how TVs worked in the 70s and how fast it was to change the channel *sob*


Even in the 90s, you could just power it on and it would show image near-instantly. Warm-up time and channel switch time were all firmly under one second. With the exception of cable TV set-top boxes, which were separate devices and first to include the ridiculous boot times and delays, that still would seem blazingly fast compared to what we have today...


Those "instant ON" TVs worked by sending a small pulse of power through the tube/valve circuitry. There was a legal stink when a Zenith TV set caused the Texas Capitol to burn in 1983. The "expert" testifying for Zenith claimed that no other TV burst into flames like that. The Texas AG had gotten copies of sealed testimony by that exact same expert in dozens of other home fires. The expert got busted for perjury.


Go back in time a little more and there was definitely "warm up" time for electonics. Tubes had to get up to operating temp, etc. When I was a kid I remember turning on the TV about ten minutes before my dad got home so it would be warmed up and ready for him to watch the evening news.


Let's just say the 70ies to 80ies were peak of consumer electronics, then? With the exception of digital photography, computers, mobiles...


Well, that sounds accurate. Especially when it comes to heavier hardware, such as car, trucks, farm equipment... Digital photography, computers and mobiles are young, but arguably, in 2023, they're past their peak.

My take is this: the peak is at the point when a given product category existed long enough to be thoroughly developed, but short enough that value engineering didn't kick in yet to ruin it.


I used to have cable TV without box (direct stuff via analog cable) until mid-2000s, and man, 15-ish years forward, the order of magnitude slower channel zapping still bothers me as hell.

(To be fair though, nowadays we have UHD/4K channels, which need way more bandwidth than the SD TV quality from 2000s. I wonder if this could be fixed with a more powerful CPU in those boxes? maybe it could, but might increase power usage?).


The immediacy of analog is so nice compared to the constant lag of software.

Audio effects and synthesizers all have software driven versions that sound effectively identical to analog and are typically cheaper. Yet, analog has been hanging on due to the simplicity and immediacy.


Bad software is a symptom, not the cause.


Here it's more like the software - any software - is a problem. I agree with GP, and my experience confirms that adding software to something that used to work without it almost universally makes it worse in every aspect, understandability and repairability being just two major ones. On top of that, taking anything that run on old-school industrial/embedded firmware and replacing that with software using modern practices and stacks of the software industry, 100% makes the product go to shit.


The conversion is pretty much fundamentally corner-cutting of some sort or another. The digital equivalent is usually a micro-controller worth a few cents replacing dollars of bespoke-by-comparison (due to smaller economies of scale) hardware cost. The goal for the exercise is almost always "good enough" instead of trying to best the existing State Of The Art. Power usage I think tends to be one of the few aspects usually improved via digitization.


That again is a problem of leadership and not of software.


> That again is a problem of leadership and not of software.

SW has an input problem and a testability problem. On one hand, the inputs to the SW are not limited (iMessage happily accepts any image file) and testing is limited to some known inputs. Software vulnerability assesment (worst case analysis) is usually performed outside of the development process at very high costs and limited outcome.


What is the cause then?


Bad culture that views software as a necessary evil or afterthought rather than an important part of the product.


Same as industrial design then. You get the occasional Braun, Herman Miller or Apple, and a vast number of nondescript silver/beige/black boxes.

It's probably true of lots of aspects of product design - if it's not driven from the top, it's mediocre.


Yeah, unless the engineers are using the product themselves. People in general seem to take care of their own tools. Much harder to get them to look after a product they don’t use themselves.


I'm solely consuming EN content. And if it's from another country, it's only whats leaked by big media. It make me wonder how much good content could be translated.


I think you're saying, how much other good content is out there that I'm missing out on because I only read English, and it's a good point.

However, English has become the (now ironically named) lingua franca of, at least, the more educated parts of the world, and many people who are most comfortable in their native languages are still often translating their best work into English in order to see it more widely read. This is often the case with scientific papers, for example.

Perhaps England's biggest gift to the world was its language.


Worldwide colonialism wasn't exactly a "gift", but I must admit it has been advantageous to me, personally, for English to be as relatively universal as it has become as a result. ;-)


> (now ironically named) lingua franca

The original language which was actually called wasn’t really French it was a creole/pidgin language used in the Mediterranean mainly based on Italian and Occitan dialects.

Greeks and others just called all Western European Franks even though they didn’t really interact with people who actually spoke French (only used in the Northern half of modern France back then) that much.


I'm from Poland, and there's a chance I was going by those specific immobilized trains--I lived in Wrocław and used Koleje Dolnośląskie (Lower Silesia Railways) to "scout" the area (there are some beautiful places around Wrocław).

And I only learnt about this whole train "vendor lock-in" from HN. Otherwise I either wouldn't know about it, or I would learn about it weeks or months later.


Newag issued a statement since, denying all allegations and saying that it was their competition which "hired hackers to slander them".

I've met q3k because we used to work at the same company and briefly on a project together. Not the kind of person I would suspect of participating in a conspiracy of this sort and Newag's statement generally reads like "we didn't think we would get caught".


While I haven't met the guys in this case, I am familiar with their work.

Additionally, I am fairly certain they are not stupid enough to not have kept detailed, forensic-quality records of their actions and whatever they dumped. Sure it may not stand up in court as evidence but it will be more than enough to show that they didn't pull this out of nowhere


Unfortunately for Newag, other than in the court of public opinion, firmware deliveries count as written evidence.


Part of their statement says(loosely translated):

"No hacker can tell, based on the content of the digital record alone, who is the author of the digital record in question"

Boy oh boy. Either they're not singing their firmware (which is a serious indictment in and of itself) or proving that it was them all along will be trivial, but the ones signing off this message are unaware of this.

Overall they got caught with their pants down and handling it badly as evidenced by the fact that they don't even have a scapegoat prepared.


"Pants down" situation aside, if the firmware is not signed or verified in any way, then isn't it prone to "neutrino bit reversal", potentially causing Bad Things?

I have no idea how those systems work and what guarantees they provide, but this would be hair-raising...


You can have checksums without signatures. Not that either a commonly checked after installation so most systems are still vulnerable to random bit flips.


^I think this is being downvoted because of poor reading comprehension skills. Please note that the parent comment is in favor of the hacking group.


Thank you for pointing this out - I reread the post and can imagine now how someone would read it differently than I intended.


I read and reread it and IMHO it's perfectly clear.


Dieselgate isn't a good comparison because in Dieselgate the equipment functioned normally from the user's point of view.


I think it is because the equipment changed dependent on context. In Dieselgate the cars changed their engine management when they got into a test cycle...


Dieselgate was about cheating environment sensors. This is more like DeereGate, locking out external service shops but even when you are supposed by law to allow them service (and even after providing them 20k page service manuals which they are supposed to follow to make appropriate service, but you lock them out anyway).


From my reading, they also seem to have seeded apparently random failures into the product, with a hidden reset key combo, even for those using them as support. Possibly to make themselves look good (our products may break randomly, but at least we fixed the "problem" quickly) going into the tendering process for support.


That Watergate building name had created quite the meme!


I thought it was a good reference. In both cases, the manufacturer placed illicit, hidden code that that could (and probably should) get it in trouble with the law.


And under test it even performed “too well”


functioned normally? Intoxicating people with fumes is hardly what the user wants.


Some car users even do special modifications for "rolling coal", so that they can intoxicate other people with fumes.


Great advert for free and open source software.

As with dieselgate, this suggests you basically cannot trust anything containing software. Can't trust it to follow regulations. Can't trust it to do its job.

Can't trust the software. Can't trust the institutions that write the software.

All very "late stage capitalist software development".


Hell, even if governments are squeamish about requiring code to be fully open and public, they can still require the manufacturers to privately submit to the government all code that powers public infrastructure (like trains), to be made available to any relevant party upon request.


An organisation that is prepared to write "sabotage" software would have no problem deploying software that is different to the software they submit.


Implying that's an impossible obstacle. Reproducibility is a thing.

Make it so code needs to be reproducibly buildable. Only reproducibly buildable artifacts can be deployed on hardware. Document the whole process.


Compile the code yourself?


Right. Mandate that the software is delivered with CI pipeline running in the client's environment with 100% reproducible builds and verify checksums.


Doesn't mean it's not a step in the right direction. Any transparency is better than zero.


> can still require the manufacturers to privately submit to the government all code

I wonder if companies purchasing trains could put code disclosure in the purchase contract? I wonder if, in aggregate, train purchasers or car purchasers could fund an independent code storage vault and pay a small premium to fund that code vault organization?

In other words, if purchasers wanted this and valued this, they would demand it in purchase contracts and fund it.


code escrow in general should be much more common.


then you just need to bribe the code reviewer(s). open source is still the better answer, good luck bribing every member of the public who could potentially read public code.


That would work only on paper. The financial interests involved are huge.


All the more reason for governments to insist.


I'm all for free and open source software, but what would you suggest here? That train operators will download code from the internet and install it on their trains?


Clearly not. A reasonable expectation might be though that if you want to sell your multi million pound products to a captive public sector, you have to publish all the source code and the means to build the binaries.


Open source means just that - it doesn’t imply one sort of distribution mechanism over others.


Yes. Once it's signed by somebody accredited to review it for safe train use.


The same way technical diagrams for roads, bridges and other public infrastructure are public.

In most OECD countries food needs to be labelled with a full list of ingredients.

Your GP can read scientific papers about the efficacy and risks of a new treatment.

(Yes, many papers are paywalled but that's irrelevant compared to secrecy)


It doesn't actually need to be open source. If they published binaries that would be enough to analyze.


How is this different from companies like Apple or John Deere that DRM components and brick the device if repaired by "unauthorized" technicians?

(I think both are equally egregious personally, but I know there's a lot of support here for Apple, so I'm curious how people reconcile these. I don't want to make this a religious war about Apple, but those practices in general regardless of which company is doing it).

Is it the secrecy that makes it different? i.e. if the train company were honest about it then it would be ok?

Or is it the scale that matters? Trains are big and expensive, while phones are small and cheap, so it's ok? (that wouldn't work for John Deere but would for Apple)


> Is it the secrecy that makes it different? i.e. if the train company were honest about it then it would be ok?

IMHO mainly that and clearly those trains are required to be designed in such a way that they could be repaired by a third party (either by law or by contract based on how the situation is described).

Apple provides (nor is required) no such guarantees. Also it has more or less legitimate reasons for its design decision (making it harder to reuse stolen parts).

> equally egregious

I certainly disagree almost completely. With Ape you know what you’re getting and can make an inform choice. Also it’s a completely different type of product. Trains have various regulatory, safety and maintenance requirements which are irrelevant for consumers devices. Screwing with the software controlling trains can literally kill people..


You make some pretty good points. Especially the "you know what you're getting" is very strong. Basically the difference between fraud vs not fraud. Thanks!


An angle you may not have considered is passenger safety.

Imagine if this happened to an airliner in flight: there'd be criminal charges for sure, not to mention huge damages and lawsuits from the families of the dead if some of the control systems locked up in mid-air.

Trains are not quite as susceptible to disaster arising in the course of operations as airliners, but a Newag Impuls 45WE runs at up to 160km/h in service with up to 218 people on board. (Their speed record is considerably higher.) A sudden breakdown in service is at a minimum going to cause timetable havoc and knock-on delays for other trains and at worse could lead to a mass casualty accident.

(John Deere tractors don't usually carry 200+ passengers and Apple computers don't usually get deployed in safety critical situations. So, different!)


Thank you for putting it so clearly! I feel this angle got somewhat lost in the whole discussion.

For instance, has anyone really analyzed, what would happened if the trains were running at 160km/h at midnight, on Nov 21? I mean, they might have, I don't know. But that would be pretty funny--having documentation for what's very likely a tort (well, it's Polish counterpart), if not an outright criminal act.


I'm not a fan of Apple's practices, but there's some aggravating elements to this. Apple doesn't brick your device if it it spends time at a repair location, for instance. Apple also doesn't simulate failures on synthetic dates to force repair.


I wouldnt be so sure, one can think of https://en.wikipedia.org/wiki/Batterygate as an example of a little bit of "bricking"


While I don't think Apple handled that well, the intent is clearly different. In fact, the situations are so different, that I have to wonder if you are a troll.

In battery gate, 1-2 year old devices were starting to reboot at low state-of-charge (typically <30% battery). Apple issued a software update that fixed the reboot problem... hmm, how does software fix what smells like a hardware issue? Well, the underlying cause was that aged batteries could not supply enough current and would brownout the CPU (which caused the reboots). The fix was to throttle the CPU a low SoC, which avoided the brownout--so they "fixed" a real problem that I experienced.

My feeling is that Apple owed customers like me some compensation. But, it is clear that the performance throttling was not just an arbitrary "fuck you". Rather, it was a misguided attempt to save the cost of warranty battery replacements in a way they thought customers wouldn't notice.

The current situation couldn't be more different. Here, the manufacturer has added software from the factory to create fake error codes. The hardware is working perfectly fine, but when the train sits in certain locations for too many days, it will pretend to have a hardware failure. During certain months of the year, the train will fake a hardware error. There is no possible explanation for this, except as a "fuck you" to the customer who wants to use 3rd party service.


> it was a misguided attempt to save the cost of warranty battery replacements in a way they thought customers wouldn't notice.

Sounds like fuck you to me.


Functionally it is similar, but trains are very critical civil infrastructure. In the case of John Deere such fraud can also have a serious impact, but it does not affect the public in the same way.

If they want to explicitly claim exclusivity on maintenance or a certain enforced product lifetime, fine, it is a nasty practice but fair enough. But not making the operators aware of these conditions, when they knew months beforehand what would happen when they lost the tender, and while it was seriously affecting the public later on, that is criminal in a way that is not comparable to Apple's practices for instance.


UPDATE: Polish train maker denies its software bricked trains maintained by competitor: https://news.ycombinator.com/item?id=38579098


This kind of thing reminds me of 737 max debacle.


Outside the obvious issue in this article I found the following statement horrendous:

> it has to be taken apart, the parts sent to the various manufacturers, checked, sent back, the train put back together again and tested

Instead of having one public company mastering the art in its entirety everything is split with contractors. A good example of a successful way to do that (but slowly dying thanks to capitalism) is SNCF operating everything in a massive warehouse https://www.youtube.com/watch?v=SeRH2M2Z-ms


But was this sabotage by an insider at the manufacturer, or something deliberate by the manufacturer?


There wasn't just one manufactured failure, but multiple different ones. Refusing to help would also point towards intentional malice. Why would you sell a product, then refuse to assist, unless you've intentionally designed the product to fail so only you would know how to make it work again?


The manufacturer lost the bidding process, so quite reasonably (if you look at it in a limited fashion) said "Fine, let SLS do the work, you're on your own".

Arsehole-ish, but not illegal. All the hidden lockouts on the other hand....


The hidden lockouts containing GPS coordinates of competitors' repair facilities should be more than enough to establish criminal intent (in my armchair non-lawyer opinion).


They knew that SLS would not be able to do it.


I don't think it's assholeish for someone who's not getting compensated in any way to not help out. It's a business. They have an active incentive to NOT help.


It's not about "not wanting to help". It's about placing logic bombs of "if vehicle is at this gps coordinates of a competitor, engage self-destruct". Hackers actually did extract such coordinates from train firmware.


unless we have the entirety of the context for this code and the 20,000 pages of service manuals, i do not accept at face value that it's this simple


Any kind of GPS coordinates, especially those of competitor facilities in the firmware of a train is proof positive that something really bad is going on.

Context and manuals are just so much smoke and fail to obscure the facts.


Considering that the situation this was named after had _very_ specific timing, state and sensor values coded in a defeat device, I'd say that having the mapped the gps coordinates of your competitors im the firmware of your product is pretty damning.

Nevermind the poorly executed "if day => 21, month => 11, year => 2021", which was conveniently setting a failure which wasn't actually present.

It'a probably not that simple, but it's not that complicated either. If you make something engineered to fail without there being a failure present, that's clear malice.

Imagine buying a car, you own it until the warranty runs out and the the manufacturer's workshop moves (say there was a fire/flood/sinkhole/industrial disaster, and they had to) and the car would refuse to move since it's not being serviced at the official location anymore.


There’s literally a hundred reasons why code like that could exist. My point is there is probably another hundred thousand lines of code and we have no idea how the few lines we see are being used.


To what end? So they can sell more trains? That makes no sense.


It seems like the trains were programmed to cease functioning if they spent more than 10 days at the GPS coordinates of maintenance shops not owned by the original manufacturer.

This would force the government to rely exclusively on that manufacturer to then fix these trains and perform all future maintenance.


They wanted to prevent third party repair services from being able to repair their trains, so that they could keep those maintenance contracts for themselves.


After sales support, as in spare parts and maintenance, is a big part of income for manufacturers of heavy equipment, as such machines run for a loong time given parts and maintenance. To me they really did not want to lose on 'subscription money' in the form of service contracts they missed out on. It came close to the operator coming back to them to fix the trains 3rd party seemingly couldn't.


>The train manufacturer, Newag, also competed in the tender to carry out the maintenance, but the manufacturer’s bid was about 750k USD higher and the tender was eventually won by SPS, which offered to carry out the maintenance of 11 trains for around 5.5 mln USD.


Just thinking outloud. But if you made it so your competitor couldn't fulfill their servicing contract, then the entity taking out the contract might just very well come to you to solve the problem. You might not win the contract on price, but win it by default because you made it impossible for anyone else to complete it.

That is until your scheme is uncovered because you left the GPS coordinates of your competitors workshops in your code.


More sanely (not to be confused with likely!) the courts will decide that since this is something only the OEM can do, it must done at no charge as part of normal warranty work.


These trains will be used for decades. Normal warranty wont cover anything of note.


Warranty should cover this - if the manufacture won't let it be fixed by someone else than in should be free.


Every once in a while there comes a point where the discussion of high-currency-shorthand pops up:

>5.5 mln USD. U$5.5m? Not saying I'm more correct than anyone else, but the former seems outlandishly long.


mln is from the Polish original.


5.5 millidollars?


Vendor lock-in for maintenance has massive financial incentive, as was relatively clear in the article, even going so far as to cite some explicit numbers that are relatively big money when projected across the scale of an entire fleet.


How does that make no sense ? That's the whole point of a business.


The idea that does not make sense is that this would increase train sales, not the idea that selling more trains would be good for business.


Considering that the "sabotage" was intended to bring the company extra revenue by having non-faulty parts replaced and by requiring maintenance to be carried out by them and never third parties, it aligns with the company's own interests too much so to say "some employee did this without authorization".

"Deliberate by manufacturer" 100%.

The scarier part is that had this happened in the United States, DMCA would likely have protected them from prosecution, and the government might be liable for damages.


If the manufacturer did it, doesn't it still fit the definition? It's something like "deliberately causing something to fail", regardless of who does it.


While I believe intentions were malicious, it's very easy to argue that

1. it's not failing, it's disabling

2. it's a safety feature - "SPS can't safely maintain these trains, so we have a safety lock out if they attempt it"

3. there is a ton of stuff that works this way - even Harley Davidson motorcycles require authorized maintenance and the bike's computer won't accept repairs unless a proprietary tool is used


Newag was required by contract to provide accurate service manuals so that competitors could safely maintain the trains. This was not a "just take your car to dave, he knows some stuff". For SPS and other competitors this was like "you need to show every certification that exists and certify all your tools to prove that indeed you can service those cars, or you will be foreclosed due to fines". Plus, they were provided ALL service manuals, like 20k pages to follow to the letter.


i wouldnt be surprised if this info was somewhere in those 20k pages, and perhaps if the procedures were actually followed, stuff like GPS based lockouts wouldn't happen


Ah yes, on appendix 35 of section C, "do not store the train in your service yard specifically or it will stop working"


The article covers this, and says the information about the lockouts was not in the manufacturer provided manuals.


According to who?


Directly in TFA, matey:

  > Newag explains that the train were
  > blocked by a “safety system” – but in
  > the 20,000 pages of  instructions, it
  > is in vain to  find even a mention
  > of it.
No mention whatsoever in the maintenance documents. It then becomes prudent to question the intentions and fitness of the company behind such a product.

This episode puts even John Deere to shame. I'm imagine JD are enjoying themselves right now on this Friday afternoon.


On #2, that's sabotage. Also, on #3, that's sabotage too.


The higher-up the decision went, the worse it is.


In a properly functioning country the responsible persons should already be imprisoned. Some governmental agencies were aware of that for at least half a year, but failed to act. The fact that source code was not immediately dumped and analyzed is the evidence of malevolence, corruption and intentionally putting people's lives at risk.

Welcome to the dark side of Poland - where citizens don't matter.


"the responsible persons" ... hmmm. Who would that be?

The programmer who implemented the code? Do you think they thunk these tricks up? They was just following orders.

The manager of the programming team, who set these tricks as things that needed to be implemented? Again, just following orders.

The "Cxx" Title people who directed that there be "some protection" in some way that got implemented as what we see? Did they specify these measures? Did they say "it should break if serviced by a competitor?" Unlikely. Thye wouldn't know how to be that specific, probably.

Some middle manager, maybe a committee meeting, sketched out a "DRM" scheme with the specifics? What do you imagine that meeting looked like? "We've got a directive to secure the systems from outside tampering, what does that mean in terms of how the machine behaves?" Or does that bring us back down to the engineers again?

... the responsible part here isn't a person, its the company as a whole. Just as it took the collective efforts of everyone to make the train, it took their collective efforts to make it wrong.

Corporate Death Penalty; perhaps. make it plain that we will no longer tolerate sill shenanigans like this.


> "the responsible persons" ... hmmm. Who would that be?

But that is the thing. We do not know who is responsible without an investigation.

We don't need to guess. The local responsible agency should get a warrant and take a copy of their code repo and their internal comms. And then they need to spend the time (call in experts if needed) to figure out what happened and who was involved.

If it is normal code development you can find all the paperwork which documents the change. If they tried to disguise it, (which they might have, or might not) then that is some maffioso stuff and you take the tools police use to break up organised crime groups. You take a low level person who you can incriminate and you flip them. You show them that you have enough to send them to a prison for years and offer them the opportunity to cooperate.


> They was just following orders.

In itself that’s not enough to be considered innocent.


IMO responsible is enginer and everyone up management chain + possibly peers if engineer part of a team. And those people should have a trial.


> Again, just following orders.

I seem to recall there was a trial in the forties of some relevance to Poland about this sort of thing.


That would be at least everyone knowingly involved with who is a professional engineer.


Everyone in FTX except SBF should be innocents then. They was just following orders.


Well, justice takes time and this is a complex novel case. I would rather have a system that is right than prematurely put innocent behind bars. However, if the allegations turn out to be true, which seems to have a decent probability, they could charge them with a criminal offence.

There is Article 254a in the Polish Penal Code. If you obstruct critical elements of infrastructure such as trains, you can face between 6 months to 8 years in prison.


Corruption is not just the dark side of Poland, but the entire west IMHO


The west is the least corrupt part of the world.

https://en.wikipedia.org/wiki/Corruption_Perceptions_Index#/...


It baffles me that people don't realize just how bad it is in non democratic countries. Russia has FSB extorting shop owners for protection money, and even an occasional assassination is nothing particularly interesting there. Chinese companies have party cells in management. Venezuela or many African countries need to hire foreign contractors (sometimes Western :) ) so that their heads of state and other VIPs do not get killed by their coworkers. The Red Sea has actual pirates. Lebanon failed to remove kilotons of explosive ammonia nitrate for years, until it eventually blew up the capital. I could go on and on, but you can see how this compares to "train company bricked a train and it's a major scandal".


perception != reality


There are reasons for why we can assume that the west is the least corrupt area of the whole world.


But for the average person, perception does influence how likely they are themselves to take part in corruption, and that in turn does influence the total amount of corruption in a country. Actual corruption is very hard to measure, so perception of corruption is a reasonable thing to research since it is in many ways "the best you can do" to get some measure of actual corruption.


Perception is a proxy for reality, given that you can’t measure the latter.


Note from that wikipedia article

>The Index only measures public sector corruption, ignoring the private sector. This, for instance, means the well-publicized Libor scandal, Odebrecht case and the VW emissions scandal are not counted as corrupt actions.


Because corruption is when state power is abused. When private companies do illegal shit it’s just a crime.


There is no requirement for it to be "state" power, please look up the definition.


Okay, whatever. Pretty sure they private sector corruption isn’t worse in the west than in other parts of the world but if you want to disagree I won’t be able to change your mind


I wanted to get numbers on this, but naturally it's not feasible to get accurate numbers on corruption happening. I found the Corruption Perceptions Index which seems to be the closest we have in quantization. By measuring perceptions of corruption, as opposed to corruption itself, the Index may simply be reinforcing existing stereotypes and cliches though.

But according to their results the "west" has the least perceived corruption.

https://en.wikipedia.org/wiki/Corruption_Perceptions_Index


The other 3 directions are corruption-free?


can't speak for the other continents but there is very little corruption in Antarctica, so I guess if you go South enough, it is actually better


Yes, the west is corrupt. The rest is worse though, and less ashamed of it.


[flagged]


Nixon never went to jail.


And perps WERE arrested and put in jail for the Benghazi attack.[0]

[0] https://en.wikipedia.org/wiki/2012_Benghazi_attack


Seven others did go to prison though.


The programmer appears to be programmed.


[flagged]


The article has nothing to do with pollution.

The reference to Dieselgate might be because a geofencing condition was found in the code running the trains that caused certain misbehaviour when they were at the manufacturer’s competitor’s maintenance depots.


Much like detecting if their cars was setup in an emissions testing scenario and changing the engine behaviour.


I accept being pulled up - I should have read the article!


TFA is talking about train software that deliberately renders it nonfunctional when it detects it is being serviced by a third party. This isn't about environmental concerns.


But there’s an end game to batteries given they have a decade+ useful life, and most can be recycled and reused. You belch out the diesel and your only option is to belch out more.


It is clear you haven't even clicked on the article.

Shame on you. This deserves a ban from HN. Just reading the headlines and contributing to a discussion about that article as if you actually read it is deceptive and wrong. It's killing our ability to meaningfully discuss content when people just read the headlines. If you don't want to read articles and instead post about what you think the articles must be based on headlines, go elsewhere on the internet.


> Shame on you.

This is not appropriate.

> It's killing our ability to meaningfully discuss content

Hyperbole doesn't help discussion either. As my sibling comment says, just downvote and move on.


he accepted he made a mistake, chill out. downvote and move on


I did - see below!



No, it isn't. Six points with no comments don't count as already discussed.


But why wasn't the new submission merged with the existing one? The urls are identical.


Nevermind malware, not using seL4 should already be a crime in this context.


We have rovers on Mars and satellites and probably nuclear warheads using RTOS of all kinds and in cases even Linux, but sure seL4 is the only OS conceivable for those cases, obviously !

This is a case of fraud, industrial malfeasance and just plain dishonesty. The software component of the story and its security measures are not even at play. Sure they are probably shit (given the date parsing ...) but even FreeRTOS would make an amazing OS *IF USED PROPERLY *.


>using RTOS of all kinds and in cases even Linux

Absolutely, and thus there's obvious room for improvement.

>This is a case of fraud, industrial malfeasance and just plain dishonesty.

In practice, this amounts to critical infrastructure sabotage, which fits into terrorism.

If the train network experiences issues, the whole country is impacted.


That seems totally orthogonal; seL4 can run a program that checks GPS and sabotages the system just like anything else.


A train manufactured by a Polish company suddenly broke down during maintenance. The experts were helpless – the train was fine, it just wouldn’t run. In a desperate last gasp, the Dragon Sector team was called in to help, and its members found wonders the train engineers had never dreamed of.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: