Hacker News new | past | comments | ask | show | jobs | submit login
Never trust a system that seems to be working (twitter.com/foone)
497 points by pkilgore on Oct 17, 2022 | hide | past | favorite | 320 comments



Great article, made this exact same mistake in space exploration before learning to transmit demand and not supply. There are so many principles like this I learned from playing through the game. One area I've been focusing on improving is trying to anticipate how any given subsystem might go wrong and add even simple circuitry to detect condition and alarm/signal to the factory dashboard. Has cut down on the times where I realize some part of factory hasn't been working properly for like 5 hours. Applying this to software development has made me better about getting better about how I use rollbar logging


This is why 4-20 mA is a common signaling standard in industrial automation. 4 mA means zero and 20 mA means one. 0 mA means broken transmitter!


It is also why white traffic lights no longer mean "go".

White means "the red glass lens in front of the incandescent bulb shattered again".

Related, from train signals:

> The invention of the track circuit in 1872 by William Robinson permitted automatic block signals. Most simply, a low voltage battery current runs up one rail and down the other. As long as current flows from the battery through one rail to the relay and back through the other rail, the relay remains energized and routes current from another battery to the green lamp of a clear signal. But when a train is present, the circuit is short circuited by the steel wheels and axles of the cars; a broken rail or an open switch will also break the circuit. As a result the relay is opened and the signal displays red. The system is arranged so the failure of any component results in a red indication or a dark signal (which must be interpreted as the most restrictive aspect that signal can display).

https://www.trains.com/trn/train-basics/abcs-of-railroading/...


Train air brakes are similar in failing safely for the same written-in-blood reasons:

https://en.wikipedia.org/wiki/Railway_air_brake


Some military trucks are designed the opposite way on purpose - because they’d rather have no brakes and be able to move than be stuck on the battlefield.


> Some military trucks are designed the opposite way on purpose - because they’d rather have no brakes and be able to move than be stuck on the battlefield.

Do you have any details on that? Or a source I could read? I'm pretty curious now.

I drove trucks in the military - specifically, FMTV, HEMTT, HMMWV, and 5 tons. None of them had the reverse brakes you describe.


The deuce/half air-over-hydraulic brakes fail open (loss of air pressure leaves you with hydraulic brakes alone, loss of hydraulic leaves you with no brakes at all). However, there is a failure mode that leaves the brakes full on if the air cylinder jams.

At the time it was an acceptable trade-off; later military vehicles have more in common with trucks than tanks, I suspect.


Losing brake boost in a failure condition is the typical failure mode in most vehicles with hydraulic brakes. A hydroboost failure in modern medium trucks reduces braking performance just as much as an air pack failure in the deuce and is just as scary..

The brakes, unboosted, are supposed to still be capable of stopping the vehicle. This is also why hybrid and electric vehicles with regenerative braking, that normally use brake-by-wire with a pedal feel simulator, fail through to using the PFS as a direct unboosted hydraulic cylinder.

Losing all brakes in a hydraulic circuit failure is just because those trucks are so old they only have single circuit hydraulic brakes. Part of the A3 upgrades was changing over to a modern dual circuit hydraulic system.

So basically, the issue is that medium trucks are in a weird in-between. Not heavy enough to require full air brakes, but heavy enough that some of the safety assumptions we make about light vehicles are potentially less true.


They have a way to change tire inflation to suit the terrain. I wonder if they can change the failsafe direction of the brakes to suit the situation.


That’s a great example of how you always have to design for a context.


Yeah, any design is going to be various trade-offs, and you have to understand them (and how they may relate to your mission) when determining what to do and what would be a bad decision.

For example, cars have "fail open" brakes but have independent cylinders so that it is relatively hard to have all four wheels fail at the same time (older cars had single master cylinders) - and one of the tradeoffs is that people want cars to get going immediately and not wait for brake cylinders to "charge up".

As more and more things get computerized, you have more options for "intelligent failover" - anyone who used to drive an old manual car knew you could slow down using the engine - but now the computer could detect brake failure and do things for you to help recover.

And as seen from other comments, the modern US military prioritizes "on-road safety" over "under ongoing attack ability" - partially because of changing strategies I suspect, but also from the realization that the US military kills more members of the US military each year via accidents, etc, than anything else.


Pretty much every car made since the 1980s has had dual-circuit brakes with usually one front and one rear wheel cylinder on each circuit. Volvo being Volvo took it further with their 2-series cars in the 1970s and 1980s, where the front discs had four-pot calipers configured as two sets of two-pot calipers. Each braking circuit had one half of each front caliper and one rear caliper, so in the event of one half of the brakes failing it would still stop in a straight line.

You can use engine braking in automatic vehicles too, by simply changing to a lower gear.


There is one subtle difference between automatic transmissions and manual ones and that is the manual one will allow a shift into a lower gear even when engine damage is likely.

An automatic one will not, and will hold the current gear until car speed falls within shifting range.

Ask me how I know :)


Yeah, newer ones can be too smart for your own good (but not theirs) - older ones had no problem with you slamming it into reverse or park at freeway speeds - ask me how I know!


I am reminded of a friend who changed his steering wheel plastic trim, in the process he bumped the wheel position sensor out of calibration. No worries, trigger a recalibration. Uh oh, won't recalibrate.

This of course meant the entire cars safety and stability control systems, ABS and traction control were all disabled. It drove like a tank.

The fix, obviously, was to replace the brake booster primer pump (hybrid car). It had worn enough to no longer recalibrate. Still perfectly functional, and would have kept working for a long time, bit it wouldn't quite meet the factory expected parameters so it failed it's test.


According to the other guy that founded the Range Rover forum I run, in the 1990s when they were getting P38s as traffic cars the training for "really really stop right now" was to throw it into reverse at 70mph and plant the throttle.

It appears that ZF 4HP20/4HP22 gearboxes can do this an unlimited number of times without apparently being damaged, but I'm not trying it on either of mine.


I have always wanted to try that! I'm not going to try it on any car I own either. It is all just super high risk, and I just don't have a spare car laying around that has the beans to pull that off.


Ahhh, the classic money shift.


Depends on the gearbox. Really modern ones won't, anything more than about ten years old will downshift if you select a lower gear.


Well, I have a 2000 Honda that won't do it. And I had an 03 Kia that also won't do it.


Yeah - but as we saw with the floormat/unintended acceleration, many people don't think of it in an emergency (all the fatalities could have been avoided by shifting into neutral in those cases, might have blown up the engine but wouldn't have crashed/killed).


It's something you're taught to do as part of your driving test, or at least you were when I sat mine.


> For example, cars have "fail open" brakes but have independent cylinders so that it is relatively hard to have all four wheels fail at the same time (older cars had single master cylinders) - and one of the tradeoffs is that people want cars to get going immediately and not wait for brake cylinders to "charge up".

And emergency brake that's wholly separate circuit.



Not in the US. US trucks are built using standard truck parts, including off-the-shelf air brakes that fail safe.


they didn't always fail safe.

I remember a house at the bottom of a steep mountain grade that got plowed into on a regular basis especially during that time.

EDIT: hmm... seems freezing might still be a way to cause failure?


Failure at the bottom of a steep hill is a different type of failure caused by brake fade due to the brake drum swelling with heat as the truck goes down the hill.


But I believe at some point air brakes had to be reversed by law.

In other words, the brakes would apply on loss of pressure, instead of via pressure.


Similarly train brakes are failsafe. They operate by compressed air. Being connected to the locomotive and hence control allows them to be released. If the carriages become disconnected, brakes get applied.


We have white traffic lights now in Canada for "transit advance", when buses are allowed to proceed before the rest of traffic.


Yes but it's also a different symbol - a vertical line rather than the full circle (at least in my experience).


Also common in Europe (CH, D).


Would they have breaks (gaps) to segment the track into distinct stretches of conductors?

How far down the line would the supply voltage run (or how far away would a train complete the circuit) before resistance losses degrade the system?


Yes, insulators were (and still are) used between track blocks.

The maximum length of a block with this system (versions of which are still used today) varies greatly depending on local conditions. I've seen 2k to 10k feet as a general performance range.


Another example is how garage door opener trip-beams work (the sensors that detect if something is in the way before lowering the door).

The transmitter and receiver are wired in parallel and connected to a power supply through a series resistor. When the receiver receives the signal from the transmitter, it pulls the supply voltage down, which then causes the transmitter to stop transmitting, which then causes the receiver to stop pulling the supply down, and the cycle continues. The end result is that if there's no obstruction, the supply voltage pulses and the system checks for this. If there's any other type of condition, the signal is either 0 or VCC. Basically, the system is designed so that false positives are extremely unlikely since that's the worst case scenario.

There's some capacitors in the transmitter/receiver to control the frequency.


Gas central heating boilers use something similar to control the gas solenoid - the solenoid is fed through a biggish capacitor, which has a diode and a capacitor to "smooth" the pulses from the microcontroller. If the microcontroller output latches itself on, the capacitor charges fully, there are no more pulses, and the solenoid drops out.


That sounds very complicated. What advantage does it have over wiring the two in series and having an open circuit indicate an obstruction?

edit: it gets you protection against one of the two failing closed, duh


Alarm systems use the "End of Line" resistor concept. The alarm triggers when a sensor is opened and current flow stops. Closed is ambiguous because it could mean power is flowing through all closed sensors OR there is a short (alarm panels handle shorts). A resistor allows the panel to consider 0 Ohms a short and some specific number of Ohms as closed and working. It apparently also provides a way to detect tampering of the wires.

https://www.alarmsystemstore.com/pages/all-about-resistors


Also why a standard keyboard has a Break key: used to be that a telegraph station would have a bell that rang if it detected a broken connection (a “break condition”), then people figured that deliberately inducing a break was a handy way to alert the remote operator. I think the first automated telegraph systems actually used current-loop signalling, but I’m not sure about the first practical ones; in any case, the concept carried over into modern (well, 70s-vintage) asynchronous lines—most UARTs nowadays have a break interrupt that you can enable.


I have a standard keyboard and it has no break key.


To clarify, by a “standard” keyboard I mean a (usually US) QWERTY PC keyboard with 101 to 105 keys (depending on the amount of Win/Super and Menu keys and the presence of an additional punctuation key to the left of Z). Those have three rows of three keys each above the arrows (not counting any “media keys” such as Power); the topmost one of those is PrintScreen / SysRq, Scroll Lock, and Pause / Break, though some of those labels may be omitted. I honestly have never seen a full-size PC keyboard without those. (How am I supposed to hit Ctrl-Break or Win-Break otherwise?)

Strictly speaking, there is no Break key in the same sense that there is no SysRq key—there are only special Break and SysRq scancodes that AT and PS/2 keyboards send instead of Pause and Print Screen when Ctrl or Alt respectively is depressed (thus the combined labels), while USB keyboards don’t even have those, they just send normal modifier sequences. But I don’t think that’s what you had in mind :)


I know what you meant. I was implying that a 101-105 key keyboard has not been standard for a decade or more.

I would be willing to bet that none of the ten most popular/best-selling computers in the country for the last 5-10 years have had a key that says “break”.


The 10 most popular/bestselling computers in the last 10 years definitely would include the Dell optiplex line that comes standard with a full size keyboard that has a Pause/Break key.


Nope, they’re all laptops.


You're talking about laptops which you're right, are probably more popular than desktops, but taking a look right now at a few different laptops near me shows they have break as a function key. Take 3 data points as you will.

I'll take that bet and offer up a break key from my active keyboard if I'm wrong.


Even on laptops, there's still usually a Pause/Break key, it's just merged with something else such that you have to use Fn.

For example, on most modern Thinkpads, Pause is Fn+P, and thus Break is Ctrl+Fn+P.


It's also named "Pause key", but one could argue that if your keyboard doesn't have any of these two keys, it cannot really be called "standard".


Interesting. That makes sense, but what’s the reason for using current instead of voltage as the variable?


Charge is conserved whereas voltage (i.e. electric field differential) is not.


High impedance (low current flow) signals are more prone to electrical interference. This is particularly bad when you're running long wires across a factory with big electric machines running. And if your signals are low impedance (significant current flow), then you get voltage drop across your long wire, so you can't use voltage-based signaling.


Current flow is independent of wire length, the sender only has to turn up the voltage high enough to overcome the wire resistance.


This happens because the real insulators are better approximations of ideal insulators, than real conductors are approximations of ideal conductors.

In electrical circuits made with normal conductors and insulators, e.g. copper wires and organic polymer insulators, it is possible to ignore the currents that branch from the wires and close through the insulators, because they are extremely small (e.g. in the femtoampere range), but frequently it is not possible to ignore the voltage drops along the copper wires, which are caused by the resistances of the wires, and which may be in the millivolt range.


I have a bachelors degree in electrical engineering and work experience in hardware design and in the back of my mind I've always known this to be true, but this is the first time in my life I've ever heard anyone say this explicitly. EE pedagogy is badly broken in places.


> EE pedagogy is badly broken in places.

Generally, the biggest problem with math, physics and chemistry is that if you miss just one lesson (no matter if you physically miss it, or in worst case miss out understanding a concept for a couple minutes), you can easily end up "left behind" for years. With history, arts, languages, a number of computer science topics or most of biology you can either skip out on a certain subject or catch up easily on your own, but not getting a specific part in math/physics/chemistry and lacking the resources to follow up is setting oneself up for exponential learning gap later on.

The societal problem is that we've not set up schools or universities to deal with this "failure mode". Privileged kids can have after-school teaching or siblings helping out, but most will sooner or later resign and hate these subjects for the rest of their lives - and that is incredibly sad.


If only we had the technology to somehow record lessons for later viewing or for refreshing knowledge later, and additional catch-up materials & deeper dives into the topics. This seems like extremely low-hanging fruit.

Why do I get a sneaking suspicion that few in the machinery of education want this, as it is a fast-track to continuous accountability?


Indeed, when discussing theory we love pretending that there's no voltage drop from the wires, which can really bite you when you get into design or deep failure analysis. All good technicians and engineers learn this reality fairly soon, and it's one of the things we check especially when dealing with indeterminate failures.

As much as I love academia, I really wish there were more instructors with real experience that could say things like, "for math purposes assume a zero-loss conductor" while still emphasizing that real life is not so simple.


Maybe it has to do with the culture at specific schools. As I mentioned in another comment, my instructors loved to include wire resistance in their problems, and always explicitly stated the zero-loss assumption when applicable.

We did have a healthy incubation program, where instructors helped companies with product development, so perhaps that is what brought the real-world experience.


As another EE, I had the opposite experience. Voltage drops due to wire resistance and transmission losses were drilled into our heads at any possible opportunity.


Difference between focusing on digital vs analog, maybe? As another EE, I vaguely recall this being pounded into us more in power systems than any of my digital electronics courses.


Sounds like you missed the “actually, all those digital wires are transmission lines” class :)


You will also get voltage drops from the sensors using power along your current loop as well, no? Whereas current will be the same along the entire loop all the time.


Also, there's greater immunity to interference and poor connections.


1-5 V is also common. You can convert between the 2 with a resistor.


That's one reason, but the big one is that it makes it easy to remote-power the sensors. It gives them a 4mA budget to Do Their Thing even when outputting a zero.


I would say the independence of conductor length to voltage drop since current is measurable at any point in the wire is more important. another big advantage to it is that it also is resistant to noise. (though you should be using grounded on on end TSP anyway) nowadays it's pretty common to use cat5 and read a bunch of things from the sensors though via various protocols. 4-20mA sure is a lot simpler to program against though.


or why we build a stop command with normal-closed contacts. a broken wire results in the same signal as a "stop" command.


Semantics are really important in these situations. We have a system with an E-stop (emergency stop) circuit. Any break in the loop and the system will shut off. It needed to work together with one particular vendors board that insisted on having a "hardware enable" signal that it could control. Rather than adding an additional circuit, someone decided to have "hardware enable" flip a normally open relay in the E-stop circuit. Technically this "works" because enable clears the E-stop condition and allows the thing to function. From a practical point of view this has caused no end of odd problems, particularly when diagnosing issues where things fail because the meaning of this signal has been compromised.


This reminds me of what I've seen happen with the internal health-check endpoint often added to APIs. Simple at first, but eventually there are endpoints to cause the health-check to synthetically fail for deployment purposes; the health-check is also monitoring downstream dependencies; the health-check becomes much more and much less than it was intended to be.


If you're doing three-wire control [0], it also means you can add additional "stop" buttons by wiring them all into the circuit in series.

A nice-to-have addition to your table saw is a big red stop button you can reach with your foot so you don't have to take your hands off of your work pieces to shut the saw down if things have gotten more interesting than you expected them to. Three wire control makes this super easy to add (once you've found the big red mushroom-head n/c switch, of course).

[0] https://twcontrols.com/lessons/control-wiring-3-wire-control...


I think for a foot pedal, it'd make more sense to have an enable rather than e-stop. Altough, that also depends on if you only work from one location or many. (Presses in industry often use such a design for hands, so that it ensures you hands aren't in the press.)


Also issue with golang (+ protobufs) where you can't distinguish a populated zero value from the default "not present" value.


This drives me absolutely bonkers. We had a situation in a machine learning context where we should have been treating the output of a detector as NaN (it was deliberately opting out), but instead it was falling back to zero.

Last I checked (this was years ago) the way to deal with this is to wrap it in a sub-message (which are actually nullable), which is just kinda gross.


I thought proto3 has support for optional scalar fields now (https://github.com/protocolbuffers/protobuf/releases/tag/v3....), and proto2 always had it


C# linqs method - first or default has the same problem


The method itself just uses the default value of the corresponding type. If you have a sequence of ints, and you cast it to Nullable<int> first before applying FirstOrDefault, then you can distinguish between null and 0.


Thats why JavaScript has both undefined and null!


Lua does fine with only a nil, although it gets a tad annoying when you actually want to store a nil value in an array and end up creating a hole instead. (Arguably, the solution to that problem is to use your own marker object and not a nil when you need some sort of unspecified or default value which is not a literal absence of one.)


Do we really need to differentiate between undef and null?


Not necessarily, but it does allow a level of robustness to be able to distingish between values which have deliberately been assigned null, and values which have not been initialized correctly.


That's why in our monitoring we used nagios scheme + 1 (0 for invalid, 1 for ok, 2 for warning, 3 for crit, 4 for unknown)


One reply tweet notes "a complex system always operates in a failure state", but to find more discussion on this point it's worth noting that this is a restatement of:

> "The Fundamental Failure-Mode Theorem (F.F.T.): complex systems usually operate in a failure mode."

-- John Gall, General Systemantics (aka Systemantics, aka The Systems Bible), 1977 < https://en.wikipedia.org/wiki/Systemantics >

Searching for "fundamental failure mode theorem" will provide interesting further insights on this.


The older I get and the more systems I encounter, the more I'm convinced everything is in a state of failing. The question is only how long before it becomes noticable.


One of the consequences of this is when something goes very wrong, and the people involved say it really shouldn't have because something like 4 separate things have to fail at the same time for that to have happened.

What they don't realise is that, thanks to the FFT, 2 of those 4 things actually failed over a year ago and the system has been continuing to work ever since without anyone ever noticing. One of the other things actually fails intermittently a couple of times per month, and while a couple of people have noticed that and reported it, either it's not been investigated at all, or when it was investigated everything looked fine and the report was put down as an unexplained glitch. All that was actually needed was for the 4th thing to fail, to cause this "can't happen" failure mode to suddenly ruin everyone's day.


All of this is further exacerbated by the modern business take on capitalism of "optimize away work that doesn't positively produce".

Just because you're paying someone to go check and they don't find anything 99% of the time, you don't get rid of that. If you do, there goes the pnly sensor capable of propagating that signal to the rest of the control network. Applies equally well in electronics/circuits/or human domains.


If only C-levels and Boards viewed such jobs as a good excuse to provide someone with a living as opposed to poor worker utilization hurting their bottom line


Experienced sysadmins see the occasional chatter of problems in logs and emails as the pulse of the system, the EKG hooked up to the patient letting you know how things are going.

The right level of logging is usually enough to be annoying but not so annoying you need to reduce it. That means you're hopefully thorough enough in the logging that you're reporting what you care about and the things you see are the the things you can manually fix or ignore, and fit a frequency level you can live with.

Until the stuff you're managing becomes too large for that to make sense anymore, because that only scales so far (but surprisingly far for diligent admins). Then you need to move to something more complex, which is a big job and requires an entirely different way of reviewing (like Prometheus). You're probably better doing that from the beginning, but it's time and effort but everyone can take, or may not have existed at that time.


> because that only scales so far (but surprisingly far for diligent admins)

Surprisingly far. We log terabytes of data a day, most of it is unnecessary. I've been trying to tell people that it is not a good idea but so far I've had limited success. It's easy for people to just "printf" everything (we are lucky if logging libraries are used with proper levels).

Right now, we have the most expensive 'add' operation in history. K8s pod logs some data. That is eventually noticed by fluentd, which is watching the filesystem. Fluentd sends to our logging servers over the network. That goes to kafka, then is shipped to the system that actually does the indexing. And then, there's a query running, which will then count the number of times that string appeared and updates a counter.

All of which could have been avoided if the apps just added +1 to a counter, and exposed that as /metrics so Prometheus could scrape.


I don’t know if we worked for the same company but your logging pipeline sounds almost identical to the logging pipeline at my last company. That said, logs are definitely abused often, usually all it takes is an engineer to say to themselves “well, I know this gets logged 1000x a second per host, but, someday during an outage I could potentially use this!”, without realizing that one log call can cost thousands or more a month. Tools like ElastAlert unfortunately dont help in that it makes people comfortable using logs as a fail signal. It seems like the best way to get teams to limit their logging is to give them the exact dollar amount their log call costs; ie, “this one log line accounts for 5% of all log traffic, well that costs $X a month, and it’s only been queried for N times”


mtail is handy for such cases, when you really only care to extract a few metrics from the logs. https://google.github.io/mtail/

Generally though I've found once you start aggregating logs you find many more uses for them.


Time to market. Nothing else matters, use whatever stack/library works and whatever I can copy/paste of stackoverflow to get it to work, we can fix the problems in software as we go and we only have to support it for 2 years then we can make another.


'complex systems usually operate in a failure mode' has a somewhat different meaning from "complex system always operates in a failure state".

They usually operate in a failure mode because it's easy to be deceived by the superficial appearances of complex systems, thus it seems like a big bother to drill down the layers, until a failure has occurred. As excellently demonstrated by this 'Factorio' player.


Thank you for introducing me to this rabbit hole. Here's a NYTimes article from 1976 -- note, many issues with the OCR, but still interesting and entertainingly written:

https://www.nytimes.com/1976/12/26/archives/why-nothing-work...


Same author wrote an entire book, The Systems Bible, on the topic.


Also related: How Complex Systems Fail https://how.complexsystems.fail/


Thanks! I read this years ago and have been sharing the concept with colleagues, but couldn't remember where it came from.

It's a really useful concept to keep in mind when designing or maintaining complex systems.


The number of interactions of a system with n components increases according to O(n^2).

The probability of an interaction not going wrong decreases exponentially with each interaction.


This is so spot on it's universally true


If your code has side effects, it only ever seems to be working.

----

> the newbie says "aww, why isn't it working?"

> the intermediate says "yay, it's working!"

> the expert goes "hmm, why is it working?"

I've also seen this as

My thing isn't working and I change X and now I get error J instead of error K, so now I have to change X back because it's still not working.

Where K is a "better" error than J, such as K = failed to connect and J is Server 500 error. With J you are at least talking to a webserver.


Working with beginning programming students, "Yay, now we have an error message that makes sense" has become my catchphrase.


"Read the error message, no wait don't close that go back" when helping people is a definite feeling.

Take a deep breath. Take off your headset. Go outside. Never look at a computer again. Start driving a bulldozer. Life is good.


I’ve lamented here before, but the number of technical people who absolutely freeze up at the sight of an error message is daunting. Being able to interpret errors correctly is a huge step up in this industry; to me it feels like table stakes, but clearly isn’t.


> I’ve lamented here before, but the number of technical people who absolutely freeze up at the sight of an error message is daunting.

"freeze" is still a much better reaction than "quickly dismiss without paying attention". I can understand "freeze"; it's possible to learn and train from there to "good, well done, now read everything and think". I do not understand "quickly dismiss without reading and paying attention".

(I know about the studies for "people dismiss something that isn't getting them their goal", but I still don't understand the mindset that can dismiss something like that even when theoretically trying to figure something out.)


Part of the problem is that we've trained everyone, especially nontechnical people, to ignore and dismiss stuff.

You're using a program. A thingy pops up or unexpectedly shows up that interrupts you.

80% chance it's some stupid ad or newsletter thing. Dismiss with prejudice.

19% chance it's a "oopsie poopsie we did and whoopsie, the server monkeys are working VERY HARD to fix this, sowwy!!" and then the app still works apparently just fine, or if it doesn't you just force close and re-open the app.

1% of the time it's an ad again.

In the remaining floating point error's worth of %, it's an actual actionable thing a user could action to fix the issue.

I don't really blame people for just predicting that a pop up will be useless because it probably will be.


Usually error messages are somewhat nice, What I have problems with at work is "error 395" and such. Where the documentation for the error codes can't be found easily.

(I hate progress openedge)


It's amazing the number of junior engineers I've taught to simply read the error message properly.


Or a junior dev will see "Error 422: thing_id is required" and instead of updating the request they sent to include thing_id they'll report it as if it's a 500 Internal Server Error until you pick up the ticket and read the error message to them.

Of course, they didn't actually put the error message in the ticket, they'll write "backend crashed" and put a screenshot of the front-end crashing because they didn't do any error handling at all.


Or its close cousin: "Yay, now we have a different error message!" When something is totally broken and you have no idea why, this is a promising signpost.


Similarly, the joke that the engineer’s second worst nightmare is, “it doesn’t work but it should”, while their worst nightmare is, “it works but it shouldn’t”.


That’s the surest way to nerd snipe some people. This code doesn’t work, and yet it does. After reading the commit history, it in fact never worked, and yet it has been up until yesterday. What in the actual fuck.


My favorite was C code that was roughly (all functions and variables h:

  int bar;
  
  ...
  foo()
  {
    if(bar) {
      doX();
    else
      doY();
    }
  }
Note that uninitialized globals in C are implicitly zero-initialized.

doY() had been broken for years but nobody ever set bar to be true; doX() was nevertheless being called. Setting bar to zero caused the buggy doY() to happen. Setting bar to one seemed to work, but would occasionally segfault.

Then someone explicitly initialized bar and got a linker error. Turns out in ye olden days of C, before "extern" was a thing there was a feature where variables that were initialized in no more than one compilation unit would be all coalesced by the linker. Before doY was broken, someone defined a function named bar, which then caused bar to have a non-zero (i.e. true) value. Someone later added some broken feature to "doY" and it passed all the tests because changing doY() didn't have any effect.


You're missing the } and { around your else, btw. I'm only mentioning that because I immediately saw it and got ready to read a horrifying story involving the text "#define else".


oh gosh I am.


I thought that was the bug on my first read. Naming collisions suck.


The buggy behaviour of A caused buggy behaviour of B that caused buggy behaviour of C, just so happened to align enough to "look like it is working".

Double fuckup if the thing is backups and your data is gone. Reason why we check not only "does the backup job finished" but also "does the size looks right", because backup with 0 files still returns OK...


Haha yeah, been there. I forget all the details, sadly, but there was some long-present bug with how the frontend architecture word in the dev environment that would keep the changes from being propagated. But every member of the team had set up a workaround so that, on their machine, it would load the correct assets some other, undocumented way, so they wouldn't notice the bug.


I used to do industrial tech support and this gives me the raging anxieties.


I have an exceptional ability to fix things: often I can fix things before determining why they are not working.

This is immensely frustrating because I like to understand mechanical and electrical devices and take pride in my craft, so if I fix something too easily I don't trust that the fix will hold without a mental model of the thing.


No news is bad news.

I don’t like code that compiles cleanly or runs cleanly on the first try. The worst is when I know that code isn’t close to done yet. Without errors my list of actionable tasks is low. I know there are a certain number of problems, and if I can’t see any it’s because they’re invisible, not because I don’t have any.


I came here to say this exactly. There's nothing more stressful to me than when a lot of freshly-written C++ compiles without error or warning on the first try, and then seems to run without issues. I get the distinct feeling that there is a problem that I'm just not seeing yet.


Eh, that heavily depends on language. Some have better tooling to get most of the typical errors out before the compile.

That being said I'm always suspicious and start to fiddle with them if I see test + code pass on first time


Reminds me of Jurassic Park (book, I think?); nobody realised that dinosaurs were loose and breeding, because the system was doing:

if dinosaurs < expected { alert("escaped dino!"); }

and nobody anticipated dinosaurs > expected


"Life, uh, finds a way." Dr. Ian Malcolm anticipated it, but the park science lackeys dismissed his idea.


If I remember correctly, Malcom did nothing but harp endlessly about how the park was doomed to failure due to various vague non-sequiturs.

All we know is that he predicted fence integrity would fail somehow. We don't know what else he predicted, maybe he just got that one right by chance. We don't know if Hammond changed the park after the paper to invalidate Malcom's reasons for predicting fence integrity failure; if he had done this, Malcom would be wrong despite having predicted fence integrity failure.

We sent people to the fucking Moon _and_ brought them back home. I'll be damned if humans can't solve Jurassic Park.


Malcolm was an author surrogate for the purpose of going on weird abstract rants which were mostly nonsensical even before they injected him full of morphine.

I vastly prefer reading books in general, but with Jurassic Park they really cleaned up a lot of rough edges in the movie. The kids are way less annoying too.


I read the book as kid after seeing the movie. I don't remember it so well, but one scene I liked in the book absent from the movie was the "compies" killing Dr. Hammond with narcotic bites.


Oh man, I really liked that one too. I couldn't find it with a quick search, but iirc they added that scene in one of the later movies (although maybe not with Hammond).


That was one of the system's invariants.

assert (dinosaurs <= expected)



There are two good lessons here:

1) Robust system design involves identifying the parts of your system that are mission-critical and always monitoring them. NASA missions have great automation and a 24/7-staffed mission control.

2) If a system failure can result in massive secondary damage, isolate that system. Warehouses receiving orbital payloads should probably be nice and far away from the base you care about.


> Warehouses receiving orbital payloads should probably be nice and far away from the base you care about.

As mentioned elsewhere in the thread, shades of 'The Moon is a Harsh Mistress' by Heinlein.

All your base are belong to dust.


Sorry, having never read that story (nor played Factorio since it went interplanetary) I don't understand the reference.


> 1) Robust system design involves identifying the parts of your system that are mission-critical and always monitoring them. NASA missions have great automation and a 24/7-staffed mission control.

You should alert on critical parts but you should monitor anything and everything you can. It might be critical in finding out why system broke later on. Easier said than done for hardware but easy for software


While I have not really played SE (past initial launch, it was too daunting to build the ship in space, etc) I have run into these kind of "issues" in Factorio. While they can have disastrous effects, I often really enjoy finding the core problem and finding a solution to make it "fail-safe".

Sometimes I wish I could go back and wipe all my Factorio knowledge and start from scratch again, most importantly refusing to use blueprints from the internet except for basic things like balancers. Finding early/mid/late game malls/blueprints sort of ruined the game for me. Min/maxing is fine by myself but once I'm "competing" against the internet or feel obligated to find the most efficient/best green/red/blue circuit factory, or science packs, etc it really ruins the game for me and makes it feel more like a job.

I got a good thousand or two hours out of the game before I hit that point and someday I want to try playing it again with self-enforced limits on what I'm "allowed" to get from the internet and what I need to just figure out on my own. The first game I played was pure bliss (and I played with Bob's mods and a number of others, yes that was stupid for my first playthrough but damn it was fun), I'd love to recapture that.


I want to reinforce your plan. Somehow I played factorio for a few hundred hours before finding online communities and blueprints. It was a wonderful experience of discovery and exploration. I came up with a few "novel" designs during that time that I'm still proud of.

Once comparing my own work to the best online it took a lot of joy out of the game. But I found that the memory was short lived, once I cut myself off from those online communities it was easy to fall back into the "let me pull out my sketchbook and solve this problem" mindset. I think half the fun is that design process.

I'm well north of 1000 hours now, and still discovering new ways to play and solve problems. I've come full circle and now build spaghetti bases because they are so visually fun.


I've heard this story repeated many times now and I'm glad to hear people are able to get back into it after having the joy sucked out. I think next time I play I will refuse to use a "main bus" until much later in the game if at all. That was the one of the biggest joy-killers, trying to plan for a mega-base essentially from the very start. It's not fun to plan out your bus and leave lots of room for red/blue circuits or more science packs while you are still building MK1-level things. That or trying to setup production lines with room for MK2-MK3+ versions of things from the start. It's complete overkill especially since there is no advantage to having a bus next to where you start. Better to situate it further away where minerals are more dense/plentiful and later in the game when it actually makes sense. Looking back I can't believe how stupid it was for me to setup 4 lanes of iron, 2 of copper, and a handful for each circuit type from the very start of the game, it sucked all the fun out but I didn't notice the reason.

The game largely felt like I was stuck in the same cookie-cutter, always trying to never have to tear up lines or rebuild elsewhere (trying to do it perfectly from the start). That was easily half the fun of my initial runs, realizing I needed to expand or rebuild instead of trying to perfect the build order.


> I think next time I play I will refuse to use a "main bus" until much later in the game if at all.

I mostly played through without any internet advice until I shared with a friend who did use internet advice and the main bus was his advice. Thankfully I already had a base so I tried this for a specific process/product (rockets ofc but still). I think a lot of “late game” stuff by experienced players will likely converge towards a bus since you’ll be already learned scale and order by then, but maybe with less precision in layout. The problem is that you can’t un-see it once you start building with it once.

I agree that fuck perfect from the start, build at the right scale at the right time. It’s way more fun. Conversely, if you like multiplayer games, this is a game where it’s really fun to play with people who are new. You can use their naïveté to override your knowledge or build completely different looking based each time. It’s fun to guide them through rediscovery (eg wtf why did all the trains dead lock!)


I found I enjoyed the process of putting together a mall from a blueprint, but leaving the rest of the factory up to me. I find the logistics and planning part of factorio more interesting than the "build up all the parts you need to make the factory" part.


It’s really interesting how factorio problems are pretty much the same as multiprocessing/distributed systems problems. It just shows how universal and fundamental the idea of “work” and “workers” are. An engineer on my old team came up with a credit system to solve this problem, the receiver issues credits to the sender, which only sends when there is a credit available - this ends up with the same better failure mode that the article discovers and allows you to overlap communication with work.


That is the Token Bucket [1] algorithm. Variations can easily be coded to be distributed (use Redis), or account for multiple receivers etc.

[1]: https://en.m.wikipedia.org/wiki/Token_bucket


Semaphore, Dijkstra, around 1962-1963


Wow, Factorio is approaching Full Time Job levels of detail and work required to keep everything running. Incredible game that I will never play, I feel I’m back at my job!


To be fair - this specific issue is with Space Exploration, a mod that's specifically for people who decided that Vanilla didn't hurt quite enough.


Usually games or hobbies that feel like a job are much more fun because there's nothing wrong happening if you fail at it, or if one day you decide not to show up.


Jobs get pretty fun too if nothing seriously bad happens if you fail at it. That's why so many people who are financially independent keep working.


Definitely with you on your second point, but not on your first. Failing is extremely stressful for me!


And you don't have to fill out a bunch of paperwork whenever you do or change anything.


Exactly why I stopped playing the game fairly quickly.


This is a mod that seems extremely boring and tedious. The base game is not like this.


> This is a mod that seems extremely boring and tedious.

It doesn't have to be, though. I'm currently playing Space Exploration + Krastorio 2. Krastorio modifies the early/mid game, Space Exploration meddles a little with the early game, but most of it is late game.

There are a few more inputs required for things so they can get more complicated. And with Krastorio the early game is extended (you have "burner" versions of the science lab and assembly machines). So yes, earlier game is a little slower. On the other hand, it gives you stuff to make up for it. You don't require as much science(at least, before space) and there are things like core miners, which provide infinite resources so you don't have to keep moving miners. Also new power generation.

I guess that it depends on what you want to do. Base Factorio you either play until you launch the rocket, or you move on to megabases. If you want to do that, it's perfectly fine. However if you want to keep playing, it's a lot of the same, only bigger. Space Exploration essentially _starts_ once you launch the rocket. I don't want to do megabases so it seems ok with me.

Also, spaceships :)


Burner things is from space exploration too. Krastorio doesn't change the very early game.


It definitely can be. I don’t think it needs to be as long as it is. With 5 space science categories, each with 4 tiers, and each tier requiring 4 unique data cards with one common ingredient and one catalog recipe for the tier, one insight per category, there are ~125 intermediate products to manufacture that don’t really pose any new interesting logistical challenges. For example, once you set up iridium processing, the 20 intermediates you need to manufacture to finish all 4 tiers of materials science are pretty straightforward. Sure, dealing with the scrap generated in the process of making those intermediaries is a logistical challenge that becomes harder the deeper you progress, but they probably could have presented that logistic challenge without requiring you to set up production for 25 separate products — most of which are ONLY useful for making materials science packs.

I have really enjoyed the puzzles it has posed in the form of circuit network logic for automating rockets and spaceships. The issue described in TFA is just the tip of the iceberg. My factory has deadlocked in a number of interesting and unique ways over the course of the 600 hours I’ve played thus far. I’ve enjoyed the mod. It has some very interesting ideas and challenges. It’s a shame those puzzles are hidden behind so much slog.


That's very subjective. It also adds a ton of new gameplay components that push factorio players to better think about how to manage their logistics which is fun for some players (not all for sure)


Space exploration is a mod which has extra automation challenges compared to the base game, and it's probably got some of the most interesting ones compared to other mods which tend to just have the same basic concepts as the base game, just with many more steps piled on, which are probably more on the tedious side (pyanodons mods being the most infamous in this regard).


For most of these things, you just make a blueprint and stamp it down repeatedly.


Same goes for work. If you join a well oiled machine that just seems to be working great, you may never understand how it works, since you won't be exposed to various people diagnosing issues. Join a place that used to work and is now creaking, or a place that never worked, and there's more pain but also more learning.

Also when does a Factorio system ever work? There's permanently a pressure for it to do more, stuff stuck on the wrong belts, not enough of some input...


If you use a calculator and then carefully design out the logistics & geometry of your factory, you can build stuff that basically "just works" once you iron out the missing inserters & power poles. You get bottlenecks when you focus on one product at a time instead of looking at how the system as a whole functions.


> Also when does a Factorio system ever work? There's permanently a pressure for it to do more, stuff stuck on the wrong belts, not enough of some input...

Sounds like a growing SaaS company.


I think this is the classic "when it is working well, no one notices"


Just want to note that the novel noted in the first post, The Moon Is A Harsh Mistress, is a great read. I think I read it after seeing it recommended in Mary Robinette Kowal's Lady Astronaut series, but it's a nice balance of technical and science fiction with politics, almost like a condensed version of KSR's Mars Trilogy. It's mostly (not entirely) devoid of the more typical Heinlein sexism/objectification.


It seemed to me to be a handbook on how to conduct a revolution. Much as Heinlein's Stranger from a Strange Land was a handbook on how to start a religion (hello Scientologists).


It's been a long way since I've read it, and it certainly had an interesting theme of politics and oppression. What bugged me was the description of the resistance movement: it pretended to be cleverly divided into cells, but ultimately the computer knew everything so it was pretty questionable security theater, and it didn't seem the book was particularly self-aware of that fact.


My favorite is, after fixing a system in prod, asking "how did that possibly work in the first place?"


I've totally asked that many, many times. But sometimes the answer is actually quite simple: no one was ever paying attention to the outputs of that piece of the system.


well, I'm the sort of person who digs around logs of ostensibly working systems just to find problems, which has been very useful in identifying long-tail problems (like, 1% of your network cables are blipping but the application just runs slower instead of failing).


Reminds me a lot about the fascinating read of Knight Capital, and the $440M-in-28-min bug that lost them 75% of their equity.

Reminds me a lot about Knight Capital bug that cost them $440M in 28 min.

> Rogue orders seemed originated from the new RLP router code, but no one could pinpoint the bug.... they reverted to last-stable.... and even more trades executed than before.

https://www.henricodolfing.com/2019/06/project-failure-case-...

I remember a PDF that went into even greater detail, was a very good read.


Isn't the lesson NO SIGNAL != 0

More universally, 0 isn't null.


Tests that output booleans (pass/fail) are an antipattern.

Tests and their dashboards must distinguish “the testing system worked and the test failed” from “the testing system failed”.


I dunno, I think regression tests are inherently boolean. Either it passed or it didn't. The code path that the system takes to get to a green light or not is arbitrary. If you start using an enum to return the result of the test, folk tend to add more and more enums over time until it's overly complicated. Processes return an 8 bit integer, but 99% of the time all anyone checks is whether it was 0.

There's certainly practical benefit to being able to quickly triage a test failure. Tests that only output booleans are annoying. When there is a failure, it's helpful to have some out-of-band information about why it failed. That's typically provided via console output, but you can make whatever complex logging system you want.


If a regression test fails, you alert the programmer who kicked it off and maybe roll back the checkin.

If the regression testing system has an error while running a test, you page the oncall SRE and maybe try to rerun it. You certainly don’t roll back any checkins.


Most testing frameworks distinguish between failure (aka assertion violation) and error (invalid or nonexistent test result).

But there may be mistakes writing the tests (false success result). To check the tests, I break the implementation, then check that they fail.


Whenever I write a non-trivial amount of code that appears to work the first time, I'm immediately suspicious. I probably spend more time testing that code than I would have if I had broken and fixed some things along the way.


Also a good reminder why exceptions and optionals exist. Bonkers that "I didn't get a response" defaults to 0.


It's not that "no response" defaults to 0, it's that in Factorio's circuit network, "no signal" and "signal with a value of 0" are exactly synonymous.


One thing I hated about working with Go. POSTed JSON bodies missing a field could not be distinguished from those where that field pointed was present and pointed to the empty value for that (an empty array or string or something). This may be entirely work-around-able and simply a failure of how that job worked, but it was a whole thing.


In this case, instead of, say, an int, you could have the field be a pointer to an int.


And now you've traded one problem for another.


Side note, does anyone know why the author has written "F*ctorio" instead of "Factorio"?



the problem is that while it also breaks search for the inflammatory assholes that want to ruin everyone's day, it also breaks search for everyone else.

it's a strange problem, to be sure. "I want to speak publicly about something in the market-place square, but in such a way as to avoid the nay-sayers and rabble-rousers.' -- this seems like a problem that was fixed in ye days' of olde by forming secret collectives and 'meeting down by the docks', I guess that's akin to what federated services offer?

note : I find federated social networks to have the same flaw, it's my belief that good things should be archived and catalogued for the public good -- so I have a hard time getting behind the idea of federated spaces that are protected from archival efforts -- but I do acknowledge that curation of a group is one of the best ways to reduce harassment.


> it also breaks search for everyone else.

Yes, and that sucks. But you can blame the assholes for that; they're the ones who ruin it for everyone else.


[flagged]


Personal attacks will get you banned on HN. No more of this, please.

https://news.ycombinator.com/newsguidelines.html


In the self-driving context, this reminds me strongly of Tesla Autopilot. Good enough to work most of the time, but likely would end up in greater overall injuries/accidents per mile if actually enabled at scale.

Waymo, Cruise, Aurora, and others are doing it the right way.


Well, I was driving home yesterday, and the integrated navigation map of my car was randomly rotating 180 degrees every few moments, just long enough to be noticeable on-screen, but then flipped back.

I have no idea why (something in the compass integrated in the car? was GPS being hacked?). But the thought I kept having was "If I had a self-driving car, what would be the impact of this?". All I could think of was horrendous disasters for me and the other cars around me on the road.


A proper self driving car will have more signals for orientation than just the compass.


Exactly, sensor fusion is so important.


We once learned a good lesson about this, as well as recognizing the real threat that ESD poses. We once had a terrible failure that was the result of a main signaling line being damaged by ESD such that it had much higher resistance than it should have. Our signal levels were too close such that it danced over the line a few times and got interpreted as the opposite value, but only sometimes. It also taught a good lesson that when the robots do become self-aware and take over the world, it's gonna hurt like hell.


This is a very dramatic illustration of one of the benefits of a pull vs push-based architecture.


I’ve built a pretty successful software monitoring business off of this basic premise.


Statisticians learned long ago that "missing" needs to be treated special. It should be either an entirely separate signal or a "sentinel value" riding on the existing signal that everyone knows and couldn't possibly be a normal operation value. Which is why sentinel values are usually a huge bunch of 9's or the maximum possible value of the field or something.

Interesting that the sentinel value is zero in this case. In data analysis that's usually a terrible sentinel value, but here it's the most practical one.


This sounds close enough to Golang where a null value is basically a zero value. I think that would be another million dollar mistake that would need fixing later in the language


"This Tweet has been deleted" -- does anybody have a mirror? Maybe @dang can update the main link to that.


https://threadreaderapp.com/thread/1581643415850098688.html

https://archive.ph/cToUl

but foone doesn't like the orange site so we probably shouldn't link (another advantage to running your own site, you can make refers you don't like see goats)

https://threadreaderapp.com/thread/1582089029096382464.html


[dead]


Seems they're tired of people in HN threads misgendering them, and have already tried diplomatic solutions. You can find it if you go straight to their profile. I honestly haven't seen anything like that in this thread, but maybe I'm not looking hard enough...


A more narrow lesson: this is why all SQL statements involving NULL are always false.

> in any case, losing power means the transmitter stops transmitting.

> and here's the fun part: your circuit which controls that inserter is set to insert "if [ICE] < 8000"

> and GETTING NO SIGNAL AT ALL counts the same to it as ICE=0.

> 0 is < 8000.


In industrial automation, one old and popular analog signal scheme is the 4–20 mA current loop [1]. Why 4mA instead of 0mA? Because that way, you know 0mA specifically indicates a fault caused by a failed sensor, break in the line, etc.

[1] https://en.wikipedia.org/wiki/Current_loop


A CHECK constraint treats NULL as true.


Spoiler alert! (lol)

I'm in the middle of reading this book and I figured this would (finally) happen next chapter. I'll be looking forward to this pivotal moment.


Someone just figured out how erlang's gen_servers work and why you should use call instead of cast unless you really know what you're doing


I think "never" is too polarized, maybe "be skeptical of" is a better suggestion.


Threadreaderapp of the tweets (still live): https://threadreaderapp.com/thread/1581643415850098688.html


Not familiar with this particular game. But in real-world systems design the default should be (?)

  boolean deliver = false;
  while( polling ){
    deliver = readyToReceive();
    if( deliver ) {
       send();
    }
  }


If 'readyToReceive()' returns a bad value you could have the same problem. Imagine it is an async external call, it fails, and whoever wrote the error handling did "return 'error'".

Now you have the same problem.


They don't teach that in college.


Tweet seems to be deleted now. A brief summary of what it was:

* In Factorio (game) you can have space stations that send supplies to your ground base

* If the supplies are not caught, then they cause damage to your base

* To avoid sending supplies that cannot be caught, you can use logic controls

* A common approach is to have the station say "if (ground supply < X) send"

* This fails if the ground supply loses power, as no signal is interpreted as 0 and 0 < X

* Thus, the system will appear to work until your ground base loses power, at which point it will be destroyed

* A better system is to have the ground base use logic to say "if supply < X send signal" and the station to say "if signal received, send". This way, a power failure fails safe instead of fails active.

With a fun callout to https://en.wikipedia.org/wiki/The_Moon_Is_a_Harsh_Mistress


That's also easier way to design it in Factorio. When I was playing with Space exploration mod that's precisely what I did, the base sent "shopping list" and the provider sent stuff on that shopping list


As I and others have previously pointed out [1] foone uses they/them pronouns and feels particularly frustrated when their posts get featured on HN, because users here seem unable to honor this simple preference of theirs. It looks like they deleted their tweet out of frustration with todays latest round of misgendering.

I hope people on HN will learn to respect all members of our community. Yes that involves not assuming every person online is a man!

[1] https://news.ycombinator.com/item?id=32978438


I did a quick search on the comments page for "he"+"him" and found nothing referring to foone, and was about to congratulate HN for getting it right, and then thought to scroll to the bottom of the page and found a dozen dead/autocollapsed comments, most including the wrong pronouns. At least they're where they belong.


the hateful comments about them having mental illnesses and so on are also terrible (i.e. noduerme in this thread)


.oi


[dead]


Doesn't that broadly seem like a problem? Given the preferred pronouns here are "they/them", which is what people ought to be using if they don't know, the problem stems from people inherently assuming "he/him" for any technical work. That's a problem beyond just trans/gnc issues.


Given that for non-city dwellers, interactions with transgenders are rarer than rare. Given that the assumption that anything non explicitly female is male is still very common. I suspect foone will be dealing with this issue for longer than they’d like.


[flagged]


We've banned this account for ideological flamewar. Please don't create accounts to break HN's rules with.

https://news.ycombinator.com/newsguidelines.html


> no one is getting dehumanized

[citation needed]


"Don't feed egregious comments by replying; flag them instead."

https://news.ycombinator.com/newsguidelines.html


A bit of an off-topic: I only ever saw white traffic lights in the US and Canada (specifically, pedestrian traffic lights) and they confused me a lot at first - the rest of the world uses green. The icons displayed (hands) are confusing too: the rest of the world uses an icon of a man walking or standing.


The white crossing indicator lead to a hilarious situation for me once. I am colorblind and always assumed the walking guy is green just like car signals. One day when a person I was hosting from Indonesia was walking in downtown San Francisco with me, we were talking about traffic conventions. He said to me “Here things are very easy. You just wait for the white man and then you cross!” I thought he was making an poor-taste joke about following the locals when traveling, and it took quite a while to clear everything up! And I learned something new about my own country’s traffic signals.


The hand is to mimic that of a traffic cop. When they want you to stop, they hold up their hand to indicate that you need to stop.

It used to be that you had to know english in order to understand them as well...many of the signs used to use the words "walk" and "don't walk"


I guess my point is that it was still confusing. It took me a day or two to acquire a reflex. It obviously doesn't matter in the long run - you just get used to it - but I wonder why would US & Canada go a different way with regards to so many small things (think turn signals on cars being red for american cars as another example)? What would be the rationale? It's as if the western hemisphere was completely cut off from the rest of the world and had no idea how others were doing things. I don't quite understand it.


>I wonder why would US & Canada go a different way with regards to so many small things (think turn signals on cars being red for american cars as another example)?

In many cases the US were the first to mass-manufacture things to some kind of standard, so the question is really: why did the rest of the world choose a different standard. Often they have a good reason (better design, maybe), but the US also has a good reason for not switching (inertia, existing tooling, people understand the existing standard).

The US was the first place that traffic light and crossing signals were installed. The reason they didn't go with the standard the rest of the world uses is that it didn't exist.


The curse of innovation: you might end up being stuck with it while the rest of the world learns from your mistakes.

Case in point: Ma Bell. And things like signing the credit card receipt.


> In many cases the US were the first to mass-manufacture..

Traffic lights? Not so many actually?


And standards don't adapt? To me this feels wrong and backwards


They do when there is good reason, but there isn't always good reason.

In fact, a theoretically better standard could actually be worse in some situations if people are accustomed to a different standard. Imagine if America had settled on red for go and green for stop, and then tried to change the standard to match the rest of the world. It would be a calamity as some drivers continued to adhere to the old standard that they knew.

Stop signs were originally yellow, so such a change isn't unprecedented, but in that case they also contained the word STOP so the colour wasn't critical to understanding the sign. Standards can more easily change when there is backwards compatibility available. More recently, some jurisdictions have started adopting traffic light shapes (square = stop, diamond = caution, circle = go) but retain coloured lights for backwards compatibility.


They do sometimes.

Like, the WALK / DONT WALK crossing signals that were around in my youth were slowly replaced with the HAND / WALKING PERSON ones, I'm sure because they're better for people who don't read English.

But also there is a cost to change, and it's often not worth doing if the benefit from the new system or standard isn't a lot better than the old.


When ATMs were first implemented they would dispense money before or at the same time as giving your card back.

This resulted in premature conclusion errors. They went to the ATM to get money. They got their money so they left... forgetting their card.

When ATMs were updated they fixed the design error. Now the card popped out and had to be removed before money would dispense. This resulted in a spike of people leaving with money hanging out of the ATM because they had been trained that removing their card was the end of the task. (Which is why money now gets sucked back into the machine if it hasn't been removed fast enough).

The point of the anecdote (other than that you hire HCI experts before implementing an interface) is that implementing a new and objectively better system doesn't necessarily result in an objectively better outcome when replacing an incumbent worse system.


Indeed, sometimes they do and sometimes they don't.

Why doesn't the rest of the world (outside the US) have a 3rd middle brake light (CHMSL), as was mandated in the US in 1986?

My guess is because, despite measurements of ~22% crash avoidance at the time, that number turned out to be between 0% and 4% in practice. Maybe the rest of the world didn't care about 4%, or maybe they thought it was as stupid then as it turned out to be.

But the CHMSL standard in the US has refused to die.


"It's as if the western hemisphere was completely cut off from the rest of the world and had no idea how others were doing things. I don't quite understand it."

Seems like you understand it fairly well. I can't speak for Canada, but the U.S. mostly ignores how other parts of the world do things.


Since this is HN, and I'm a pedant: reflexes can't be acquired, since they are not processed by the brain, but by the spinal cord, without involvement of the brain (e.g. pulling back the hand when touching a hot surface).

What you're thinking of is called procedural memory, which helps you perform a task without being consciously aware of it (hitting a fast baseball, looking left/right when crossing a street).


This is quite far from being the case.

Dr. Pavlov's seminal book is generally translated as Conditioned Reflexes. Saliva in response to a bell is a reflex, and it's an acquired one.

Not falling flat on our face when we step forward is also reflexive, as is catching something thrown to us. Both are acquired through rather lengthy processes.

There are many similar examples.


The term reflex has changed over the years.

Back in 1649, when Descartes formulated the concept of a reflex, it was used to describe lower animals, to support his notion that they were automata without a mind on their own. The word reflex originated from the "reflection" of the sensory input into a response. The physiological backgrounds were not yet known.

In the early 19th century, Hall narrowed the definition of a reflex to be a "involuntary action of a muscle or gland in response to the stimulation of a receptor neurone which does not depend on the existence of consciousness".

Sherrington, in 1904, narrowed the definition further, introducing the concept of the "reflex arc", a hardwired pathway between receptors and effector muscles.

Pavlov later did a strange thing and widened the definition, which is the source of some confusion. Nowadays, Pavlov's Conditioned Reflexes are mostly called Conditioned Responses, to avoid this exact confusion. NB: there are people arguing that conditioned is a mistranslation, and that it should mean conditional, which makes more sense in the context.

Pretty much every current publication uses the term "reflex" for involuntary, inborn responses, detected by receptors, transported to the spinal cord by afferent nerve fibers, processed there, and the motor signal sent through the efferent nerve fibers to the effector muscles. I only found the term "conditioned reflex" in papers discussing Pavlov's experiment.


:D I have a few of the "Walk" (in white) and "DON'T WALK" (in orange) signals around me in PNW.


I may be biased because I am used to the hand but I find the walk and hand shapes are very distinctive compared to the walk and stand.

However green on walk would be nice.


Clearly everyone should adopt the (East) Berlin standard: https://en.wikipedia.org/wiki/Ampelm%C3%A4nnchen

Very visually distinct, walking and standing, and green/red.


Any kind of figure triggers the walk reaction from me (or it did until someone finally clued me in on a trip to Europe).

My working theory had been green figure means cross, red figure means cross with caution, like a flashing red hand in the US.


I find them far less distinct than a walking person and a hand...and if I was colorblind that would be a large problem for me (and colorblindness was the reason for distinguishing the shape in the first place, so makes sense to have very different shapes rather than both "stick figure with hat").


I always assumed the pedestrian light was white to avoid confusion with the green traffic light for parallel drivers.


I would think you'd specifically want to avoid displaying lights of the same color as "go" for auto traffic, for purposes other than telling auto traffic to "go". So I'd be inclined to avoid green for pedestrian "go", if I were designing such a system. But maybe that's not actually a problem, IDK.

> The icons displayed (hands) are confusing too: the rest of the world uses an icon of a man walking or standing.

Is a hand held up, palm-out, not a signal for "stop" in places other than the US or Canada? I wouldn't assume it's universal, but I'd expect it's got broader reach than that.

Besides, again, I'd personally assume it'd be better to have very different icons for "walk" and "don't walk". Not two different depictions of a person.


It can also mean “Greetings, Earthling”.


Don't know about Eastern Canada, but over in Western Canada, the only white signal light is the walking person symbol for cross walks. Otherwise, it's Red, Yellow, Green lights for the traffic, and a red hand or white walker for crossing the street.

Just figured I'd mention that.


I don't know that I've seen all of eastern Canada, but of the parts of it that I have seen the exact same is true.


I’ve never seen a white traffic light, but pedestrian crossing lights are often white/red still.


Are you suggesting I'm doing worse on Google Captchas because I consider pedestrian crossing lights as a form of traffic lights but Americans don't?


Possibly. I think people in the US strongly associate "traffic" with cars and don't generally think of the crosswalk indicators as "traffic lights".


As an American I totally agree with that. I would never think to include the cross walk lights.

If you asked for “signal lights” I could see it. But not traffic.


Best choices to pick on captcha are not what you think is pictured but what you think most people think is pictured.


Since about 60% of the visitors of most English-speaking websites are American, I assume that requires finding the American answer. But that requires me to jump not just a language barrier but also a cultural barrier to figure out nuances like "what constitutes a traffic light? Do pedestrian lights count? Do the poles and cables count, or just the light-emitting part?" "What constitutes a street sign" "Is a Mecedes Sprinter with windows a bus (it's one of wikipedia's example of a minibus)?".

In the end the known unknowns aren't that bad, just assume laziness and choose whatever checks the least boxes. But the unknown unknowns are regular culture shocks. Like pedestrian lights not being considered traffic lights.


Buses are bad, but I feel trucks are even worse. Or actually vehicles in general. What does a Venn diagram of cars and trucks look like? Are Vespas and trikes motorcycles?


I’ve never had an issue, but I’m American. So if it’s expecting American centric answers that would make perfect sense.

So many not what most people would pick, but what Americans would pick?


And they call it "green", despite the fact that it's obviously white.

I agree with you. When I'm in the US I keep scanning for a green light, so it takes me a while to find "the white man" telling me it's OK to cross.


We detached this subthread from https://news.ycombinator.com/item?id=33235298.


the rest of the world uses green.

Only if you define the rest of the world to mean the very small percentage of places in the world you have been.

From Wikipedia:

In some countries, instead of "don't walk", a depiction of a red man or hand indicating when not to cross, the drawing of the person crossing appears with an "X" drawn over it.

Some countries around the Baltic Sea in Scandinavia duplicate the red light. Instead of one red light, there are two which both illuminate at the same time.


I've been to all continents except maybe Australia. Only North America has the weird white light for pedestrians and a red palm (which also blinks, which is very counter-intuitive). I can assure, most countries of the world use red and green icons of man standing/walking for pedestrians.


I've been to all continents except maybe Australia

Maybe Australia? You don't remember if you've been to Australia?

Congratulations! I've never been so drunk that I didn't know which continent I was on.


It's a retrofit of the older WALK/DONT WALK signs.


It was deleted because it was posted here.


Why is discussion on a different website so distressing? Are there people from HN harassing you on twitter?


It’s trendy to hate HN by some on Twitter.

*edit to provide context. this isn’t unique to twitter/hn. it’s trendy on reddit to hate tiktok and instagram. it’s trendy on 4chan to hate reddit.


Sure, except in this case foone has explained exactly why they dislike their stuff showing up on HN.


i edited my comment to provide a little more context but just because someone provides reasons doesn’t mean it’s not influenced by the trend. reddit’s trend to dislike tiktok has valid points but it’s also fueled by the trend. i’ve criticized hn here on hn so this isn’t me trying to defend the site.


They also seem to hate Factorio itself, which is the reason that while they play it and tweet about it so much, they mangle its name. I don't understand it, but everyone is free to remove what they want from their own Twitter I guess.


FWIW re the mangling: According to foone, the reason they blank out the name is to avoid it showing up on searches for the game (automated or otherwise), which they say has led to a subset of fans of the game sending them transphobic comments because of the past drama and the presence of a trans flag on their twitter page.

But as you say, people can do what they like.


That is not the reason I had been told and I can't easily check this (can't search -- that's the point). It is not unlikely that I have been led astray.


yes, people from HN have harassed, insulted and misgendered foone on Twitter. it's also happening in this very thread. (i.e. noduerme above)


[flagged]


Some people dislike interacting with the HN community for one reason or another. I imagine that's the case here. This person definitely isn't the only one.

Getting a post or tweet picked up by a large community who doesn't see your stuff regularly can be a lot more stressful than speaking directly to your audience (in this case, direct Twitter followers) who have a lot more context on your work.


In this case, there’s no need to imagine. Their latest Tweet has the answer.


I can only see 1 comment in this thread that uses "he" to refer to a person, and it refers to a different person, so I'm not sure I follow.


There are plenty, but you may have to turn on showdead and scroll to the bottom and uncollapse to see them.


It seems to be about misgendering on HN?

I never understood why it's such a big deal. When people guess wrong about my gender, ethnicity, preferred spoken language, etc, I correct them.

I guess if they were to argue with me about the correction and insist their guess was right, that would get old fast and I'd be as frustrated as foone.


It happens much more frequently by people with malicious intent to them. All you have to do is say once that misgendering is annoying and them trolls will latch on and do it intentionally to harm you



Same reason StackOverflow/Exchange's moderator team self-destructed 3 years ago after the Monica case. New pronouns war. The end of our species.

If you're not with us, you're with them.



This tweet has been deleted... I should have checked sooner, had too much trust that it'd still be working... I feel like there is a joke in there somewhere


Oh cool, a random tweet from the middle of a twitter rant.

Unleash the chaos monkey's just in case, not to get too confident.


What drama with the Factorio devs is he talking about at the end (and namesearchers?.. I don't get Twitter sometimes)

I've never heard anyone say anything about the Factorio devs except praise their productivity and professionalism.


foone's pronouns are they/them


One dev said in one post that they like Uncle Bob.

I think that’s it.


Well, that and a number of follow-up comments from the same developer, such as this one: https://www.reddit.com/r/factorio/comments/o2ly6f/friday_fac...

A lot of people have (understandably) lost patience with the "just asking questions" style of rhetoric, and don't appreciate others making excuses for it.


I don't get what's bad or "anti-trans" in this post?


The Uncle Bob from clean code(er)?|architecture?


Yes.


> "saying the name of the game uncensored is going to get namesearchers and I have a transflag in my name so guess what kind of response I'd immediately get?"

I believe kovarex made some anti-trans statements once, and gamers in general are prone to making abuse pile-ons, so it's a mitigation against that.


> I believe kovarex made some anti-trans statements once

Nah, publicly advertised for and defended some right-wing Youtuber. [0]

[0] https://knowyourmeme.com/memes/subcultures/factorio


Might as well cancel most programmers while we're at it. It's insane to lambast him for this when all he did was recommend some of Uncle Bob's videos on project management and programming.


Uncle Bob? Stating he is just "some right-wing youtuber" sounds mildly reductive, the guy co-authored the agile manifesto[0].

I won't go as far as defending his views, but recommending his talks and books seems pretty normal to me.

[0] https://en.wikipedia.org/wiki/Robert_C._Martin


He doesn't even have a whole "getting cancelled" section on his Wikipedia page yet, can't be that bad.


Kovarex promoted Uncle Bob in the past, who has generated a drama vortex of his own.


Hmm, what does he mean "I woke up the next day"? Did he leave Factorio running, or did it get an online component while I wasn't looking?


Probably left it running? I've definitely left my factory to idle to build up resources while I'm gone, when I'm playing on peaceful mode.

You could also run headless version of the game on a server and just connect to it for the same effect.

Remember - Factorio is just an idle game with extra steps.


Ah, makes sense, thanks.


*they, this is why they hate their tweets being posted here


[dead]


It's done by some people intentionally to be offensive, and it's impossible to tell if someone doing it is doing it as an insult or out of ignorance. And, believe it or not, there's enough people out there that do it intentionally that assuming ignorance would be incorrect more often than it'd be correct


Really? Huh, that's shitty.


Go around your office using "she" to refer to your male co-workers and see how long before someone gets cross.


The chance you get the gender correct when you know the real name, the voice, the look and the behavior of a person is significantly higher. Thus it makes more sense to raise an eyebrow when someone misgenders in real life.


the tweet was deleted :/


Wonder why it was deleted



Tweet was deleted?


What does it have to do with Factorio?


You mean apart from the Factorio screenshots and the Factorio gameplay specifics?

The thread's start is here, btw: https://mobile.twitter.com/Foone/status/1581643197427523584


The linked tweet is part of a larger thread which details an issue with how they set up transmitters in factorio which led to the destruction of the entire base. The thread then pivots to how it applies to other areas.


Ah thank you, that makes sense now. I'm not too used to how Twitter is organized.


The tweet has been removed?


Foone has asked repeatedly that HN not accept links to their threads. They've taken to deleting tweet threads that make it here.



so many dark patterns

"click here to download PDF" -> "log in to download PDF" -> [ unbelievable, privacy-eviscerating Twitter permissions, email ] --email--> You're almost there, click the emailed link to download the PDF! --click link--> you must go premium to download the PDF!

And then no less than two additional "Your PDF is ready!" emails which also redirect to the "go premium" page.


[flagged]


"Eschew flamebait. Avoid generic tangents."

https://news.ycombinator.com/newsguidelines.html


I think that should have gone over my head as I do not remember anything in the book to be libertarian. Maybe the final part where they are trying to set up the lunar government, where the libertarian stuff fails in practice?


[flagged]


You've been using HN repeatedly for ideological battle. We ban accounts that do that, so please stop doing that. It's not what this site is for, and it destroys what it is for.

https://news.ycombinator.com/newsguidelines.html


[flagged]


You posted three flamebait comments to this thread alone. We ban accounts that post like that. No more of this, please.

https://news.ycombinator.com/newsguidelines.html


[flagged]


Was thinking the same. Once you exceed like 5 tweets in a thread, it becomes a small article and you would've been better off just writing an article in the first place. If you check the thread reader below you can see just how unstructured and incoherent it is compared with a simple article.


If you're genuinely curious:

> I have ADHD: it makes writing blogs hard but Twitter easy. So it gets quickly tiring that every time I'm posted there [to HN] that the comments immediately go "why isn't this a blog? Twitter is a bad site for this kind of writing and I hate it blarg"

https://twitter.com/foone/status/1440695857699962893

This exact comment of yours is a prime example of why foone hates it when their tweets are posted here.


Thanks, that's interesting. I can see why it's easier for him to tweet in that case. But still doesn't change the fact that an article is easier to consume for the reader.


Foone is a them, not a him.

And yeah, you're right, it's definitely easier for the reader, but the author isn't required to make it easy for their audience -- if it was, Judith Butler wouldn't Judith...Butle and James Joyce wouldn't James Joyce. Foone writes tweets, 149k people have chosen to read their tweets. Personally, while I agree a blog'd be nice, I'm just glad foone makes their knowledge available.


How does that make sense? You write the same amount of text in the end.


The dopamine hit from click the send button can help overcome the ADHD tendency to wander off task. Plus, even if you have half an essay posted to Twitter and still get bored, it's much more difficult to just leave it halfway finished than to leave an unposted blog post in your drafts. And besides that, I'm really not sure why people get so upset at people putting content into the world. Wasn't the point of the internet to make it easier for people to share their knowledge with the world?


Personally I don't care too much.

But on the other hand I have ADD and there's a chance I get bored with a twitter thread and move on before finishing it while a blog post allows me to keep it open in a tab and read it in steps.


Foone also has a few hundred Tweet Twitter rant about why they are using Twitter, so at least they are consistent.


[flagged]


[flagged]


foone's pronouns are they/them




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: