The last time I looked into the Post Office scandal, the impression I got was that "we just
had too much faith in the IT system!" was a convenient story, but the high-ups knew full well about the problems, and full well that they were persecuting innocent people, but did it anyway to cover up their own failings. If so, people need to see substantial jail terms for this. Perverting the course of justice is no misdemeanor. There's even allegations that Paula Vennells knowingly lied to Parliament
There is substantial evidence that the people in power in the Post Office Board dismissed the evidence from the sub post masters because they were, more often than not of Asian descent (Indian, Bangladeshi, Pakistani,...). So the Post Office Board had a mental block to joining the dots because they were racist.
When they were eventually confronted with the overwhelming evidence of hundreds of false convictions they doubled down because they lack any sense of morality.
So while it is a sad technical tale it is also an indictment of British Board room culture.
The whole sorry affair has been covered by Private Eye magazine for over ten years.
We can and do reasonably infer mental states from actions.
If someone knows they are sending innocent people to prison, benefits from doing so, and continues doing so we can reasonably infer they lack morality.
Most internal mental states that have any impact on the actions of an individual are leaky. That is because they affect actual real world actions.
So given a set of actions you can infer a likely set of mental states — this is also done in the court room. Can you always infer with 100% accuracy and with every colorful detail fleshed out? No. Do you typically need to? No.
And we have we gone through that process to look at the issue from all sides and hear arguments from all parties? No.
You don't need to know intent to fire or punish someone anyway. In this case we can dismiss people for improper action, without demeaning them personally.
If you read my post again you will notice I didn't say anything about the situation at hand. I just refuted your general statement that one cannot reason about mental states from the visible actions.
I agree that one should be cautious with early accusations or hopping on trains just because they are rolling.
Yeah, but my comment doesn't has to be. I am allowed to go into your direction, generalize your statement or put it into a different environment and examine it. How else should people discuss things?
Why say punishment via rules can solve this, in other words, putting people in cages at the end of the day (or in other cases, taxing a permitted behavior), instead of advocate for structurally removing that power from them?
Am I inferring correctly that you are suggesting I may cause a perverse incentive, wherein they would be more motivated to cover it up due to the potential of punishment? I don't think that applies, as the punishment I'm suggesting is for the cover-up and knowing prosecution of innocent people, so the incentive would be to not do that. Not to mention they already seemed fully motivated to continue saving face anyway
Having rich people face actual material consequences to their life, rather than "punishments" like losing a job or honor, would hopefully reduce corporate malpractice, yes, as they now know that they are not invincible (obligatory note that I am talking about criminal responsibility, not suggesting we do away with LLCs or something)
I think that might be an effect, but I’m not sure. If something like this were to happen again, the thing I would most want would be whatever rules would most speedily lead to any prosecutions against subpostmasters being sopped and any convictions being overturned (with just compensation for the wrongful convictions).
I think to achieve that it has to be better in expectancy for someone who knows about the problem (and widely believed by that person too) to act sooner to resolve the problem than to try to cover things up, but I don’t really know what rules would help. For example, maybe an alternative thing that would work would be some kind of whistleblower rules like in finance[1].
In the hypothetical where I am stood with my hand on the lever choosing between stopping the scandal sooner and letting executives off entirely or having more wrongful convictions for longer but later throwing the executives in jail, I will choose the former every time, even though it feels unjust.
[1] In finance the whistleblower rules are designed to incentivise people to not be accomplices to crimes done by their superiors/coworkers. The incentives must be high because, no matter what the rules say about it, whistleblowers will expect to not work in finance again after blowing the whistle. The rules work because those large-scale financial crimes generally are hard to without one’s competent colleagues finding out.
Your question makes a good point. Punishment is only effective at the bottom of the funnel.
A well designed system, whether a computer or an organization should always have controls that guide things to where they should be. That means carrots and sticks.
In a place like a post office, that means separation of duties, audit, and clear lines of accountability. Things fail when nobody asks awkward questions or looks.
I do think that leadership responsible for a debacle like this must be punished. Their incompetence destroyed lives of innocent people.
In principle there was already some separation of duty as solictors in England owe a duty to the Court regardless of who pays them. However it seems not to have been enough in this case. Hopefully will be looked into.
The problem here was that as many sub-postmasters were co owned by a husband and wife with children. There was an enormous incentive for one of the sub post masters to plead guilty and say, "my wife had nothing to do with it." Doubly so as the sentences would often be on the cusp of the "suspendible range", a guilty plea combined with contrition is much more likely to get a judge to suspend your sentence if he can.
Of course, this "plea-bargaining" is standard practice in the states but we aren't used to it in the UK.
The issue is that no barrister would allow their client to plead guilty without giving them a little canned speech about not pleading guilty if they didn't do it and having them, "sign their brief".
I meant the solictor of the Post Office -- solicitors in England are both classified as officers of the court and owe duties to the Court. So amongst other things Post Office's own solicitors should have declined to act in any way that could mislead the court.
After a lifetime of fixing bugs I’m often surprised that computer systems work at all. I had something similar to the failed post office system affect me personally, and everyone just rejects that the computer system could be at fault, even when presented with evidence, as it too too much work for them to try to understand.
> I’m often surprised that computer systems work at all.
I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work?
I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else.
And like you say, people treat the computer system either as all knowing and infallible or as a scapegoat.
People outsource their critical thinking all the time, but with computers the failures are more egregious. There is less recourse to correct things, and information propagating through systems can make the error more persistent. Also: people take the verdict from a computer as a given even when reality and computer disagree.
I've used this story a couple of times before on HN, and if it hadn't happened to mother's coworker, I would have thought it a preposterous joke. Mom's colleague was buying something during her lunch break, and had her card rejected. Got the same result in another store, so obviously thought her card had just broken.
Went to her local bank branch to ask what's wrong. The clerk took the card, looked up from her screen and against the evidence in front of eyes declared: "You are dead." Once that kind of flag starts to make its way through our computerised systems, it is very difficult to reverse. It took several weeks to get the error corrected, and in the meanwhile other systems that trusted the bank as a source of truth had picked up the information and started to apply it to their accounts.
Eventually it came to light that a full namesake's estate had had their accounts finally closed, and the clerk doing that had done only a name search when applying the "account terminated, owner deceased" flag to the accounts. Bank never took responsibility for their process failure. They fought for a good time to even have to accept that there had been an error in the first place.
Which incidentally is not entirely unlike the situation in this perversion of justice.
I had a similar experience as your moms colleague.
I transferred 20k from savings to checking. During the process I think what happened is the web banking page froze and allowed me to hold the enter key on confirm that repeated several times. I ended up with 200k in my account that came as a negative 180k balance to my saving account sent to my checking account.
(values rounded for story simplicity)
This meant that not only did their online portal not have controls to make sure that internal money transfers are confirmed before processing. They also had no controls to make sure that when an internal transfer is done that the money is there to be transferred.
I waited a couple days expecting a call. Nope. At the end of week I call them. Get transferred around a bunch. The whole time everyone is saying everything is good nothing is wrong, there is no problem. I 100% certain every single one of them was lying through their teeth while they go the FBI on the phone because of the automatic assumption that the computers are right and I am automatically a criminal. There is no way all those people saw a -200k balance in a saving account and thought it was fine. (BTW that means it is standard bank policy to lie to a customer when something is wrong)
So what happened. They immediately zeroed my accounts and froze all my bank activity for three months while they "investigated" Then they only returned 15K of the original 20k that I moved so they effectively stole 5k from me. I spend another 6 months trying to get that money back and did not succeed.
I would have been screwed if I had not been born poor and was familiar with the banking system corruption. I have my income,spending,saving and investing split among different banks and banks accounts in case of something like this ever happened.
+1 to the advice to have different accounts with different banks. I've done this for a bit over 10 years now: checking account at one bank, savings at a different bank, credit card at yet another different bank, yet another credit card at yet another different bank, joint checking account with spouse at yet another different bank ...
My spouse did run into some nonsense some years ago due to her checking and savings accounts being at the same bank and automatically linked and mixed-up. Banks are always offering you super convenient awesome features with "protection" and "connection" and just PLEASE NO - that's how one mistake, that was that bank's in the first place, takes down all your available funds. You'll probably get most back, eventually, but do you really want to live with that for some weeks/months ...
Here is a posting[1] from a thread about a somewhat related experience with a bank more than 30 years ago, from the RISKS Digest, 1988:
"Someone, in this case a customer though in other cases it has often been an employee, complains that something is badly amiss with some system, receives no redress, and finds that their reports are ignored. ..."
"Which incidentally is not entirely unlike the situation in this perversion of justice."
So true. I went through the Wikipedia article, and couldn't escape this feeling:
1) Some errors were made (faulty IT system).
2) Handling of said errors expands enormously into various reports, IT vendor denying their system is faulty, legal proceedings, criminal convictions "tainted" & overturned, etc.
To the point where original errors almost seem irrelevant within the bigger frame of subsequent events.
Imho all the mistakes were made in the processes concerning 2), and 1) is a tiny side-show at most. IT systems can fail, period. What matters is how that's handled.
Which of course also applies to human cognitive processes.
We’ve had millions of years of evolution to prune out problems, but in many cases the problems were papered over by a kludge that introduced other problems, which were then kludged, etc. No wonder the complex systems we build are such a mess. They’re a reflection of ourselves.
They’re not yet a reflection of ourselves, only of our thought processes. They’re flawed, as are we, but we’ve been ruthlessly battle-tested in the only arena that matters.
With medical devices you have to act on the assumption that the software will fail and that mitigations should be in place to deal with the consequences of failure.
To me, modern computers, in particular, are an enigma, and I am an electronic engineer specializing in digital systems.
Imagine something with 200 billion individual parts with individual functions (and that’s just the cpu) that is presumed to work perfectly flawlessly billions of times a second for a decade.
By contrast, the crankshaft on your car goes through a bout 1 billion revolutions before it is worn out. Your heart might beat 4 billion times before you die, if you are really lucky.
The precision and reliability we take for granted from modern computing hardware is dumbfounding.
>> It boggles my mind, because I have seen how software gets built. How can it possibly all work?
Wait till you uncover what the SW runs on :). From math, processor/memory, rtl design, verification, tools that generate the circuits to the fabs that create the ICs.
And when I submit this comment it appears for others to view.
It is a miracle this entire structure works. It never ceases to amaze me.
Even the stuff that's built to rigorous standards is mostly built with a serials of failsafes because the same issues of being able to trust the thing apply. But it's fine because like you note, failsafes limit the effect of any individual failure.
Beyond that too, I think it's just important for people to realize that most things just don't matter that much. Everything we do eventually has to reach human scale and at human scale, there's so much room for slop and correction that most failures don't matter. Missing a paycheck because the payroll computer dropped your row is awful, but society can be built around allowing people to recover from that. Late payment forgiveness, forbearance, procedures around cutting a new check early.
I think the biggest danger about the "presumption that computers are working correctly" is in it's ability to erode our understand of the fact that "to err is human". It's too easy if you assume the machines are perfect to assume that any errors are therefore malicious, and so it's easy to go from a system where you have "oops my tax software messed up, no big deal I'll file an amended return" to "the only way for a mistake like this to happen is to intentionally put in bad data, so you must be guilty of tax fraud". Or even worse "the computer won't let me do it". I understand you want computers to have check in place that prevent the overriding of procedure and you don't want to give your low level employees the ability to override procedure at will. I also think that EVERY system should be built such that it is possible with a certain combination of people on site at the point of contact to override ANY check in the system to make the state of the system match what it should be. My specific example in this case was a problem with a prescription at the pharmacy. The script had been partially filled because they had stocking problems, but instead of going into the "partial" state, it was closed as filled completely. The tech knew what had happened and knew we still were owed the rest of the script. The pharmacist knew what had happened and agreed we were owed the rest of the script. The store manager agreed. And yet no one could change the state of the system. The only option was to get a new script from the doctor for a new fill entirely. That's the danger of assuming everything is working correctly all the time.
While the metaphore is great, it's actually stand on a myth.
That's all the more interesting in such a context to evocate it. We often take for granted what feels like aptly grounded easy to grab information. I think that is where LLM feels the most as a terrible threat to human societies, because they can at large scale provide customized very convincing utter bullshits.
I would add some nuiance and state that a lot of popular software is broken to the point that it's main function (the reason that the software exists) is not working.
Some examples:
-- The websites of various airlines (TAP, Vueling, KLM) have often had bugs in them (in the past 2 years when I used them) to the point that they don't allow you to book tickets or log-in with your account. I would say they where broken more often than not.
-- Instagram and Google have been deactivating people's accounts (automatically) for no apparent reason. I've had this happen with Instagram 3x in the space of 2 months after which I just gave up and stopped using it.
-- Skype, no explanation needed for those who use it.
-- Teams fails in many amazing ways no matter how often they "update" it. I've heard stories about the mute button not actually muting, Teams suddenly reversing the order of the letters in your sentence as you're typing in a chat, it just shuts down in the middle of a call, unresponsive, etc etc etc. The amount of productivity these bugs in this software burn worldwide must be significant and the real cost to companies in the form of wasted man-hours is likely higher than whatever they pay in licensing fees to Microsoft.
-- Telegram can't handle an external monitor (the scaling gets all messed up causing you to have to restart Telegram to fix it).
-- iCloud takes between seconds and an hour or so to sync files even when all devices are on a good connection.
To add some meta irony to my previous post: as you can see in my previous post the formatting got all mangled. The new-lines (enters) that I typed got ignored upon posting. I cannot think of any good reason for that "functionality" from a user perspective?
Also I can't edit that post anymore to fix it.
And to think that HN is an example of relatively good software that seems un-bloated, super stable, fast and seems to always work :P
Most people have a tendency to take the easy path and think stuff is perfect rather than accept everything works "approximately", although we see again and again proofs of the contrary.
But as an engineer I find this natural and efficient! You should prioritize and some bugs are less important than others. If anything more effort should be put in education, health and fundamental research rather than fixing all bugs in all software.
Sadly this often happens because often times the machine works 99% of the time. So when the thing starts failing in 1% of the cases people will dismiss it because it worked in the previous 99% of the time. People have trained themselves to trust the results so questioning it does not even enter their minds.
The system itself does not actually do what it says it is doing (The Operational Fallacy).
This people and engineers as well often forget. Just because the system says it does X does not mean it actually does it. People would do well to cultivate a healthy skepticism towards those complex IT systems. They might not fail most of the time but they inevitably will.
The rapid growth of the software industry is both the reason for the unsalvageable mess and for companies being able to spend obscene amounts of money to somehow keep shit together. At least that's how I feel about it.
A big part of the problem with these prosecutions is that the prosecuting authority is the Post Office itself. Until this case came along, I had no idea that the Post Office had these powers.
So effectively, the Post Office was acting as both complainant and prosecuting authority.
"The Post Office Investigation Branch (IB) is the oldest recognised criminal investigations force in the world. For more than 335 years it has worked to detect offences against the post and prosecute the perpetrators of these crimes."
Anyone can bring a prosecution in England. In theory it provides a way to hold the powerful to account but it's very expensive so almost all private prosecutions are actually by corporations.
It's true that anyone can bring a prosecution. It's also true that the Crown Prosecution Service can take over a private prosecution, and then drop the case.
Obviously, the CPS should have done this in the Post Office case. During the relevant period, the CPS was underresourced and overloaded (it still is now).
This is the first I've heard of this scandal. I'm curious what the major problems were with the software that caused the discrepancies. Dropped connections? Wikipedia didn't go into much detail with that.
I agree, I find it hard to read about because it makes me so angry. Four people driven to suicide, hundreds of lives ruined. It took them so long to hold all the inquiries and decide on compensation that dozens have died before being acquitted or receiving compensation.
And I've yet to hear mention of the people who weren't prosecuted because they paid the excess out of their own pocket. From the stories I have read, it seems that many who were prosecuted had tried that but ran out of money. Many more must have successfully dealt with the "shortfall" and are surely due their own restitution now?
Every transaction the system touched should be inspected, because I'm sure there were plenty of sub-postmasters who took the "shortfall" at face value and paid up without even thinking about questioning the system.
Yeah, there are schemes on this and Sir Wyn Williams seems keen and driven to make his enquiry as large as needed. So there is some hope we will learn more. But let's be clear here. Nothing can compensate the reality of what these people had to deal with.
Its a major problem with many of such issues. Some stuff is just really horrible, often even without any clear approach of how to fix it. It then gets even more problematic if people react to this by retreating into easy stories that act as mental fig leaves. Especially once these leads to clearly dysfunctional attempts to solve the issue, making the situation even worse for those effected.
Determining who to trust with this (on what topics) seems to be an incredibly hard problem with no obvious solution. And unfortunately a lot of people either seem to delegate to those that make nice sounding promises or ignore the issues all together. Even the statement you just made could always turn out to be a "noble lie" aimed preserving personal sanity at horrible cost for others. After all, having psychological support, or being able to deal with this, says nothing about what kind of solution they would produce. In all likelihood those people to delegate to will turn out to be in large parts depressed, detached or psychopaths. So i dont see a way around verification.
Having agency in a society ( through democracy, markets, shaping narratives in discussions and many other ways) might just require adults who deal with the horrible aspects society produces in a responsible manner. Which includes not designing solutions for a false peace of mind. After all, you yourself (as in every individual) are the one responsible for the consequences you produce.
Personally i hope this might be a matter of outer alignment, as in a problem of not agreeing on shared goals that you can apply to problem solving in a coordinated manner. Which has to include the ability to verify process and outcome (which unfortunately in itself are highly safety critical) and the unwillingness to accept false but pleasant conclusions with high costs for others.
I think the issue wasn't the bug but the lack of care? Pretty much any large system develops bugs during the development but they get noticed and fixed. These bugs got noticed and not fixed.
Tldr, distributed ledger provided by a third party with unauditable access went out of sync and branch managers were thrown under the bus for imaginary fraud because computer says so.
My usual presumption is that the computers are not working correctly, as I've seen way too many cases where they in fact are not working correctly. I don't even believe Google Maps when I'm driving, I always sanity check the route and double check street names/exit numbers when it tells me to turn. I've been burned too many times.
As a programmer that works on drivers, it's my experience that hardware almost never works correctly. "We'll fix it in software" might as well be the official motto of the hardware industry.
As a firmware engineer I’m horrified at the presumption that hardware is always operating correctly. Many times I’ve been told “your new firmware is broken” and the real problem is something as mundane as an improper capacitor or resistor value.
Also just look at the errata sheet for any microcontroller. Hardware bugs are so prevalent we have a special term and semi standardized document to describe them.
There's no indication that any of the components are exceeding their operating temperature range. It's just that the case becomes uncomfortably hot. It's hard to measure the peak external temperature of the case, so there can't really be a simple hardware-level failsafe for that.
But thinking about a court case, what should be the presumption? Should it be up to the prosecution to prove that the RAID controller / network card didn't corrupt the data or that the GPU did the math correctly? Or should it be up to the defense to prove that it didn't? And how would each side demonstrate their case to a reasonable level of certainty?
As others in this thread have said, for bits of the system that have clearly defined parameters and get extensive testing (by like, millions of people), I think you can begin with the assumption that those bits are correct: you can assume the CPU is adding numbers correctly, that the compiler compiled the code correctly, that the spreadsheet executed the formula it was given correctly, and it's up to the defense to prove otherwise. Bugs in the CPU / compiler / spreadsheet software would get caught pretty quickly.
But for bits where the right answer is not clear, and / or doesn't get extensive testing -- the formulas inside the spreadsheet, custom-written software -- the assumption should be that it does not work, and it should be up to the prosecution to demonstrate that the software is working correctly.
Mathematical verification of both hardware and software should be mandatory, at least as long you rely on it for the law. Simple as that. For example, how can you trust a photo, when tons of computational photography has been applied to it, and you don't know the exact amount the algorithms can change the photo?
That's not really economic at present, hopefully it will be one day. There are a very small number of verified programs, no mainstream operating systems and no major application suites. We just don't have the technology for it yet.
It does not seem very practical to throw out all digital evidence (eg photos and videos) because there is some chance of misprocessing. Who even records in analog today? What corresponding standard would you demand for analog evidence? Even in analog there were film issues, processing issues, noise, smudges, tampering and uncertainty, people have been dealing with these sorts of confounders for a long time.
We don't generally require "mathematical certainty" for other types of evidence. Personally I think GP was proposing a reasonably balanced commonsense perspective.
Common sense does not get you very far in a world where any video or audio can be fake, and you cannot really tell which one is and which one isn't. Common sense is officially done for. Bring on the math.
Come now; imagine a world where nobody could ever be convicted without mathematical verification of all components of evidence. Nobody could be convicted of anything, since even if we had 10 people and three 4k video cameras all see someone pull out a gun and shoot somebody, we can't have mathematical certainty even of the three video cameras, much less the unreliable wetware memories of 10 people. We might as well give up having trials entirely.
I rather rely on the memories of 10 people than on 4K cameras which can be hacked. I don't mind relying on a 4K camera for something like tennis, to decide if a ball is out or not. But if it is just up to software if I go to prison for 10 years or not, I very much DO mind.
The rule is either "on the balance of probabilities" or "beyond a reasonable doubt". Neither of which was well-applied in the Horizon cases. "Mathematically proven" is almost always the wrong standard for legal cases, as it doesn't speak to the quality of the proof.
The relevant standard ought to have been "beyond a reasonable doubt", but obviously even those who did know better managed to convince the court that doubting the computer was unreasonable. Apparently without manually checking the output against reality, even when the defendent brought their own numbers?
I strongly suspect that the racial discrimination that's being revealed by the press played its part in that -- no need to worry about the computer, or even about reality, if one's mind is set anyway.
That's easy to say, until you've tried to write a provably correct piece of software or tried to do model checking.
I have tried to build models for the schemas that databases have, not their implementation, just the schema, and it is difficult AF. Using z3 for this might as well be trying to start a fire with a pair of rocks.
What exactly have you tried to do there? Isn't a schema already a model? What properties did you want your model to have?
Yes, in general it is difficult, because then software becomes mathematics. And that's on top of the software doing something useful and what the user wants.
But I think we are now getting the tools to get there. You will still need to become a mathematician, though.
> What exactly have you tried to do there? Isn't a schema already a model? What properties did you want your model to have?
Yes, a schema is a data-model, and a database can tell you whether the row you are trying to insert satisfies the constraints, but it can't tell you whether an instantiation of the database exists, nor can it tell you how to construct it.
There are 2 issues here, first proving that an instantiation exists; and second proving that there's a sequence of insertions that satisfy the constraints. Notice that some tables can have non sat constraints.
This becomes significantly harder when you are dealing with recursive definitions, cycles in the dependencies, foreign keys that refer different tables across the same constrained columns, type compatibility, and on and on
In other words, the space of all databases described by a schema may not even be constructible :upside_down:
Ok, I understand. So first you want to see your schema as a property on a database (which is just a bunch of tables with data, I guess), and see if there is actually any such database, or if the property is always false. Second, assuming that there are any such databases: Given certain commands which operate on databases, is there any sequence of commands which transform the empty database into a database such that the schema property holds for it? Actually, you probably would want the property to be true for every intermediate step as well, right?
Sounds interesting! Although in practice, solving this is probably not that important, because if you cannot come up with some examples of constructing databases for your schema and application, then the schema is the wrong one pretty much by definition.
You needn’t think of databases as tables and data, you may think of them as trees of structs, ie an object relational model and show satisfiability of the model by constructing a tree of dependencies.
You can do it incrementally for parts of the model, but that doesn’t guarantee that long range dependencies are satisfiable.
Yes, you then need to show that it is incrementally constructible, which is a different can of worms… the tree view above isn’t actually fully correct because you duplicate entries mapping to the same object. Because of this duplication, the tree may refer to an object that can only be inserted later.
Oh and the encoding must be able to accommodate arbitrary length arrays and arbitrary objects … which makes usage of SMT solvers difficult to say the least.
While I agree that for the average company this isn’t useful, and the average engineer won’t shoot their feet on purpose, what I am doing needs to accommodate all those scenarios because I have no guarantees of sanity.
Now I am interested in what exactly your use case is :-)
Maybe you would want to model this first in a general purpose interactive prover such as Isabelle, and proceed from there. But in the end you seem to be looking for a push-button algorithm for proving the consistency of a particular logic, and I can tell you, that's probably not gonna end well.
I can't say much more unfortunately, not yet at least.
I have semi-sketched the idea in Dafny instead of Isabelle. Direct encoding of arrays makes it difficult, but I am thinking iterative deepening might just work. I am working on a Z3 approach for this, and I am crossing my fingers.
If Z3 doesn't work, I will try emitting Dafny code instead and hooking up to the compiler.
You are treating hardware like software by arguing with the experience we have made so far with a given model. Environmental conditions as well as it being different physical objects is a huge problem for hardware. The data about error cases is also simply not recorded. You get at best statistical reliability with small sounding chances for errors that are still happening all the time.
You can also not make any statement about reality by arguing with feasibility.
edit: I am unfortunately not qualified to tell you how feasible ways to deal with reality in this case could look. Which doesnt mean that i cant identify some error cases. You also require different levels of competence for designing a rocket and recognizing a non functioning one.
> the assumption should be that it does not work, and it should be up to the prosecution to demonstrate that the software is working correctly.
Phrased differently, you want them to prove there aren't any bugs. You can't prove a negative so this is a non-starter. There will always be "but what if it requires specific timing" or "maybe it relies on some other software being installed" or "maybe the computer had a virus". It's basically the same reason the government has to prove you're guilty; because you can't prove you're not guilty.
I think the onus has to be on the defense to prove some kind of a bug exists. The interesting question is whether that implies that the software provider should be required to hand over the source code in discovery. It seems unreasonable to force someone to go on a bug-hunt where their future hangs in the balance without access to the source code. Ditto for providing some kind of a test instance; I can see companies being leery of providing binaries to someone trying to publicly prove there are bugs.
What you say is completely true in my experience. Comically so.
However when using a computer application and noticing unexpected and erroneous behaviour, what proportion is the time can the bug be traced back to hardware vs software?
I’m going with about 1 to 1000. I think that’s conservative. Do you see it differently?
> As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug.
> But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to.
That's not a distinction that's relevant to this article. By "computer" it means a system composed of hardware and software (and which is almost always what "computer" is used to mean):
> The presumption that computers are presumed to be operating correctly, unless there is evidence to the contrary is what lawyers call “a presumption of evidence”.
> This means that a court can be satisfied that a relevant fact can be established just by computer records, unless there is evidence that the computer is not working properly.
> And so when the computer record shows, for instance, a financial shortfall by postmaster or postmistress, the court will accept that as evidence of an actual shortfall - unless the defendant can show that the computer was not operating correctly.
> In short, when the computer record is the essence of a prosecution case: computer says guilty.
>it's my experience that the computer itself usually works correctly.
Oh you sweet summer child. Hardware has so many little "quirks' or hacks to make it work correctly and it often doesn't! It's often expected that really low-level, "touching the metal" software makes up for these little bumps that happen quite often.
It does happen to Hardware. Here are two older examples:
“The Intel Pentium series of CPUs had two well-known bugs discovered after it was brought to market, the FDIV bug affecting floating point division which resulted in a recall in 1994, and the F00F bug discovered in 1997 which causes the processor to stop operating until rebooted.”
You are describing the exact same experience people who arent programmers have with software. It usually working and being almost always problems with the user.
It also leads to the same horrible misjudgements as in the article mentioned above. If you dont know what to look for you will come to erroneous convictions by ruling out the error cases you do know.
> As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug.
Wow there are some patronising people responding to your comment.
I think this is mostly true for 95% of software and the experience of software developers. But CPUs are full of bugs that only manifest when certain code is run - perhaps the order of instructions or when in certain states. Often these are hidden/fixed in software - compilers/drivers/microcode etc.
> As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug.
By hardware do you mean CPU?
Also, these days, "hardware" is still something that's either got a microcontroller, FPGA, or some other thing that is eother running or was designed with software. The area is quite muddy, especially if you deal with hardware systems of a collection of various computers and hardware components.
And I'm not so sure things aren't CPU or IC bugs that get bubbled up. The amount of Heisenbugs and various other transient bugs make me question that.
In my experience with instrument control systems, the bug could be anywhere from a cable to the UI or anything in between. These systems had to be developed in strict layers to help isolate bugs and also workaround them.
What do you mean by works correctly? By what standard do you say the software gets things wrong? After all it's an indisputable fact that computers only ever do exactly what you tell them to.
> Your experience is quite limited. Too limited to have a meaningful opinion on this subject frankly.
I am having quite the big problem with this getting downvoted. The poster is spot on here and this needs to be communicated. Not doing so is irresponsible. Overconfidence in this regard is extremely dangerous and there is no nicer way to say this.
You can not reasonably assume that you are able to judge something works reliably without understanding how the error cases look. Especially if these are capsuled from you and or you dont know what to look out for.
This is at the very core of why something like the post scandal was possible. People repeating the same mistake in this very thread is just really bad. The only thing worse would be not telling them this.
Please don't cross into personal attack. If you know more than someone else, that's great, but in that case please share some of what you know—without putdowns or snark—so the rest of us can learn.
No one else but you can provide you with experience. Hardware has both bugs (defects) and it fails, sometimes in ways not gracefully. An experienced multidisciplinary developer is certain to encounter them at some point. Depending on the discipline this may be a very common occurrence or it may be rare. Devs working at either extremes of scale are more likely to encounter these problems. Defects may go unnoticed due to software workarounds of all things
A very common problem is bugs caused by cosmic radiation if you want an example. But bugs in silicon are also not rare and really difficult to catch. You really got to stick your neck out and hope you can actually proof that through stuff like debugging with electron microscopes. Especially if it becomes an issue of responsibility between different departments. The monetary stakes make this all the more difficult.
Hardware bugs are simply capsuled better so people dont recognize them as such. Its the same mechanism at play as for people who assume software normally functions. They are also just unaware of the inner working.
Stuff is just really complicated and people should manage their expectations. I get that going from assuming to have a solid base to no such thing existing is difficult to handle but there is no sensible way around that. Not doing so gets you stuff like the article.
The presumption becomes problematic mostly when the operator of the computer is the same party who is being persecuted. 1) It is difficult or impossible for any outsider to inspect the workings of the computer, 2) the operator has every incentive to fake the outputs of the computer or hide the errors.
Related issue, but from the opposite perspective: some years ago I remember the copyright trolls using their own logs of the bittorrent swarm as an evidence in courts. Faking of this data would have been both very easy and pretty much impossible to detect by an outsider. Also they had a massive financial incentive to do so. The fact that this "evidence" was considered in the courts seemed outrageous to me.
While this is a question of what the court should presume, should we ask what does a software developer presume or doesn’t:
A software developer would presume proper working of software if it is being
used widely. I would presume JavaSDK or pandas library to be working properly, inspite of the fact that pandas that 3.5K bugs file on GitHub issues. But if I am thinking about integration a less known library or using a less known software I am less confident about its working.
Given enough eyeballs, all bugs are shallow. As such, they are complained about by thousands of people, for years.
Less ironically, one can expect widely used features of widely used software to have been tested in realistic conditions so whatever bugs there might be are at least known. But it takes an insane amount of testing to cover the phase space even of a moderately complex piece of software: look how large is the test suite of SQLite.
Apparently the only working approach to less buggy software is to compose software from small, self-contained,isolated, well-understood parts, so that you can reason about the whole more easily. This is the philosophy behind the original Unix. This is the philosophy behind functional programming. This is partly the philosophy behind actor systems (even though reasoning about async interactions is harder). I expect to see more and more of these in practice.
Yes. Bugs are there, but either they're shallow, well known and worked around, or have low impact for the majority of people using them.
The epitome of that was utf-8 support for a long time: there were a huge number of bugs, but alaphabet based languages would be mostly ok, and most devs would know it was a mess and work around that as much as they can.
To me one should expect any software to be broken in numerous ways, and take the time to check what happens around the area that are critical to them.
> compose software from small, self-contained,isolated, well-understood parts, so that you can reason about the whole more easily.
That looks awfully close to the micro-service approach as well. The approach can be good, but it still gives me pause.
> That looks awfully close to the micro-service approach as well. The approach can be good, but it still gives me pause.
Microservices aside, GP's statement seems to imply composition is like addition wrt. complexity. Unfortunately, in reality it's multiplication. Complexity multiplies with each moving part. There's hardly a better example of that to what we inherited from "original Unix" - chaining small, focused tools in pipes turns into a write-only mess after around the fourth pipe.
Software complexity isn't simply a divide-and-conquer problem, because we pay a penalty for each individual piece, and each piece of glue that connects it with other pieces. That's the whole reason we rely on abstraction and generalization so much - the ability to take a bunch of "small, self-contained, isolated, well-understood parts", put them in a box, and stick a higher-level name to it. Recursively.
Yes, composing self-contained parts is mere multiplication: if you have two sequentially connected components with m and n states, the composition has no more than m × n states (in practice much fewer).
But if you have two tightly coupled components which can arbitrarily jump into each other's code, you have (m + n)^2 states.
A four-stage Unix pipe is easy to split and thus analyze and understand, step by step, because each step is reasonably small, and has only one input and one output (presumably). Spaghetti code may be impenetrable, and only work by coincidence.
As they say, any sufficiently complex system always operates in a partly degraded state. This applies to complex software as well.
Microservices are not even about structuring your app; they are about the particular kind of deployment structure for your app. You can make a highly modular app along the lines of DDD and separation of concerns, and deploy it as a monolith for simplicity, or cut it into 2-3 large deployables. (Sadly, there are cases when a monolith remains tightly coupled, but the coupling is implemented as RPC, and the whole thing is purported to be "micropservice architecture".)
How does that work? I'm not a lawyer. I'm a lazy programmer.
So, does the defense get to subpoena the logs, and depose a programmer to explain why some random request returned a 502?
Does the defense get, like, red team access?
Although I'm a lazy programmer, I do think a lot about failure modes. If my freedom is on the line, I can probably think up a few (orders of magnitude) test cases. Happy to provide a shell script or python or golang binary to check. Heck, if it comes down to it I'll provide a stack of fortran cards. Do I get to see the source? That "few" is probably going to move a decimal place or 5.
I get volume is not reasonable in the eyes of the court. Scale is how you(I) find problems, but whatever. Gimmie source, gimme access, I doubt I could beat Knuth, but with my freedom on the line I could give him a run for his money.
How do CVE's count? If I find evidence of log4j do I auto win?
This makes me think of the Jepsen tire fire talk. The system probably mostly right. But there's no way it's correct. What's a "fair" level of access to prove that?
The real problem is this isn't a "right" that the postmasters could exercise. You're accused of stealing money. You're not paid very much in the first place and have only limited legal aid, and may even be in prison (on remand) already. There's no "defense team", you only have a solicitor assigned to you at the police station. You're also not a programmer. How are you going to argue that the database which shows you stole money wasn't working correctly?
In the US at least it is hard to bring source code into the trial, even though you technically have a right.
I remember one case where the FBI installed malware via a browser exploit on thousands of computers to get their IPs and one defendant tried to get the source code but the government managed to keep it secret.
>Perhaps also relevant would be United States v. Lafon Ellis [1], where the defendant sought access to the source-code of a DNA fuzzy-matching service.
I'm glad at least in this instance the defendant got access to the source code. He was being helped in pursuing this by EFF so I assume there were volunteers that helped with the analysis. But what if your case is lower profile? What if you can't afford to hire 10 programmers for 3 months to write test cases?
Also I wonder how such a request would go down in a UK court.
The post office found that money was "missing". It went after the sub-postmasters. The sub-postmasters didn't know that it was a systemic issue, they were told to plead guilty to reduce their sentence.
Unless you have a credible lawyer, and have a clue that something in the system might be wrong, then you're not going to know.
> So, does the defense get to subpoena the logs, and depose a programmer to explain why some random request returned a 502?
In theory yes, but you can get some idea of how it worked in practice from the transcript of the trial of Seema Misra [1]. This was one of the few cases where the postmaster pleaded not guilty and expert evidence was presented to the jury. The defence complained throughout the trial that their expert did not have adequate access to the underlying computer records. New documents were identified during the course of the trial, and the defence applied unsuccessfully for the trial to be stayed at the close of the prosecution case.
There already exists rules for certainty in law. After all just because you were seen by fourteen witnesses found with the murder weapon and admitted to the crime doesn't mean you are 100% guilty.
It could have been your identical twin doing those things as a silly example.
That doesn't mean the prosecution loses.
The goal is to meet that certainty which is by definition below 100% but high enough that society accepts it.
David Allen Green is a great commentator on law in the UK. During the Brexit period, he was a contributing editor to the Financial Times, with similarly insightful analyses of how Boris Johnson et al were misusing the government's legal powers.
Anyway, David also posted this to his non-gated "The Law and Policy Blog" at [1].
There are more comments there, including this one from me (written for his blog's legal/non-technical audience, not for the HN crowd!):
"There has been a big shift in the practice of computer programming in the last 25 years. In the “old days”, software was delivered on floppy disks or tape, and pretty much had to be correct first time.
Nowadays, everything is on the web, “agile” means the functionality gets delivered incrementally in 2-week sprints. It is regarded as ok to leave out the difficult cases or ship bugs, provided they don’t affect too many users too much of the time. In other words, the software is never really finished.
With that in mind, it becomes simple to rebut the presumption of correctness: if the company’s tech organisation describes itself as “agile”, using “scrum” etc, its software – by design – contains bugs and omissions, both known and unknown.
Fortunately, modern development practices means evidence for this will be in the dev tickets in GitHub, Azure DevOps or whatever other systems their dev teams use."
I had a visceral negative reaction to the idea that computers should be presumed to be working correctly, but I was having trouble articulating in precise words what was wrong with it. I mean, obviously we get work done with use computers every day and they seem to work alright. A commenter at that link does a good job:
>To treat the operation of an extensive software system as being a mechanical device is as gross a category error as treating an airliner as being merely a collection of nuts and bolts. In either case we can satisfy ourselves that the elementary parts are performing within specification without having gained any confidence that the complete assemblage will function properly under all conditions to which it will be reasonably exposed; indeed we won’t even be confident that we understand what it was exposed to. Even showing that it has passed a collection of tests isn’t adequate: one needs to see what unfixed issues are being registered against it and how they are being resolved (fixed? Swept under the carpet to keep the numbers below since arbitrary threshold?) The opposing side in case hinging upon accuracy of computer records should have the right to access the issue database, and absence of a well-run system or a large number of issues in it (or evidence that issues have been “scrubbed”) should be taken as a strong argument that the computer records are unreliable.
>Much could be learned by the software industry from the culture of aviation safety, where even minor incidents are followed up by detailed investigations intentionally run in a non-judgemental way and the findings distilled into public reports and recommendations. Similarly, one of the best legal developments in recent years has been the increasing tendency to publish detailed judgements.
This sort of thing has been going on since ... well, forever. I was reminded of this post [1] by Phil Agre in the RISKS Digest from 1988:
"I've noticed that responses to attempts to complain about uncorrected technical problems tend to take certain recurring forms ...
What has happened? Someone, in this case a customer though in other cases it has often been an employee, complains that something is badly amiss with some system, receives no redress, and finds that their reports are ignored. So they make noise about it. The response?
1. Misplaced accusations of uncooperativeness. ...
2. Condescending lectures about how `incredibly complicated' the systems in question are ... "
As a developer I never assume computer systems are working correctly. Quite the opposite in fact. If something isn't on the 'happy path' I assume something about the systems involved isn't working correctly and go check the facts for myself.
The Windows task bar auto-hide feature has been broken (spontaneously decides to quit hiding, but doesn't adjust the desktop viewport accordingly, so the bottom of any app window is obscured by the errant taskbar), since its first beta in 1994.
Computers, as in hardware, work as designed. Are there bugs in the software run on them ? Then the title should be about "bugs" or logic errors, not "computers"
Having seen this in action, it’s fascinating how fast humans flip from “this weird new computer is automatically wrong” to “computer is wrong? Haha nice try.”
I can see why they changed it. I expect in last 3 decades amount of computer evidence grew up to amounts where there is no practical way to prove computer was operating correctly.
If I present a digital photo as evidence it would be dismissed by defense lawyers to easily. Then basically any digital evidence would be dismissed the same way.
I can see how it was easy to manage in the 80’s and not anymore in 00’s.
While it's increasingly clear that Lessig's legacy will be as (to borrow a phrase) the internet's own failson, he is far more famous today for CC than Eldred v. Ashcroft.
This is contrary to the principle of presumed innocence. This is presumed guilt. The burden should always be on the prosecution to prove guilt, and if that requires the prosecution to prove the correctness of hardware, models, software, or data, then that is what should be required. And if the prosecution cannot prove the legitimacy of evidence, then than evidence should be disallowed.
The standard does not have to be beyond all doubt. The standard is fine at beyond reasonable doubt. If the prosecution can replicate consistent evidence for and against the defendant using the same computer systems with simulated data, then that is probably enough to clear reasonable doubt.
But the burden of proof should absolutely always be on the party wishing to use "evidence" to take someone else's property, freedom, or life.
Assuming it's right is assuming guilt. That is the wrong principle.
> This is contrary to the principle of presumed innocence.
No it isn't. It just means that some situation involving information from computers is "circumstantial evidence".
If programs have to be proved correct, that basically means data from computers could never be introduced as circumstantial evidence.
For instance, guns are normally assumed not to go off unless someone pulls the trigger. If a member of the Crips guns down a Bloods member across the street, he could claim that his gun accidentally went off the moment he was inadvertently pointing a gun in the direction of the enemy gang member. If he could prove that true, then it could be ruled an accidental death. If the defense lawyer does not make that argument, it will be assumed that there was nothing wrong with the gun, and thus someone pulled the trigger.
Taking this to absurdity you can create a new obfuscated program for each case that presumes guilt based on whatever evidence is could be in principle be decided with a computer, maximizing the burden on the defendants.
Maybe another principle would be that a program is assumed to function as proven in the past. The further a given program is shown prior to be correct, the more the burden of proof shifts to the defendant to show otherwise.
If bugs were easy to find and solve there wouldn't be an entire highly paid industry to do it.
Add to that malicious actors actively trying to work against systems (bank/post office/medical system etc) and it becomes pretty hard to prove that both the software and the data referenced by the software is in fact correct.
Now think about the last time a critical piece of data was recorded on paper with a pen/analogue typewriter and both parties got a physical copy of it.
Last time you made a cash withdrawal did you have to go to a teller at the bank, fill in paperwork, have two copies of that paperwork exist and then that paperwork was digitised or was the entire process digital beginning to end?
Think about how you, if you were an expert witness, would prove that an transaction made on a banking website was not a computer fault.
Do you show a code audit which meets some standards?
Do you have the bank's CISO explain all of the security measures?
INAL but I'm reasonably sure the defendant only has to prove that a system with the same security and audit trail has in fact been faulty and your evidence goes out the door or at least the validity of it decreases.
Thanks for the detailed answer. I think however that the real reason that there is a big industry to find and solve bugs is because of decision made when building and designing the software in the first place. For example if software was built in a robust way with care taken to refactor code along the way then maintenance and bug fixing would probably be a lot easier and may not need an entire industry to support it
I would hope so but I don't think that is the case, there is a portion of bugs, including security vulnerabilities that are actually hardware related but has a workaround in software.
That and no matter how hard you try there will always be logic errors in programming that will need to be fixed.
