Observation of vote counts is only a 1 to 1 relationship. You can't know if the whole election at scale is safe based on watching one person count votes. Perhaps it might feel safe to some, but that doesn't mean it is.
Moreover, physical observation doesn't scale. Only a handful of people can observe due to the physical limitations of meatspace. A simple attack could be to "DDoS" the count observations by crowding them out with your agents.
In my country, there are 50,000 polling stations [1] and the ruling party has 100,000 members [2].
The polling stations are open 7am-10pm, so 15 hours, before being driven to a central location for counting.
Thus, many polling stations don't have observers from the party at all - let alone the 2-4 observers needed to keep a constant watch from the first vote to the count.
Polling stations aren't where the count happens, the ballot boxes are pooled at one site per constituency and counted there.
This is particularly noticeable at the spread-out constituencies like Na h-Eileanan an Iar, where the count has to wait for ballot boxes to arrive by boat or, if the budget is available, helicopter.
The ballot papers themselves are traceable, so if you're going to stuff the box you need to add the relevant number of ballots to the record and fake their issue to particular voters.
They're also in the custody of ballot officials; in the UK this is one area where I genuinely trust the ability of those people making up the system to operate genuinely according to the rules. We've even managed to hold fair elections in occupied Belfast.
Switching to e-voting would turn the whole thing over to known insecure consumer PCs and known suspect contractors like Diebold and Capita, all of whom I trust about as far as I can throw them; I trust the small army of ordinary people who make the paper count happen far more.
Maybe next election I'll volunteer as an observer.
The paper solution is not ridiculous, it's perfectly workable.
The electronic solution is the one where malicious actors can substitute key parts of the process and things may be vulnerable to attacks that nobody has thought of (Meltdown/Spectre passim).
You would presumably have to compromise a lot of ballot boxes in order to influence an election. This would require a coordinated conspiracy potentially involving thousands of people.
If you can pull that off you might as well say fuck it and take over the country with force.
There are currently 20 political parties in parliament representing vast ideological differences. They couldn't even agree on what pizza to order if their life depended on it lol.
the observers at polling places are not highly ranked party members making deals in smoky rooms. they're almost certainly residents of the tiny area being polled, or the polity conducting the count.
certainly shady deals between a couple of folks at the tops of parties is a possibility, but collusion between the thousands or tens of thousands of local volunteers of both parties?
You don't even know how votes are counted yet you're trying to argue about the subject. Guess what, you can't DDoS polling stations with your "agents" and competing parties do not collude. Collude to do what? Lose the election? This isn't a joke.
Each polling place only has about a thousand votes, making it a process that I can easily audit completely. The results for each polling place are reported online, so I can check a part of the process from beginning to end. If you have just ten people or so, each auditing one random polling station, you can easily get to a statistically meaningful assurance of the election’s integrity.
Moreover, physical observation doesn't scale. Only a handful of people can observe due to the physical limitations of meatspace. A simple attack could be to "DDoS" the count observations by crowding them out with your agents.