Isn't it interesting that governments can get their act together and come up with big distributed systems crunching tens of Gbps in real time but still continue to implode spectacularly when it comes to large-scale webapps (e.g. Obamacare, UK NHS systems, UK taxation systems.)
> when it comes to large-scale webapps (e.g. Obamacare, UK NHS systems, UK taxation systems.)
There's absolutely no reason to believe the spy tech isn't as poorly implemented and terrible value for money as the examples you mention. Given their secretive nature it's even more likely to be the case I'd suspect.
This is my thinking as well. A lot of what spy tech is, is the ability to throw a warrant or NS letter into someone's face and force them to open the gate. The mythical uber-competent security services seem to be a pretty questionable premise. The few criminals caught via these tools are almost always the low hanging fruit of unconnected crazies, who seem to be harmless cranks and not real terrorists. Five eyes didn't stop or predict Snowden, Assange/Manning, Boston bombers, the rise of ISIS, Putin's advance on Crimea and Eastern Ukraine, the massive cyberwar Russia and China have declared on us, etc. In fact, all these things were major surprises to these security services, at least from what we can tell.
There's a lot of federal tax money spent on what is cargo cult science and cargo cult intelligence. The NSA has the ability to buy a lot of PhD's but they are stuck with the same batch of incompetent and profiteer federal contractors the rest of the government is stuck with.
It can't help that the Secret Service is losing top talent to private security firms like Academi (aka Blackwater), which draws a parallel with parts of the tech industry.
It makes me wonder exactly how much worse it is in the Intelligence industry. I agree with others in this thread, given the secrecy involved I expect it to be orders of magnitude worse than what we see in the private sector.
It also makes me wonder about the private Intelligence industry, about how much better they must be than their government counterparts, and about how they work together with government to "fight terror" and other demons. Combine that with the idea that government IT is losing talent to private firms in the same way... Maybe we should be more worried about private Intelligence firms than we currently are.
These days too many of them are a bit intimidated by the mere presence and aura of people in uniform, intel people, cops etc...Years ago, many of them had served in the military or a had a very good understanding of what they can really do (not what they claim they can do) how they worked/the types of people behind them(JFK, Churchill etc) so were rightly sceptical of their claims. I mean, has anyone ever met a cop/spook/politician who didn't want more power to do X, more money to do Y, more people, more secrecy etc?
One of the main problems for intel agencies is that they are merely one conduit to policy-makers, amongst many these days. Ultimately it is those policy-makers who need to (and increasingly are incapable of) using this information for strategic decision-making...Tactically it is easy to say "arrest/shoot that guy," because they get kudos and less blame if some thing goes wrong. Strategically it is harder to say "don't shoot that guy, keep him alive and in 5 years we can negotiate with him." It took the British a long time to learn how to do this in Northern Ireland, and it was only when they did that they managed to start to move towards peace with the IRA. I'm not sure would that be possibly in the bloggy, politically polarised, short-term, tabloid, Sky/Fox News type of world we live in these days.
After all, even if they did predict the rise of ISIS, Boston, it is the policy-maker who must decide what action to take. Difficult when:
a) There are sooo many warnings coming across their desk
b) Limited resources
C) Not many real policy instruments available. For example, the US public would not have been up for attacking for acting on intel available about ISIS 18 months ago, thus that policy option was thus not available.
>The few criminals caught via these tools are almost always the low hanging fruit of unconnected crazies
This is true of the criminals caught via FBI counterterrorism's more traditional police work that ends in civilian courts.
It remains entirely possible that the intelligence establishment enjoys some success in penetrating more sophisticated groups, but doesn't write press releases about it (since it will be acted upon with further covert operations to get further into those organizations, or with secret assassinations.)
Why does every post on this subject always have someone come in to slam home the notion that the government is always inept in everything they do? Its really good propaganda for keeping the public pacified. But it doesn't reflect reality. The government as well as its component organizations are not monolithic, and are made up of both bafoons and geniuses. Based on conversations I've had with people that have been part of the apparatus, the CIA is known to be somewhat incompetent when it comes to computer tech, but the NSA is phenomenal.
Stop apologizing for the government, whether conscious or not.
Because they keep proving it over and over again? (at least in certain problem domains they do.)
Government activities are generally important to a lot of people, if not because said people are direct consumers of the alleged benefits then because they paid for them. We keep paying and they keep ballsing it up, and everything we know about the theory of government (in an economic sense) tells us this is a design flaw and, ergo, not something that can be expected to self-resolve.
And it's not that they're totally inept all the time (although ineptitude is a recurring theme) - sometimes they do get good stuff done. The much harder question to answer in these cases is how efficient was it? How much tax money was shat away in admin/stupid compliance BS, and is all that wasteful pomp really a necessary price to pay for democracy? Could there be another way?
It's all very contentious, high-stakes stuff and if you disagree strongly enough with it all to wilfully not comply you'll soon find the whole system is propped up on various threats: they'll take your money, your home, your pride - they may even take you. And they'll do it all under a banner of moral righteousness that turns it into you vs. "society". How could that ever not be a talking point?
I talk about it most days in one way or another, and not because I'm some horrible right-wing fanatic who wants to wipe my ass with money and glee in the suffering of others, but because it's an incredibly fragile, wasteful system that perpetually sits on a knife edge and I suspect does just as much harm as good. It's misguided in so many ways and so, so hard to change that all I can do is moan about it and hope that my moans contribute in some way to getting the volume of the debate loud enough to think that something could change.
> And it's not that they're totally inept all the time (although ineptitude is a recurring theme) - sometimes they do get good stuff done. The much harder question to answer in these cases is how efficient was it? How much tax money was shat away in admin/stupid compliance BS, and is all that wasteful pomp really a necessary price to pay for democracy? Could there be another way?
I think this largely misses the mark, for several reasons.
1) The problem with government isn't ineptitude. At least the federal government is probably average to above-average in terms of competency. But what's the competency level of the IT folks in your average Big Corp where IT isn't a front office department? It's not very good. Take those not very good IT folks and then make them do projects 10x the size and complexity of what your typical Big Corp handles, in areas where the tolerance for error is nil (health care, benefits), and political concerns make things like iteration and building MVP's intractable.
2) It's easy to talk about "admin/compliance BS" because you don't have to point to any actual procedure that you might have to defend as being BS. It's not clear to me that the federal government has any more internal red tape than your typical Big Corp.
I spent a short time at the FCC, after working as an engineer in the wireless sector, and it was pretty eye-opening. On one hand, it was quite lean and efficient for what it is tasked with dealing with. At the top it's five commissioners, a few very experienced policy advisors, and a rotating staff of interns doing grunt work. On the other hand, the necessary complexity is staggering. For every decision, there are dozens of stakeholders. Things that were no-brainers from a purely engineering point of view became multi-facted problems combining social justice issues and economic issues.
I think engineers, software engineers in particular, confronted with complexity, seek to eliminate it. Something is hard? Do less. That's usually just not an option with government.
>Because they keep proving it over and over again?
The incredibly complex and expensive purchasing/hiring system that seems to fail every single time exists because we don't trust government employees to spend our money. Perhaps we shouldn't. But you cannot deny that the system designed to prevent improper spending is the cause of a hell of a lot of improper spending.
Government agencies are often legally bound to purchase from the lowest bidder even when they know the product will be "technically correct" but extremely low-quality. Because the regulations and processes are so complex, it is incredibly expensive and difficult to sell to the government, meaning the few companies that actually manage to do it have higher costs and less competition. Of course prices are high.
Sure, the system is necessary to stop incompetent/malicious actors from wasting money or throwing contracts to their cronies. But it also stops a lot of smart and diligent people from getting the resources they need to complete their projects.
If you want government agencies to be able to purchase high-quality goods and services at reasonable prices, then you're going to need to loosen the chains. And that probably means some fraud. And that probably means some agencies buying slightly nicer stuff than the absolute cheapest thing that would work on paper. But the overall effectiveness of government might end up higher, and its cost much lower, if you were to eliminate the "government services" industry and open it to the wider market.
Honestly, government is pretty much the same as a big company. The wild card is that you have a strata of extra executive management that may be brilliant and fighting the professional workforce or incompetent but capable of screwing things up.
I work for a large state government. Plenty of Dilbert moments, but important stuff gets done. There are people all over the place that have dedicated their life to what they do and are masters of their craft or policy area. In IT, we have plenty of crap, but a few things that are world class and make a real difference in people's lives.
I've worked for small companies where the "friends & family plan" ruined key aspects of the business/workplace. I've also seen large companies with all manner of incompetence and bad behavior -- one bank leader I know actually built a business unit with no useful purpose... specifically for the purpose of having chum for a layoff.
Lowest bid policies are not just stupid, they're obviously stupid. So not only do you have an obviously stupid policy, you've got the obviously stupid system that settled on such an obviously stupid policy in the first place. Given this mountain of stupidity, it is safe to conclude that the government is not being run nearly as inefficiently as it could be in the absence of such multifaceted idiocy.
Did I say the people are dumb? No, I said the rules are dumb and the system that formulated those rules is dumb. Extending this to the people running the system was an inference you made, not something I actually said.
As it turns out, the current rules are actually very effective if what you want is a system tailor made to produce large and profitable overages after any danger of losing a contract to a competing contractor has been mitigated.
HN used to have a libertarian lean to it back when it was a smaller community of programmers who were starting companies like Dropbox and AirBnB, people discussing SICP/Lisp, and willing to fill the homepage with Erlang articles to scare aware newbs. Now that it has been fully transitioned into mainstream site and much more like reddit, it is much less libertarian and more /r/politics.
So that generalization doesn't apply so cleanly here anymore.
There's no need to merely suspect. You can be (depressingly) sure thanks to Thomas Drake, who blew the whistle on the NSA for not only not doing things the right way, but for spending gargantuan sums of money to do things in a way so wrong that the only thing done at all was something that flatly violated the Constitution.
Well said. And even more of a reason to insist they respect your privacy. Even if you think "you don't have anything to hide," you still don't want your personal information held on a system designed by anyone with that government contractor level of QA/QC.
My pet theory is that it has to do with outsourcing and the widespread belief in the private market that software development is a product just like any other that you buy on the market.
When your average intelligence agency has a million dollars to spend on a project, it will keep a house full of smart people and tell to forth and create, paying their bills and keeping their backs. We might call this the academic model, if it wasn't so far from how academia works today.
When a civilian government agency (or any large private entity for that matter as they are completely similar in this regard) have a million dollars to spend on a project they immediately start phoning the big boys: We've got a million to spend on this project, can do do it?
It is all a bit exaggerated of course, but in this made up example one will get a working, if perhaps shoddy product, and one will get a server filled with very important powerpoints while the product itself tends to be late and slow.
Or to put things in a simpler way and less of a rant: I suspect it is a lot easier to spend a million dollars on a highly specialized system, than to do the same on a very generic system anyone could build. The corollary is that the same intelligence agency would fail in the same way a private or civilian organization would when implementing a new payroll system.
Different command structures could be part of the cause. Obamacare was a non-classified civilian project, under the leadership of civilians. The primary fumble was (due to corruption) using a federal-friendly (corrupt and ineffective but known to be loyal) contractor rather than an actual development firm.
NSA etc are military operations with honeycombed classification schemes and cellularized access to information. Even if there were failures and waste, we'd never hear of it. I assume that corruption plays less of a role since the concern is on effectiveness, but I have nothing to base that on.
The military isn't that well organized. Anyone here can name at least a dozen snafu from the last decade. A purchase, an invasion, a process a project. You name it.
At the heart of the military organization, there is an essential core of operations that ensures that anything that absolutely, positively must get done can be done. The remainder is simply there to exercise and maintain the massive required capital investment while it is functionally idle.
The gigantic failures exist so that when there is an actual need for aerospace engineers to respond to an actual threat, rather than the theoretical projections based on the data from the last war, it won't take 25 years to train them up from scratch.
The failures are not quite as horrible when you realize that it could be more expensive in the long run to do nothing at all. But then the questions turn toward the amount of pre-emptable core capabilities we need to maintain. For a long time, the doctrine in the U.S. has been the ability to sustain simultaneous nonnuclear war in two separate theaters plus nuclear second-strike capability sufficient to create a MAD deterrent. That is expensive. And since we haven't actually needed that much war materiel since 1945, that leaves a lot of room for waste.
It could be argued that the limited regional wars of choice that the U.S. has pursued since then have simply been capital maintenance, to prevent the military from decaying into expensive uselessness. I think in its deepest backrooms, the generals still gauge their effectiveness by whether they could fight both Russia and China at the same time.
For a long time, the doctrine in the U.S. has been the ability to sustain simultaneous nonnuclear war in two separate theaters plus nuclear second-strike capability sufficient to create a MAD deterrent.
I understood it to be a bit more specific than this: sustain conventional war against our two strongest non-allies.
In any case, it's also my understanding that we've been maintaining a level far higher than what would be required for this.
That's just not true, the world isn't a Tom Clancy novel where the US versus China is decided by some magic bombs that suddenly appear.
Look at the Iran-Iraq war for the level of casualties that some potential enemies are capable of taking (half a million fighting age males dead on Iran's side). North Korea having 9 million military age personnel brought up in a personality cult. Let along China and the size of their armed forces.
The USA lost four thousand in Iraq, two and a half thousand in Afghanistan. The UK lost nineteen thousand men dead on the first day of the Somme in 1916, do you really imagine a Western nation losing that many for any conflict today? To be honest even in a war of national survival I doubt it.
We lack the political will to sustain conventional war. The military knows it and acts accordingly.
I'm not certain what aspect of my reply you're disagreeing with. Are you saying that it's not true that we've been maintaining the ability to fight more than two wars? Assuming that's the point...
Your reply seems focused on the quantity of soldiers. This is an important factor, but I was primarily considering overall spending, which obviously includes lots of capital expense, logistical stuff, etc., aside from the personnel. I believe that the level of investment in military hardware and the like is significantly higher than would be required for a successful war on two fronts (assuming that there exists such a thing as "successful war" these days).
The US military is designed to win by a modern version of blitzkrieg, not by grinding out a victory in a war of attrition. They want to win like Germany did in the Battle of France, not like the Soviet Union did at Stalingrad.
It's why the Iranians had over 1 million casualties taking on Iraq. And several years later, the US rolled over the Iraqi military with 1,000 causalities.
So sure, the US can't successfully invade China, but it would sink the entire Peoples Navy and defend Taiwan, Japan and S. Korea.
World War I and horrors like the Somme were a product of decades of training, indoctrination and religious connection to national policy.
We don't do that anymore... Because a full total war is too expensive -- when the US suffers a loss of 20,000 men in a day, a button will be pushed in a silo somewhere and cities will vaporize.
You'd assume that if a GCHQ/NSA IT project went spectacularly off the rails (and I can virtually guarantee it has at some point or the other: you'd have to assume so given how often it happens with defense projects we know about) we wouldn't hear about it anyway.
Actually the TRAILBLAZER project was the NSAs attempt and it was a total failure, costing around 3 billion dollars. They abandoned it in 2006. SAIC the contractor that "built" it for the NSA got pretty rich though.
"Trailblazer was a United States National Security Agency (NSA) program intended to develop a capability to analyze data carried on communications networks like the Internet.
In 2005, NSA director Michael Hayden told a Senate hearing that the Trailblazer program was several hundred million dollars over budget and years behind schedule. In 2006 the program was shut down, after having cost billions of US Dollars. Several anonymous NSA sources told Hosenball of Newsweek later on that the project was a "wasteful failure".
The new project replacing Trailblazer is called Turbulence."
- http://en.wikipedia.org/wiki/Trailblazer_Project
It's much easier to write software that does something, than it is to write software that does the right thing every time. If an NSA system fails to see 10% of traffic, that's acceptable. If an NHS system incorrectly rejects 10% of claims, that's a disaster.
Ha, yeh right. They have proven themselves just as bad as mangaging those projects. In fact it is one of the only areas where the UK Parliament Intelligence and Security Committee has really given them a bit of a kicking.
While the other replies here have some very good ideas, there might be another reason for this (perceived? real?) difference that is far simpler: running a webapp on the scale of Obamacare/NHS is probably a much harder job than tapping the internet backbone.
For the NSA/GCHQ/etc, the actual engineering task of setting up some DPI boxes, beam-splitters, and other wiretap equipment is not particularly complicated on the engineering side[1]. The difficulty with that kind of project is access and legality. Even their massive DB in Utah, while (very) large, is probably a fairly simple schema compared to a many business DBs.
Compare that to a healthcare website, that has to implement many random features mandated by law in a way that is compatible with a giant list of "standards", medical jargons, insurance requirements of all sorts, financial regulations, to name just a few of the details. And it has to work, because lives are going to rely on it. While this kind of website might be simple in concept, the random, contradictory, poorly-specified, "design by committee" project requirements are going to be nasty.
[1] crypto may be different, but very little of what the NSA is currently doing has anything to do with crypto. They don't even need side-channel attacks when it's all in plaintext... sigh
Government department may be incompetent, private defense contractors are some of most capable organizations in the world. Different US government operations are being increasingly privatized. One difference between private contractors and government organizations, is contractors are not held accountable to oversight (to congress for example) as the government entities are.
They fixed Obamacare's website and they probably have even more incentives to fix the spying applications (if they are broken)... catching "terrorists" might be a low priority goal on their list which would explain why the reporters say that the NSA surveillance played little role in foiling terror plots.
It's not that the Intelligence Community has their act together any better, it's that they have more money to throw at the problems. Less regulation and oversight probably "help" things, too.
"First, the data stream is filtered through what is known as MVR (Massive Volume Reduction), which immediately rejects high-volume, low-value traffic, such as peer-to-peer downloads. This reduces the volume by about 30%."
So all ye folks can start disguising your packets as BitTorrent porn downloads now :)
It's also how they tried to mislead the public by saying NSA only collects "4 percent" of the world's data" or something - but "forgot" to mention that data could be emails, chats and urls (essentially the content itself), while the majority of the data is stuff like Youtube/Netflix content and torrents.
If you're at all important/a threat, they're paying particular attention to your porn habits so as to bolster their dossier on you. This is for later public release/blackmail purposes.
Why specifically? I always considered this a relatively safe way to transmit encrypted data. Assuming you have secure key exchange worked out, which is always the harder part anyway.
Ah, I guess I should have qualified my statement with "assuming the underlying crypto of truecrypt is secure, does sharing the ciphertext via bittorrent introduce any further side channel attacks outside of compromising the key exchange?"
It must suck being the people assigned to damage control on these continual document releases.
I wonder if NSA and GHCQ are changing program code names as they become compromised, or if at some point they just changed all code names regardless. I suspect the latter. If that is the case, I imagine it was probably a bureaucratic nightmare of biblical proportions.
