Hacker News new | comments | ask | show | jobs | submit login
State paid $22K each for Internet routers (wvgazette.com)
148 points by Lazare on May 11, 2012 | hide | past | web | favorite | 131 comments

To be clear, from the article, the routers themselves cost $7,800. And, if I'm rolling out a state infrastructure, and wondering what I can put in place for the next 10 years to serve as a foundation, you could certainly do worse than the 3945 - it's a very flexible ISR, and, all-things being equal, it's probably not worth the hassle of putting 2921s in some locations and 3945s in others. Who knows how much bandwidth you'll want on these high-speed fiber connections 5+ years from now - the 3945 is rated for 350 megabits/second (with features), the 2921 tops out at 75 megabits.

Amortized over 10 years, I would have chosen the 3945 everywhere versus sticking 2921s in some places (the ISR that would have been an alternative) and 3945s in others. Single Security Policy. Single IOS update Policy. Zero doubt as to what features will run in a particular location.

I think what most people have difficulty with is that they are comparing this decision to roll out a state communications infrastructure with the fact that they can go connect a $60 linksys wrt54G in their house and serve a dozen people without breaking a sweat. And get wireless as well! The issues involved in scaling that across the state, while looking to the future, and managing all that gear is a different challenge though.

I don't see any huge scandal here.

That seems like a pretty silly argument to me. The library with 4 computers would be just fine with the $60 router. How is it any harder to manage that gear compared to managing the completely unneeded equipment that the local employees have no clue how to run.

Not to mention spending $14k on upgrades for many places that don't need that particular upgrade. I don't care if it's a little simpler to buy all the same, it's absurd to spend millions and millions of taxpayer dollars on something that is completely unnecessary.

I'm not really sure where it came from, but the Slashdot summary states that a lot of these things were never even un-boxed. If true, that's an incredible waste and far-outweighs any convenience or stability benefits gained by using homogeneous hardware.


Interestingly enough - I can see both sides of the argument. When I was building the IT infrastructure for SSN, I ran the company's IT and it's dozen employees on a $75 linksys (and a half-dozen Poweredge Dell Server) for the first 5-6 months. Eventually, we grew to around 20 employees, picked up a data center colo-cabinet for our customer apps and needed a bit more robustness, so I purchased a $300 used Cisco 2621xm, a couple T1 WICs for $500, and ran the company for the next year on that 3 mbit pipe - the company got to around 50 employees before we moved. For the first two years of its life I spent a grand total of $875 on routing hardware and 50 employees were able to get their work done (including code pushes to our data center) over that 3 mbit pipe.

When I handed over network engineering, though, our last purchase order for networking equipment (with around 500 employees) - was north of $500K (including very hefty Cisco Discounts) (This isn't including the Data Center infrastructure - by then we had colo space in three data centers - just the corporate infrastructure for three buildings)

It's hard to explain without living through it - but the decisions that make sense when you are small, and you aren't paying your network engineers $150K/year (fully loaded), and you aren't trying to figure out how to handle pager duty, and warehouse and deal with RMAs, and support various types of hardware, and worry about rotating inventory, and dealing with upgrade cycles, and manage security, and patch levels, and remote administrations - not even considering future feature enhancements and performance (75 mbits sounds great today - but what about the future?) - when you get to scale, the CapEx (the capital cost of the hardware) starts to have fewer consequences on a technology investment (particularly over 10 years) than the other elements. Not to mention that there are also political issues (Layer 8) associated with differing levels of services/features for small population centers versus large population centers.

Yes - this does result in seemingly ludicrous situations like a 4 computer library being run on a ISR capable of running 350 mbits/second without breaking a sweat - but in 5-8 years from now, some technology administrator for the state will take over that infrastructure, and I can guarantee you that they won't be thinking "My God, we're over provisioned on our networking equipment" - but instead, "Thank goodness I have a few more years of runway before we have to replace all this gear."

I realize I haven't fully fleshed out the argument as to why it might make sense to put these hefty branch routers into smaller locales, but hopefully it doesn't sound silly.

Hey - at least they didn't roll out 6509s in all these libraries. Now that would be something I could rant against. :-)

These problems with managing at "scale" btw, are one of the reasons why consumerization of IT is so popular. The reality is that individual decisions made at small scale, can sometimes be orders of magnitude more efficient than by the enterprise. Example - Our company doesn't have a policy regarding mobile device OS upgrades - which means they don't have to worry about standardizing on a particular mobile device, or test "authorized apps" (or even have authorized apps) against the OS upgrade, or manage the OS updates, etc... Basically, a few early adopters will do an OTA upgrade, watch it for a few days, and if their device doesn't brick or crash more frequently, and the forums seem to indicate it's a good version - they'll send out a note to an internal alias and everybody else will just upgrade their device. Absolutely zero corporate overhead required.

This is true of all sorts of "edge" infrastructure - Skype clients, IM clients, Laptops, backups (now that we have backblaze/crashplan), etc...

Deploying at scale loses all sorts of that efficiency, and should be reserved for "Core" things like your exchange server, filer, and networking. For everything else - try to empower your users - they'll probably be better at it than you will be doing it centrally.

This is a great thought - when talking about a growing business. Comparing it to a public library in a very small town (just over 6k) doesn't make any sense though. For the money, the library can buy a new router every year for next couple of centuries and keep up with current technology. Or we could have bought them a router and then given them $21,940 for new computers, books, facility upgrades etc.

Remember - these decisions weren't being made by the library, or for a 500 person company, they were part of a $126 million dollar infrastructure upgrade for over 1000 sites for the state of west virginia. The point I was trying to make was that the decision that makes sense for that library (individually - say the local librarian was given $25,000 and given freedom to spend it however they chose - they would probably do exactly as you suggested - drop $60 on a linksys and the rest on computers/books), no longer makes sense when you are rolling out $126 million worth of equipment. The most realistic alternative decision would have been to downgrade the library from a $7800 Cisco 3945 to a $2800 Cisco 2921 (an ISR with less capacity, but reasonably similar features and management). The Library would then have been on a 75 mbit/sec platform for the next 10 years. They wouldn't have received any of the savings to spend on books/computers, etc... They would have just gotten less capable networking equipment.

I could probably defend either decision (Going heterogenous 2921 (small) + 3945 (big) vs homogenous 3945 across the state) - but I know the one that would let me sleep easy for the next 10 years.

That's pretty circular logic. If they were given a $60 Linksys then it wouldn't be a $126M investment. It'd be a $1M investment.

The problem with this argument is that the blanket purchase only provides giant imagined future capacity for the smallest users. If they will actually need this quality of a router in 5-8 years, then the big users will definitely need even bigger ones. But apparently the state is OK with the biggest users not having that much breathing room. So why do the smaller ones need it?

The first thing that comes to my mind is that in 5-8 years, as the State of west Virginia starts to run out of headroom in the larger locations, they can selectively upgrade that 5% or so that require something larger, while leaving the other 95% on the previous iteration.

Also - one thing I've learned about networking sites (real world experience, two companies that went from a dozen to 500+) is that once you are able to satisfy one person with sufficient pipe - that same bandwidth is usually sufficient for the 20-50 people. The reality is that need for bandwidth is very bursty.

And in 5-8 years, there will be more powerful equipment available at lower prices for that 5% of infrastructure. And you still have 1100 5-8 year old routers at all your locations that might not be capable of running whatever physical layer link we're using 5-8 years from now.

Future proofing is one thing. This is like buying a 747 for a route that serves a dozen passengers a flight to prepare them for the future.

> The first thing that comes to my mind is that in 5-8 years ... they can selectively upgrade that 5% or so that require something larger, while leaving the other 95% on the previous iteration.

In other words, they'll be in the exact same position that they were evidently trying to avoid from the start.

The thing is that these libraries are not going to grow to several hundred seats in the next 5-10 years, like the company you worked for. They will remain at their level of 5-20 seats. As such there is no need, nor will there ever be, for such strong infrastructure in a small communal library. They'd easily be able to make do with buying a 100$ router every second year.

This smug, platitudinous self-serving drivel is all that a network executive needs to know to misspend millions of dollars. You conveniently neglect to mention remote network management facilities that will handle configuration and monitoring of the routers that could have been purchased instead of a fully loaded 3945, including the $487 CISCO router that was mentioned in the article. How do you justify this preposterous expense? "What if the state decides on an unprecedented expansion of library facilities?" In this economic climate? "What if homeland security needs emergency facilities and this is the only site available?" You didn't read the article where two identical enterprise level routers were installed in the same little neighborhood. What about the efforts to engage ham radio operators--West Virginia is full of them. So much for the big picture. You live in a world where automating the administrative aspects of networking hasn't kept up with the network devices themselves. At least that's what you tell your superiors. It's as if remote network management facilities don't exist and you're still upgrading IOS on routers state-wide by hand with TFTP. Don't tell me: you support one-size-fits all cable management for the state too.

Yes exactly, homeland security will need emergency facilities for the next terrorist attack and they can conveniently use the libraries. It's a good thing they had an ex-homeland security person to manage these tough decisions and prepare the libraries for the coming internet armaggedon.

This is pure, unadulterated nonsense. There are plenty of ways to manage everything you mentioned with a tiered approach to the IT needs of the municipalities of the state without resorting to grossly over accommodating every location. Hell, they could've even used some of that money for regional IT management positions, putting people back to work.

Shame on you for justifying this nonsense in any way shape or form. This is textbook waste ala bureaucratic laziness.

When smart and experienced people tell you that things work differently in large orgs and govs, they don't primarily mean that in a negative way (...that the system is broken), they mean that as a matter of fact (...that the system is completely different).

It's not broken, it's not "fixable", it's just the nature of it all.

Let me put it to you this way, if they purchased $60 dollar routers, there would be a scandal about the incompetence and lack of forward vision at the top of HN right now.

Ten years of IT wages seems a bit more costly than some fancy hardware, particularly when you remember the pension.

Does that in any way change his argument that the money could have been put to better use?

I don't think it does. In the example put forth in the article, the library had only four computers! In the case of just this one library, they could have had: ipads, kindles, more computers, more books and other media.

I think it's kind of a no-brainer that this is a prime example of incompetence in government.

They would also need to hire someone to manage those devices which would cost maybe 2x the router year after year. It'd suck too (not going to get a competent person in that position).

My company provides free I.T. services, support, and sometimes equipment to non-profits and other community organizations. We would be happy to be the on-call techs for stuff like that.

I would be surprised if there wasn't somebody in their area that does the same.

Yes, because his argument was that it could be done with cheaper hardware and some IT-smarts (aka salaried employees managing it).

> ...and some IT-smarts...

I don't think he said that. For one, cheaper hardware doesn't imply increased management costs; for two, the more expensive hardware didn't come with a support contract included in the price (at least, it's not mentioned in the article); for three, his comment reads to me that they could have optionally put the money towards employing people, to put people back to work -- i.e., it was one example of a better way to spend the money. I don't see anything in his comment that implies, "cheaper hardware and some IT-smarts...".

Some dude is checking (or should be checking) on the PCs in that library anyway.

Hell, a college kid could do the job.

>This is pure, unadulterated nonsense. There are plenty of ways to manage everything you mentioned with a tiered approach to the IT needs of the municipalities of the state without resorting to grossly over accommodating every location. Hell, they could've even used some of that money for regional IT management positions, putting people back to work. Shame on you for justifying this nonsense in any way shape or form. This is textbook waste ala bureaucratic laziness.

And this BS attitude is part of the reason why the US has one of the worst internet infrastructures in the western world. Let's just use what is adequate for today and save some pennies, sure cable internet (or 640K) should be enough for everybody.

Not to mention that you're suggesting micromanaging and needlessly complicating the municipality IT infrastructure for marginal gains (compared to IT staff wages). Not to mention ignoring the "economy of scale" and mass-ordering benefits.

The benefits from economies of scale and mass-ordering still apply to much lower priced hardware.

The IT staff wages will still have to be paid because hardware needs support regardless of the unit price.

>The benefits from economies of scale and mass-ordering still apply to much lower priced hardware.

If you break the uniformity of the order, to lower priced for some and higher priced for others (for which the lower priced won't do) then you break the economy of scale benefit.

Instead of buying 100 same machines, you then buy a 100 machines of A/B/C etc types. You still buy bulk (within every machine class), but not as "bulk" as before.

Sure, the savings from buying bulk the more expensive unit may not totally offset the savings of buying units with different price points, according to each deployment's specific needs. But uniformity has a lot more benefits, too.

>The IT staff wages will still have to be paid because hardware needs support regardless of the unit price.

Sure, and nobody argued that. What we said is that since the price of the hardware is a drop in the bucket (marginal) compared to the total cost (mainly the IT staff costs), then it doesn't matter. Paying 30,100,000 compared to 30,020,000 is not much of a difference (this one is 0.2%, but even if it was 5-10% it would not matter much).

Also there will be savings in the staff training (the have to only learn one unit), fixing costs, etc.

The most important reason though, is that we are very bad at anticipating future needs in internet infrastructure. Better get something that has a lot of headroom for future needs.

If you are putting routers in a school that might serve a few thousand students and teachers, and a branch library that might be serve 5 patrons, you really can't afford to adopt a heterogeneous hardware strategy just to make admin easier.

It is surely more cost effective to use the cheaper router, and replace them when necessary than to buy the uber-model that will still be state of the art in 10 years.

Bucket system needs into three categories (large, medium, small) and find three routers that best manage those needs. Hire a full-time government worker to facilitate categorization of each installation location and manage the installation. Replace upgrade as needed.

Yes - that's actually a reasonable alternative. I'm hoping that someone did the math, and looked at Option #1: $7800 Cisco 3945, $2800 Cisco 2921, $900 Cisco 1921 vs Option #2 - $7800 Cisco 3945 across the board) and then compared it with the costs of having to prematurely replace those 2921s/1921s when they realized the 75 mbits/second wasn't what it was in 2012 (or some service modules with cool feature (Telepresence?) wouldn't work in the 1921)

Another thing to realize - they probably get one chance to upgrade their infrastructure like this. Sticking a 3945 in now means they can save money on upgrades for the next 10+ years.

Government money also seems to sometimes dry up as priorities are shifted - who says there'll be money over the next 10 years for upgrades.

I think you meant homogeneous, as in all the routers being the same?

Yes! sorry, I meant homogenous. Even a small state like West Virginia is too big and the needs are too diverse for a one size fits all approach.

The nonsense about $60 linksys devices was addressed in the article. Did you bother to read it? CISCO quoted a $487 router. You expect the taxpayer to foot the bill because you're too lazy to keep track of which router you've installed at each location? I deal with bureaucrats like you all the time. This is the last place I would have expected to find them.

The bureaucrats I deal with would buy the various routers at best prices and then proceed to build a software system to track which router was installed where that would cost > top of the line routers and would be built by a company friendly to the person making the decision. Initial costs would be low to win the bid but many "required changes" would ramp the bill up. They would create software jobs and get optimal router pricing - if that's not efficiency I don't know what is !

Those systems exist--CISCO has been selling them for years. Or have we identified another area ripe for disruption?

I can assure you that any existing software would fail to meet the requirements (hint - primary requirement being related to/friends with/lobbying the decision maker).

You're right if nepotism is a requirement. But if we could suspend our cynicism long enough to consider available enterprise products (which is a tremendous strain admittedly for anyone who agrees with Paul Graham about "enterprise" computing) then as a comment below points out, CISCO sells Monitoring as a service and there are other vendors that provide this.

CISCO also sells Monitoring-aaS, which will do discovery, monitoring and upgrade for $xxx/device/yr.

The other thing is that the bids had to be submitted very quickly or they risked not getting any funding at all. In that light, they can't really be blamed for not doing research into the most efficient allocation of money! (To be clear, I blame the stupid bid process that rewards rushed decisions and punishes anyone who stops to think.)

Yes, I have been there, bid by fax with 6h turnaround in one case but that was only a few thousand.

Fix the process.

It hardly seems like managing gear would be all that difficult at this kind of margin. Hire someone full-time at 100K annually who manages the systems and upgrades when necessary.

"West Virginia Homeland Security chief Jimmy Gianato, who's leading the state broadband project, defended the $24 million router purchase last week, saying the devices "could meet many different needs and be used for multiple applications."

"Our main concerns were to not have something that would become obsolete in a couple of years," Gianato said. "Looking at how technology evolves, we wanted something that was scalable, expandable and viable, five to 10 years out. We wanted to make sure every place had the same opportunity across the state."

Wow. I've spent some time in West Virginia libraries, and yes the internet connection was slow, but it's not due to the router. And in most towns, libraries and schools are not going to be serving anyone outside their walls with some kind of WAN. Best case will be public wireless near the building.

And how is it that the "West Virginia Homeland Security chief" is "leading the state broadband project"? Seems like bureaucratic overreach, technical ignorance, and budget authority all wrapped in.

Why not spend $50 for a router for libraries and something less than $22k per school?

These routers are pretty good choices for the schools. For the smaller sites it's hard to say without knowing their exact requirements for the future. For example this might be related to your question about the Homeland Security Chief being involved. Libraries are multipurpose buildings. In an emergency they could be used as an emergency response command center. Perhaps the library is used as a polling place and they have specific security requirements. The state may have decided it was in their best interest long term to equip all government buildings with a baseline level of equipment.

I think part of the reaction to this story is people coming to terms with how much real network equipment costs. It's definitely not cheap. You're paying for features, service, long-term support, upgradability, reliability, etc. Some of these routers will probably be in use 15 years from now. I've still got a Cisco 2500 series router chugging away. Next year will be it's 20th birthday. This stuff is built to last.

What decent router costs $50? That blue soho router your thinking of is for your mom's house not a statewide installation.

A router that can handle 5-30 clients at current internet speed standards within a library setting. Yes, bigger installations could benefit from a $20k router, but the vast majority of these routers are a complete waste of money in WV outside of Charleston and Morgantown.

Unless there is some big backbone being built that can service a mostly small-town and very mountainous state, spending $20k (each!) now so that your routing infrastructure within schools and libraries won't be too slow in a couple years is kinda crazy.

I run my office network off of a $50 router quite happily. But that's because I'm right there to take care of it. Earlier this year, for example, our network got a fever:


If I needed to cover 1,000 widely scattered locations for the next decade, there is no way I would try to do that with a bunch of consumer-grade stuff. Especially when trying to manage a bunch of existing T1s and a conversion to fiber.

For a library that only has four computers... I don't see the difference from my Mom's house.

The difference is that the library doesn't have nearly as many devices as my house.

Am I the only person that is (not really of course!) tempted to put on official-looking clothing and confidently walk in with a laptop and Cisco console cable, nonchalantly telling the librarian their "Internet box" "needs a service upgrade".

And then quietly replace it (after dumping the config and replicating the configuration) with a $100 half-depth Supermicro server bought off Ebay, running OpenBSD?

And then walk out, like the Grinch after he artfully puts Cindy Lou Who to bed, with a nice new 3945 for my own use?

EDIT: forgot to mention, I need a CF card reader also to get the config off the Cisco flash.

If you then sold the router on the second-hand market and gave the money to the library to spend on materials that they actually needed, your plan could do some real good.

At least you'd be creating some value with those tax dollars.

Thinking the exact same thing -- there are plenty of libraries to choose from...

Considering you're talking about pulling the config off flash, which is where the OS and not the config is stored, I'm not too worried about what you plan to do.

From the article: "I'm not an expert on the technical side," [Gianato] said, "but these have all kinds of capabilities and applications."

Earlier: "Gianato acknowledged that he didn't heed Dunlap's advice or wait for an evaluation.

'The routers already had been bid out,' Gianato said. 'I think John was looking at our needs now, not looking at our needs into the future."


So not only does Gianato admit he doesn't know anything (which means he's not even suffering from the Dunning-Kruger effect), but he also admits that he's not sure what his IT advisor was thinking about, but, eh, it doesn't really matter because the people who are the experts obviously don't think about the long-term and I'm not going to bother asking them about it anyway, because as we all know, bids are binding contracts. Right.

I'm surprised this article actually does a decent job covering both sides of the story. When you look at how flexible the ISRs are, and the likelihood these devices will be in service for probably the next 10 years or more, it's not not the worst investment they could make. They're actually well sized for the high schools and other larger sites. Obviously any 3900 series is going to be overkill for a little library but you do have to look at support costs, service contracts, future requirements, future upgrade costs, etc to get the full picture. This is not a 'bridge to nowhere' situation.

You can get an ASA 5505 firewall with all the features you'll ever need for a small library installation for about $1200 retail (hell of a lot less if you're buying, say, 50 of them), and any network engineer who can't handle ASA configuration in his sleep should be fired immediately for gross incompetence.

There is no excuse for this insanity.

I have a buddy who sells routing equipment at this level. Doesn't have a degree in Fahrenheit, but just cleared a mil in his bank account, owns his 6 bedroom Connecticut home outright and keeps a beach house in Mystic. He's 38. I'm 36, a doc, and still paying off student loans and working for Uncle Sam.

Private sector FTW. Please, please I hope everyone who reads this begins to understand that governments are not meant to allocate resources. The idea (promoted by politicians, mostly democrats) that we should give the government our money and they will spend it wisely "for the betterment of all" is purely, simply and fundamentally flawed.

Out government should be small. It should perform a few basic functions like making sure we are secure, our trading interests are promoted and our banking and financial markets are healthy. It should educate our children to the best standards in the world. Finally it should make sure we are energy independent and our food and water is uncontaminated. That's pretty much it.

Please don't make this a democrat/republican debate. This is the crooked way that all of our US government is spending our tax money. We need corporate money out of politics if we want to prevent these outlandish contracts from being awarded to the corporations that donate enough money to campaigns.

It's not a democrat/republican or big/small government thing. It's about corruption of the system.

Out of your comment, I agree most with that our government needs to help educate our children. Without an educated population, things only get worse.

how is this not a big/small government thing? We gave away billions of dollars in a stimulus program with little to no oversight that is very clearly getting wasted here.

While the republicans are certainly guilty of lots of spending and many things, democrats are clearly the party that is guilty of more spending and even bigger government.

There are plenty of countries whose governments are far to the left of the Democrats who spend money much more wisely, so I don't see it as inherently tied. For example, the Scandinavian countries have much larger governments proportionally, but generally spend more wisely, and have more transparent bidding/provisioning of their state-funded services, so they get more service and less waste in return for their money.

Thank you. Finally some common sense to counter the libertarian anti-government sentiment. Whenever they tell you that government run schools are failing, ask them why the government run schools in other countries are performing so much better. Whenever they say that government can't allocate resources efficiently, point them to the governments that have balanced their budgets. It is possible. You don't have to throw the baby out with the bathwater. The solution to bad government is not to abandon it, but to work ever harder to improve it.

Getting the money / corporate influence out of government would be the place to start.

They probably do spend more wisely, but that doesn't necessarily mean they are spending on something they should be. For example in this case buying the cheaper routers would be spending more wisely, but could pretty easily be argued that this isn't something we should be spending money on. Certainly not at the federal level. Why is my tax money in Tennessee being spent on routers in WV?

There are a few components to this particular screw-up. 1 is big government spending (with little oversight) and 2 is one idiot in WV with a big budget who failed miserably at spending it.

But it could pretty easily be argued that it's something that we should be spending money on. The existence of an argument against a course of action doesn't, on its face, preclude taking that course of action. Such a line of thinking is, well, bizarre.

"1 is big government spending (with little oversight)"

The obvious problem here isn't "big government" spending. It's spending without oversight. This problem occurs all the time everywhere, in sectors both public and private. To claim that private businesses are immune to this by nature of their ability to fail and thus inherently a better mechanism than government is an absolute farce.

Take ownership of your government. To view the elimination of government as a solution to bad governance is both a lazy approach to problem solving and toxic to proper governing.

Private sector isn't immune, just not as bad.

Nobody said anything about elimination of government, just less of it.

It's not a big/small government thing because you are being played by what people told you the two party system is about, i.e. republicans are for small government and democrats are for big government. It's a bunch of bullshit.

I'm sorry to burst your bubble, but our current presidents and congress members do what will appease the general population in the very near term, no matter what they are supposed to do according to their party lines. They are all equally guilty of spending.

You know, the TSA in the airports, which has a larger budget than the FBI? Guess who created and nurtured that entity? Your small government friends. And, who spent so much money on the wars in Iraq and Afghanistan that it is bankrupting the country. We can't even get involved in another war right now because we simply can't afford it. It is questionable if we will ever be able to lead a war like that again, due to budget constraints.

Then, we have the wall street bailouts. Which, by the way, aren't republicans supposed to be the only friends of wall street? Not if letting the banks fail mean that you are going to have to line up for the bread line tomorrow morning.

The big government people aren't exactly cherubs either. How many RIAA lawyers have been appointed to the Justice Department by the current administration? I think it's 5, so far. Enjoy additional Internet censorship.

Sorry to be such a downer, but things are just getting worse and the current two party system is a complete facade. I'm envious of countries like Germany that are seeing a surge in pirate party support. Unfortunately, it's going to take decades to undo this mess and, frankly, a large majority of baby boomers are going to need to expire naturally before the voting population comes back into balance.

democrats are clearly the party that is guilty of more spending and even bigger government.

Let me guess... military spending doesn't count as "more spending". The $800 billion allocated for 10 years of stimulus is a paltry sum compared to the $700+ billion we intended to spend every year on military, and we all know that money is spent very efficiently.

Meanwhile, federal spending has increased every year under both parties for the last two decades, so you don't really have a leg to stand on with regard to your "democrats are the big government spenders" FUD.

Are you kidding? Look at the budgets (or lack thereof) under Bush and Obama. They aren't even in the same neighborhood. I'll be the first to agree that there is a lot of waste in defense spending. My first job was at a defense contractor working with the air force. I've seen it.

Just because there is waste in defense, doesn't mean there isn't more elsewhere. Defense is one of the few things are government is actually tasked with. All the other shit we waste money on isn't part of what they are supposed to be doing. Roads and armies, that's all I expect. I don't need the government to feed me or wipe my ass.

consider this statement politically agnostic, but military spending is the closest thing we have to govt subsidized R+D in this country. there several enormously useful and successful military inventions that no commercial company would have touched.

gps and the internet are two of the most notable and visible examples with astronomical infrastructure costs. no private company would undertake a risk like this yet both have substantially improved society. the microwave and digital cameras are two others that a private company may have eventually created.

this isn't an argument for military spending, just the fact that without a government body that focuses entirely on applied research and development the military is the next best thing.

As a Canadian I find the persistance of the "small vs. big" government narrative in American politics bemusing - there is no small government party in the states, only branding.

They wouldn't be buying any routers without the federal dollars to build fiber so it's kind of a moot point in this situation. Anyone who has worked in the private sector can tell you this type of stuff happens constantly. Too many technically ignorant people making decisions is the problem public or private. A private contractor could have sold them on the same exact package for the same reasons. Maybe even charged more for it.

As for your core point about the role of government... just look at the top 10 economies of the world. Most are not libertarian paradises and they're doing OK. It seems to me both models could probably work if executed properly. I doubt one model is inherently right and one is inherently wrong. Most likely some hybrid is the best solution for everyone.

If you had worked in a big company environment, you would know that this kind of nonsense happens all the time, in both private and public environments. 30000$ routers are pocket money compared to some of the things I have seen (millions of $ spent to develop and maintain in-house software that cannot do 10 % of what a basic open source solution would do).

When you reach a certain size, and you have many competing groups, all with their agenda with the same company, it becomes difficult to allocate resources correctly, especially in a domain that is not considered "core of business" (IT being a common case, sadly).

you should see the crazy amount big companies spend on IBM tools. people pay big money for clearcase.

licenses for clearcase cost enormous amounts. yet clearcase is missing many of the features found in open-source (and free) distributed version control systems. meanwhile, there is no free alternative to the routers, which may actually be necessary at some of the installed locations.

this doesn't include the huge server costs to run clearcase after you buy the license, which aren't necessary for distributed version control systems like git or mercurial. and big companies often spend more money buying other IBM tools to integrate for which free alternatives exist...

the amount of money spent on this purchase is fucking peanuts compared to what many large private sector companies spend on IBM software that lacks most of the features found in your favorite open-source alternative.

Very well said. I agree with your point that governments aren't great at allocating resources, and should concentrate on the basics.

I disagree with the sidebar that it's mostly Democrats trying to take our money. Both parties do it, just with different priorities and in different ways, with all the accepted terms used to discuss it.

In other news, JP Morgan just posted a $2 billion loss due to their hedge trading unit. One of the paragons of the capitalist system. Big and old and supposedly "successful" and supposedly run by very smart people who deserve very high pay because what they're doing is very hard and technical, and only they know how to manage resources efficiently, unlike say the government. Let's not even get into the paychecks of all the CEO's, traders, partners, huge sums of money for what in many cases involves just hiring other people and telling them what to do, pointing out the obvious, etc. Meanwhile the little people, the people that do the vast majority of the real and necessary work in our communities --- teachers, doctors, repairmen, cops, firefighters, road crew, waiters, delivery folks, carpenters, electricians, etc. --- get the least amount of pay. Because that's "efficient". If that's an efficient allocation of resources, maybe that's a bad thing.

I think we agree on the idea that government should be small, or at least as small as possible. And we agree on the roles it should play. But I don't buy into the argument that government is inherently bad at allocating resources efficiently and private businesses are inherently good. I've seen way way too many examples that contradict the latter. I think it ultimately comes to individual situations and the choices of individuals, what information is available to them, their goals, where their financial interests pull them.

> Gianato said putting the same size router in every school was about "equal opportunity."

A little known fact in the tech community is that a $7,800 router provides exactly 16.016 times more opportunity to a student than a $487 router.

All the tubes have to to be the same size. It's only fair.

Spending other people money is so easy. But $22k apiece instead of the $485 solution quoted in the article - that's madness. It can't be just gross incompetence.

I guess someone involved in the decision got some kind of kickback, gift, or will have a nice cushy job offer waiting for them at Cisco when they get fired as they should be.

Remember this article next time some liberal ask to raise taxes and justifies this by saying how stimulus money is important, and why everyone should have the same opportunities (quoted in the article - as if a 22k router gave any kind of opportunity to people browsing the net in a public library in WV).

And cry when they get trashed at the end of their lifecycle without having ever been connected to optic fiber.

"Remember this article next time some liberal ask to raise taxes..."

Unless I misread, the article is talking about West Virginia. I would hardly call WV a bastion of liberal thinking.

Never attribute to malice (or political leaning) that which is adequately explained by stupidity (or incompetence)

You're right. This isn't a "liberal" failing. Wasteful government spending is an affliction that spans the entire political spectrum.

Democrats controlled two of West Virginia's three House seats, both Senate seats, the Governor's mansion, and the state legislature (which they have held since 1928).

One of West Virginia's two current Democrat senators, Joe Manchin, was governor at the time of the stimulus and appointed Mr. Gianato.

The executive branch of the federal government in 2009-2010 was controlled by the Democrats, along with both houses of congress. The federal stimulus was passed with no Republican support.


The entire chain of custody from the grant through request for bids through the eventual approval was controlled by Democrats. The only people involved in this fiasco without a (D) after their name are those people who will have their taxes raised in order to pay for these routers, because of course they were not paid for with existing revenue.

This is very much a failing of Democrat administrations at a federal, state and local levels.

Democrats in West Virginia are hardly liberal - they're sort of a party on their own compared to the Democratic party elsewhere in the US.

He did actually say "Liberal" and not Democrat. There is a bit of a distinction there. Generally the Liberal party in the US isn't so Liberal and the Conservative party isn't so Conservative. I mean when you get down to it, in US politics...nobody's really much of anything. To peg it all down on a bunch of labels adds about nothing to meaningful discourse, because the labels don't mean anything.

wasteful spending is an affliction that spans the entire spectrum of human enterprise, both governmental and corporate.

Sure but in the private economy it is self correcting. Businesses that waste money go out of business.

Governments that waste money just raise taxes, borrow more money, or print more money. Corrections when they do come are often catastrophic in nature.

Not to mention this was federal stimulus funding.

I love West Virginia, and there is so much more important the state government could do with this kind of money if they're going to take it outside of "Homeland Security". As in, better health care or safety for the huge population of coal miners.

why do military closets cost 5000$ a piece? because it costs a sh#tload of money to sell the military a closet. i don't know where i read this but it is a great analogy.

i have been - in my time (2000? 2001? 2002? 2003? 2004? ...) - involved into selling the national austrian television their first video streaming solution (think: free (as in free beer) windows media streaming server) - it took man-years. man-years cost money.

I want to punch you in the face. ORF's IPTV stuff has been 100% unusable until the Mediathek came along.

I could further elaborate on the reasons why this matters to me, but I'm just relieved that it works OK now.

yeah, i felt guilty for years. glad it's gone now.


I'm not a programmer or technical person, I'm an end user who uses R, LaTeX and processing now and again. I hack an awesome spreadsheet. I have a degree and I do a professional job.

I could be a manager in a public service or an elected representative. Perhaps in a small rural authority.

Given the lack of consensus in this discussion thread, and evidence in the article of a similar lack of consensus between the various offices involved, how am I supposed to reach a decision?

Is there a case for some kind of planning toolkit or requirements estimation software? Is there an opportunity here?

PS: I and other colleagues did once have to help a senior manager spend out a £250k capital grant in 10 days. The idiotic spending deadline was due to delay in award of funds in a competitive funding round. We did ok but could have done better with another month or two to think through detailed requirements.

You use your intelligence. The Cisco person says you can do what you want with $500. Then do you continue to spend $30k instead? Do you think your small town library is going to grow into a college campus (as the other pro-mega-router people are expecting)

OK, I have to admit that looking at the picture in the article, I personally would have tried to vire some of the money for the super-router into a less fancy one and some decent all-in-one lcd desktop pcs for that table. The space saved by the all-in-ones could have provided a 'quiet table' for someone (and the PCs could double as DVD players).

I would have to make a spreadsheet (good at those) of all the sites and then include a rating for current traffic and then expected traffic growth over the life of the project. Yes, a library with 4 PCs is not going to have a lot of growth...

> Given the lack of consensus in this discussion thread, and evidence in the article of a similar lack of consensus between the various offices involved, how am I supposed to reach a decision?

1. You ask people who have already done this, like hotels who deploy thousands of networked sites. 2. You hold a bakeoff at a few dozen sites and see what works.

Yup, in my previous job that is exactly what we did, we found other Colleges who had done similar stuff and asked them how they did it.

I noticed some went for a 5 year sort of rolling provision making assumptions about increased traffic (but certainly no gold plating like in the original article) and others just bought what would work now and dealt with changes as they arose.

I have to say that the former strategy seemed to provide a more uniform system with less 'catches'. People don't share budgets so I have no idea which came in cheaper.

> Our main concerns were to not have something that would become obsolete in a couple of years

No you fool! These are designed to handle large loads, not protection from obsolescence!

Interestingly, the CEO of Cisco is from West Virginia and went to WVU - I believe he's one of the largest donors to WVU. (only know this because I'm also from WV, not suggesting any conspiracy here)

Sounds like a buffoon was in charge of the operation. Gianato suggesting that the unused/unneeded T1 interface cards could still be used for "video conferencing, wireless Internet and "voice over Internet protocol." is just painful to read. I am not sure how they can be used for such things if you don't actually stick them in a router!

I would still like to see a breakdown of the costs and what services are provided with the routers before I call it a total waste though. If it was just the routers then it was a waste. If it includes other things like a long term warranty & onsite service, then maybe not so much.

Some one should request a detailed FOIA.

West Virginia Homeland Security must be planning for a truly unprecedented expansion of libraries and schools, based on the capacity of the routers and the number of them allocated to sites with a handful of users. This is heartening, especially during a period of massive cutbacks to public education. ;)

When a public school/library buys a 22K router with government money, sounds like an overkill (handout) of massive proportions. Someone's cousin here was probably on the Cisco + Verizon sales team. This is frustrating.

I remember a few weeks ago PB made a comment that at least social networks and alike make it easier to spread the news, which eventually will lead to greater good (http://news.ycombinator.com/item?id=3890543). At least that's true, but doesn't feel like it's enough.

Hacker News, you continue to disappoint me. Every time I read an article, I know the comments are going to be filled with people arguing for the sake of feeling superior. And once again, they are.

No. This is not okay. No twisted sense of reason that only you have makes it okay. A $60 consumer router that needs to be upgraded every year is still cheaper than this. Even with support costs. People have been rolling out tiered infrastructure for years.

I describe this as 'funny money'. The government is spending other people's money.

I imagine the government worker that signed off on this purchase worrying a lot more about a dollar increase in the local lunch special than millions spent on equipment.

It is also funny money because the marginal cost of producing one of these routers is more like $500, not $20k. It is just a transfer payment to the company, it isn't the same as if the government burned $20k worth of barrels of oil. Nothing near that amount in resources was actually wasted.

Not quite. Cisco will probably take this purchase as a signal to produce more unnecessarily large routers. Those people paid to make the unnecessary routers could have used their valuable skills elsewhere. They may convince more people to learn how to build unnecessary routers instead of something more valuable. It's waste all the way down.

Not exactly, unless you think those people will literally use their salaries from the activities you described to buy something and go out and burn it. It might be a mis-distribution, and it might spur misguided investment in an area, but it isn't all squandered to the tune of exactly $20,000 or anywhere close to that. The marginal cost is a much more accurate measure.


Most of this is in practice amounts to the justification of bureaucrats too lazy to do a proper analysis. I work with them and this is exactly the generalized rubbish they spout--it NEVER gets deeper than this. Details are never forthcoming. Instead we have self-serving nonsense about the difficulty of maintaining databases of router configurations. It's as if remote management facilities don't exist--you tell your superiors that you're still upgrading IOS with TFTP, so you need all the routers to be the same.

You haven't mentioned the article once.

What do you think the probability of expanding a small library from four users to 50 is in this economic climate? Did you read that the stimulus was for the installation of fiber? Instead the state got routers equipped with T1 modules.

i live in a developing country. I am thinking what would happen here if a state offical had 24 million in his hands to spend on the infrastructure? The official may be corrupt and this thing may never go to proper bidding. So official and a company would share say 10 million as the profit. But 14 million would still go to something useful.

In this story, at least 20 million of the money seems to be completely wasted. (Higher capacity router does not mean better internet access for 4-computer lib.)

Wait, not finished. So our corrupt official has 2 million in his hands. He may buy a car or a house, but probably he would spend it for something useful. So much of the value would return to the market. This is also true for the vendor company. But in the US case, the value is gone. it is like you burned the cash.

Moral of the story: stupidity is more dangerous than corruption. You may want to have a corrupt offical than a stupid one.

Finally, that stupid would never have survived in the relatively corrupt but wild political system of my country.

In politics, stupidity is not a handicap

Yours is the most bizarre, vaguely chauvinistic straw man rant that I've ever read.

where is the straw man?

Time to connect the entire neighborhood to those machines and share the cost /administration.

Well... at least they got high-powered routers for $22K apiece. But, it's a shame there isn't a law on the books that selling the government something at 50X the market price / need is fraud.*

* Yes, I realize this would bankrupt half the military contractors as well.

Can't blame them for selling them something they asked for. Who's to say they were even told what they were for. This is government waste and incompetence, nothing more.

You are correct on the military contractors as well though. Lots of government waste and incompetence there as well.

I see a lot of posters argueing about the type of cisco router that should have been used or if a soho Linksys router should have been used instead.

Assuming that the government has some rules that requires them not to deploy soho or open source products and they have to spend the money on some big brand company: how come Juniper was not taken into consideration (or other known network vendors)?

I would argue that Juniper could provide cheaper equipment of simmilar of higher spec (let's ignore for a moment if that is oversized or not) - and it becomes even cheaper if you consider the simpler licensing terms and upgrade support.

Isn't the government supposed to run open tenders? Is cisco mandatory in the US?

Verizon has successfully siphoned a bunch of money from the govt and by extension, taxpayers, yet again.

Anyone familiar with telecom and living in PA, WV, other states that have Verizon as the ILEC, knows this.

What piece of technology will stay relevant `5-10 years out`?!?

This is why I'm a libertarian.

Most of these places don't know what the box is and when Johnny down the street comes in to fix their internet it will be replaced by a blue soho router. There will be theft and waste and probably 20% of these will be missing within a couple years.

A librarian will notice a computer missing, but an expensive router replaced with a cheap one won't even be noticed for years.

Similar story in India would be like "State paid $20k each for Internet routers worth $20"


somehow i doubt these sites have multiple uplinks, so they certainly don't need BGP routers. even if they're doing IPSec between some locations this can be handled by an old piece-of-junk $100 computer running BSD or nix, e.g. with isakmpd and pf.

stuff like this is why america's infrastructure is crumbling: dodgy contractors looking for an excuse to upcharge municipalities for irrelevant gear and services.

Here in Australia, our local DMV (RTA) is all MacBook Pros, all running Windows XP in virtualization. Would love to meet the dipshit that authorized this travesty.

You really want to advocate installing "old piece of junk" routers across the entire state? Have you ever managed a project this large? And if so did you use old piece of junk equipment?

$500-1k appliances with Atom CPUs running pfSense, please, not $100 clunkers with spinning drives and dust-laden fans.

Anything more than that for a moderately large library or most schools is an outlandish, shameful waste of money and equipment.

This is absolutely disgusting to me. The people that run big and small government in this country are complete idiots.

The price includes maintenance and support correct? It is not 22k for the metal only?

it's sad, but nothing new. either verizon is lying about who asked for the specific model, or some inexperienced network "architect" decided not to do the due diligence to spec out what was required, and just asked for the moon.

The price includes maintenance and support correct? Its 22k for the metal only?

Smh, so much waste.

Regarding planning for the future, this seems like someone who was used to buying machinery was put in charge of buying technology. Sure, in five years the library might need a gigabit of bandwidth, but by then the Linksys WRT may well handle it for $100. It's not like you're buying a work truck, which may well serve the government for 5, 10, or even 15 years (assuming low usage), or a lathe that might be used intermittently for 35 years.

There's a lot to be said for uniformity in deployment, but that's why you would define, say, three tiers of supported hardware instead of kitting out everyone with top-of-the-line.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact