
State paid $22K each for Internet routers - Lazare
http://wvgazette.com/News/201205050057
======
ghshephard
To be clear, from the article, the routers themselves cost $7,800. And, if I'm
rolling out a state infrastructure, and wondering what I can put in place for
the next 10 years to serve as a foundation, you could certainly do worse than
the 3945 - it's a very flexible ISR, and, all-things being equal, it's
probably not worth the hassle of putting 2921s in some locations and 3945s in
others. Who knows how much bandwidth you'll want on these high-speed fiber
connections 5+ years from now - the 3945 is rated for 350 megabits/second
(with features), the 2921 tops out at 75 megabits.

Amortized over 10 years, I would have chosen the 3945 everywhere versus
sticking 2921s in some places (the ISR that would have been an alternative)
and 3945s in others. Single Security Policy. Single IOS update Policy. Zero
doubt as to what features will run in a particular location.

I think what most people have difficulty with is that they are comparing this
decision to roll out a state communications infrastructure with the fact that
they can go connect a $60 linksys wrt54G in their house and serve a dozen
people without breaking a sweat. And get wireless as well! The issues involved
in scaling that across the state, while looking to the future, and managing
all that gear is a different challenge though.

I don't see any huge scandal here.

~~~
dclowd9901
This is pure, unadulterated nonsense. There are plenty of ways to manage
everything you mentioned with a tiered approach to the IT needs of the
municipalities of the state without resorting to grossly over accommodating
_every_ location. Hell, they could've even used some of that money for
regional IT management positions, putting people back to work.

Shame on you for justifying this nonsense in any way shape or form. This is
textbook waste ala bureaucratic laziness.

~~~
sliverstorm
Ten years of IT wages seems a bit more costly than some fancy hardware,
particularly when you remember the pension.

~~~
thaumaturgy
Does that in any way change his argument that the money could have been put to
better use?

~~~
readme
I don't think it does. In the example put forth in the article, the library
had only four computers! In the case of just this one library, they could have
had: ipads, kindles, more computers, more books and other media.

I think it's kind of a no-brainer that this is a prime example of incompetence
in government.

~~~
bricestacey
They would also need to hire someone to manage those devices which would cost
maybe 2x the router year after year. It'd suck too (not going to get a
competent person in that position).

~~~
thaumaturgy
My company provides free I.T. services, support, and sometimes equipment to
non-profits and other community organizations. We would be happy to be the on-
call techs for stuff like that.

I would be surprised if there wasn't somebody in their area that does the
same.

------
mikescar
"West Virginia Homeland Security chief Jimmy Gianato, who's leading the state
broadband project, defended the $24 million router purchase last week, saying
the devices "could meet many different needs and be used for multiple
applications."

"Our main concerns were to not have something that would become obsolete in a
couple of years," Gianato said. "Looking at how technology evolves, we wanted
something that was scalable, expandable and viable, five to 10 years out. We
wanted to make sure every place had the same opportunity across the state."

Wow. I've spent some time in West Virginia libraries, and yes the internet
connection was slow, but it's not due to the router. And in most towns,
libraries and schools are not going to be serving anyone outside their walls
with some kind of WAN. Best case will be public wireless near the building.

And how is it that the "West Virginia Homeland Security chief" is "leading the
state broadband project"? Seems like bureaucratic overreach, technical
ignorance, and budget authority all wrapped in.

Why not spend $50 for a router for libraries and something less than $22k per
school?

~~~
dfc
What decent router costs $50? That blue soho router your thinking of is for
your mom's house not a statewide installation.

~~~
mikescar
A router that can handle 5-30 clients at current internet speed standards
within a library setting. Yes, bigger installations could benefit from a $20k
router, but the vast majority of these routers are a complete waste of money
in WV outside of Charleston and Morgantown.

Unless there is some big backbone being built that can service a mostly small-
town and very mountainous state, spending $20k (each!) now so that your
routing infrastructure within schools and libraries won't be too slow in a
couple years is kinda crazy.

~~~
wpietri
I run my office network off of a $50 router quite happily. But that's because
I'm right there to take care of it. Earlier this year, for example, our
network got a fever:

[http://nerdfeed.needfeed.com/blog/2012/02/our-network-
has-a-...](http://nerdfeed.needfeed.com/blog/2012/02/our-network-has-a-fever/)

If I needed to cover 1,000 widely scattered locations for the next decade,
there is no way I would try to do that with a bunch of consumer-grade stuff.
Especially when trying to manage a bunch of existing T1s and a conversion to
fiber.

------
patrickgzill
Am I the only person that is (not really of course!) tempted to put on
official-looking clothing and confidently walk in with a laptop and Cisco
console cable, nonchalantly telling the librarian their "Internet box" "needs
a service upgrade".

And then quietly replace it (after dumping the config and replicating the
configuration) with a $100 half-depth Supermicro server bought off Ebay,
running OpenBSD?

And then walk out, like the Grinch after he artfully puts Cindy Lou Who to
bed, with a nice new 3945 for my own use?

EDIT: forgot to mention, I need a CF card reader also to get the config off
the Cisco flash.

~~~
smithian
If you then sold the router on the second-hand market and gave the money to
the library to spend on materials that they actually needed, your plan could
do some real good.

------
waiwai933
From the article: "I'm not an expert on the technical side," [Gianato] said,
"but these have all kinds of capabilities and applications."

Earlier: "Gianato acknowledged that he didn't heed Dunlap's advice or wait for
an evaluation.

'The routers already had been bid out,' Gianato said. 'I think John was
looking at our needs now, not looking at our needs into the future."

\---

So not only does Gianato admit he doesn't know anything (which means he's not
even suffering from the Dunning-Kruger effect), but he also admits that he's
not sure what his IT advisor was thinking about, but, eh, it doesn't really
matter because the people who are the experts obviously don't think about the
long-term and I'm not going to bother asking them about it anyway, because as
we all know, bids are binding contracts. Right.

------
jsz0
I'm surprised this article actually does a decent job covering both sides of
the story. When you look at how flexible the ISRs are, and the likelihood
these devices will be in service for probably the next 10 years or more, it's
not not the worst investment they could make. They're actually well sized for
the high schools and other larger sites. Obviously any 3900 series is going to
be overkill for a little library but you do have to look at support costs,
service contracts, future requirements, future upgrade costs, etc to get the
full picture. This is not a 'bridge to nowhere' situation.

~~~
nknight
You can get an ASA 5505 firewall with all the features you'll ever need for a
small library installation for about $1200 retail (hell of a lot less if
you're buying, say, 50 of them), and any network engineer who can't handle ASA
configuration in his sleep should be fired immediately for gross incompetence.

There is no excuse for this insanity.

------
niels_olson
I have a buddy who sells routing equipment at this level. Doesn't have a
degree in Fahrenheit, but just cleared a mil in his bank account, owns his 6
bedroom Connecticut home outright and keeps a beach house in Mystic. He's 38.
I'm 36, a doc, and still paying off student loans and working for Uncle Sam.

------
dstroot
Private sector FTW. Please, please I hope everyone who reads this begins to
understand that governments are not meant to allocate resources. The idea
(promoted by politicians, mostly democrats) that we should give the government
our money and they will spend it wisely "for the betterment of all" is purely,
simply and fundamentally flawed.

Out government should be small. It should perform a few basic functions like
making sure we are secure, our trading interests are promoted and our banking
and financial markets are healthy. It should educate our children to the best
standards in the world. Finally it should make sure we are energy independent
and our food and water is uncontaminated. That's pretty much it.

~~~
epoxyhockey
Please don't make this a democrat/republican debate. This is the crooked way
that all of our US government is spending our tax money. We need corporate
money out of politics if we want to prevent these outlandish contracts from
being awarded to the corporations that donate enough money to campaigns.

It's not a democrat/republican or big/small government thing. It's about
corruption of the system.

Out of your comment, I agree most with that our government needs to help
educate our children. Without an educated population, things only get worse.

~~~
clarky07
how is this not a big/small government thing? We gave away billions of dollars
in a stimulus program with little to no oversight that is very clearly getting
wasted here.

While the republicans are certainly guilty of lots of spending and many
things, democrats are clearly the party that is guilty of more spending and
even bigger government.

~~~
_delirium
There are plenty of countries whose governments are _far_ to the left of the
Democrats who spend money much more wisely, so I don't see it as inherently
tied. For example, the Scandinavian countries have much larger governments
proportionally, but generally spend more wisely, and have more transparent
bidding/provisioning of their state-funded services, so they get more service
and less waste in return for their money.

~~~
clarky07
They probably do spend more wisely, but that doesn't necessarily mean they are
spending on something they should be. For example in this case buying the
cheaper routers would be spending more wisely, but could pretty easily be
argued that this isn't something we should be spending money on. Certainly not
at the federal level. Why is my tax money in Tennessee being spent on routers
in WV?

There are a few components to this particular screw-up. 1 is big government
spending (with little oversight) and 2 is one idiot in WV with a big budget
who failed miserably at spending it.

~~~
cantankerous
But it could pretty easily be argued that it's something that we _should_ be
spending money on. The existence of an argument against a course of action
doesn't, on its face, preclude taking that course of action. Such a line of
thinking is, well, bizarre.

 _"1 is big government spending (with little oversight)"_

The _obvious_ problem here isn't "big government" spending. It's spending
_without oversight_. This problem occurs all the time everywhere, in sectors
both public and private. To claim that private businesses are immune to this
by nature of their ability to fail and thus inherently a better mechanism than
government is an absolute farce.

Take ownership of your government. To view the elimination of government as a
solution to bad governance is both a lazy approach to problem solving and
toxic to proper governing.

~~~
clarky07
Private sector isn't immune, just not as bad.

Nobody said anything about elimination of government, just less of it.

------
sbov
> Gianato said putting the same size router in every school was about "equal
> opportunity."

A little known fact in the tech community is that a $7,800 router provides
exactly 16.016 times more opportunity to a student than a $487 router.

~~~
tatsuke95
All the tubes have to to be the same size. It's only fair.

------
guylhem
Spending other people money is so easy. But $22k apiece instead of the $485
solution quoted in the article - that's madness. It can't be just gross
incompetence.

I guess someone involved in the decision got some kind of kickback, gift, or
will have a nice cushy job offer waiting for them at Cisco when they get fired
as they should be.

Remember this article next time some liberal ask to raise taxes and justifies
this by saying how stimulus money is important, and why everyone should have
the same opportunities (quoted in the article - as if a 22k router gave any
kind of opportunity to people browsing the net in a public library in WV).

And cry when they get trashed at the end of their lifecycle without having
ever been connected to optic fiber.

~~~
gjkood
"Remember this article next time some liberal ask to raise taxes..."

Unless I misread, the article is talking about West Virginia. I would hardly
call WV a bastion of liberal thinking.

Never attribute to malice (or political leaning) that which is adequately
explained by stupidity (or incompetence)

~~~
gwright
You're right. This isn't a "liberal" failing. Wasteful government spending is
an affliction that spans the entire political spectrum.

~~~
chmod777
Democrats controlled two of West Virginia's three House seats, both Senate
seats, the Governor's mansion, and the state legislature (which they have held
since 1928).

One of West Virginia's two current Democrat senators, Joe Manchin, was
governor at the time of the stimulus and appointed Mr. Gianato.

The executive branch of the federal government in 2009-2010 was controlled by
the Democrats, along with both houses of congress. The federal stimulus was
passed with no Republican support.

<http://www.nytimes.com/2009/01/29/us/politics/29obama.html>

The entire chain of custody from the grant through request for bids through
the eventual approval was controlled by Democrats. The only people involved in
this fiasco without a (D) after their name are those people who will have
their taxes raised in order to pay for these routers, because of course they
were not paid for with existing revenue.

This is very much a failing of Democrat administrations at a federal, state
and local levels.

~~~
jaredmck
Democrats in West Virginia are hardly liberal - they're sort of a party on
their own compared to the Democratic party elsewhere in the US.

------
franze
why do military closets cost 5000$ a piece? because it costs a sh#tload of
money to sell the military a closet. i don't know where i read this but it is
a great analogy.

i have been - in my time (2000? 2001? 2002? 2003? 2004? ...) - involved into
selling the national austrian television their first video streaming solution
(think: free (as in free beer) windows media streaming server) - it took man-
years. man-years cost money.

~~~
quink
I want to punch you in the face. ORF's IPTV stuff has been 100% unusable until
the Mediathek came along.

I could further elaborate on the reasons why this matters to me, but I'm just
relieved that it works OK now.

~~~
franze
yeah, i felt guilty for years. glad it's gone now.

------
keithpeter
HN-ers

I'm not a programmer or technical person, I'm an end user who uses R, LaTeX
and processing now and again. I hack an awesome spreadsheet. I have a degree
and I do a professional job.

I _could_ be a manager in a public service or an elected representative.
Perhaps in a small rural authority.

Given the lack of consensus in this discussion thread, and evidence in the
article of a similar lack of consensus between the various offices involved,
how am I supposed to reach a decision?

Is there a case for some kind of planning toolkit or requirements estimation
software? Is there an opportunity here?

PS: I and other colleagues did once have to help a senior manager spend out a
£250k capital grant in 10 days. The idiotic spending deadline was due to delay
in award of funds in a competitive funding round. We did ok but could have
done better with another month or two to think through detailed requirements.

~~~
amalag
You use your intelligence. The Cisco person says you can do what you want with
$500. Then do you continue to spend $30k instead? Do you think your small town
library is going to grow into a college campus (as the other pro-mega-router
people are expecting)

~~~
keithpeter
OK, I have to admit that looking at the picture in the article, I personally
would have tried to vire some of the money for the super-router into a less
fancy one and some decent all-in-one lcd desktop pcs for that table. The space
saved by the all-in-ones could have provided a 'quiet table' for someone (and
the PCs could double as DVD players).

I would have to make a spreadsheet (good at those) of all the sites and then
include a rating for current traffic and then expected traffic growth over the
life of the project. Yes, a library with 4 PCs is not going to have a lot of
growth...

------
GreyTheory
> Our main concerns were to not have something that would become obsolete in a
> couple of years

No you fool! These are designed to handle large loads, not protection from
obsolescence!

------
jaredmck
Interestingly, the CEO of Cisco is from West Virginia and went to WVU - I
believe he's one of the largest donors to WVU. (only know this because I'm
also from WV, not suggesting any conspiracy here)

------
Klinky
Sounds like a buffoon was in charge of the operation. Gianato suggesting that
the unused/unneeded T1 interface cards could still be used for "video
conferencing, wireless Internet and "voice over Internet protocol." is just
painful to read. I am not sure how they can be used for such things if you
don't actually stick them in a router!

I would still like to see a breakdown of the costs and what services are
provided with the routers before I call it a total waste though. If it was
just the routers then it was a waste. If it includes other things like a long
term warranty & onsite service, then maybe not so much.

~~~
frankydp
Some one should request a detailed FOIA.

------
ChristianMarks
West Virginia Homeland Security must be planning for a truly unprecedented
expansion of libraries and schools, based on the capacity of the routers and
the number of them allocated to sites with a handful of users. This is
heartening, especially during a period of massive cutbacks to public
education. ;)

------
kirillzubovsky
When a public school/library buys a 22K router with government money, sounds
like an overkill (handout) of massive proportions. Someone's cousin here was
probably on the Cisco + Verizon sales team. This is frustrating.

I remember a few weeks ago PB made a comment that at least social networks and
alike make it easier to spread the news, which eventually will lead to greater
good (<http://news.ycombinator.com/item?id=3890543>). At least that's true,
but doesn't feel like it's enough.

------
tylermenezes
Hacker News, you continue to disappoint me. Every time I read an article, I
know the comments are going to be filled with people arguing for the sake of
feeling superior. And once again, they are.

No. This is not okay. No twisted sense of reason that only you have makes it
okay. A $60 consumer router that needs to be upgraded every year is still
cheaper than this. Even with support costs. People have been rolling out
tiered infrastructure for years.

------
8ig8
I describe this as 'funny money'. The government is spending other people's
money.

I imagine the government worker that signed off on this purchase worrying a
lot more about a dollar increase in the local lunch special than millions
spent on equipment.

~~~
cma
It is also funny money because the marginal cost of producing one of these
routers is more like $500, not $20k. It is just a transfer payment to the
company, it isn't the same as if the government burned $20k worth of barrels
of oil. Nothing near that amount in resources was actually wasted.

~~~
msellout
Not quite. Cisco will probably take this purchase as a signal to produce more
unnecessarily large routers. Those people paid to make the unnecessary routers
could have used their valuable skills elsewhere. They may convince more people
to learn how to build unnecessary routers instead of something more valuable.
It's waste all the way down.

~~~
cma
Not exactly, unless you think those people will literally use their salaries
from the activities you described to buy something and go out and burn it. It
might be a mis-distribution, and it might spur misguided investment in an
area, but it isn't all squandered to the tune of exactly $20,000 or anywhere
close to that. The marginal cost is a much more accurate measure.

------
luminaobscura
i live in a developing country. I am thinking what would happen here if a
state offical had 24 million in his hands to spend on the infrastructure? The
official may be corrupt and this thing may never go to proper bidding. So
official and a company would share say 10 million as the profit. But 14
million would still go to something useful.

In this story, at least 20 million of the money seems to be completely wasted.
(Higher capacity router does not mean better internet access for 4-computer
lib.)

Wait, not finished. So our corrupt official has 2 million in his hands. He may
buy a car or a house, but probably he would spend it for something useful. So
much of the value would return to the market. This is also true for the vendor
company. But in the US case, the value is gone. it is like you burned the
cash.

Moral of the story: stupidity is more dangerous than corruption. You may want
to have a corrupt offical than a stupid one.

Finally, that stupid would never have survived in the relatively corrupt but
wild political system of my country.

~~~
doyoulikeworms
Yours is the most bizarre, vaguely chauvinistic straw man rant that I've ever
read.

~~~
luminaobscura
where is the straw man?

------
forgottenpaswrd
Time to connect the entire neighborhood to those machines and share the cost
/administration.

------
sakai
Well... at least they got high-powered routers for $22K apiece. But, it's a
shame there isn't a law on the books that selling the government something at
50X the market price / need is fraud.*

* Yes, I realize this would bankrupt half the military contractors as well.

~~~
clarky07
Can't blame them for selling them something they asked for. Who's to say they
were even told what they were for. This is government waste and incompetence,
nothing more.

You are correct on the military contractors as well though. Lots of government
waste and incompetence there as well.

------
timurlenk
I see a lot of posters argueing about the type of cisco router that should
have been used or if a soho Linksys router should have been used instead.

Assuming that the government has some rules that requires them not to deploy
soho or open source products and they have to spend the money on some big
brand company: how come Juniper was not taken into consideration (or other
known network vendors)?

I would argue that Juniper could provide cheaper equipment of simmilar of
higher spec (let's ignore for a moment if that is oversized or not) - and it
becomes even cheaper if you consider the simpler licensing terms and upgrade
support.

Isn't the government supposed to run open tenders? Is cisco mandatory in the
US?

------
patrickgzill
Verizon has successfully siphoned a bunch of money from the govt and by
extension, taxpayers, yet again.

Anyone familiar with telecom and living in PA, WV, other states that have
Verizon as the ILEC, knows this.

------
mehulkar
What piece of technology will stay relevant `5-10 years out`?!?

------
jeffpersonified
This is why I'm a libertarian.

------
graylights
Most of these places don't know what the box is and when Johnny down the
street comes in to fix their internet it will be replaced by a blue soho
router. There will be theft and waste and probably 20% of these will be
missing within a couple years.

A librarian will notice a computer missing, but an expensive router replaced
with a cheap one won't even be noticed for years.

------
suhastech
Similar story in India would be like "State paid $20k each for Internet
routers worth $20"

------
conformal
wow.

somehow i doubt these sites have multiple uplinks, so they certainly don't
need BGP routers. even if they're doing IPSec between some locations this can
be handled by an old piece-of-junk $100 computer running _BSD or_ nix, e.g.
with isakmpd and pf.

stuff like this is why america's infrastructure is crumbling: dodgy
contractors looking for an excuse to upcharge municipalities for irrelevant
gear and services.

~~~
dfc
You really want to advocate installing "old piece of junk" routers across the
entire state? Have you ever managed a project this large? And if so did you
use old piece of junk equipment?

~~~
sk5t
$500-1k appliances with Atom CPUs running pfSense, please, not $100 clunkers
with spinning drives and dust-laden fans.

Anything more than that for a moderately large library or most schools is an
outlandish, shameful waste of money and equipment.

------
shane_mcd
This is absolutely disgusting to me. The people that run big and small
government in this country are complete idiots.

------
dfc
The price includes maintenance and support correct? It is not 22k for the
metal only?

------
spydum
it's sad, but nothing new. either verizon is lying about who asked for the
specific model, or some inexperienced network "architect" decided not to do
the due diligence to spec out what was required, and just asked for the moon.

------
dfc
The price includes maintenance and support correct? Its 22k for the metal
only?

------
PaulAnunda
Smh, so much waste.

------
georgieporgie
Regarding planning for the future, this seems like someone who was used to
buying machinery was put in charge of buying technology. Sure, in five years
the library might need a gigabit of bandwidth, but by then the Linksys WRT may
well handle it for $100. It's not like you're buying a work truck, which may
well serve the government for 5, 10, or even 15 years (assuming low usage), or
a lathe that might be used intermittently for 35 years.

There's a lot to be said for uniformity in deployment, but that's why you
would define, say, three tiers of supported hardware instead of kitting out
everyone with top-of-the-line.

