
Why a one-room West Virginia library runs a $20,000 Cisco router - cpeterso
http://arstechnica.com/tech-policy/2013/02/why-a-one-room-west-virginia-library-runs-a-20000-cisco-router/
======
nikcub
I have first hand experience with this, although in Australia. My first job
was working for a IT development and solutions shop (this is over 17 years
ago) where they won the bid to rollout the internet and network gateways
across a large number of Australian government schools.

We were doing similar work for private school at the same time, and the
process for the public and private school could not be more different.

For eg. in the tender process, which I was part of, for private school we
would use cheaper Taiwanese routers. For the public schools it 'had to be
Cisco'. The only time we ever used Cisco, outside of large enterprise clients
with 1000+ seats, was with public school tenders. The private school would get
a $500 white-label router, the public school s would get a $6-15k Cisco router
with additional VPN module costs.

We would charge a higher consultant rate on the Cisco jobs, and would bill 3
days instead of 1. We won the contract because we were 50% cheaper than the
other tenders who all wanted to install 3 and 5-series routers. How we got to
the point of being able to even tender is another story that involves somebody
in our organization sleeping with somebody at the government organization. The
other tenderers were accustomed to dividing the work up amongst themselves at
inflated prices, they didn't even know who we were and we received a lot of
abuse for breaking up their little scheme.

(Edit: a further idea of how this worked, the 3 people in the gov office
responsible for tenders all had very nice cars and holiday homes while the
rest of the office was working away on below-average wages. You could see what
was going on just by looking at the car park)

So a dozen of us roll out hundreds of these routers in public schools and
after a month we find that we rolled out the wrong version of IOS, one that
was vulnerable to a simple security attack. Instead of forcing us to upgrade
all the routers remotely, or out of our own pocket, we instead won another
few-million-dollars worth of work to send a person out and apply the upgrade
to each router (which took 5 minutes, we charged a full day plus travel).

The routers weren't even being used properly - the topography was net
connection -> cisco router -> internal server -> switches. The internal server
would do all the DHCP and everything else. These expensive routers were being
used as bridges, although they were pitched as having 'forward compatibility'
incase the school wanted to implement features such as user accounts (they
did, although again they used a custom server, not the router).

When these projects are audited there is nobody who is technically competent
enough to make an argument against who would be on the side of ditching or
shrinking the projects. Some of the smarter teachers knew what was going on
but didn't mind since they got access to fancy equipment (we would create user
accounts for them).

My first, and not my last, experience with government bureaucracy and budgets.
I would estimate that the private schools got more out of us at a tenth of the
cost. Since then I had an even worse experience with the government health
department, where 6-figure invoices were written and paid for goods that
didn't exist (that department has since been broken up and the subject of a
large corruption enquire). No surprise that I became very anti-government size
and spending.

Edit: to add, we were so 'disruptive' to the backdoor deals that we were
uninvited from conferences, kicked off panels, not invited to the mixer events
where gov buyers met providers, a couple of years later we lost our
accreditation[0] temporarily until we appealed to the Government Minister. On
site we would be locked out of network cabinets, not given IP information for
the net connections, etc. Our jobs were made difficult by competitors and
others because of our pricing and methods. They couldn't figure out how they
didn't get rid of us, because they didn't know that one of our guys had a
solid relationship with somebody at the government (that person wanted to
clean things up). Usually we wouldn't have been allowed anywhere near these
projects and if we did win one we wouldn't be allowed back in for not playing
with the system.

[0] 'accreditation' for government tenders needs to die, it is a formal method
used to keep honest operators out of what is essentially a cartel. The
Australian government is getting better in this regard[1], they now have an
open tender website but I believe it still requires some form of accreditation
that has a person in a department standing between application and approval.

[1] although not too much better. NBN Co., a government owned company that is
building a $40 billion nation-wide fibre network recently suspended tender
process because every single bid came in over the expected price:

[http://www.theaustralian.com.au/business/in-depth/high-
price...](http://www.theaustralian.com.au/business/in-depth/high-prices-force-
nbn-to-suspend-cabling-tender-process-and-look-
elsewhere/story-e6frgaif-1226031831542)

~~~
juiceandjuice
It sounds like your company ripped off the public schools, and that's why you
are anti-government?

~~~
bd_at_rivenhill
The fact that the government is not competent to spend taxpayers' money wisely
seems like a good reason to be anti-government. After hearing enough anecdotal
evidence on this topic (which very few people have both the capability and
desire to thoroughly investigate in order to properly quantify it, thus the
reliance on anecdotal evidence), it starts to appear as if such incompetence
is an integral feature of all governments once they reach a certain size.

~~~
vidarh
I think it's an integral feature of all organizations that reach a certain
size without sufficient level of audits at lower levels. And given enough
large organizations, there will be problems.

Given the number of large government entities, there's a lot of room for bad
oversight.

But on top of that there is also often a lot more openness when the auditors
does their jobs in government departments and these things are discovered, and
even if not, the press often has a lot more access (e.g. freedom of
information requirements).

In a lot of companies, a lot of borderline behaviour will instead just lead to
people getting fired, especially when there is no evidence.

A new CTO at a company a friend does work for "decided to part ways" with his
company a while back, for example, after migrating a substantial part of their
service to a new, totally over-engineered and untested platform that failed
spectacularly after they'd very publicly spent a massive amount of money on
it. Incompetence or corruption? Probably impossible to prove.

And to the public there was no real sign anything was wrong: The launch was
highly publicized by the company, and they never told anyone about the
negative effects later. Only people with insight into internal sales numbers
until they made emergency changes to revert some of his more harmful decisions
would know.

And for him it likely doesn't even matter that he was effectively fired: He's
either a useful fool, or a useful accomplice, for someone who will happily
make use of their network to help him land somewhere else. And many potential
hirers will only remember the positive press attention prior to the launch and
will have no way of knowing about the problems.

There are a lot of stories like that in private business too.

------
trout
Visualizing this conversation..

IT Director to Sales Guy: "We just got $20M in grant money for getting
broadband across the state. Can you get me some numbers?"

Sales guy to engineering team (partner or internal): "Hey I need lots of
boxes. They've got 1300 sites."

Engineering team: "Ok.. what do they need?"

IT Director: "Pretty sure my network guy says everything has to have redundant
power supplies and at least 1 ethernet connection. To do a survey for each
site would take over a year due to bureaucracy, and I've got 3 months on this
grant"

Engineer to Sales guy: "Ok I built out those routers. Do they really need
redundant power supplies everywhere? 3900's seem big."

Sales guy: "Ya, that's what they said. Anyways this came out below budget.
Thanks!"

IT Director: "Looks to be under budget, meets our needs, thanks!"

.. meanwhile IT management/engineers aren't involved. Somewhere, someone
didn't slow this project down to do due diligence. That or somewhere buried in
some document is a requirement for redundant power supplies, but that sounds
less likely the case.

~~~
intended
> Somewhere, someone didn't slow this project down to do due diligence

In it a nutshell.

If anything, there should have been multiple flags raised.

1) A ball park figure should have been estimated on what this should cost,
broken down by equipment type.

2) That should have made it clear that switches would be fine for multiple use
cases, and the overall estimate should have dropped. *

3) Back and forth with provider as haggling over components proceeds. Massive
attention to detail would be shown here.

4) First quote received from Cisco. Due diligence proceeds, issues raised.
Customers says NO, just to buy negotiation room. (fine they have a grant,
maybe not so aggressive)

5) Updated quote, go back to 4 if issues.

6) Quote acceptance. Now monitoriing of execution.

There should have been many opportunities for flags to have been raised, even
before the quote was received. And most definitely after it would have crossed
the ball park figure.

* It looks like the clause for redundant power was what tripped it up. Someone said we need X, and then somehow it became the TRUTH for all routers/switches.

So its entirely possible (if not likely) that Cisco said "based on your
constraints, these are the only components which match your requirements", and
the Government bought it.

Which is face-palm/head-desk territory.

------
DanielBMarkham
What's worse than overpaying millions for a few routers? Overpaying billions.

At the federal level, specifically to prevent such things from occurring,
there's this huge byzantine procurement process. A process that is literally
worth tens of billions of dollars to game.

So if you have product foo and want to see a zillion copies of it to the feds
instead of some freeware or cheaper solution? 1) Convince somebody on the
inside of the procurement system that yours is the best product, 2) they write
up the specification so that no other products will qualify (many times just
copying directly from your brochure) even though it's technically an "open"
bid, and 3) have somebody with lots of procurement knowledge help guide the
paperwork through the system. At the end of the day, it's all just paperwork,
no matter how much money is involved.

And that's just what I've observed in IT. I'm guessing IT is the worst, since
all the products somewhat look alike. But I'm not sure. Quite frankly, its way
too depressing to think about much.

BTW, the best way to do #1 is simply hire people retiring out of the
procurement system. Yep, there are laws against direct hires, so you hire
somebody from DoD to help shepherd a Commerce Department contract, or a DHS
procurement expert to help with a DoD job, and so forth. If you do this
correctly, the poor schmucks left handling the paperwork will be so happy that
you can offer extensive support in making sure everything is done correctly
that this is another huge plus in favor of your getting the bid.

~~~
stdbrouw
The trouble is that you actually do need some of this flexibility in the
procurement process. If your department needs Photoshop, you don't want some
goofy non-technical auditor telling you that he googled around and thinks GIMP
will do just fine. Or you want one particular contractor because just last
year they did an amazing job on something that was pretty much exactly what
you need.

~~~
jimmaswell
gimp generally would do just fine, wouldn't it?

------
redshirtrob
I'd like to know how much of that $5 million savings would have been burned up
in the capacity studies. My guess is a lot.

Given the choice between a fixed cost and an unbounded cost (being the
capacity study referenced throughout the article) I'm not surprised they went
with the fixed cost.

I'm not saying WV didn't get swindled, but I can sure see how this might have
happened:

WV Rep: Do we really need these $20k routers for all our locations?

Cisco: We really couldn't say what your exact needs are without a proper
study.

WV Rep: What will that cost?

Cisco: It's really hard to say. We'll have to visit all of your locations and
speak with the IT Manager there. We'll have to measure average and peak load.
Of course, we'll want to plan for future growth so we're not at this same
point in two years...

The real crime is that they didn't open the RFP to multiple bidders. A little
bit of competition can go a long way.

~~~
vidarh
Here's a capacity study for you:

E-mail all the IT managers, and ask them:

\- How many users do you have? \- What is the total size of the community you
serve? \- Do you need new equipment?

That would have quickly revealed that many of them had so few users that these
routers were total overkill, and had so few _potential_ users that even 100%
simultaneous usage in some cases would not max out the capacity, and that some
of them did not have a need for new equipment.

I agree with you that multiple bidders would be essential. But a tiny little
bit of due diligence and investigation on the behalf of WV would have gone a
long way.

That said, sometimes this is what backfires too: I've seen a government
contract recently where a lot of extra work was carried out to meet the
requirements of a "security consultant" that refused to sign off on a system
because of "issues" that were clearly generated only to ensure that he could
pad his list of "problems" he identified to help justify his fees.

~~~
pbhjpbhj
Wouldn't the central procurement already know the max capacity of the lines
(DSL, fibre, whatever) that had been installed, already know what
routers/switches were in use and already know the populations being served?
That's pretty basic audit information that I'd expect could be pulled up from
a central database lickity-split.

Eyeballing those figures should be all you'd need - but yes if there was no
central record of past procurements an email survey (or better yet online
using a survey system) would apparently be pretty quick.

------
rachelbythebay
Oh, the FCC E-Rate program. So much corruption, so much waste. There were
people who would purchase something for a school with the most % of kids on
free or reduced price lunch, since that was used to set the "FCC match"
percentage. So, if 90% of the kids were on that plan, we'd only pay 10% of the
actual cost.

So, they'd order something for the school with the highest percentage and
would then park that equipment there for a year (to fulfill the requirement
that it must be used there). A year later, they'd move it somewhere else.

I wrote about this particular router instance last year and linked it with my
own FCC story from the year before that.
<http://rachelbythebay.com/w/2012/05/08/router/>

------
WestCoastJustin
Probably going against the grain here, but what about the operations staff who
are charged with supporting these 1,164 routers? What about warranties and
parts replacements? There is a price for standardizing on a piece of hardware
at such a large scale. When you standardize, sure there is going to be
hardware overkill in some places, but in the big picture this is just a price
of doing business at this scale. What is the price of service calls out to
this location when the hardware is down? You have a tool-chain that supports
this hardware, you have experts supporting the network (security, patching,
etc), and you have contracts in place to make sure it is operational.

~~~
rdl
There's a point in standardizing, but they probably should have standardized
on 2-3 different branch router products, vs. one.

~~~
WestCoastJustin
Yeah, I agree. I'm just saying I can kind of see where they are coming from.

~~~
prawn
Who, Cisco? I wouldn't be excusing their actions in a hurry.

Everyone in this deal should be advising as though they are spending their own
money. You can spend a lot of money, but would you if it were you own and you
had other things to consider in a budget?

~~~
theorique
Do they get a bonus or other benefit if they act as if they are spending their
own money, or alternatively, punished if they do not act as such?

In other words: if there's no incentive, what difference does it make?

~~~
prawn
I'm saying they should. Not that they will.

------
nicw
This almost happened at the California State University (CSU) level. An RFP
went out, Alcatel bid $22MM, Cisco $123MM.

[http://www.networkworld.com/news/2012/102512-cisco-
csu-26371...](http://www.networkworld.com/news/2012/102512-cisco-
csu-263711.html)

""Everybody had to comply with this spreadsheet," he said. "Every campus had
two border routers, two cores, and two server farm switches. All the vendors
had to propose exactly the same solution" based on the average number of
servers deployed at each CSU campus. "All of this is based on exactly the same
data to all of the vendors. It's exactly the same formula for all of the
vendors.""

------
meaty
This doesn't suprise me.

I am related to an ex-sales exec at Cisco UK. They are a bunch of shysters who
will desperately oversell anything to anyone. The person in question was
responsible for selling such kit to managers who didn't even know what it was
but they apparently needed it. You know the sort who populate senior positions
in the public sector. They are rife in the healthcare and council sectors in
the UK. Incompetent morons waiting to be milked for our cash.

The asshat is now selling VoIP and video conferencing solutions to medium
sized businesses (via their executives) which is the next cash cow he can
milk.

------
homosaur
Corruption is awesome when you're on the nipple end of the government tit.
Nice work if you can get it, Cisco.

~~~
fnordfnordfnord
Cisco may not have anything to do with it. It may be down to a salesman in the
middle who is on commission. I once saw CCTV cameras deployed with Jena
lenses, I guess for similar reasons. The auditors (if there were any) weren't
knowledgeable enough to know better.

edit: After further reading, Cisco did in fact shamelessly participate in this
gigantic boondoggle.

~~~
scotty79
In my country government officials are usually paid to be this stupid. I can't
imagine USA government officials are this stupid for free.

I know ... never attribute to malice what can be sufficiently explained by
stupidity, but come on! Someone sure got at least a nice car out of it.

~~~
tdoggette
I chuckled. It is, in fact, quite easy to imagine that American bureaucrats
(as a collective) are that stupid. I'm not discounting the other possibility,
though.

~~~
intended
Whats somewhat sad though, is that for the majority of my time, America was
described as the place where things "just worked".

To be fair, a lot of things do "Just work". This may just be the time
Americans look to their government and decide to clean house and rehire
competent people into it.

~~~
homosaur
They can hire all the competent people they want, the system itself has become
broken. There's too little representation and too much gerrymandering. There
needs to be profound systemic reforms to even begin to address what's wrong
with the US government.

------
jeffdavis
I was surprised how much blame this article lays at the feet of Cisco. What
did they do except sell the customer what they wanted? Sure, they might have
been more noble if they put on the breaks in the name of the taxpayers, but
that could hardly be expected.

It takes a lot of nerve for someone who just wasted my money (federal dollars)
to try to blame it on the other party for fulfilling the order.

~~~
dalke
Did we read different articles? The one I read says "The report finds plenty
of blame to go around. The ultimate cause of the fiasco, it says, was the fact
the grant implementers did not conduct a capacity or use study before spending
$24 million." This does not place the ultimate blame on Cisco.

You say "what [West Virginia] wanted", but the article shows how Cisco was
unable to show that they knew what WV wanted. For example, Cisco claims that
WV specified internal dual power supplies but there is no record that WV ever
had that requirement. Instead, "what WV wanted" is defined only as being
whatever WV signed off on at the end.

As to if Cisco's behavior is expected or not, the Ars article says: "[Cisco]
had a moral responsibility to propose a plan which reasonably complied with
Cisco's own engineering standards" but that instead "Cisco representatives
showed a wanton indifference to the interests of the public in recommending
using $24 million of public funds to purchase 1,164 Cisco model 3945 branch
routers."

Ars even links to section 5A-3-33d of the West Virginia Code. The likely
relevant clause is "(B) Performance in violation of standards established by
law or generally accepted standards of the trade or profession amounting to
intentionally deficient or grossly negligent performance on one or more public
contracts;", and the auditor's report shows examples of how Cisco's actions
were likely in violation of their own published standards.

This is not a crime. As the article says, the strongest remedy regarding Cisco
is that Cisco might be "barred from bidding on future projects."

~~~
jeffdavis
"Cisco was unable to show that they knew what WV wanted"

Yes, they have a contract. That is the best indication possible that the
customer wants what you are selling.

If I trust someone else with my money, and they transfer that trust over to a
salesperson, then they have violated my trust.

I don't want to hear about "blame to go around" and "the salesman should have
sold us less stuff". Maybe apply some shame or sanctions to the salesperson, I
suppose, but I really don't care unless there was some kind of kickback
involved.

Allowing the stewards of your money to redirect blame onto Cisco for selling
too much stuff seems so overwhelmingly naive that I don't know where to begin.

~~~
dalke
This is a modified version of the Prisoner's Dilemma. If there is trust then
overall costs go down and the number of sales goes up. This is a net win for
both sides.

If one side does not follow standard business practices, which the auditor
argues happened here, then that one side may get better benefits. But this is
actually an iterated prisoner's dilemma, and the standard tit-for-tat strategy
of the Prisoner's Dilemma makes good sense. That is, shut out Cisco from
future contracts for a period of time.

This encourages future vendors, who want repeat business, to stay in a regime
of trust and relieves WV of having to pay for staff with full domain expertise
on all contract-related matters. A role of the auditors is to identify these
problems after the fact and use debarment as a way to disincentivize this
vendors from this behavior.

This isn't naive. It's basic game theory.

------
mrb
Cisco: "the criticism of the State is misplaced and fails to recognize the
forward-looking nature of their vision"

My sarcastic retort would be: "no, _you_ fail to recognize the forward-looking
nature of their vision because you did not recommend this other even more
expensive and more powerful router!"

------
jessaustin
Gosh, it seems like state bureaucrats can't figure out how best to spend
money, and don't care to hire anyone who can. How might one's political
preferences change in response to this shocking revelation?

~~~
mpyne
Well, think about it economically. What's the incentive for those who are
skilled and capable to go into the public sector instead of private
enterprise. Elections suck, the press sucks, your fellow public servants suck,
having to deal with the general public, etc. etc.

Some days I wonder how we don't seem to encounter tons more corruption and
incompetence at all levels of the U.S. government.

------
marcoperaza
Caveat Emptor. Cisco is out to make as much money as they can, they have no
duty to save the government money. West Virginia's procurement process was
irresponsible and wasteful and now they want to shift the blame to Cisco in
order to cover their own incompetence. If you overpay for something, it's your
fault.

------
shalmanese
Previous discussion on the issue with a lot of the same arguments that have
been rehashed here: <http://news.ycombinator.com/item?id=3962029>

------
emperorcezar
I grew up in West Virginia. I also was pretty involved in the computer
operation at my high school. The "head" of IT was a math teacher they suckered
into doing all the work.

Anyhow. I remember when they decided they needed a domain server. What they
eventually came back with was a two year old model HP tower server that ran
over $4k. It was insane. They could have gotten something new for much much
less, but all the explanation I could get was that they had a contract and had
to buy the items listed in that contract.

There is a culture of corruption and it is prevalent in not only the public,
but also the private sector.

Another story is when I was in a diner years later and heard my old social
studies teacher asking for a bribe from someone to get a job in the coal
mines. I felt pretty bad because I believe he got his money and thus his power
from years before when I told him to buy Google and others stock.

------
crazygringo
> _Getting any of the money back seems unlikely at this point, but the
> legislative auditor does have one solid recommendation to make. The State
> Purchasing division should determine whether Cisco's actions in this matter
> fall afoul of section 5A-3-33d of the West Virginia Code, and whether the
> company should be barred from bidding on future projects._

Cisco is a public company with a fiduciary duty to make money. You should
_assume_ that companies are going to try to fleece you whenever possible.
Unless they committed some kind of contractual breach or fraud, they're not
the problem.

The fault here is clearly with one or multiple people in the government. This
is the kind of gross misconduct for which it feels like firing isn't even
close to enough -- this kind of managerial incompetence really deserves at
least a short stint in a white-collar prison somewhere.

------
guynamedloren
This makes me sick to my stomach. The worst part is that there's nothing we
(the taxpayers whose hard earned money went to to pay for these things) can do
about it, nor is there much we can do to prevent it from happening in the
future. I really really hate to admit it, but that's the truth.

~~~
pekk
Did you consider any of the tools of democracy? Although it can take time,
that should work just fine if most people actually agree on what to do.

If you are in a minority of the population which wants to unilaterally impose
its will on everyone else because they are more important, however - then I
guess that democracy is no good and only an armed coup will do. Let's hope
that people don't start thinking that way, because then we are all in trouble.

------
ripter
'The auditor's office sent off a fairly testy e-mail to Cisco noting that the
3945 routers were not appropriate for most West Virginia deployments—even
according to Cisco's own literature. "I would appreciate an explanation as to
why you believe the 3945 routers are not oversized and misconfigured for
hundreds of locations," the auditor concluded, "and, thus, a significant over
expenditure of millions of dollars for Cisco equipment."'

Did I miss some part where they contracted Cisco to visit every location and
assess it's needs and build an order according to that?

If you order a bunch of 3945 routers then why is it Cisco's fault for selling
you a bunch of 3945 routers?

------
bluedino
Remember the wasteful spending back in the UNIX workstation days - a Sun 10 in
every administrator's office for e-mail, and nobody ever had the training to
use the things?

~~~
hmottestad
My high-school admin bought a Mac Pro with dual CPU. It was powerful enough to
run all the school servers at once, yet he only used it to remote desktop into
the real servers.

~~~
cynwoody
Those who can, do. Those who can't, teach. Those who can't teach administrate.

~~~
pekk
And I suppose none of this applies to private industry...

------
jlarocco
Having worked on a government contract before, this doesn't surprise me at
all.

They were going to spend all the money the feds gave them. Whether they spent
it on over powered routers or "consulting" or something else, it was all going
to be spent by the end of the project.

There are enough projects overbudget that being on budget looks really good.
And there's always a concern with being under budget that next time the feds
won't hand out so much money, and you might really need it then.

------
rlpb
From the article:

> the legislative auditor concluded that the company "had a moral
> responsibility"...

Since when do corporations have any kind of responsibility, other than to make
as much profit as possible? Expecting anything else is futile. If something
needs to be fixed, it is in mechanisms to restrain this kind of behaviour
where it is in the public interest. Demanding that corporations have morals is
not the solution.

~~~
mgkimsal
If corporations have the concept of 'personhood' attributable, they can
probably have the concept of 'moral responsibility' attributed as well.

------
SpikeGronim
I saw the exact same thing with Cisco in the NY public school system. They
gave my high school something like $50,000 of equipment in a rack to support
two machines that submitted grades. We stripped it for parts and put it to
better use, it had some really nice switches in it. But it was clearly
misspent public money. At least we did put it to use in classrooms, though not
as they intended.

------
njx
This could have been executed by the process "Wired RFP". just look for the
this term and you will see references where government RFPs are wired to
specific vendor. In other sense, no other vendor may qualify to bid based on
the RFP requirement.

For e.g RFP bidding/winning criteria is based on point system.

1\. Price accounts for 25% 2\. Referrals from similar orgs 40% and so on

you see, when the price in the RFP accounts less then it is an indication that
the RFP is wired to a specific vendor.

A small business who is just starting may or may not have 3 or more referrals
from a public school or university.

In one of the RFP process I sat, the RFP had a typo where they had literally
spelled out the name of a competitor company. I pointed out this in the pre-
bidding call and the personnel acknowledged that it was a mistake.

Government RFP are a waste of time because it is the last step the government
does in their purchasing process. They have already chosen the vendor and most
of the time the vendor helps them write the RFP.

Fixing the RFP process may eliminate some of the overspending.

------
iamjason89
"The auditor began digging, speaking to many people in West Virginia state
government who had been involved with the project. The Department of Education
told him that it "did not request or require that the routers for the state's
schools have internal dual power supplies. Education would not have made this
requirement because unless a school has two power sources the feature of dual
power supplies would have no use." A network engineer for the Department of
Education confirmed that he had not requested such a feature."

Education would not have made this requirement because unless a school has two
power sources the feature of dual power supplies would have no use.

Oh is that how redundant power supplies work? You need two power sources, eh?

~~~
caw
For full redundancy, yes.

You get redundant power supplies by plugging both into the same power feed.

However, you can gain some extra redundancy by plugging each power supply into
a different circuit, and reducing the common point of failure (shared power
source).

~~~
nickzoic
Two UPSes.

Although I've got a good war story on that one too, I had a machine with
redundant PSUs emit a stream of smoke from one of the PSUs. It was happily up
and serving while smoking out the machine room until the operator (rather
sensibly) grabbed both power leads and pulled, at which point both fire and
ssh went out ...

------
gritzko
Well, just another me-too story. When I was somewhat younger I had to "deploy"
one Cisco product which was definitely a result of an "incentivized" sale.
That sweet sweet setup involved login-password authorization (assisted by a
Cisco router) where the user had to type in "public" and "1" as a password
because... the enterprise actually had no end-user accounts and absolutely no
possibility to introduce them...

As far as I understand, such an incentivized sale was a common practice. Once
the management somehow achieved consensus (i.e. the right balance of risks,
benefits and favors), engineers were left with making things appear OK.

I am inclined to think that Cisco made a strong bet on this kind of a business
model at some point.

------
Tichy
Startup Mountain camp anyone? This sounds like a great place to host a hacker
convention now?

------
MertsA
Just nit picking but there are several places in the article that are
incorrect such as

"The West Virginia legislature at peak times can have over 600 internal users
and numerous guests accessing "multiple Web servers, up to eight simultaneous
live audio webcasts, multiple SQL servers, and multiple Google search
appliances located in the Legislature's server farms." Despite all this, the
legislature doesn't even use a router but instead runs a cheaper Cisco
switch... and it has never exceeded capacity."

It's pretty clear that not only does the author not know networking basics,
his sources don't either.

~~~
papsosouid
Why is that clear? Cisco sells plenty of "layer 3 switches", they are switch
hardware with basic routing functionality available in the IOS. It is entirely
likely that the quote is referring to a catalyst 29xx or something which is
like 1/10 the cost of the routers the article was talking about.

------
joshlane4
this kind of nonsense is what people want to prevent the federal government
from doing...bring on the sequester!

~~~
tzs
If this is the kind of nonsense you want to stop, that approach is ineffective
to the point of idiocy.

~~~
crusso
Since the sequester is really just a reduction of an increase and not really a
cut, how is it ineffective? Seems like it's the only thing that even kind of
looks like a cut that might go into effect in our lifetimes.

Funny how the Feds were fine with a 2% increase in payroll taxes for consumers
but a 2% cut in the increase of their budget is going to bring on Armageddon.

~~~
pekk
I don't understand why people speak so fondly of things like education cuts.
Reducing the absolute size of the budget and increasing revenue are both
important for continuing to reduce the deficit. But almost every real problem
with government is caused by the relative allocations of funds and how
departments are managed, not the mere existence of funding for things like
education or public health.

------
mich41
Reminds me of new EU members. Joining the EU was advertised as source of funds
for development and catching-up with the West and now a benchmark of
politicians' "performance" is how much funds are they able obtain for their
electorate - the more is spent, the better.

In extreme cases one didn't even have to bribe officials to get bogus
contracts because if the city didn't find a way to spend all the money it was
offered, it would simply get less.

------
lifeisstillgood
There is positive corruption and nevative corruption

Positive corruption does the right thing (build roads, install broadband into
poor communities) but adds cost

Negative corruption distorts right thing so much it simply does not happen
(communities never get proper triads or sewage because they are the wrong
tribe)

The broadband is the thing - good. If you don't want government to overpay I
suggest Cisco routers is the last thin to look at - start with military
contracts

------
js2
Meh, I'm much more upset by the mountain top removal going on in WV, clearly
visible in the satellite picture of Clay, WV included in TFA.

~~~
jrockway
You can blame Cisco for that, too. See, one of their interview questions is
"How many teaspoons would it take to move Mt. Fuji," and they needed to do a
small initial study.

------
noonespecial
It seems absurd but I'm going to have to side with the state/library on this
one. It can literally cost more to make a choice to use the cheaper unit than
to just buy the big one.

"Some libraries need smaller routers. Ok. Which ones?" could easily run in the
millions before any routers were purchased. Unfortunately, that's just the
nature of government purchasing.

~~~
jessaustin
This is bullshit. They could have put the network engineer from the Department
(the guy who confirmed he had not requested dual power supplies) on this for a
week, and he would have had a solid estimate for far less. Put him on it a
month and he could have had an itemized order with the exact models to order
from Cisco or from any of its competitors. If he were busy with other duties,
any other network engineer in the state's employ could have done the same. If
they were all too busy, a consultant could have been hired on for a month's
engagement at two to three times the FTE cost.

~~~
damian2000
Agreed - it sounded like the Cisco guy just spec'd up the absolute maximum
needed for one or two locations, then just decided to roll out the same spec
across all 1000+ locations, obviously trying to maximise sales. Cisco was
probably shocked when the government approved it all.

~~~
prawn
"Cisco was probably shocked when the government approved it all."

Wouldn't that imply they hadn't succeeded with the same ploy before? I think
that's unlikely!

------
ericcumbee
"did not request or require that the routers for the state's schools have
internal dual power supplies. Education would not have made this requirement
because unless a school has two power sources the feature of dual power
supplies would have no use."

What if one power supply fails? I expect better from ars.

------
chayesfss
a company like cisco or microsoft trying to rip off state/local governments?
color me not shocked.

~~~
joenathan
What does Microsoft have to do with this?

~~~
paganel
> What does Microsoft have to do with this?

I don't have a source handy, but around 2004-2005 Steve Ballmmer came in a
short visit to my country (somewhere in Eastern Europe), he had a even shorter
meeting with the Prime Minister of that time and sure enough not 6 months
later a Government programme was launched and all the schools and public
libraries in my country had to be equipped with Windows machines. This was
around the time when people thought that tools like Open Office actually had a
chance of toppling the MS Office suite.

~~~
poloniculmov
Hey fellow countryman! What good would it do for kids in our country to learn
Open Office, an office suit nobody uses in a workplace?

~~~
mgkimsal
30 years ago, why would anyone have bothered to use MS Word when nobody used
it in the workplace? Everyone used WordPerfect or WordStar.

If you had a generation of kids who have experience with openoffice, it'll be
much more likely that openoffice would be something that would be adopted over
time - newly formed companies using it, gradual shift away from MS Office,
etc.

MS knows their history, and will continue to seed generations of users with
low-cost/free stuff to keep their dominance going. If they quit doling out
free copies, something like openoffice would gain a larger foothold in schools
and eventually businesses in < 10 years. Would MS Office be entirely replaced?
Of course not, but it would not be the default/automatic choice for everyone
without question.

~~~
poloniculmov
I don't think that's public education's role.

~~~
mgkimsal
What isn't? Indoctrinating students in MS Office? Or OpenOffice? Or Apple
products?

~~~
poloniculmov
Exactly. It should just teach the industry standard, which in this case is MS
Office. It's not the education systems' role to push for open source
solutions.

------
yaddayadda
"For those keeping score at home, this means that 75 $20,000 routers are
depreciating in a state police warehouse somewhere in West Virginia." So we
should be seeing 74 Cisco 3945 routers on eBay very soon. (My commission for
this idea is 1 Cisco 3945 router.)

------
hiddenfeatures
Maybe I should get myself checked, but is anyone else thinking about the big
opportunity here?

You wouldn't need more than your average crow bar to get into this thing. I
bet they don't have a decent alarm system (if any).

Seems like a relatively low-risk, high-reward situation to me.

------
jgoney
Before I read the article, I was hoping for a feel-good piece about how
someone hacked together a massive WiFi system using such a router to supply
Internet to some isolated community in rural West Virginia. It didn't turn out
to be exactly that.

------
mgkimsal
Didn't read the entire post - is there some reason the state or local
municipalities can't sell this stuff on ebay, get lower-powered hardware to
match their needs, and pocket the diff for the locality's coffers?

------
EvanAnderson
I haven't something quite this egregious happen firsthand, but it fits the
pattern for the interactions I've had when working for a "partner company"
reselling big-name networking gear.

------
yergi
This is so old that it was on the frontpage of Digg before Reddit was popular.

------
yason
Nothing is as expensive as free money.

------
papsosouid
This should surprise no one. This sort of nonsense is the entire reason cisco
exists as a company. Their products are underpowered, overpriced, and their
software is incredibly unreliable. Scamming big entities with closed bid
processes into buying millions of dollars worth of equipment is cisco's bread
and butter.

------
wilfra
Nice commission for that sales rep.

This is the dark side of enterprise sales. Microsoft and many others have
surely done many such deals over the years - and paid out handsome bonuses to
the reps/execs who made them happen.

------
g2bsocial
Burn in hell, CISCO, bunch of pricks.

~~~
g2bsocial
OK, the "Burn in hell" comment added little value, and so the downvotes from
Mods understandable, but take into account I'm a West Virginian, and this is
genuinely upsetting to us.

