IT Director to Sales Guy: "We just got $20M in grant money for getting broadband across the state. Can you get me some numbers?"
Sales guy to engineering team (partner or internal): "Hey I need lots of boxes. They've got 1300 sites."
Engineering team: "Ok.. what do they need?"
IT Director: "Pretty sure my network guy says everything has to have redundant power supplies and at least 1 ethernet connection. To do a survey for each site would take over a year due to bureaucracy, and I've got 3 months on this grant"
Engineer to Sales guy: "Ok I built out those routers. Do they really need redundant power supplies everywhere? 3900's seem big."
Sales guy: "Ya, that's what they said. Anyways this came out below budget. Thanks!"
IT Director: "Looks to be under budget, meets our needs, thanks!"
.. meanwhile IT management/engineers aren't involved. Somewhere, someone didn't slow this project down to do due diligence. That or somewhere buried in some document is a requirement for redundant power supplies, but that sounds less likely the case.
In it a nutshell.
If anything, there should have been multiple flags raised.
1) A ball park figure should have been estimated on what this should cost, broken down by equipment type.
2) That should have made it clear that switches would be fine for multiple use cases, and the overall estimate should have dropped. *
3) Back and forth with provider as haggling over components proceeds. Massive attention to detail would be shown here.
4) First quote received from Cisco. Due diligence proceeds, issues raised. Customers says NO, just to buy negotiation room. (fine they have a grant, maybe not so aggressive)
5) Updated quote, go back to 4 if issues.
6) Quote acceptance. Now monitoriing of execution.
There should have been many opportunities for flags to have been raised, even before the quote was received. And most definitely after it would have crossed the ball park figure.
* It looks like the clause for redundant power was what tripped it up. Someone said we need X, and then somehow it became the TRUTH for all routers/switches.
So its entirely possible (if not likely) that Cisco said "based on your constraints, these are the only components which match your requirements", and the Government bought it.
Which is face-palm/head-desk territory.
At the federal level, specifically to prevent such things from occurring, there's this huge byzantine procurement process. A process that is literally worth tens of billions of dollars to game.
So if you have product foo and want to see a zillion copies of it to the feds instead of some freeware or cheaper solution? 1) Convince somebody on the inside of the procurement system that yours is the best product, 2) they write up the specification so that no other products will qualify (many times just copying directly from your brochure) even though it's technically an "open" bid, and 3) have somebody with lots of procurement knowledge help guide the paperwork through the system. At the end of the day, it's all just paperwork, no matter how much money is involved.
And that's just what I've observed in IT. I'm guessing IT is the worst, since all the products somewhat look alike. But I'm not sure. Quite frankly, its way too depressing to think about much.
BTW, the best way to do #1 is simply hire people retiring out of the procurement system. Yep, there are laws against direct hires, so you hire somebody from DoD to help shepherd a Commerce Department contract, or a DHS procurement expert to help with a DoD job, and so forth. If you do this correctly, the poor schmucks left handling the paperwork will be so happy that you can offer extensive support in making sure everything is done correctly that this is another huge plus in favor of your getting the bid.
Given the choice between a fixed cost and an unbounded cost (being the capacity study referenced throughout the article) I'm not surprised they went with the fixed cost.
I'm not saying WV didn't get swindled, but I can sure see how this might have happened:
WV Rep: Do we really need these $20k routers for all our locations?
Cisco: We really couldn't say what your exact needs are without a proper study.
WV Rep: What will that cost?
Cisco: It's really hard to say. We'll have to visit all of your locations and speak with the IT Manager there. We'll have to measure average and peak load. Of course, we'll want to plan for future growth so we're not at this same point in two years...
The real crime is that they didn't open the RFP to multiple bidders. A little bit of competition can go a long way.
E-mail all the IT managers, and ask them:
- How many users do you have?
- What is the total size of the community you serve?
- Do you need new equipment?
That would have quickly revealed that many of them had so few users that these routers were total overkill, and had so few potential users that even 100% simultaneous usage in some cases would not max out the capacity, and that some of them did not have a need for new equipment.
I agree with you that multiple bidders would be essential. But a tiny little bit of due diligence and investigation on the behalf of WV would have gone a long way.
That said, sometimes this is what backfires too: I've seen a government contract recently where a lot of extra work was carried out to meet the requirements of a "security consultant" that refused to sign off on a system because of "issues" that were clearly generated only to ensure that he could pad his list of "problems" he identified to help justify his fees.
Eyeballing those figures should be all you'd need - but yes if there was no central record of past procurements an email survey (or better yet online using a survey system) would apparently be pretty quick.
The grant implementers should also be fired for cause, to wit, incompetence).
There is something more fundamental, there were probably schools that had nothing close to reasonable internet access. (Likely a lot of them were in places where they can't get great internet at home either.) The masses were fed up with it, especially as they thought their kids were being left behind so they beat on their representatives and they just got a blob of cash to fix it, costs be damned. You think it was going to make the parents happy when they announce that it's going to take 7 years to get DSL to their child's school?
It's a monster of our own creation. No idea how you solve it... the people that doll out the money have no idea how to do the work or how to estimate if an estimate is good or not; they're just too far away it and the public's memory is too short term.
They probably did open an RFP to multiple bidders.. That's just game theory though, you think the bidders want to leave money on the table? Every where I've seen government bids produced, they increase the costs.
So, they'd order something for the school with the highest percentage and would then park that equipment there for a year (to fulfill the requirement that it must be used there). A year later, they'd move it somewhere else.
I wrote about this particular router instance last year and linked it with my own FCC story from the year before that. http://rachelbythebay.com/w/2012/05/08/router/
If you have 5 office buildings with 50-500-500-500-2000 people in them (respectively), then standardising is probably a good idea since now you can use the same company to fix problems in all the builds and not have to take special care with the small building.
However. If you have 30 builds with 5 people in each. Sending a new, preconfigure, router by fedex is so much cheaper.
That is how my phone company treats me. If my home router dies, or has an odd problem that they can't figure. They just send me a new router by mail and I switch it out with the old one and send that one back.
What's the price?
Not a lot.
We are state employees, and as such, are paid state employee wages. I am compensated for mileage, but beyond that, there is no charge to the individual districts. Part of the cost of these routers was a 4 year Cisco Smart Care plan which replaces the hardware within 24 hours at no additional cost. What that really means is, they ship it to me and it's my responsibility to replace and return the failed part. Much of what I do is remote configuration changes and very little on-site time.
Everyone in this deal should be advising as though they are spending their own money. You can spend a lot of money, but would you if it were you own and you had other things to consider in a budget?
It's reasonable for an enterprise deployment to try to standardize on some components, which costs something in size-inefficiency. It's ok to spec a 2-5x bigger product for some sites if most other sites are bigger, and if the big/small products are wildly different, etc. But that's clearly not the case here.
Arguably the state should have had in-house competence to spend money on the taxpayer's behalf, or should have engaged a contractor with fiduciary responsibility to design/spec the work (and excluded from bidding).
In other words: if there's no incentive, what difference does it make?
Next task: make the case standardizing on Ferrari 599s for all government vehicles. All of you arguments will hold in that cases as well.
""Everybody had to comply with this spreadsheet," he said. "Every campus had two border routers, two cores, and two server farm switches. All the vendors had to propose exactly the same solution" based on the average number of servers deployed at each CSU campus. "All of this is based on exactly the same data to all of the vendors. It's exactly the same formula for all of the vendors.""
I am related to an ex-sales exec at Cisco UK. They are a bunch of shysters who will desperately oversell anything to anyone. The person in question was responsible for selling such kit to managers who didn't even know what it was but they apparently needed it. You know the sort who populate senior positions in the public sector. They are rife in the healthcare and council sectors in the UK. Incompetent morons waiting to be milked for our cash.
The asshat is now selling VoIP and video conferencing solutions to medium sized businesses (via their executives) which is the next cash cow he can milk.
We were doing similar work for private school at the same time, and the process for the public and private school could not be more different.
For eg. in the tender process, which I was part of, for private school we would use cheaper Taiwanese routers. For the public schools it 'had to be Cisco'. The only time we ever used Cisco, outside of large enterprise clients with 1000+ seats, was with public school tenders. The private school would get a $500 white-label router, the public school s would get a $6-15k Cisco router with additional VPN module costs.
We would charge a higher consultant rate on the Cisco jobs, and would bill 3 days instead of 1. We won the contract because we were 50% cheaper than the other tenders who all wanted to install 3 and 5-series routers. How we got to the point of being able to even tender is another story that involves somebody in our organization sleeping with somebody at the government organization. The other tenderers were accustomed to dividing the work up amongst themselves at inflated prices, they didn't even know who we were and we received a lot of abuse for breaking up their little scheme.
(Edit: a further idea of how this worked, the 3 people in the gov office responsible for tenders all had very nice cars and holiday homes while the rest of the office was working away on below-average wages. You could see what was going on just by looking at the car park)
So a dozen of us roll out hundreds of these routers in public schools and after a month we find that we rolled out the wrong version of IOS, one that was vulnerable to a simple security attack. Instead of forcing us to upgrade all the routers remotely, or out of our own pocket, we instead won another few-million-dollars worth of work to send a person out and apply the upgrade to each router (which took 5 minutes, we charged a full day plus travel).
The routers weren't even being used properly - the topography was net connection -> cisco router -> internal server -> switches. The internal server would do all the DHCP and everything else. These expensive routers were being used as bridges, although they were pitched as having 'forward compatibility' incase the school wanted to implement features such as user accounts (they did, although again they used a custom server, not the router).
When these projects are audited there is nobody who is technically competent enough to make an argument against who would be on the side of ditching or shrinking the projects. Some of the smarter teachers knew what was going on but didn't mind since they got access to fancy equipment (we would create user accounts for them).
My first, and not my last, experience with government bureaucracy and budgets. I would estimate that the private schools got more out of us at a tenth of the cost. Since then I had an even worse experience with the government health department, where 6-figure invoices were written and paid for goods that didn't exist (that department has since been broken up and the subject of a large corruption enquire). No surprise that I became very anti-government size and spending.
Edit: to add, we were so 'disruptive' to the backdoor deals that we were uninvited from conferences, kicked off panels, not invited to the mixer events where gov buyers met providers, a couple of years later we lost our accreditation temporarily until we appealed to the Government Minister. On site we would be locked out of network cabinets, not given IP information for the net connections, etc. Our jobs were made difficult by competitors and others because of our pricing and methods. They couldn't figure out how they didn't get rid of us, because they didn't know that one of our guys had a solid relationship with somebody at the government (that person wanted to clean things up). Usually we wouldn't have been allowed anywhere near these projects and if we did win one we wouldn't be allowed back in for not playing with the system.
 'accreditation' for government tenders needs to die, it is a formal method used to keep honest operators out of what is essentially a cartel. The Australian government is getting better in this regard, they now have an open tender website but I believe it still requires some form of accreditation that has a person in a department standing between application and approval.
 although not too much better. NBN Co., a government owned company that is building a $40 billion nation-wide fibre network recently suspended tender process because every single bid came in over the expected price:
The number of times I've had potential supplier hint at extra perks or "return favours" is fairly high, and I've not even been in positions of very substantial purchasing power.
It's often not even direct exchanges that are being hinted at either, but "simply" seemingly innocent networking: Buy from the "right" well connected sales guys and make it clear you're "flexible" about requirements, and they know they benefit from exploiting their network to the max helping you get lucrative jobs in big companies where you can spend more with him.
It's often only morals that prevents it from happening - I've been in plenty of situations while where my budgets were not huge, nobody knew enough to second guess my decisions, and I could've easily spent a lot more than I did without anyone batting an eyelid. Especially since just the very act of bloating your budget in many places makes you look more impressive and important and translates into status (and looks good on your CV...)
I went with a friend a few years back, to get a helicopter ride with someone he did consulting with in the forestry industry. He told me about how the people he worked for expensed a helicopter to go for coffee sometimes... Just for the heck of it. Nobody asked.
The assumption was that if they chartered a helicopter, it was for surveying, so any helicopter expenses were pretty much by default evidence they were doing their job, rather than slacking off and costing the company a fortune.
I tell the guy that manage the sale: "Hey, fine if don't choose us, but if you choose them you will lose ALL the money, for sure". The answer? "We don't care about lose money".
We walk away, relieved of "lost" that client.
Government departments are not.
In Australia independent audits rely on the Australian National Audit Office, which is profoundly and deliberately underfunded because of its annoying habit of discovering embarrassing incompetence and wastefulness.
Governments spend YOUR money, companies spend their own money.
On top of that, companies are subject to competition and, as cynwoody pointed out, they can go out of business. This means that the incentives for reducing fraud are greater in private business and companies that have high levels of fraud that are easily reigned in by obvious solutions will be at a huge competitive disadvantage and more likely to simply go away.
If I never buy a product from Apple or Exxon, there's no problem with that. If I don't pay taxes I go to jail.
Actually you would expect taxpayers to be even more vociferous in pushing audit requirements on government departments than shareholders are in ensuring their portfolio companies are correctly audited.
And if you discover that a company is not doing enough to prevent fraud then you can divest yourself of that company. This in itself is another form of negative feedback which exists much more strongly for corporations than for government. Because individuals divesting themselves of a public stock company will result in the stock price dropping which will have a negative impact on the company, encouraging it to do something about the problem.
And as pointed out in another reply a huge difference between companies and most governments is that countries tend to get the vast majority of their funds from taxes while companies hopefully should get their money from customers. But there are exceptions to this rule, some countries do not have any taxes.
Also, companies can go broke. Compared to that sanction, independent auditors are a band-aid.
In public sector, the audit process is incredibly painful, and you're going to end up losing a career and pension over malfeasance. I've seen it happen a few times. Once you start operating outside the lines, you're likely to escalate and will get caught.
More offensive to me are the folks who understand the system intimately and waste time & treasure legally. I've seen cases where a government used a big govt contractor to install PCs -- at cost of $120/hr for two guys and a truck. Offensive? Yes. Illegal? No.
Apart from the "too big to fail" banks...
As a consequence, when most countries release their budgets, expenses can be hidden for the next guy to reveal. And lazy assets don't burn a whole in spending. Imagine if the US government had to admit how much it was "spending" on rent (taking up land which could otherwise be used by the private sector). A $20,000 router isn't an "investment", it's a burn of $4,000 a year over the 5 years it's in use.
The reason accrual accounting is used in large companies is, when ownership and management is separated, you need to ensure the managers give an accurate picture of what they have spent.
One typical example I had is not receiving a replacement laptop when working for a bank because replacement hw is too expensive, so I could not work for essentially one month. My consulting rate would have paid for a top of the line laptop in 2 days. But those are of course two different cost centers, so it will never surface.
I've always worked with people that, even if not cutting-edge geniuses, are generally more than competent. They don't accept dodgy tender because they know the value of what they're buying, and they certainly don't overspend on gold-plated crap because our budgets are always stretched to the limit.
I would genuinely like to work for one of these mythical government departments where money flows like water. In my experience, the budget is always barely enough to fulfil our role and all these stories about profligate government waste are little more than hearsay.
Then again, I've never worked in the Health or Mining industries.
The higher-than-Europe comparison I had not heard before. The higher than private industry cost of United States government contracting (and contracting by local units of government when they spend federal funds) is dictated by the Davis-Bacon Act,
once we understand that the law's requirement to pay "prevailing wages" means, in that context, to hire union labor even when less expensive nonunion labor is available.
This article discusses some examples: http://www.theatlanticcities.com/commute/2011/11/1-billion-d...
Madrid comes in at $58m/km, Paris and Berlin at $250m/km, and anything in the USA at over $1 billion per kilometer. NYC at $1700m/km. Some of those differences have legitimate explanations, but that's a 30x difference between Madrid and NYC. The Copenhagen metro, with quite expensive Scandinavian unionized labor, also cost a mere $150m/km, and the current expansion seems to be clocking in around $200-250m/km.
(Not arguing with your basic premise that US costs are over-expensive, but a lot of that 30x difference can be explained away).
A bit of it can be, the 5x difference between Madrid and Berlin/Copenhagen/Paris seems acceptable by the differences in environment (Copenhagen's lowest point is 1m above sea level) and costs of living or complexity of the city's "cellars". The 6x increase on top of that is harder to justify.
According to the Wikipedia, the 1318 km Chinese Jinghu High Speed Railway (350 km/h) came in at about $27m/km.
The french LGV Est, also rated for 350km/h, cost ~€4b (up from an original 3.125m estimate, excluding trains) for 300km, or ~€13m/km -> $~17m/km. Including all costs (trains &al) according to the 2007 SNCF report the line cost €5.5b, €18m/km, ~$24m/km.
About £1 billion for a tram line where the end points are about 11 km apart in a city that has excellent taxi and bus services....
[NB We will have a spare aircraft carrier going "cheap" as well...]
Given the number of large government entities, there's a lot of room for bad oversight.
But on top of that there is also often a lot more openness when the auditors does their jobs in government departments and these things are discovered, and even if not, the press often has a lot more access (e.g. freedom of information requirements).
In a lot of companies, a lot of borderline behaviour will instead just lead to people getting fired, especially when there is no evidence.
A new CTO at a company a friend does work for "decided to part ways" with his company a while back, for example, after migrating a substantial part of their service to a new, totally over-engineered and untested platform that failed spectacularly after they'd very publicly spent a massive amount of money on it. Incompetence or corruption? Probably impossible to prove.
And to the public there was no real sign anything was wrong: The launch was highly publicized by the company, and they never told anyone about the negative effects later. Only people with insight into internal sales numbers until they made emergency changes to revert some of his more harmful decisions would know.
And for him it likely doesn't even matter that he was effectively fired: He's either a useful fool, or a useful accomplice, for someone who will happily make use of their network to help him land somewhere else. And many potential hirers will only remember the positive press attention prior to the launch and will have no way of knowing about the problems.
There are a lot of stories like that in private business too.
I've never understood it myself. Governments are funded by tax money, and the people involved in these companies that love overcharging the Government pay taxes. Congratulations on ripping yourself off.
Any additional taxes are diluted amongst the general population, while my sales commission is mine and mine alone...
One thing I've learned from Private Eye: many of our government contractors don't pay taxes -- or at least not British taxes, or a negligible amount thereof.
At some point people will probably need to fully fund and empower an independent audit organization staffed with the kind of detail oriented savants that will start cutting through a lot of fat.
As has been mentioned above, there are many times that people end up doing work only because they are wired to try and do it cheaply/well.
The rest of the time no one is at the helm.
"not uncommon" becomes
Aren't there any avenues to get people fired over this? Talk to a investigative journalist maybe?
edit: After further reading, Cisco did in fact shamelessly participate in this gigantic boondoggle.
I know ... never attribute to malice what can be sufficiently explained by stupidity, but come on! Someone sure got at least a nice car out of it.
To be fair, a lot of things do "Just work". This may just be the time Americans look to their government and decide to clean house and rehire competent people into it.
It takes a lot of nerve for someone who just wasted my money (federal dollars) to try to blame it on the other party for fulfilling the order.
You say "what [West Virginia] wanted", but the article shows how Cisco was unable to show that they knew what WV wanted. For example, Cisco claims that WV specified internal dual power supplies but there is no record that WV ever had that requirement. Instead, "what WV wanted" is defined only as being whatever WV signed off on at the end.
As to if Cisco's behavior is expected or not, the Ars article says: "[Cisco] had a moral responsibility to propose a plan which reasonably complied with Cisco's own engineering standards" but that instead "Cisco representatives showed a wanton indifference to the interests of the public in recommending using $24 million of public funds to purchase 1,164 Cisco model 3945 branch routers."
Ars even links to section 5A-3-33d of the West Virginia Code. The likely relevant clause is "(B) Performance in violation of standards established by law or generally accepted standards of the trade or profession amounting to intentionally deficient or grossly negligent performance on one or more public contracts;", and the auditor's report shows examples of how Cisco's actions were likely in violation of their own published standards.
This is not a crime. As the article says, the strongest remedy regarding Cisco is that Cisco might be "barred from bidding on future projects."
Yes, they have a contract. That is the best indication possible that the customer wants what you are selling.
If I trust someone else with my money, and they transfer that trust over to a salesperson, then they have violated my trust.
I don't want to hear about "blame to go around" and "the salesman should have sold us less stuff". Maybe apply some shame or sanctions to the salesperson, I suppose, but I really don't care unless there was some kind of kickback involved.
Allowing the stewards of your money to redirect blame onto Cisco for selling too much stuff seems so overwhelmingly naive that I don't know where to begin.
If one side does not follow standard business practices, which the auditor argues happened here, then that one side may get better benefits. But this is actually an iterated prisoner's dilemma, and the standard tit-for-tat strategy of the Prisoner's Dilemma makes good sense. That is, shut out Cisco from future contracts for a period of time.
This encourages future vendors, who want repeat business, to stay in a regime of trust and relieves WV of having to pay for staff with full domain expertise on all contract-related matters. A role of the auditors is to identify these problems after the fact and use debarment as a way to disincentivize this vendors from this behavior.
This isn't naive. It's basic game theory.
If Cisco wants to be treated like untrustworthy rip off artists, then they acted the right way. People will take note.
Cisco acted like a business, you can't blame them for that.
Any time someone on hn complains about some business ripping them off, some variant of this comment usually gets posted. And it completely misses the point.
The point isn't moral blameworthiness. It's sharing of information. Some people may prefer to buy from a company they believe they can trust to give good advice about which of their products will meet their needs best. Others may not care about that and will buy just on the basis of the product. A company happy to sell products they know are overkill is making a business decision, to increase profit in the short term at the risk of losing the former kind of customer in the long term.
So reacting to criticism of a company overselling by defending their right act that way in a market is confused. The goal isn't armchair moralising, it's sharing information about them so that people can make more informed choices about who to do business with.
My sarcastic retort would be: "no, you fail to recognize the forward-looking nature of their vision because you did not recommend this other even more expensive and more powerful router!"
Some days I wonder how we don't seem to encounter tons more corruption and incompetence at all levels of the U.S. government.
Anyhow. I remember when they decided they needed a domain server. What they eventually came back with was a two year old model HP tower server that ran over $4k. It was insane. They could have gotten something new for much much less, but all the explanation I could get was that they had a contract and had to buy the items listed in that contract.
There is a culture of corruption and it is prevalent in not only the public, but also the private sector.
Another story is when I was in a diner years later and heard my old social studies teacher asking for a bribe from someone to get a job in the coal mines. I felt pretty bad because I believe he got his money and thus his power from years before when I told him to buy Google and others stock.
Cisco is a public company with a fiduciary duty to make money. You should assume that companies are going to try to fleece you whenever possible. Unless they committed some kind of contractual breach or fraud, they're not the problem.
The fault here is clearly with one or multiple people in the government. This is the kind of gross misconduct for which it feels like firing isn't even close to enough -- this kind of managerial incompetence really deserves at least a short stint in a white-collar prison somewhere.
If you are in a minority of the population which wants to unilaterally impose its will on everyone else because they are more important, however - then I guess that democracy is no good and only an armed coup will do. Let's hope that people don't start thinking that way, because then we are all in trouble.
Did I miss some part where they contracted Cisco to visit every location and assess it's needs and build an order according to that?
If you order a bunch of 3945 routers then why is it Cisco's fault for selling you a bunch of 3945 routers?
They were going to spend all the money the feds gave them. Whether they spent it on over powered routers or "consulting" or something else, it was all going to be spent by the end of the project.
There are enough projects overbudget that being on budget looks really good. And there's always a concern with being under budget that next time the feds won't hand out so much money, and you might really need it then.
> the legislative auditor concluded that the company "had a moral responsibility"...
Since when do corporations have any kind of responsibility, other than to make as much profit as possible? Expecting anything else is futile. If something needs to be fixed, it is in mechanisms to restrain this kind of behaviour where it is in the public interest. Demanding that corporations have morals is not the solution.
For e.g RFP bidding/winning criteria is based on point system.
1. Price accounts for 25%
2. Referrals from similar orgs 40%
and so on
you see, when the price in the RFP accounts less then it is an indication that the RFP is wired to a specific vendor.
A small business who is just starting may or may not have 3 or more referrals from a public school or university.
In one of the RFP process I sat, the RFP had a typo where they had literally spelled out the name of a competitor company. I pointed out this in the pre-bidding call and the personnel acknowledged that it was a mistake.
Government RFP are a waste of time because it is the last step the government does in their purchasing process. They have already chosen the vendor and most of the time the vendor helps them write the RFP.
Fixing the RFP process may eliminate some of the overspending.
Education would not have made this requirement because unless a school has two power sources the feature of dual power supplies would have no use.
Oh is that how redundant power supplies work? You need two power sources, eh?
You get redundant power supplies by plugging both into the same power feed.
However, you can gain some extra redundancy by plugging each power supply into a different circuit, and reducing the common point of failure (shared power source).
Although I've got a good war story on that one too, I had a machine with redundant PSUs emit a stream of smoke from one of the PSUs. It was happily up and serving while smoking out the machine room until the operator (rather sensibly) grabbed both power leads and pulled, at which point both fire and ssh went out ...
Dual power supplies is a bit overkill. Although I remember my high school admin got a central router with dual power supply, but when you plugged both in at once (from the same source) they drew too much power on a reboot and flipped the fuse after every blackout.
If the cheapest router Cisco has with dual PSUs is a 39xx, well, I guess that's what you need.
Of course, there's an argument there that "99.999% uptime" is pretty pointless for a town library in the middle of nowhere, but that's a whole 'nother argument.
As far as I understand, such an incentivized sale was a common practice. Once the management somehow achieved consensus (i.e. the right balance of risks, benefits and favors), engineers were left with making things appear OK.
I am inclined to think that Cisco made a strong bet on this kind of a business model at some point.
"The West Virginia legislature at peak times can have over 600 internal users and numerous guests accessing "multiple Web servers, up to eight simultaneous live audio webcasts, multiple SQL servers, and multiple Google search appliances located in the Legislature's server farms." Despite all this, the legislature doesn't even use a router but instead runs a cheaper Cisco switch... and it has never exceeded capacity."
It's pretty clear that not only does the author not know networking basics, his sources don't either.
Funny how the Feds were fine with a 2% increase in payroll taxes for consumers but a 2% cut in the increase of their budget is going to bring on Armageddon.
It may just be a reduction of an increase if you look at the TOTAL Federal budget, but for many individual agencies it is an actual deep cut to their budgets.
In extreme cases one didn't even have to bribe officials to get bogus contracts because if the city didn't find a way to spend all the money it was offered, it would simply get less.
Positive corruption does the right thing (build roads, install broadband into poor communities) but adds cost
Negative corruption distorts right thing so much it simply does not happen (communities never get proper triads or sewage because they are the wrong tribe)
The broadband is the thing - good. If you don't want government to overpay I suggest Cisco routers is the last thin to look at - start with military contracts
"Some libraries need smaller routers. Ok. Which ones?" could easily run in the millions before any routers were purchased. Unfortunately, that's just the nature of government purchasing.
Wouldn't that imply they hadn't succeeded with the same ploy before? I think that's unlikely!
To me this looks like a situation where something was said verbally, and then no confirmation or follow up was given.
Wanting to get the quote out, Cisco just made it and sent it. They assumed that it, and any other issues, would get sorted out when the quote got reviewed.
The client slept. Cisco said "Good job sales team".
It is always the job of the buyer to know what he is getting, for what he is paying.
I suppose when I say, "they could have" I mean that some appropriately-empowered decision maker who had an interest in making the right decision could have either assigned his own network engineers to do the work, or could have assigned such an engineer to hire someone else to do the work. I probably shouldn't assume that such a decision maker does exist.
I though I was cynical but if this it truly "the nature of government purchasing" I am clearly not nearly cynical enough.
Q: Wouldn't the internet service provider be willing to throw in a router for free? I got one with my $20/month service. I bet these libraries are paying a bit more than me.
What if one power supply fails? I expect better from ars.
I don't have a source handy, but around 2004-2005 Steve Ballmmer came in a short visit to my country (somewhere in Eastern Europe), he had a even shorter meeting with the Prime Minister of that time and sure enough not 6 months later a Government programme was launched and all the schools and public libraries in my country had to be equipped with Windows machines. This was around the time when people thought that tools like Open Office actually had a chance of toppling the MS Office suite.
If you had a generation of kids who have experience with openoffice, it'll be much more likely that openoffice would be something that would be adopted over time - newly formed companies using it, gradual shift away from MS Office, etc.
MS knows their history, and will continue to seed generations of users with low-cost/free stuff to keep their dominance going. If they quit doling out free copies, something like openoffice would gain a larger foothold in schools and eventually businesses in < 10 years. Would MS Office be entirely replaced? Of course not, but it would not be the default/automatic choice for everyone without question.
You wouldn't need more than your average crow bar to get into this thing. I bet they don't have a decent alarm system (if any).
Seems like a relatively low-risk, high-reward situation to me.
This is the dark side of enterprise sales. Microsoft and many others have surely done many such deals over the years - and paid out handsome bonuses to the reps/execs who made them happen.