
After alleged 'bait and switch,' Nevada DMV cancels $78M IT contract - us0r
http://statescoop.com/after-bait-and-switch-nevada-dmv-cancels-78-million-it-contract
======
us0r
Audit:
[http://budget.nv.gov/uploadedFiles/budgetnvgov/content/IAudi...](http://budget.nv.gov/uploadedFiles/budgetnvgov/content/IAudits/About/AuditRpts/C18-01%20DMV%20Audit.pdf)

This one is extra bad. Not even a communication plan? They did manage to spend
$13 MILLION on Oracle shit though. The best part is the solution they sold,
which is not even working in New Hampshire, and was built on DynamicsCRM (not
portable to oracle)

The grand finale:

"The contract was amended in February 2017, which extended the termination
date for an additional year for maintenance coverage. It also increased the
maximum amount of the contract by $3 million, primarily for additional
hardware and software."

That person needs to be fired immediately and probably arrested.

~~~
CapacitorSet
Unsurprisingly, the work was outsourced to an Indian company, "Tech Mahindra".

Nothing against Indians as a people, but Indian outsourcing companies are
known for the terrible product quality.

~~~
ehnto
I just don't understand why public works should ever go outside of the local
economy, globalisation or not. I don't just want my tax dollars to give me a
new thing, I want that new thing to be built in my state. If we don't have the
skillset then let that money stimulate growth there.

Recently our government outsourced some naval ship building outside the
country at the last moment after having expressed they were going to have them
locally built. The reason for the switch was cost. To shave a million or two
off the price of a many hundred million dollar project they decided to
completely remove our local economy from the benefactors.

Private projects? Do what you like. Global times. But a governments role is to
grow the economy and welfare of its people, public works are excellent ways to
stimulate that.

~~~
mc32
Or local universities[1]. I think they had their then current staff train
their replacements --what gall.

Now, imagine if the faculty president thought "hmm, I bet we can save a bunch
if we do remote teaching. We'll get super large interactive displays in every
classroom and we'll have faculty in [some country] teach our students. We'll
save millions!"

[1][http://www.sfchronicle.com/bayarea/article/UCSF-to-
outsource...](http://www.sfchronicle.com/bayarea/article/UCSF-to-outsource-
dozens-of-tech-jobs-to-India-10414432.php)

~~~
acct1771
Anyone who's ever seen a University budget can see why they'd never do that.

Slippery slope, if you outsource/fully digitize the teachers...administration
comes next.

------
atarian
It's kinda crazy how much money gets thrown around by an org you wouldn't
think twice about. If a group of engineers from HN got together and secured a
deal like that, they could easily retire with that kind of money.

~~~
acdha
Never underestimate the complexity of those projects: I’d bet that the
requirements would fill a bookshelf, and simply navigating the hurdles to bid
& convince them that you have the resources to deliver would rule out a group
of engineers below the triple digit range.

(Yes, that’s not a good way to build things successfully. Welcome to
enterprise IT.)

~~~
Retric
In theory the requirements fill a bookshelf in practice nobody checks them
that closely outside of fairly basic automated tests for things like 508
compliance.

~~~
acdha
This is both untrue – maybe they don’t check 100% but everyone who put
something on that list is going to notice if you leave their needs unmet – and
it doesn’t matter unless you know which things won’t be checked. Otherwise you
have liability for not delivering, which probably means you take on the same
level of complexity as requested even if much of that effort is wasted or
counterproductive.

~~~
Retric
It's true that a range of stakeholders care about different things, but the
solution is to demo the project regularly to verify it fit's basic needs and
covers everyone's pet issue.

Gov software projects are on a pass fail scale, and people have minimal
expectations despite what they may hope to see. The waterfall approach of
gather a bunch of details > build something is more about budget than a black
and white must meet 100% of every specific req.

~~~
acdha
The problem is when people are restricted by policy or law from not using
waterfall. So many bad government outcomes are caused by attempts to prevent a
past mistake on a different kind of project and changing those policies is
harder than trying to muddle through any one project.

------
kabdib
Prior version of the system used PowerBuilder. That figures.

My fly-by of PowerBuilder 20 years ago made a deep impression. Like, I never
wanted to even _think_ about that hunk of garbage again. It's right up there
with PeopleSoft. If you're ever offered an opportunity to work with these
systems, just run away.

The state didn't know how to buy or develop software then, and it doesn't know
how to buy or develop software today.

~~~
j45
Experienced developers know that a mess can be made in any platform.

One issue with a position and maligning statement like the above against any
platform is that it actually enables said charlatans to charge way more for a
sub standard developer base who end up being over paid, because skilled
developers are not as readily available to fix, and ideally refactor the
system.

Hardest of all to imagine is systems that have had legitimately tens of
millions of dollars of mission critical and hardened infrastructure that
manage more complexity than many of us will ever handle.. are written in
platforms like Powerbuilder.

I have personally never used Powerbuilder. I know some ace developers who
spent a lot of time (among other things) using Powerbuilder and they have said
it's like any other language or framework - lots of messes, some good stuff
too.

Still, I won't ever stop noticing the kinds of hateful comments people throw
at others because of what stack they work in. It's sheer overlooking of
opportunities.

Comments like the above are like saying all wood is bad because we saw a house
built poorly out of wood.

This approach excludes folks from the real huge innovation projects that are
coming due as these platforms and projects are gradually renewed or
refactored.

~~~
kabdib
I've worked with many programming environments, and PowerBuilder stands out as
one of the worst. I remember exchanging emails and doing phone calls with the
developers about some howling bad bugs in their COM support, bad enough that
we wound up ditching PB despite the market it represented. Judging from the
bugs I was finding and the release notes that I was reading (which covered
issues that included how to avoid writing code that would crash the compiler),
PB was a mess under the hood.

I don't abide by broken tools, but I do respect humility and realize that bugs
and flawed designs happen. However, I will have no truck with companies that
ship buggy, crashy products and then do the whole "manage customer
expectations" dance in an obvious attempt to avoid remedial engineering. When
I used it, the product was several major releases old, and they seemed to
think they could trade workarounds and sophistry for fixing bugs.

Hateful? Probably. Undeserved? Definitely not.

~~~
j45
I agree Powerbuilder isn't the best tech - the point I was trying to make is
every tech has it's pro's and cons, and older techs have more to maintain
while trying to stay current and get ahead, just like newer platforms can
suffer from a lack of maturity. Powerbuilder didn't do itself any favors by
trying to rest on it's existing marketshare.

The kind of hair pulling I heard from buddies working on otherwise interesting
systems in PB was quite deep, several nights a week in it's height. The issue
of using an older version of PB seemed to make things worse.

Still, it seems relevant to look at a technology in it's time compared to what
else was available, and wonder if this is where today's language+framework
apologists may end up in the future when one could no longer explain away
their current favorite platform in 2018 as a flavour de jour.

My last comment is not not directed at you in any way - You articulated a
depth of experience: Opinions based on hearsay or surface interactions about a
tech is not much different in the venom spread than than judging a group on
the color of their skin based on one interaction and an underlying propensity
to find fault. It's harsh, but maybe it's not recognizable that it's a similar
muscle.

------
maerF0x0
I wish code for america would take these kinds of things on. I assume the DMV
rules are sufficiently similar that most/all of them could benefit from
opensource.

[https://www.codeforamerica.org/](https://www.codeforamerica.org/)

~~~
dboreham
I’m guessing it’s actually the humans that cost $$$ not some source code.

~~~
jachee
They spent $5M on Oracle software. That's a LOT of someone else's source code
they paid for.

~~~
matt_s
And at one point Oracle was just using open source software, adding on some
things and calling it something with a $10k/cpu cost. They may still do this.

------
aerotwelve
How does this happen?

Who in their right mind thought it was a good idea to outsource the
development of this critical system to a foreign contractor? Did anybody stop
and think about what might happen to all of that public money if the contract
is breached?

~~~
Spooky23
You can can tell from the auditors report that the RFP or other solicitation
was fubar. Many of the deficiencies there are standard contract terms that
were probably missing.

It also sounds like the project was getting done on the cheap. The actual
dollar figure is buried a bit... it sounds like they got their hands on a big
wad of cash and went on a spending spree. I bet they bought a new mainframe,
lots of of network crap, etc.

The poor procurement practices led to the state ending up with some two-bit
bodyshop that just tried to job out the work. My guess is that the state
program side is barebones and some political hack pushed this through.

~~~
dboreham
Interesting. In my teeny tiny company our insurance policy states that we must
have all contacts over a certain value reviewed by an attorney. Wondering how
a vast beurocracy somehow isn’t subject to that rule?

~~~
Spooky23
Lawyers will tell you if things are legal. They won’t generally stop you from
doing dumb, legal things.

~~~
dboreham
In my experience that is absolutely not true. Perhaps you need a better
lawyer?

~~~
avisser
A lawyer who will tell you that a hybrid cloud/on-premises solution using
Oracle isn't using today's industry-insider (read: us) agreed best practices
and might waste a bunch of money?

Good luck finding and having enough money to pay that lawyer.

~~~
dboreham
We're talking about completely different things. My comment was a follow on to
this:

>Many of the deficiencies there are standard contract terms that were probably
missing.

Lawyers exist to make sure you don't sign a contract that is missing standard
contract terms.

------
atonse
According to the audit document, the company was Tech Mahindra.

I am genuinely surprised when a 8 to 9 figure software project ever succeeds.

But I'm also amazed that the government spent $25mil before they realized they
got completely bamboozled. There's something shady going on here.

~~~
notyourwork
As someone who has worked with state funding contracts I disagree that this
implies anything shady. Hourly rates for consultants at exorbitantly high. It
also depends on how the project is big, fixed feed versus hourly bulk.
Regardless, the amount of hours quoted and the rates are inflated based on
politics.

It's not shady, its just incompetent people signing contracts to promise tech
companies bags of money.

~~~
frankharv
From the article: "the project was not managed by the state's Enterprise IT
Services division, and was managed solely by outside master service agreement
contractors hired through the DMV's internal IT department."

If that is not shady I don't know what is.

~~~
matt_s
It's not shady, its politics. The DMV IT department didn't want to deal with
"corporate" aka Nevada Enterprise IT. Perhaps Nevada IT was too busy digging
out of a debacle with healthcare IT systems to be bothered with DMV stuff.

------
greatamerican
I feel like I could provide the Nevada DMV with whatever it needs for
significantly less than $78 million.

~~~
wavefunction
Only one way to find out! There's so much opportunity to improve the "non-
sexy" work of governments and institutions and the interface between them and
the people they help/serve/administer that it's crazy there isn't more start-
up attention focused there.

~~~
craftyguy
In reality, there is very little realized opportunity to improve the 'non-
sexy' work of governments and institutions, because you're up against either
big names like Oracle or <insert official's nephew's name here>.

~~~
wavefunction
I won't argue that Oracle isn't optimized to meet and take advantage of public
institutional procurement policies but if you've ever seen one of Oracle's
offerings, it's easy to surpass their quality and given their rates, trivial
to come in cheaper.

My mom was an instructor with the community college system in Colorado and
about ten years back they rolled out an Oracle based-and-delivered student
enrollment product that failed during its first enrollment after being
deployed because it had been set up to only allow 100 concurrent connections
to the state-wide persistence solution. Oracle was of course 'earning'
something like $90,000 every day the outage continued despite it being their
own direct fault.

I can't stress enough how incompetent or over-priced Oracle solutions are for
pretty much anyone.

------
osrec
$78m over 5 years for 25 (probably average) developers and a few licenses is
excessive. Not sure what massive tech problems the Nevada DMV is trying to
solve, but I doubt it's much more than a DB cluster with a web app or two
thrown in front of it. Tech Mahindra's developers may be average, but their
marketing department seems to be exceptional!

~~~
danudey
Integration with existing/external systems is likely a huge part of the cost.
If the DMV integrates with the police systems (so police can run
license/registration), they need to be integrated. If they have license plate
scanners that do the same, they need to be integrated (or replaced). Point of
sale terminals for things like license renewals might need integration.

No system exists in isolation.

------
pxeboot
This type of waste occurs in all levels of government. My last employer was
spending over $1,000,000/year maintaining some horrible SharePoint sites that
were rarely used.

~~~
vkou
This sort of problem is not limited to government. Private companies waste
money on all sorts of dead-end nonsense - all the time.

It's why IBM and Oracle are still in business.

~~~
jjeaff
In all my years in tech, I have literally never heard a single person (that
wasn't part of a flashy ad campaign) exclaim that IBM or Oracle offered their
company a solution at a good rate, got it setup in time and that it works well
and solved their problems.

~~~
nraynaud
I think I have read one anecdote: Apple stores and Oracle. I read somewhere
that Steve Jobs was happy with the point of sales software and that there is
some Oracle in it (and that Jobs and Ellison knew each other)

Only positive Oracle anecdote I know of.

~~~
kgwgk
They didn’t just know each other:

“Ellison's idea was to buy Apple and immediately make Jobs CEO.”

“Jobs proposed what Ellison called "a more circuitous route." He would
persuade Apple to acquire Next, and then join Apple's board.”

“After Jobs was CEO of Apple again — interim CEO initially — Ellison joined
Apple's board.”

[https://www.recode.net/2016/5/13/11672932/larry-ellison-
stev...](https://www.recode.net/2016/5/13/11672932/larry-ellison-steve-jobs-
oracle-apple-usc)

------
snowpanda
I'm sure that money will find its way back to America, in political donations
to be exact. Kind of gullible to think these 'failed to deliver' accidents
keep happening on accident after half the budget has been spent.

------
j45
I would love to see how much of the purchasing and decision making committee
was composed of people who didn't understand technology, let alone how to
procure and implement technology.

At the highest levels, there is a self-preservation disease permeating most
organizations where CxO's try to fake it till they make it with the IT/Systems
decisions they take on, and it turns out like this.

------
marktangotango
Ah the good old responsibility dodging and blaming following legacy
modernization project gone off the rails. At least they got out spending only
$27 million. The state says bait and switch, I wager the contractor never got
requirements or basic documentation for the project.

I’m at a company now that’s planning one of these projects for a core system.
I’m looking forward to having a front row seat!

~~~
us0r
Read the audit i posted above. The contractor didn't even show up LOL

------
notananthem
Tech Mahindra contractors, not surprising. An acquaintance of mine came in on
a job with them- they pay all their guys salary which is kind of crazy. That
guy was doing better than the hourly contractors with family time and pay.
Something sketchy is up with them.

~~~
megy
Not that odd. A lot of contracting companies do that, then bill you out at a
day rate.

------
i_feel_great
I am currently working on a largish COBOL codebase which the company has tried
4 times to port to Java, each time by porting all at once. It cannot be done
all at once.

------
bovermyer
It's stories like this one that make me want to pivot and go work on projects
for the government. I KNOW I can contribute something useful.

------
linkmotif
I wish there was a collection of these.

