
NYC mayor demands $600M refund on software project - abennett
http://www.itworld.com/software/179037/nyc-mayor-demands-600m-refund-software-project
======
kenjackson
_CityTime was launched in 2003 at a budget of $63 million, but costs swelled
dramatically as the project stumbled along for nearly a decade._

From $63M to $600M? I would have thought that ppl may have noticed when the
costs doubled or tripled the budget. Almost 10x the original budget? That type
of oversight must go pretty high up.

~~~
bmelton
I've been on the wrong end of a couple of projects like this, and in my
experience (not necessarily indicative of this case) the customer can be VERY
at fault.

I've been dragged into 'feature acceptance meetings' that literally dragged
out for weeks when testing of a new chat component devolved into a days-long
discussion on whether the graphic in the header (provided by the customer) was
appropriate. The unspoken issue proved to be that they didn't want us to use
'barren trees' in the picture of their building. We switched the photo used
from one taken in winter to one taken in summer. This took something like 400
man-hours to resolve, after which we achieved signoff, but at the expense of
having met budget.

Another instance on the same product was feature creep to the extreme. We had
originally won a firm-fixed contract, meaning we would build products a, b and
c for x dollars, and if it cost us more, it came out of our pocket. If it cost
us less, we pocketed the profits. The customer decided, on day 1, that they
actually wanted a lot more features than were described in the project scope
defined, and that we won a legally binding contract to perform. Out of
accomodation, we changed their product to a time-and-materials contract,
meaning we had a fixed markup on labor and a minimal markup on hardware /
tangibles. This then involved them dragging the project out for years longer
than anticipated and signing invoices all the way, then griping at us on the
budget near project completion and asking for a bunch of free labor (which is
illegal under their new contract). The problem in that case was that the
project management and senior execs changed repeatedly. We were at first
working with their Office of Science and Technology, then with Information
Services, then with Public Affairs. Each of these department changes involved
swapping our equipment out so that we could have appropriately charged-back
equipment to the appropriate department, which meant weeks of non-work. Each
department had a different spin on what they wanted out of the project. We
spent 8 months and thousands of man-hours implementing a very elegant services
oriented architecture that could surface all of their legacy systems to a
single web portal. Then we changed departments, and they literally told us to
turn that feature off, and instead focus on a visual redesign. We implemented
a single sign on service that we integrated with our product, but could be
extended to work with literally every system in their enterprise. It was
abandoned due to a department change.

During the first phase of the project, we'd worked with the customer to build
out their data center. They wanted a new data center because the division that
owned the project was going to be adding a bunch of other programs to
integrate with what we were building. They spent over a year on construction,
converting a service garage into a data center. They bought millions of
dollars worth of EMC storage hardware and Cisco routers. Then they realized
that they didn't have any budget planned for out-year maintenance, so they
moved the venue. That was somewhere in the realm of $50 million dollars that
was literally just thrown away, as the last I heard, they were re-converting
the data center back into a garage.

In short, government contractors catch a lot of flack, because they control
the visibility of these things. The contractors often can't speak to details
of these projects due to security clearances or NDAs, and at the end of the
day, the government is always happy to blame the vendor. It's well practiced,
and designed with a variety of scapegoat clauses built in.

~~~
protagonist_h
what? it took 400 man hours to replace a logo?

~~~
gnaritas
400 man hours to _decide_ to replace a logo, big difference. That's what you
get when you design by committee.

~~~
bmelton
Exactly right.

Replacing the logo itself took all of about 30 minutes (receive new picture,
digitally retouch, overlay onto background image, resize, play around with
transparency, save to png.)

Also, bear in mind, the 400 hours is 400 man-hours, which is a cumulative
number based on the number of people involved. If you've never done business
with government, let me assure you, it was a lot more people than necessary.

~~~
zaidf
Absolutely. The man hours can easily pile up when you do a conf call with a
half dozen folks where majority of them are on mute 90% of the time.

I recently did contract work for an agency and had instances where a simple
spacing or image resizing took days of work because the client or the agency
would understand and then forget that I must crop and/or resize images for
them all to be of one size.

------
n-ion
Maybe some of you know, maybe some of you don't - but the real hero here in
bringing these people to justice is Juan Gonzalez
<http://en.wikipedia.org/wiki/Juan_Gonzalez_(journalist)> He did the reporting
that led to the federal indictments. Went on to win the 2010 George Polk award
for exposing them.
[http://www.democracynow.org/blog/2011/2/22/juan_gonzalez_win...](http://www.democracynow.org/blog/2011/2/22/juan_gonzalez_wins_2010_george_polk_award/)

------
thematt
_Although New York has "received a working system that will advance our
management ability ... because the project was apparently tainted by fraud and
kickback schemes, the City must be made whole"_

...

 _The city is also planning to launch a request for information (RFI) process
in January in search of alternative timekeeping systems._

Wow. So they eventually got a working system. Now they're throwing it away and
going to purchase a new one? Meanwhile turning 83 contractors into FTE's to
maintain the existing system? This seems like complete craziness for something
as commoditized as timekeeping software.

~~~
kenjackson
Timekeeping systems are actually extremely complicated. EXTREMELY
sophisticated. The level of detail it takes to get one right is painstaking,
and it is key to a city's survival. It's complex enough that you wouldn't
understand it, nor would any auditor.

For that reason, I just started a new startup company: TimeKeep NYC. We
specialize in timekeeping software for NYC area cities. Looking for employees
now, especially ones who aren't snitches. Based on past products in this
market, I think we can do quite well.

~~~
clistctrl
I'm curious, what are some of the complexities involved in a timekeeping
system?

~~~
ams6110
Things like varying overtime rates depending on time of day, accumulated
hours, whether it's a holiday, different rates for depending on specific work
assignments, etc. It's not as simple as a flat rate x number of hours. But
it's nothing that hasn't been done before, either.

~~~
smokeyj
At first I was like, "wow this would take me five minutes to program". Now
it's looking like a solid 20 minutes.

------
scubzero
Worked for subsidiary of SAIC contracting out to NYC government over a decade
ago. Trust me when I say this is nothing out of the usual... project will go
order of magnitude over budget constantly.

As for the requirements that drive these projects, typically they are written
specifically to present the appearance of an open bidding process while
allowing the departments to know ahead of time who will actually implement the
contract because they are so specific in certain areas that only one contract
can possibly fulfill them.

------
jarrett
Those who say that the city's payroll needs are complex are entirely correct.
A city government that size will inevitably have a mind-boggling number of
variations in the business rules that the software must model.

Yet $600 million? I've built accounting software, and though it wasn't as
complex as what NYC needs, I'm 90% sure $600 million is astronomically higher
than what a reasonable price would be.

What about the original price of $63 million? That still seems awfully high to
me, but I'm not sure. Does anyone have an opinion on that?

~~~
kingkilr
I've never built a time keeping system, but this is how I put that number in
perspective. For $63 million I can hire everyone who's ever been a Django core
developer (people I've worked with, and who are excellent) for 5 years, at
500k a year... and there'd be money left over.

~~~
protagonist_h
Building a time-keeping system for large city like NYC is many,many times more
work than building web-framework like Django. It's not a technically
complicated problem, but it has a different kind of complexity. That
complexity comes complex business requirements, which have to be analyzed and
implemented piece-by-piece.

------
hvs
Some background on this:

 _Mark Mazer, CityTime consultant accused of scamming millions, has checkered
past of investigations_
[http://articles.nydailynews.com/2010-12-23/local/29441786_1_...](http://articles.nydailynews.com/2010-12-23/local/29441786_1_citytime-
project-mark-mazer-sexual-harassment)

 _CityTime Crook Pleads Guilty As Defendant List Grows_
[http://gothamist.com/2011/02/11/citytime_crook_pleads_guilty...](http://gothamist.com/2011/02/11/citytime_crook_pleads_guilty_as_def.php)

 _Mark Mazer Indictment [scribd]_ [http://www.scribd.com/doc/58405089/Mazer-
Mark-Et-Al-S2-Indic...](http://www.scribd.com/doc/58405089/Mazer-Mark-Et-
Al-S2-Indictment)

------
BadassFractal
How complicated must that piece of software have been for it to cost that
much? For that kind of money it better be at least as complex as any OS out
there.

~~~
almightygod
my guess is most of the time was spent in analysis and consensus building
being that it was for government. The code is surely atrocious

~~~
InclinedPlane
I would wager good money that you could supply thedailywtf.com with entries
every day for a month from the code base of this project, easily. When you do
the math it works out to something like $100 million a year, which implies a
fairly large development team (hundreds of people, at a bare minimum).

It's really easy to waste money on big development projects of this sort. The
bigger the development team the harder it is to manage, the less well it is
managed the less productive it is in aggregate, and thus the less productive
it is per salary-dollar. And these are not small effects. A small team of very
talented devs can quite easily compete head to head with a giant team of
hundreds of devs, precisely because of these scaling and management problems.

------
vaksel
that seems a little excessive even at 63 million...I'd bet if you gave a YC
startup $1 million, they'd be able to do the project in 1 year.

~~~
brendoncrawford
See: <http://news.ycombinator.com/item?id=2716446>

~~~
vaksel
with 1 million, you can afford to pay a thousand bucks to have someone
translate that for you

------
peter_to
In case anyone wants to read up on the CityTime project, the VillageVoice had
a pretty lengthy spread on it in January.

[http://blogs.villagevoice.com/runninscared/2011/01/this_week...](http://blogs.villagevoice.com/runninscared/2011/01/this_week_in_th_90.php)

------
bane
Funny, I did part of my M.S. coursework in Enterprise Management using SAIC as
a case study in Enterprise Management Failures.

------
bluedanieru
Michael Bloomberg might have a point, but he really isn't in a strong position
to complain about shitty overpriced software. Glass houses and all that.

~~~
tkahn6
What position are you in to declare the Bloomberg terminal 'shitty overpriced
software'?

~~~
mdda
Just so people can put this in perspective, a Bloomberg terminal license costs
approx $1750 per month, with a two year contract. The software basically gives
users a fairly raw interface to financial data, analytics and an email/IM
system.

In a sales call, the Bloomberg sales reps will demonstrate endless
functionality of the terminal : there's a huge amount there. The dirty secret
is, though, that most people use only 4 different functions (different people,
different functions, of course). The big switching cost is that the email/IM
is entirely proprietary : Everyone has to have a BBg terminal, because
everyone else has one.

The software is probably as well written as you would expect from a company in
a monopoly position. One thing in their favour : If they fundamentally changed
the way the 'MSG' button worked, 50% of their users would complain.

~~~
tkahn6
Full disclosure I'm an intern at Bloomberg.

You're essentially correct on all points but clients tend to use many more
than 4 functions though significantly less than the total number.

And with regards to IM and email, it should be noted that any user of a
Bloomberg terminal can IM any other user anywhere in the world (there's no
'friending' or anything). Many, many trades are conducted over IM or 'IB' as
it's called.

The most accurate analog I can think of to the Bloombeg terminal is Emacs. It
has everything and the kitchen sink (News, dining suggestions, email, IM,
sports updates, etc.) and clients are extremely particular about
functionality. I have a cousin that's been a subscriber since 1997.

And with regards to code quality, I'm not sure how much I'm permitted to say,
but the first Bloomberg terminal was released in the 1980s (this was before
SQL existed). Also they maintain the largest private network in the world
(they run a line to every terminal)... so all the things that go with that.

~~~
heydenberk
> The most accurate analog I can think of to the Bloombeg terminal is Emacs.
> It has everything and the kitchen sink (News, dining suggestions, email, IM,
> sports updates, etc.) and clients are extremely particular about
> functionality.

Of course, Emacs is _free_. Do you think Bloomberg is susceptible to
competition from a much lower-priced and much leaner startup?

~~~
RickHull
Bloomberg terminal is a classic example of the
<http://en.wikipedia.org/wiki/Network_effect>

