
Baltimore police will pay consultant $176k to maintain Lotus Notes system - linkmotif
http://www.baltimoresun.com/news/maryland/baltimore-city/bs-md-ci-lotus-notes-20180808-story.html
======
mikeash
Two years of support for $176,800? What a bargain! As long as this old system
gets the job done it seems like an extremely cheap way to go.

~~~
Justsignedup
Bargain? That's practically highway robbery (taking from the developer that
is)! A developer making shitty modern react apps can make 180k a year with a
bit of effort.

And I have no idea how many lotus maintainers there are left in the world.
Definitely not that many.

~~~
madeuptempacct
Good, senior, full-stack devs make $90,000-125,000 in flyover states atm. And
I do mean good ones. A lot of people are stuck at $80,000ish.

$180,000 for a front end tech lead is unheard of in said states. Architects
make maybe $160,000.

~~~
scarface74
What do you consider a “flyover” state? From the major cities I know that are
not on the west coast, New York City, or DC, $125K is around the top for a
“Senior Developer” and $145K is near the top for architect level skills who
don’t have any direct reports.

~~~
randallsquared
$125K is near the top for a Senior Developer in Charlotte, SC, circa 2014
(flyover? Probably). Probably higher there now, and definitely higher in NYC
or DC.

~~~
scarface74
Same thing I’m seeing in Atlanta. I haven’t seen too much of a change since
2014.

------
chadash
Summary for those in EU: The Baltimore police department uses Lotus Notes
software for keeping track of cases.

> _Officers tap into its databases to track criminal investigations, check
> arrest data, log ballistic test results and identify “troubled officers,”
> along with various other uses. There are millions of records and roughly 150
> databases built into the system, each designed to address different unit and
> personnel needs._

They started using LN in 1996. Now, all of their stuff is built on top of it
and it's hard to switch. So they are paying $177K for the next two years of
software support.

Summary over. Question of mine: is this a particularly high rate for two years
of support for an entire police department? I can't imagine there are many
specialists out there who know how to support lotus notes, let alone all of
the custom data they have layered on top of it. And it seems like it would be
a reasonable amount of work to maintain.

~~~
jandrese
For an enterprise support contract of that size it's dirt cheap. Try to build
a system with 150 databases on a cloud provider or using your own cluster and
you'll spend way more than that. VMWare licenses alone would blow that figure
away if you rebuilt it in a modern style. If you were crazy enough to spec
Oracle for those databases then it would be many times that cost per year.

~~~
mixmastamyk
VMWare and Oracle in the cloud? "Out of the frying pan, and in to the fire."

~~~
toomuchtodo
No different than Lambda and Aurora. Never marry yourself to the vendor.

~~~
scarface74
You’re always married to your infrastructure. That’s like developers writing
“repository classes” just in case one day they want to switch from Oracle....

Statistically no one ever switches out their architecture. The risk is to
great, the reward is too small, and the number of potential regressions is too
high.

How did all of those people who wrote thier website with Angular 1 fare?

~~~
toomuchtodo
Are you going to rebut all of my comments about risk management in this manner
(antagonistic, usually vendor lock in, but it's been about other risks as they
relate to cloud infrastructure)? Because I work in risk management for a
financial utility, this is what I do for a living: ensure developers and other
practitioners don't make poor architectural and technology decisions.

One of your own comments [+] exemplifies the consequences of making a poor
technology choice ("Amazon has been trying to move off Oracle for 5 years and
it’s still expected to take two more. Moving off of an infrastructure is
hard......"), and I'm pretty sure the cost of that choice and the resulting
migration to remediate is expensive AF.

[+]
[https://news.ycombinator.com/item?id=17729107](https://news.ycombinator.com/item?id=17729107)

~~~
scarface74
Was it a poor choice at the time? Apple decided to base the ITunes Store on
WebObjects in 2001. They certainly didn’t have “vendor lock in” since they
owned WebObjects, how does that decision look now in 2018 that they are still
running on WebObjects?

You’re always stuck on your architecture and you’re always at risk of a
technology becoming obsolete.

Look at all of the companies that are still stuck with an older version of the
JVM because they are afraid to upgrade - and thier whole stack is open source.

Unless you personally audited every single line of source code your developers
wrote, I guarantee you that your architecture is tied to some assumption that
will make it hard to migrate in 5 years.

~~~
toomuchtodo
We audit every single line of code written by our developers. We audit every
line of libraries they use. We audit their architecture, their infrastructure,
and their plan to move from cloud to bare metal at anytime. Our applications
run for decades at a time.

We automate wherever possible, of course, but there are some tasks humans are
required for.

It doesn’t hurt for developers here to heed the warnings from professionals
who deal with this every day. While the scale is orders of magnitude
different, the underlying fundementals are the same.

~~~
scarface74
And you can 100% guarantee that all of your infrastructure dependencies won’t
be obsolete in 5 years or any company or open source project won’t be
abandoned and that every vendor you depend on will always be acceptable and
that you can switch with low friction and no downtime?

Does your company have the luxury of building everything in house?

~~~
toomuchtodo
Yes. And we can if we want to, yes. We’ve taken over open source projects we
rely on, but most anyone can do that. We’ve built open source versions of
seven figure commercial software just to ensure we have the skills in house to
do it.

You don’t need to have thousands of employees or billions of dollars in
revenue to accomplish 60-80% of what I’m describing. I’m simply arguing to
take measured steps at critical junctions in your software’s lifecycle.

~~~
scarface74
I can tell you with 90% confidence that if you’re working for a financial
institution that offers bill pay, your entire bill pay infrastructure is
dependent on a third party provider. I know the provider.

If you offer an iOS app, you’re dependent on closed source software. Your
project management software would be a beast to migrate, etc.

Are you really suggesting that every company should “take over” every open
source project they depend on and build “open source versions of commercial
software?”

~~~
toomuchtodo
I feel like I’ve made my thesis clear, and I’m not willing to argue it
further. My apologies.

------
sonofblah
This article is irresponsibly written, because the two-year term is buried in
its middle.

The headline reads "will pay consultant $176,800," which will be parsed as
"per year" by, what, 99.9% of readers?

The first sentence is no better: "pay $176,800 to a computer software
consultant."

As others have mentioned, this is almost certainly a bargain--I can't imagine
any new system being quoted without a price of 'millions.'

------
dougmwne
Clickbait title. The article itself seems mostly positive, rational, and
informative but the poor citizens of Baltimore are getting squeezed for ad
dollars instead of being kept properly informed about local events. This is a
nonstory and the journalist knows it. Baltimore PD is getting a great deal
supporting their legacy IT while they convert to a modern system that will no
doubt cost many times more to purchase, integrate and maintain. At just $44/hr
the consultant is being underpaid for the area if they're full time 1099.

~~~
walshemj
That's only just over 20% premium to the fulltime UK salary I could get in
London which pays a lot less for developers than the USA does.

A average contractor in London would be making about $160k

~~~
mywittyname
The contract is for two years. So it's actually a 10% discount.

------
disillusioned
Well, in three years when the headline is "Baltimore police pay $240M for
failed Lotus Notes migration," then I'll click. But sometimes maintaining the
legacy system is the lesser of evils, especially in government systems.

We built a custom management system for a state agency here 10 years ago. The
state still pays us around $40k/year in associated improvements, changes based
on new legislation, reports, etc. Far less than paying $1M-$2M to rebuild the
thing, or more. Total cost of ownership for the system is ridiculously low,
all things considered. And yeah, it's written on some old crufty PHP, but it
still does the job and they use it every day.

~~~
rhizome
This thread is hilarious in how many people apparently have blinkers on about
the "find a need and fill it" advice that populates every BizDev-oriented
thread here. It makes me wonder how many have _any_ experience in software
maintenance.

~~~
icebraining
What comments are you taking about?

~~~
rhizome
The ones implying, "gawd, why does this kind of business even exist?"

------
denysonique
Trying to access the site from UK:

 _Unfortunately, our website is currently unavailable in most European
countries. We are engaged on the issue and committed to looking at options
that support our full range of digital offerings to the EU market. We continue
to identify technical compliance solutions that will provide all readers with
our award-winning journalism._

~~~
copperx
I could get in with no problems from the EU.

[http://www.cnciweb.com/index.html](http://www.cnciweb.com/index.html)

~~~
Max_Mustermann
They are talking about the Baltimore Sun website.

------
ChuckMcM
One of the things I do not miss in the slightest from my time working at IBM
was their use of Lotus Notes databases all through the processes they used in
finance. In hindsight it is a text book example of a tool that gives the
installation great freedom in customizing and tailoring the system to their
needs, but provides very little in terms of configuration management or
evolved system visibility.

What really surprises me though is that IBM doesn't have a tool for extracting
the resulting system out of Lotus notes and putting into something else. They
could have used it internally to modernize their finance systems, and they
could sell such a tool to the consultants who would bid on modernizing the
police department's systems.

I think the complaining about the $176K as an expense is silly (its pretty
cheap) the complaint that they aren't investing in modernization is more
critical.

In terms of ways to change the world with software, these sorts of business
process systems are a place where there is always room for improvement it
seems.

~~~
slantyyz
At some point, didn't IBM move Domino to a DB2 back end? I would think moving
data off of an RDBMS like DB2 would be easier than the legacy Domino engine.

I remember there being an Notes-ODBC driver, but I don't remember if it was
any good.

~~~
v-erne
Three was a brief moment when this was a thing. But it really did not catch
on. They deprecated this feature few years ago (by deprecate I mean the "IBM
deprecate" which is ignoring feature in future releases so maybe people will
forget about it).

Data is not the problem here - it always was the hakish way most notes app was
created. How can any consultant / it dep. keep up when there are hundrets of
apps done without any prior design by "Citizen" developers.

------
cl0ckt0wer
Did anyone click the link to gaze on the vendor's beautiful website?

[http://www.cnciweb.com/index.html](http://www.cnciweb.com/index.html)

~~~
bdcravens
Not the prettiest, but nice and fast, and no Javascript. Nice to know I don't
need the browser to process an entire framework just to load an h1 tag.

~~~
acdha
Yeah, 0.3 seconds to render is less time than it usually takes just to load
and parse React:

[http://www.webpagetest.org/video/compare.php?tests=180809_58...](http://www.webpagetest.org/video/compare.php?tests=180809_58_c87086ddc494c13692d08ae04a79ac1a-r:1-c:0)

------
scriber
Unfortunately, our website is currently unavailable in most European
countries. We are engaged on the issue and committed to looking at options
that support our full range of digital offerings to the EU market. We continue
to identify technical compliance solutions that will provide all readers with
our award-winning journalism.

------
CodesInChaos
[http://archive.is/QZ3q9](http://archive.is/QZ3q9)

~~~
raesene9
ta Codes

------
PaulHoule
As old as Lotus Notes is, I don't think anything quite like it has come on the
market since.

~~~
mr_toad
Most organisations seem to stumble around using a combination of unstructured
Word documents, file systems and email.

When they find themselves totally at a loss they’ll try and shove the whole
mess into a document management system, but a combination of unstructured
documents and organisational inertia means it usually does more harm than
good.

There’s SharePoint, but that’s best avoided.

Some places, with a bit more technical skill use wikis, and issue tracking
systems like Jira, but even they aren’t quite what Notes was in its heyday.

I sometimes wonder why there isn’t a compelling successor to Notes built on
CouchDB.

~~~
chid
I wonder why you say that SharePoint should be avoided. I get that for small
businesses it's a lot of administration to worry about (and ties them to using
Microsoft software) - however they can use O365. What features would make a
compelling successor that SharePoint doesn't (and what are even alternatives?)

------
ekovarski
""" The consultant is charged with system upkeep until the department finishes
crafting “a 21st century holistic technology platform,” he said. The final
technology plan will be delivered to the court on Dec.1."""

crafting? holistic? platform? 21st century?

Hmm, me thinks this "platform" is doomed from the onset if you plan on
crafting a holistic platform plan by Dec 1, 2018 after still running a system
crafted 22yrs ago.

~~~
redisman
This "holistic" platform is the one that'll end up costing 20M when the Lotus
Notes thing would've worked fine

~~~
ekovarski
Holistic at 20m?

Canada tried to bring to the table a holistic payroll system, known as
phoenix, which cost them 1b before they decided to scrap it. :)

~~~
NikolaNovak
FWIW, there's nothing "holistic" about Phoenix - it is centralized payroll
(same as Regional Pay System it replaces), but the HR is completely
distributed amongst departments (running PeopleSoft, SAP, legacy or bespoke
apps of various natures and kinds), and Pension, central index, etc are
various separate monolithic applications as well. A very large percentage of
problems can in principle be traced to the fact it is the very opposite of
holistic system (note, I am in no way claiming that consolidating disparate HR
systems into one, enforcing data cleanliness, and creating a "holistic" system
could be done before heat death of the universe, for sum of money less than
totality of our GDP, and resulting in anything else than chaos, confusion and
disorder... just clarifying Phoenix was emphatically not an attempt to create
such a "holistic" system :)

~~~
ekovarski
LOL - It was more tongue in cheek but you are right, Phoenix, wasn't an
attempt to create a holistic system.

Usually when I hear projects being spun as "holistic" it means to me that
folks haven't done any basic groundwork yet on the requirements and its simply
the pie in the sky approach to apps, we want it all in <12mos for half the
cost.

------
alexc05
I was thinking that it sounded a bit high but OK for a 1 year contract, but
when I read the article turns out it's over 2 years.

At $88k per year that's totally fair.

~~~
steego
> I was thinking that it sounded a bit high but OK for a 1 year contract

If you think that's expensive, you really have no idea what other
organizations spend on IT projects. Do you have any idea how much the NYPD
spends in IT? I once interviewed a Dev Team Leader who worked at the NYPD.
Based on the projects he worked on, I can easily say they spend magnitudes
more than Baltimore PD, but they also have a much wider range of systems.

------
Ftuuky
BNP Paribas still used Lotus Notes in some departments. And every year it pays
a lot of money to COBOL consultants as well.

~~~
nwsm
I work at a company where one group in IT makes AngularJS apps, another writes
industry machine learning models, and a few others are using Lotus Notes.

People will use what works

------
wallflower
OT - Someone smart once said that Lotus Notes was an email/groupware app
wrapped around one heck of a robust sync engine.

As an even more OT comment, I always loved the password entry box for Lotus
Notes. When you typed in your password, it would do a simple animation of
cycling through hieroglyphic symbols.

~~~
justboxing
What's OT?

[ Please don't dismissively downvote. Google search for OT yields a wide
variety of expansions, none of which seem to explain this abbreviation.
[https://www.google.com/search?q=OT](https://www.google.com/search?q=OT) ]

~~~
openasocket
It stands for "off topic"

------
russtrpkovski
Helluva deal for the Baltimore police!

------
theshadowknows
Notes and Domino are actually fairly capable from my understanding. We used it
until very recently to support hundreds of retail locations, thousands of in
house applications, and corporate mail services. IT has been slow long
replacing it over a couple of years but it was working well. The email and
messaging part is fine, it’s really the calendaring part that was giving us
the most grief.

------
MithrilTuxedo
Given the number of FOSS hospital management systems out there, are there no
similar attempts to create FOSS software for the common government problem of
law enforcement? Some of our Federal agencies might be able to help with this,
saving state and local agencies buckets of money over their various bespoke
solutions.

~~~
nwsm
If you go with name brand software, you can find someone to support it 20
years later.

I doubt the same is true with most FOSS

------
saagarjha
It's kind of sad that so much information is stuck in proprietary systems that
need to be continually maintained, even if the original vendor of the software
has left. With something open usually there'd be a quick and easy way to
migrate all the data off of the platform to something newer when necessary.

~~~
fencepost
Are you under the impression that rebuilding the whole thing on any of the
popular development options currently available would result in lower
maintenance costs? Or a more open system?

Let's say they rebuild it with a Ruby backend (or maybe Go? That's trendy
these days), a bunch of React (or vue, or maybe whatever's current for
Angular), Postgres (or MariaDB, maybe Oracle's MySQL, maybe even Mongo), maybe
throw it on some Linux boxes with Apache. In 20 years that's all going to
sound like "We built it on Notes and now pay $7500/month in maintenance and
update costs" along with everything that'll happen in between when ECMAScript
10 deprecates a bunch of language features.

Could it be better? Absolutely. Should the current system be kept and
extended? Heck no. Is data migration a significant issue? Only in that it's a
very complex system with a huge range of different types of structured data.
They have programmatic access to the raw data, dumping it out is logistics.

~~~
saagarjha
I meant open as in not proprietary. It's a lot easier to migrate an SQL
database than a Lotus one, because one is open and people have written code to
allow data to be exported.

~~~
v-erne
Actually its not true - notes (or rather domino) data metamodel is really
simple (except for rich text maybe). The real problem is lack of schema which
ecourages people who dont know better to build horrendous things. And if there
is no schema and no documentation then is it really notes fault that migration
is sisiphus work?

------
EADGBE
The same sort of reasoning behind my Java web apps interfacing with a database
system that doesn't understand booleans; and needing to interface through RPG
programs in order to actually get any business logic done.

As a web developer, it makes me scream sometimes. I get it (somewhat), but I
still want to scream.

------
throwaway_bmore
[https://arstechnica.com/information-
technology/2018/06/balti...](https://arstechnica.com/information-
technology/2018/06/baltimore-police-study-mandated-by-feds-finds-massive-tech-
fails/)

Some context

------
j45
I suspect no one 8nderstands the cost of building and maintain gig software
past 10 years.

Some problems have been around longer than most developers who want to rebuild
every 3 years.

Lotus Notes, Lotus Domino were among the original mobile, offline first
business apps.

------
scarface74
I have a “it’s not going to help my resume” premium for any technology that
doesn’t help me get my next job. I also have a slight “I’m going to learn a
technology that will help keep me marketable” discount.

------
chrisdhoover
It could be worse. 10 years ago the California judiciary spent over $2 billion
for a case management system that never worked.

------
bcheung
Am I missing something? Seems like a fairly mundane and common occurrence of
hiring a tech worker. How is this newsworthy?

------
qrbLPHiKpiux
I think a Wire rewatch binge is in order...

------
LarryDarrell
I can't wait to see what the ancient systems that need support in 30 years
will be.

------
manav
Isn't it still alive as IBM Notes?

~~~
v-erne
Yeah it is. Some india based IT Sweat Schop (HCL) bought it and they are going
to add node.js integration to it to make it look modern :)

------
eerikkivistik
If anyone could provide a summary, seems GDPR has really made it hard for some
outlets to render text (editors note: this was meant as a light-hearted joke,
not ignorance)

    
    
      Unfortunately, our website is currently unavailable in most European countries. We are engaged on the issue and committed to looking at options that support our full range of digital offerings to the EU market. We continue to identify technical compliance solutions that will provide all readers with our award-winning journalism.

~~~
oh_sigh
Or they didn't think it worth it to rewrite their app to selectively allow
enabling or disabling of features based on geographic region

~~~
briandear
Are people in those other regions providing revenue? If not, then why should
they make their content available in those places?

~~~
genericid
So their content is available in those places.

------
lightbyte
How does one go about getting contracts like these with the government to do
trivial tasks for exorbitant amounts of money?

~~~
jonknee
What trivial task? Maintaining a very large legacy system for something as
important as a police force sounds like a very difficult task. All this for
the price of one cop sounds like a bargain.

~~~
bena
I wouldn't say the task is "very difficult" either. The task is probably no
more or less difficult than maintaining the inventory system for a store.

The cost and job security comes from the fact that you have to do it in Lotus
Notes.

~~~
pavel_lishin
> _I wouldn 't say the task is "very difficult" either._

I wouldn't say _anything_ , given that we know very, very little about the
system in question.

~~~
bena
I mean, any difficulties would be accidental difficulties, not essential ones.
I mean, according to the article, the system already has consistency and
integrity issues. (And no one is dying because of them either.)

The point is that the job is essentially data management.

~~~
steego
> The point is that the job is essentially data management.

You might be right, but you could so easily be wrong. You really have no idea
what the job entails. The software might be great, it might be a nightmare to
maintain/operate.

What I do know is $88K/year for any consultant typically doesn't get you much.

~~~
bena
It being a nightmare to maintain/operate would be an accidental difficulty,
not a necessary one.

The guy I responded to implied that the fact that it is for the police
department made it more difficult. My position is that that has no bearing on
the difficulty of the task.

~~~
steego
> It being a nightmare to maintain/operate would be an accidental difficulty,
> not a necessary one.

Who's saying it would be a necessary one? If you inherit an awful system
that's time consuming to maintain and operate, you have a few options. You can
simply maintain it and bill for that, or you can improve the system. Sometimes
it's easy to improve the system, sometimes it's not. It all depends on the
circumstances.

I'm sorry. What point were you trying to make?

> The guy I responded to implied that the fact that it is for the police
> department made it more difficult.

Do you think that's an unreasonable assumption?

Forget whether the systems are actually mission critical or not. The only
thing that actually matters when it comes to billing a customer are the terms
of the service agreement. I've worked on government contracts where my company
was significantly fined if the system did not operate at some minimal
operational threshold. The system was definitely not mission critical, but we
built it like it was and billed them for it. (We also got a bonus for getting
the job done early)

I don't care if he's only working on the parking ticket system. If it's clear
from the service agreement that it's going to require a significant portion of
his time/energy to service it, the last thing you want to do as a consultant
is to underestimate the effort.

I have no idea if this guy is taking advantage of the Baltimore PD or not.
Neither do you. What I do know is $176K for two years is not a lot for an
organization the size of the Baltimore PD.

> My position is that that has no bearing on the difficulty of the task.

I agree. The terms of the service agreement combined with
complexity/maintainability of the system determine how much effort is
involved. Again, life/death doesn't dictate the price. The service agreement
and expected level of effort do. However, knowing a project is for an
organization like the Baltimore PD, it's not crazy to assume they might be
more demanding than other customers. State and local governments can be a real
pain the ass sometimes.

My position is you're pretending like you know this guy's business. You don't.

~~~
bena
> I have no idea if this guy is taking advantage of the Baltimore PD or not.
> Neither do you.

Never mentioned this. Never said boo one way or another.

Read what I actually wrote and not what you wanted me to write.

I said nothing about whether the cost is justified or not. I said the cost
comes from getting someone who can program in Lotus Notes. That's all I said
about it.

It is an unreasonable assumption to think that who you're building for makes
the actual task more difficult.

> I agree.

Then what's up? You're agreeing with the only thing I'm actually saying. You
had one guy saying it was trivial and another saying it was very difficult
_because_ it was for the police. I was saying it was neither. That it was
equivalent in difficulty to another data management task. If anything people
are downplaying the logistics involved in modern retail. Data management is a
serious business.

> My position is you're pretending like you know this guy's business. You
> don't.

And his business isn't dealing with the police department. Because the guy I
responded to isn't the guy in the article. So he doesn't know the business
either. You are trying to chastise me for making an assumption when you assume
I don't have experience, but no one else has claimed any experience in this
matter either so their claims are just as invalid as mine according to your
standard. They are also pretending to know the business. And so are you.

And by the way accidental and necessary difficulties are covered by Fred
Brooks's essay, "No Silver Bullet". Accidental difficulties are difficulties
we created. By choice of platform, technology, methodology, etc. So the system
being a labyrinthine mess of Lotus Notes code is definitely an accidental
difficulty. A necessary difficulty is something about the task itself that's
inherently wicked.

