

Silo but deadly: Messy IT systems are a neglected aspect of the financial crisis - vgeecc
http://www.economist.com/businessfinance/displaystory.cfm?story_id=15016132

======
SandB0x
I've worked in financial software - this is spot on. It seems to be one of
those unfortunate industries where nobody's interests are aligned with
designing decent software:

Programmers don't use their software, users (often back office admin staff)
don't have a say in the design, and the majority of programmer effort is spent
on firefighting.

Due to the heavy regulation you can't really get in there with a lean, easy to
use product. Even if you could, the data is often stored in impermeable
proprietary mainframe systems and is garbled to the extent that importing and
exporting data is handled by an army of consultants.

There's an enormous fortune awaiting companies who can create intuitive, well
designed software for financial corporations. I, however, left because there
more interesting things I'd rather be doing, and I guess most programmers feel
the same.

~~~
coolnewtoy
One of the obstacles is security. Programmers don't have the rights to even
see their software in production environments. Designers and testers have
access only to small (often not representative) subsets of data, and it's
different subsets than the users see. Moving data from one system to another
isn't easy when you don't have anyone who has enough access to look at the
data in both systems.

------
yummyfajitas
Interestingly, I'm told one reason Goldman survived the crisis and came out on
top is that their IT is excellent [1]. They invented their own dataflow
programming language, which they use in most of their trading systems.

When the crisis hit, various higher ups were able to see their risk profile,
run various scenarios, all in a matter of hours. I.e., the CEO could say "what
if X happens", run the global system, and he would see the combined impact of
X across ALL divisions.

In contrast, Citigroup simply does not have access to that information.

[1] Another reason is that their focus on short term profits meant they were
not holding many bad assets at the wrong time (most mortgage products have
long maturity times).

~~~
bwhite
Goldman does have excellent IT vs other big i-banks. (Well, now they're all
bank holding companies since they wanted (or were compelled to want, depending
on who is telling the story) to gain access to the Fed's discount window.)

But slang, their proprietary language, has a pretty mediocre reputation. Every
strat I know there hates it; anecdotal, I know, but still. It is old -- in use
for at least 15 years as far as I know -- and cripplingly slow both to develop
in and run. From what I hear, where speed matters a fair bit of C++ and even
some Erlang (!) are in production use.

Goldman is doing well because it is full of bright people who are very good at
their jobs. Last spring/summer/fall, however, Goldman had lost vast piles of
money, particularly in their Global Alpha fund. It is true that Goldman had
started to go short on housing as early as late 2007; this is why they didn't
get pummeled as badly as their big-bank cohorts.[1] The nice rebound is due to
a variety of factors, not the least of which is that part about bright people
who are good at their jobs, plus a general market rise, plus govt rent seeking
(you better believe it), etc.

[1] The mortgage-related products whose implosion wiped out the market and
many players are all short-term vehicles. Indeed, the problem was specifically
that mortgages were not treated like old school bonds with a long maturity,
but that the income stream from these mortgages was sliced and diced into a
wide variety of CDOs with a dizzying number of tranches such that they could
be securitized and treated as virtually short-term instruments.

~~~
yummyfajitas
My understanding is that while slang the language sucks (where "sucks"
sometimes means "is not C++" and sometimes is actually meaningful), slang the
dataflow system is fantastic for the organization.

The dataflow system allows higher ups to simulate possible outputs and measure
risks easily. I.e., you tweak the LIBOR, run each desk's slang code to see how
it will affect them. Then you combine the results and see what happens
firmwide.

In contrast, at most other places, you'd need to ask each individual MD to
tweak the LIBOR in whatever spreadsheets or C++ models they are using, write a
report, have your underlings combine the reports into their own spreadsheet,
summarize, and explain to you.

------
xelfer
At the company I worked at a sprinkler system failed on top of 2 racks of
servers about a year ago, they were drenched. We dried them out, insurance was
called, we got paid whatever they were insured for.

Finally, a time to upgrade this mass of beige boxes and randomly built
servers.. or so I thought.

Instead, the insurance money was put towards paying salaries.. the servers
that got flooded are still in production a year later.. and 30 of us were laid
off.

~~~
giardini
I believe that is insurance fraud and is criminally prosecutable. The
insurance company involved, if not the local district attorney,would be
interested in hearing this story.

~~~
anamax
Where's the insurance fraud. Are you saying that they didn't have a loss or
that they didn't use the insurance money appropriately?

If the latter, you're wrong. You're free to use the insurance money any way
that you'd like.

For example, the cost to repair a car back to its "original condition" is
often much higher than its value even though the car is still driveable. In
that case, the insured is free to take the money, buy the car for its current
value, which is significantly less, and pocket the difference.

If your house burns down, you're not obligted to rebuild.

~~~
bhousel
It sounds like the servers weren't actually ruined, so commenter was implying
that there was no actual loss.

What stuck me as odd about the story is how "2 racks" of "beige boxes and
randomly built servers" could come anywhere close to being as valuable as
people's salaries.

I think if people are getting laid off, it would happen one way or another,
regardless of the insurance money..

------
swombat
I worked in banking IT for 4 years, and yes, the systems are awfully messy
there.

However, I think it isn't fair to solely blame the banks for it. If there
weren't mountains of regulations to comply with all over the place, the
systems would be a lot simpler, clearer, and more transparent. The first place
to start, to make banking IT more transparent, would be simplifying and
aligning national and international regulations. Make one set of regulations
for the whole world, and make it clear and simple.

Of course, that might not be a realistic aim. First of all, it would make the
international banking system even more vulnerable to a single loophole in
laws. Secondly, it probably wouldn't be possible to do this in a way that
follows new developments in financial products closely enough. Thirdly, the
simplification itself would result not in all the current systems being
decommissioned (that might happen in 10 or 20 years at the earliest), but in
all the current systems becoming legacy but still active systems while a whole
new spate of systems is built.

Maybe the current compromise is the one that works best, despite its flaws.

~~~
dhyasama
While I agree banking and other financial regulations are onerous (and usually
tacked on in reaction to a disaster rather than before to help prevent it), I
don't agree with transferring blame because of it. If you chose to engage in a
system, financial or otherwise, than you tacitly accept it's rules and
regulations. Creating and maintaining solid systems around complex
requirements is where the real skill comes in.

------
synnik
I did work in banking for a few years. The article is accurate, especially
about the detective work involved in tracking down data problems.

But there is yet another underlying problem behind the IT systems. The
leadership of the banks I've worked with considered IT just a "cost center".
They recognized the problems with the systems, but rather than investing in
improvements, they threw blame towards IT, approving budgets only grudgingly.

The leaders in the industry need to start to respect technology and the people
behind it before anything will change.

~~~
stcredzero
Happenings at a multinational petroleum firm:

One process writes something to a database table. Another periodically queries
it out of a table. This results in an email with a text attachment being
generated. The text attachment is then retrieved from the inbox and FTP'd to
another server, where a polling process retrieves it, does some processing,
and ends up saving something into a database table.

I am not making this up, one distributed system was really put together like
this. What's more, this Rube Goldberg monstrosity broke all the time, and when
it got fixed, the techie doing the fix was lauded, since he was ensuring this
link, which was _vitally important to the business,_ continued to work.

------
chubbard
So I found this article very much with the way I see banks and financial
systems. Being that I never worked on financial IT system this was just my
hunch. But, hearing accounts of what was going on it sounded like people were
making decisions on very little data, or that the data wasn't available to
them. When the crisis hit I kept thinking over and over that this could all
have been avoided if things were more transparent. The SEC is at best over
burdened and probably more likely gamed. The oversight we have doesn't work,
and every time we talk about reform bankers moan and complain you'll ruin
everything. I call bullshit.

Given the proper financial data anyone could evaluate the risk of a company.
It's not rocket science. It's all a bunch of electronic data. We need to
rethink regulation. We need to transitions from auditors, committees and the
SEC to financial transparency. We should be able to see every trade, every
position, every loan, CDOs, and derivatives a company owns, trades, etc.
Everything can be tracked and examined because it's just a bunch of electrons.
If this sort of system existed it would be virtually free to conduct an audit,
or monitor for manipulation. It benefits both the CEO and SEC. CEO's have
better insight into what's going on inside their companies, and the SEC can
make sure they aren't manipulating markets or doing risky things at any
moment. Instead of sending out a fleet of auditors, relying on Sarbox, etc.

There's no excuse for bad loans being made. I kept hearing of loan evaluators
being told to pass bad loans just because Wallstreet wanted to securitize
them. That's ridiculous, and if they were doing that we should be able to see
it in plain view. Loan packages can be graded by a computer, the only thing a
human should be doing is checking for fraud at the most. And remember the bang
up job Moody's did rating the pile of junk. Their failings seemed like just
total lack of knowledge about what they were doing.

Which got me thinking why can't we automate the stress test on banks? If we
had such a system then we could take all of the financial data from every
institution into one big simulation. Simulate future financial products and
their impact before we create them. Test out what could happen in the market
if we introduce this. Play out what if scenarios. If we could do that then
would we need an army of auditors, and more overhead the banks don't want.
Light weight bureaucracy not over burdened with check marks, auditors,
committees, and SEC filings.

------
johnohara
I read this article 2 days ago on a flight from CA to Chicago. Sounded
completely plausible.

All I could think was "if you think this is bad, wait until you see what's
under the hood in healthcare management and healthcare reimbursement."

------
ivenkys
Data Problems in Banks are especially acute because the same "thing" is called
by different names in different systems, no one can agree on its definition or
its characteristics.

Added to that, the amount of time it takes for I.T to come up with a solution
- due to the silo mentality - the Front-Office comes up with a patchwork
mostly Excel-based solution "just for now" and well we all know how that goes.

------
riffer
<http://news.ycombinator.com/item?id=978295>

------
CrLf
This whole discussion just looks to me as IT everywhere, not just only on the
financial "industry".

