

Firms Struggle to Find Programmers Who Can Write Parallel Processing Applications - adammichaelc
http://www.wallstreetandtech.com/advancedtrading/showArticle.jhtml?articleID=207801821&cid=RSSfeed_WST_News

======
henning
I don't work in this industry (thankfully), but why work at a megafirm like
Goldman Sachs or even the less elite ones like Morgan Stanley when you could
work for a cool boutique rocket scientist company like D. E. Shaw or Jane
Street? (Jane St uses OCaml very heavily.)

You'll have trouble finding good software people as long as you make them work
deathmarch hours and treat them like glorified typists while acting like
hypermacho traders are gods.

They will have problems as long as they view technology as a depressing cost
instead of an asset.

~~~
jaydub
Great point. Having been an intern at Wall Street firm in technology,
sometimes I got the feeling that that the revenue generating branches are
viewed as the champions of the firm, while the technologists are essentially
(well-paid) stilts that hold up the business.

------
gaius
We'll see a schism form in the developer community. The elite (the people who
were doing Java in 1996, i.e. those who actively pursue language evolution)
will make the transition smoothly to Erlang, Scala, OCaml/F# and similar. In
fact these people have already done so inside their own heads, they're just
waiting for the job market to catch up with them. The masses (the "Java
enterprise solutions architects", who just learn what they need to pass the
test/get a job) will be stuck there churning out the same old same old.

------
1gor
Hm... seems to be an opportunity.

What is the best language to learn to take advantage of this market demand?
Criteria: 1) manages complexity involving multiple processors 2) has utility
libraries (or easy adapters can be written for existing C libraries 3) either
has existing mindshare in the industry or at least is not too esoteric and can
be sold there.

Any other criteria? What languages can we select?

~~~
ComputerGuru
If you don't mind being restricted to mostly-Microsoft platforms, C# is the
race horse you'll want to place your best on. It's multi-threading support is
great (the best I've seen so far w/ regards to the balance between control &
simplicity). The way its delegates (callbacks), thread pool, and invokes work
is quite impressive & powerful.

And, of course, C# can use any C++ library you throw at it w/ P/Invoke & a lot
of marshalling.

~~~
Tamerlin
Actually, you'd be even better served looking at F#, since it has the
functional programming characteristics that make functional programs more
easily parallelizable than languages like Java.

Of course, you can also use things like LINQ and asynchronous workflows in C#
to gain a lot of the benefits... but I'm really coming to like functional
programming. I wish I'd known about it when I was building parallel
applictions, it would have made the software I was developing much easier to
write.

I also wish that I could be doing that again, rather than doing mainly
maintenance work... even though I was recruited to develop parallel
applications.

------
ComputerGuru
Writing parallel-processing applications isn't easy. I run a non-profit tech
R&D organization; and we have internships where under-grad students can try
their hand at coding in "a corporate environment" with joint development,
toolchains, and the whole shebang.

We gauge the students by how much they accomplish in how much/little time and
how elegant their solution is. And at towards the end of the internship, we
invariably ask the best students to add some form of multi-threading or
parallel-processing to the application they're working on (in C++, C#, or Java
mostly) and even our best students have trouble with getting proper parallel-
processing code working (and these guys had done brilliantly elsewhere).

I personally learned it the hard way, scouring the 'net for all the different
methods until I built my own "templates" that work great for me.

------
jaydub
Last semester I took a course on parallel computing and one of the things the
Professor stressed throughout the course was that historically, parallel
machines were built focusing first only on the hardware challenges and
software/programmability was a mere afterthought. As John Hennessy related in
an interview with the ACM: “Many of the early ideas were motivated by
observations of what was easy to implement in the hardware rather than what
was easy to use [vis-a-vis programming].”

We are going to see the situation change when there is a shift towards a more
abstract/simplified approach to parallelism, where the minutia of parallelism
is handled under the hood. I.e a return to the foundations of PRAM theory of
the 70s. Otherwise, I don't see it getting much easier to find that kind of
talent.

------
thorax
This is where companies like Cilk Arts are going to get things going,
hopefully: <http://youtube.com/watch?v=FjGKLYINGsg>

_Edit_ : To provide some more context, they work on tools and extensions for
C++. Things like race detection tools and new language keywords that make it
easier to parallelize algorithms. Based on this work at MIT:
<http://supertech.csail.mit.edu/cilk/>

~~~
mleonhard
Cilk looks like a useful abstraction on top of threading libraries. It will be
quite useful for people who already know how to write proper parallel code.
Concurrency-challenged programmers will need a language that better protects
them from race conditions. I think they'll need a strongly-typed language
combined with something like Cilk.

~~~
ibster
A new company has been spun out of MIT, with the goal of commercializing the
Cilk technology.

multicore programming blog: <http://www.cilk.com/multicore-blog/>

------
astrec
Wall Street will solve this exactly as they've solved every other talent
problem - offer such ungodly amounts of money that every man and his dog wants
to enter the field; skim the cream.

------
Tichy
Are they unable to find programmers capable of doing that, or unable to find
programmers who have "5 years experience writing parallel processing
applications" on their CV?

I suspect the latter.

~~~
pmjordan
Parallel programming has been around for a long time. It just hasn't been
mainstream, and, apparently it still isn't. The people who are really good at
it are on the bleeding edge, and presumably don't fancy working for some bank.

~~~
Tichy
Not saying that nobody could have 5 years experience or more with it. But I
think if you put some faith into your engineers, they can also learn a new
technology. Surely if you hire a decent developer, he or she will be able to
master parallel programming within a reasonable timeframe.

------
iamelgringo
There's going to be a lot of interest in Erlang in the next couple of years.
It is going to take a couple of years for CIO's to realize what a nightmare
shared state concurrency is to engineer. After a while, they will be forced to
look at different languages to solve the problem.

Or, perhaps, they won't see the problem, and startups willing to use languages
with a different concurrency model will come to dominate those markets.

------
nazgulnarsil
soon we'll need a dedicated processor just to handle which data gets sent to
the other processors.

~~~
iamelgringo
It's here. Have you looked at the Cell processor architecture?

