
Why we need to create careers for research software engineers - rkda
http://www.scientific-computing.com/news/news_story.php?news_id=2737
======
magicbuzz
Working in research as a software engineer can be very frustrating. I wrote
the majority of the analysis code for one of the chapters of the IPCC 5th
Assessment and yet was told my name couldn't be put anywhere as I wasn't a
'scientist'. I also consistently see people who have never written much code
put in charge of projects where the majority of the work/research is writing
software. These projects end up with no code repository use, poor quality
control, zero testing, limited OO use - all the things that a professional
would take for granted. Worse, they don't perceive the difference between a
postdoc who can write code and a professional developer with years of
experience. So they hire the cheaper option and end up with the usual 'ball of
mud'.

~~~
tensor
Some would argue that no OO use is a good thing (assuming you are referring to
object oriented programming).

Expanding on this, writing research code is quite a bit different than writing
production code. In particular, requirements can change quickly and
drastically. This means that very open designs where you don't hide data (e.g.
as is common in OO programming) are often the best designs.

~~~
Retric
If your research code is a big ball of mud then all it's results are
meaningless. And all your research should be rejected by any competent
journal. Sadly, journals tend to ignore such huge issues.

But, hey it's not like reproduceable results are important in science or
anything.

~~~
mbreese
The more computational journals are keeping track of this. I've had a
manuscript returned for revision because I didn't have enough unit testing in
the code. Wet-lab journals, on the other hand, probably let a lot of stuff go
on the computation and statistics than they should.

------
chaosphere2112
This is pretty much my job. I develop climate analysis software at a DOE
national lab. We have a pretty reasonable career ladder for the "research
software engineer". A quick sample:

Me: Recent graduate (2012), early career. All dev, all the time.

Lead Dev: ~15-20 years experience, minor project management tasks, mostly just
dev work. Gets author credit on scientists' papers every once in a while for
his help in generating visualizations.

Principal Investigator: 30-40 years experience, created the software package
that we develop in the early 90s, almost entirely project management.

Our organization doesn't expect publications from us (though they're a perk at
raise time). There are a few conferences that we present at annually, some
posters to make, but for the most part, I just write software that helps
scientists do their jobs better. I'm just down the hall from some of the
leading minds in climate science, and it's really a great environment.

Of course, our lab varies wildly internally; any different project could have
a drastically different experience, and different departments would have very
different expectations.

~~~
jwoah12
There are no promotions between entry-level and lead dev? I don't know that
I'd call it a reasonable career ladder if the second rung is 15 years away.

~~~
dagw
Are regular promotions that important? As long as I get keep getting to do
interesting work and keep getting reasonable raises I'm quite happy with my
job title not changing more than once a decade.

~~~
jwoah12
I suppose not. I'm just used to compensation and responsibilities being tied
closely to promotions. There just needs to be some way that the organization
regularly recognizes people for doing a good job, so however that materializes
is fine.

------
ska
There is a chicken and egg problem here, also.

Many labs don't have any really competent software people in place, which
means they don't have effective mentoring and training of new people as they
join, which means they don't have a solid mechanism for generating competence
in this area (unlike the focus area of their research).

A lot of research code is writing in fits and starts by a revolving door of
post docs and graduate students, who don't understand software development,
supervised by people who also don't understand software development. The
results are really hit and miss, dependent mostly on individuals desire to
learn these things outside of their area, or on the luck of hiring the right
student or post doc who already happens to know quite a bit (but this probably
isn't why you hired them). Some labs know they have this problem but have a
structural issue with funding - they can't possibly come up with a competitive
salary. Others don't even realize it's a problem.

You end up with a lot of people coming out of these labs with deep domain
knowledge but only rudimentary software engineering skill. From an industry
point of view it becomes hard to hire many of them because they will need the
same amount of supervision and mentoring as an entry level programmer for the
first year or two, but if you're running an R&D group you often don't have
enough mentors to go around... so there is a shortage of people with 3-5 years
of industry experience and an oversupply of those with 0.

------
tfgg
As a recently departed academic, to vastly generalize, the only people who are
research software engineers are perennial postdocs who don't realise yet that
their career is a dead end and they're being significantly underpaid for their
skill level. They also won't develop their skills adequately in such an
environment, lacking good standards and mentorship.

Academia needs to stop taking advantage and accept that it should pay double,
so they can actually hire inwards from the private sector and gain good
knowledge and skills, or somehow increase the prestige and power to be
equivalent to other PI positions. I don't the latter is attainable [1], and
I'm sure plenty of people would be happy to just be paid well rather than
leave for data science/finance/building web apps.

[1] E.g. If you're in physics, and not doing physics, you're fighting a losing
battle to get respect and that lectureship/professorship. Everyone doing
coding as their main job has to pretend to do research as well.

------
rifung
While I do find it unfortunate that software developers in research are
underappreciated, I wonder if making an official title for this will really
solve the problem. One issue is that if all you are doing is writing code, you
could do the same thing in industry, and there's really no way academia is
going to match that kind of salary.

I don't really see the issue with how it appears to work right now though? My
understanding is currently if your research requires you to program, then you
program it yourself, or collaborate with others. Thus, being a good coder
actually helps some people publish more. At least this is what I saw with
people in the more practical research topics like systems research, but even
in theoretical research.

On the other hand, I suppose this is a problem for people who are not in CS.
However, I imagine these people just hire programmers to do the work in the
same way they might hire lab assistants. In any case, I'm not really convinced
just putting a new title or career track will help.. how is this career track
any different from just a normal software development track?

It seems more like the real issue is that you are hiring a postdoc when what
you really want is just a software developer, but perhaps they only want to
pay post doc salaries?

~~~
emcq
I actually have "research software engineer" on my resume and you're right, it
doesn't do much. In fact I don't think this is uncommon, at least in the US.
Myself and former colleagues indeed have no trouble working in industry
positions.

However one thing to note is that many cases, research programming has
specialized requirements that are rare for most industry positions and domain
scientists. For example, numerical analysis to ensure stability of models,
distributed computing to scale massive simulations or analytics, algorthims
and visualizations to explore datasets, NLP, ML, etc. Some of these skills
domain scientists develop but for the most part these are likely going to
require someone with a CS background.

------
ChuckMcM
summary: if you're a researcher and spend all your time writing software
rather than publishing your research career suffers.

I see three solutions:

1) Have the universities band to together to fund a software source for
research. Something that would hire software developers who would provide
tools for researchers who were with member universities. Pay a market salary
for those programmers and release everything under a sharing friendly license,
both for recognition and to aid in reproducing results in other labs.

2) Mandate a single source for all software acquisition through some software
consulting company and add riders to all research grants that provide some
number of "software supplies" for the team.

3) Get rid of the whole publish or perish paradigm and restructure scientific
research so that everyone is accorded rank based on the novelty and
repeatability of the research done in the lab.

------
stared
I think the problem is not the lack of "career path" or a title, but that the
is little credit given to code. (To the point, that usually no public code is
being shared, even if the publication bases on it!) Why?

\- sharing code may mean risk of being scooped (instead of milking the same
code for years),

\- cleaning up code takes time, which can be used for writing another
publications.

In short, as long as publications (i.e. papers accepted to prestigious
journals or conferences) are the only credit in academia, quality programming
is lost.

See my longer answer:
[http://academia.stackexchange.com/a/23238/49](http://academia.stackexchange.com/a/23238/49)

------
pepijndevos
I started studying electrical engineering after being a self-taught software
developer for a few years. I'm basically writing all the software for the
project groups I'm in, and I love it. I like this kind of code much better
than your web app. (Though Matlab is a bitch, no namespaces, array indexes
start from 1, etc.) I hope that I can make a living researching and building
electronics and software.

~~~
kctess5
I've worked both in industry and at a research institution (albeit not very
long) and I've learned a bit of a love hate relationship with matlab. It's
really awesome for prototyping complex math-based algorithms as an
intermediate step before writing a battle-ready implementation. There's a ton
of libraries so you can think in terms of high level concepts, and the matrix
math is very natural, as is Gpu integration which is a huge plus. Once it's in
code, you can iterate quickly relying on libraries, and even create a gui very
quickly. When you settle on a final implementation, you can do a 1:1 port to
the optimal runtime to make the code more modular, extensible, and fast.

Once you get used to the indexing and quirks, it's a prototyping dream. In a
day I can do what would take weeks to implement in C++ or even Python, and I
can decide if the algorithm is good enough to warrant a full port. Once I make
that call, it's trivial to decide which parts of the algorithm I should work
on optimizing for speed during the port because you can analyze perf of each
step. Granted, I do a lot of image processing which is sort of an ideal matlab
use case.

I totally agree though, writing software to solve real world problems is much
more riveting than writing endless web apps (I've done both.)

------
p4wnc6
I've interviewed for a number of these types of jobs in the last year or so.
The pay is just too poor to even think about it. One large university with a
super well-funded and growing medical school in a huge mega-city was looking
to hire someone with 5+ years of deep experience in both Python and C/C++, as
well as some familiarity or past education in statistics and machine learning,
to be the software development lead for a set of research groups. They weren't
willing to go above $90k/year.

In another case, I interviewed with a school of public health at an Ivy
university. It was in a medium-to-large city on the east coast, where the cost
of living was certainly higher than national average, but not insanely high.
Even still, they were looking to pay in the range of $75k for someone with
significant years of experience in the scientific Python stack.

Not only does this fail to compare even with start-ups that aren't located in
major tech hubs, but also when compared against salaries offered by
established companies in the same cities, it's almost a 2x pay jump (more in
some cases), for the same skill set.

To boot, these lab/university environments tend to have many of the same
political issues that any other organization has. Sometimes they are
_slightly_ more generous with vacation time and work/life balance, but not
always. But you are generally also working with a lot of research staff who
see "programming" as a nuisance that gets in the way of their research, rather
than seeing programming as the medium of expression of their research. So when
you lobby for using good tools, good workflow practices, and try to teach
people how to write reasonable software, you usually get a lot of pushback
that people just don't like doing it that way, and since you're the super low-
status programmer, your opinion generally doesn't count for much, even if you
can back it up with more substance.

You also carry some employment risk when projects are funded by grants. Even
large, well-established, multi-year grants can be suddenly changed or reduced
when there is significant political change in the funding organizations. And a
lot of the universities, at least, are unwilling to hire these kinds of
expensive programmers as permanent university employees.

You'd probably have a much better career if you instead went to work as a
programming-literate business analyst in some part of the university's
fundraising / business departments.

In the end, these places can only afford developers with very little
experience, or people who don't understand the cost of living differential in
the city they are moving to, or I guess are independently wealthy and just
prefer to be a low-status programmer in one of these labs.

Edit: Another thing that struck me -- in a lot of these university research
situations, the programming talent that they need to hire is genuinely just
harder to find / harder to replace than even the research talent you would be
working for.

Since there's a huge surplus of Ph.D. and post-doc labor, you could staff the
domain expert research side of things much more easily than the programmer
side -- especially if you need someone who is truly full-stack to do things
ranging from migrating legacy research data to a cloud platform all the way up
to designing fancy UI demos with d3.

If you look at another set of industries where the lower-level contributors
are much harder to employ than the manager -- professional team sports -- you
see that the market dictates that the lower level experts (the actual players
on a sports team) almost always command more salary than the coach. There can
be exceptions, like Phil Jackson (but even he didn't earn more than his best
players) and there will be backup players who earn less. But the general idea
seems to hold true.

This means that a lot of these scientific labs and universities will
eventually have to face the market reality that the domain expert labor is
just not worth as much as the software design labor. Or else they can just
keep on compromising on the quality of the software and try to hold on to a
worldview where scraping by with bottom-of-the-barrel software development is
good enough. It might work for them for a while longer, but I doubt it will
work for too long.

~~~
geebee
This is a really good analysis. I think that as it stands, the best hope for
filling these jobs is to get a highly talented postdoc who will accept this
role in exchange for assistance with his or her academic career aspirations
(another way to do it is more insidious, which is where a PI stretches out a
doctoral student's completion time to hang on to his or her talented and cheap
programmer a few more years).

Of course, this means getting authorship (doesn't have to be first, but it
needs to be there), recommendation letters, and so forth. It is also a
temporary situation, since what is happening here is that the postdoc is doing
software development for well below market rate in order to get a boost in
climbing an academic career ladder.

Unfortunately, this scenario makes it even less appealing for a software
developer who isn't part of the academic ladder. Because they aren't getting
any special boost, career-wise, it just becomes a demanding job with long
hours and low pay.

The setup is virtually guaranteed to result in very low quality software. Post
docs may be brilliant people, but it's unlikely they will learn to write high
quality, maintainable software in a transitional period between grad school
and a faculty position, especially since it's something they'd like to leave
behind as quickly as possible. And it's equally unlikely that someone talented
at software development will stick around in an organization like this where
they will never get the chance to lead a project and make a decent salary,
especially when those opportunities exist outside universities.

The idea of a research programmer role is a really good one but I'm not sure I
can really see faculty accepting it. It would be a major cultural transition
for them.

------
sien
Ironically we're all discussing this on a platform created by a research
software engineer 25 years ago.

~~~
johansch
Please expand?

~~~
shanemhansen
I assume this is a reference to the fact the www was created at CERN. http was
written to help scientists share documents.

~~~
johansch
Exactly, it wasn't a research project. An engineer needed to solve a problem.
He happened to work in a research institution. Had he been a researcher the
web wouldn't have happened.

------
mrdrozdov
So universities should hire Software Engineering professionals and have them
build tools for the research ecosystem? They can definitely afford it.

------
Animats
They mean "programmers who support research projects in other fields", like
lab techs. Not people doing research in software.

------
jkot
I think I have this sort of career.

I take theoretical paper and turn it into practical implementation. I make
money on consulting. Lot of independence and freedom.

I dont think it is possible to do such thing in academia. Software guys I know
in astronomy are underfunded and on short contracts. Most guys will leave for
commercial company before starting a family.

~~~
chubot
Can you give some details on what you do? I think it's pretty interesting (and
rare) that you can make consulting money by taking a paper and turning it into
an implementation. What field are you in, who do you work with, etc.? Are you
independent?

I'm doing the same sort of thing at Google. I've been a software engineer for
13 years or so and I switched over to a research group in 2014. There is
definitely a culture difference, even though Google is one of the places which
has the least "pure" research. All research teams have software engineers
pretty much, and they mostly work with the same codebase that engineers do.

~~~
jkot
I work on database software. Right now I am working on queues. I am
independent. It started as hobby opensource project, people asked me for help
and I started charging them. Now it will be 3 years since I started working
full time on this.

------
saintgimp
Dr. Pamela Gay wrote an article about the difficult state of non-tenure jobs
in research and academia: [http://www.starstryder.com/2015/11/12/dreaming-of-
success-in...](http://www.starstryder.com/2015/11/12/dreaming-of-success-in-
an-age-of-diminished-possibility/)

It's the same problem for software devs, postdocs, research assistants,
adjunct professors - basically everyone except tenured university professors.
We're not rewarding the right things.

~~~
testrun
You are missing the career bureaucrats in universities. They have won. The
vice chancellor at our university get more than AUD 600 000. Not bad for a
small university in the middle of nowhere.

------
johansch
I think this is done much better by e.g. Google already today. Academia has
have their 50 year run with software research and the end result is quite
disappointing.

I have been peripherically involved with some recent academic research
projects in Europe. The way I see it it's basically a way of keeping senior
academics occupied and also diverting tax revenues to EU companies for no good
reason. It's also a breeding ground for corruption - who chooses which
companies get these pointless deals?

~~~
thescriptkiddie
> Academia has have their 50 year run with software research and the end
> result is quite disappointing.

You might want to tell that to the students and faculty at MIT, Berkeley and
CMU who wrote the papers and much of the software powering your shiny new
macbook and iphone.

~~~
johansch
Most of them were students, not academic researchers . The sole reason they
were there was that it was the only way to get to computers.

Since you mention MIT: I guess one example is how much effort Marvin Minsky's
influence wasted in AI research and how he single-handedly killed more
relevant research in that area for decades.

What I was trying to get at though is what I have seen in the EU. Basically
they don't even attempt to do basic research - most of the money goes to
building product-like software systems, but without any realistic customer
feedback loop (they already got the funding from oblivious tax payers
thousands of miles away!). It's just wanking.

~~~
xamuel
>without any realistic customer feedback loop

This is so so true (I spent some time working as a post-doc in England). There
are tables filled up this very moment with dignified scientists arguing about
bikesheds in CRUD applications that wouldn't get three upvotes here on Hacker
News. And remember: if you can't open it in MS Excel, that makes it "Big
Data"!

All joking aside though, what universities are doing with their postdocs is
abusive and immoral and they're going to regret killing the golden goose when
they've squandered all the prestige in these positions.

------
jack9
I thought I was going to read about the real problem with the lack of research
into software development. Research software engineers is a secondary issue.

~~~
0xdeadbeefbabe
[http://herpolhode.com/rob/utah2000.pdf](http://herpolhode.com/rob/utah2000.pdf)
is about lack of computer systems research, and it touches on the problem you
mention.

------
jeffdavis
Would this new career involve some kind of real scientific credit?

------
camkego
Did they put software plus engineer in the same sentence. Uh oh!!

