
Ask HN: Where are the hard programming challenges/jobs? - nielmalhotra
I got started programming a couple of years ago because I like problem solving. The harder, the better. I know I'm not alone in feeling that way. I've gotten a bachelor's degree and learned a bit of ruby, rails, and other frameworks/languages, but I'm running into a problem. It seems like all jobs are just simple CRUD apps. I look at 99 percent of the software out there and know that I could do it if I had the time.<p>My question is how do you get to work on the hard/fun stuff? A quintessential example for me would be self driving cars. It's challenging and would have a real impact on the world.<p>I'm looking into becoming a data scientist because it's newer and seems to have more challenges. I also have a theory that any challenging computer science problem requires a lot of math. Do you need a PhD to work on these hard problems? Can anyone give advice on how to avoid a career of working on simple CRUD apps? (CRUD is a metaphor for simple problems in this case)
======
ef4
Your attitude is common, but mistaken. It's another version of "the grass is
always greener". Whether you're working on a self-driving car or implementing
Microsoft Word 2014, the fundamentally hard parts are actually the same.

The self-driving car might sound sexier if you've never done something like
that, but once you've learned the basics of how control loops work and the
quirks of realtime operating systems, that's about it for novelty. What you'll
actually spend your time on is the same old stuff: managing complexity.
Debugging, testing, designing APIs, designing user interfaces, integrating
disparate module, etc. You may actually find that it's more soul-crushing than
a basic CRUD app, because in a safety critical project you'll probably have
less autonomy, more oversight/bureaucracy, and a much slower pace.

The fundamentally hard part of any software project is complexity. It's easy
to let even the simplest project spiral into unmanageable complexity, and it's
deeply challenging work to prevent that. That's the meat of what any good
software engineer does.

There's a better way to word your original question: in what kind of
organization should I be writing software? That has a far bigger impact than
the problem domain on your happiness, level of challenge, and autonomy.

~~~
cconroy
I think there is an exception to this for people that work on programming
languages -- there the best programs written.

In my spare time I have never had more fun programming then writing
interpreters for scheme, prolog, and smalltalk. It would be amazing to get
paid to work on tomorrows languages!

------
petercooper
It's not the only way but a PhD will certainly open a lot more doors at the
"high end" research end of things. That said, even Microsoft Research will
hire "Research Software Development Engineers" who work with researchers to
_implement_ cutting edge stuff as well as migrate it into regular Microsoft
products, and only a bachelor's is required:
[http://research.microsoft.com/en-
us/jobs/fulltime/technical....](http://research.microsoft.com/en-
us/jobs/fulltime/technical.aspx) .. these sort of roles can blossom into more
challenging and more research oriented work over long periods of time.

This is a rather crude generalization, but PhDs are often demanded at high end
research positions not to prove you know your stuff, but to guarantee you have
the mettle to do research and bounce back from failure after failure.. a must
if you're working on someone else's dime. Of course, if you can find another
way to support yourself, you can do (almost) any research you like in your own
time without strings attached ;-)

~~~
nielmalhotra
If I had money and free time, then I'd love to do research on my own. Thing
is, I need a job to pay the bills :( I'm thinking of getting into freelancing
and using that to float me as I get to do more and more research.

~~~
burningion
This is the strategy I'm currently implementing. I've been doing freelance
work while working on research with physiological sensors as computer input.

You'll need two things to get started doing this sort of thing: standing out
as a member of your community so people come to you, and an understanding of
business needs.

If you really want to make the switch, shoot me an email. I'll give you some
tips on making the transition.

~~~
nielmalhotra
So, I can't find your email. I checked your profile and your blog. I can
follow you on twitter or whatever you like, but your advice would be extremely
valuable.

------
alakin
Im channeling Peter Thiel here: there is little connection between hard and
valuable. The goal of most companies is to create value, and do it without
expanding too much effort.

If you really want to challenge yourself give yourself a goal like:

1\. Decode the genome 2\. Find the correlation between all aspects of someones
lifestyle, diet, and their health. 3\. Solve poverty - maybe there is an
economic solution.

The world is full of hard problems, but you have to take the lead if you want
to work on them.

------
irahul
I think employers will be interested in hiring you for such projects only if
you have demonstrated knowledge and experience in the field. If your work
experience primarily consists of RoR web development, most employers will
pass. It's not that it's implied that if you do RoR, you won't know about
algorithms and AI, but there is a high correlation, and sorting through a ton
of RoR resumes to find a worthwhile candidate is too much work.

You can work on hard projects on your own. Once you have made substantial
contributions, you will have credentials to interest the employers. Most of
the open, hard projects have very few full time developers. PyPy has only 3 I
believe. If you have something to contribute, people will be more than happy
to have you on-board.

------
btilly
Follow the advice in [http://steve-yegge.blogspot.com/2006/03/math-for-
programmers...](http://steve-yegge.blogspot.com/2006/03/math-for-
programmers.html) and _learn math_. Once you have the skills to do the
interesting stuff, you start to see opportunities to do it in the oddest of
places. For instance I've been paid to do things like statistics and machine
learning on "simple CRUD apps".

Maybe you won't be that lucky. But if you don't have the required skills, then
you _definitely_ won't be that lucky.

Incidentally a fun place to test the intersection between math and programming
is <http://projecteuler.net/>. Besides, it is fun.

~~~
Evbn
Most machine learning work is the CRUD of math. It is plug-and-chug in
standard toolkits.

------
far33d
Hard problems are everywhere. Every company, every app, every product has
unique and difficult problems to solve.

Sometimes those problems are with product design and UX. Sometimes those
problems are mathematical. If you are successful, often they have to do with
scalability and reliability. If you work with great people there are always
hard problems in enabling productivity and building great tools. Many
companies have serious and significant challenges around security and may not
even know it. If most companies are solving simple problems, then why aren't
most companies automatically successful?

If you broaden your definitions of "hard" you might find that there are many
things that qualify. If what you mean is "hard mathematical problems" then
yes, you will have to learn some math.

------
dumby
Try looking for bioinformatics jobs. It involves big data where the amount of
data to be analyzed is growing exponentially every year, a lack of knowledge
on what to research to provide meaningful results, and huge benefit when
people come up with novel ways to interpret data successfully.

It will involve acquiring some domain specific knowledge to be truly
successful (what hard problem doesn't?), but you will be able to dive in
straight away and make huge contributions just by making current analysis more
efficient!

~~~
paulgb
I'm slightly hijacking the thread here, but does HN have any recommendations
on CS-heavy bioinformatics grad programs (not just US based)?

~~~
Evbn
Threadjacking? This question should be a front page discussion, instead of
gossipy crap.

I would love to know as well.

------
BSousa
Try the games industry!

The pay sucks compared to other development, and work hours are crap, but you
will be able to work on AI, gameplay, Graphics, physics (whatever your
experience is).

But in all honesty, if you started programming a _couple_ years ago, and
learned _a bit_ of rails/ruby, you have a lot to learn in various languages
and systems (try C, embedded, iOS/Android, learn Haskell, big data, etc)

------
31reasons
It sounds like going to a Disney Land and complaining that all rides are easy.
If you want tough and challenging rides go to SixFlags.

If you want to tackle tough challenges may be ruby/rails is not the ideal
place. Perhaps learn C,C++ and implement faster ruby or javascript interpreter
or implement a distributed filesystem in ruby or java. You can also contribute
to the rails framework.

~~~
nielmalhotra
I've learned that ruby/rails is not the place to go for tough challenges.
Agreed. My question now is where do I go from here?

~~~
31reasons
If you like computer graphics, there are also tons of interesting problems in
that field.

------
rikacomet
Lets consider Maths, as an example to answer that. Well, up till college, we
are taught to solve problems that are already solved, towards our PhD, we come
across problems that haven't been solved, they are hard, but they are not the
end of it, beyond that lies a sanctum, where you must find a problem and fix
it. Thats what entrepreneurs do, it may sound easy, but do not be fooled by
it, because being an entrepreneur it seems easy when you have found a problem
and its solution, or as Steve Jobs said " looking back, the dots connect but
20 years ago, looking forward, they did not"

Billions of people around the globe haven't even noticed many a problems, yet
a handful of entrepreneurs did, wasn't that the first part of problem solving,
and if you haven't probably noticed, you are here after all, so you might
unconsciously know the answer lays somewhere around here.

So from my eyes, whether its programming or maths, the hardest problems that
exist are the problems no one has noticed yet.

For your case, you might want to find a problem of programming, if you have
programmed for that many years, you will surely run into one, that no one has
noticed. If you haven't, as Steve Jobs also said "Keep Looking"

------
MojoJolo
I'm with you. I really like solving problems or finding a good algorithm to
solve the problem. What I can say for you is try to find a company with lots
of R&D going. I think there are many companies out there, especially startups
that are having some R&D to improve their product and beat their competition.
I think startups like Prismatic (<http://getprismatic.com>) and Diffbot
(<http://www.diffbot.com>) have a heavy NLP and machine learning research
going in the background to improve their product. I'm interested in joining
those companies I mentioned and contribute to their reaserch and development.

I think game companies also have an R&D to improve their AI, path finding,
rendering algorithms, etc.

That's it. Just find an R&D job.

------
qznc
Game engines are hard. Working on CryEngine, UnrealEngine, idTech7, or
something equivalent sounds very challenging. Games involve AI, physics
simulation, concurrency, high performance, and more. The salaries is not that
great in the game industry though.

------
ExpiredLink
"CRUD" is hard. That's why there are so many bad CRUD applications out there.
Write a good "CRUD" application and I take my hat off to you.

------
rayiner
Defense sector is full of hard problems. A PhD helps, but even with a BS you
can be working on very challenging things. When I worked at a defense startup
we were designing protocols, writing VHDL and drivers for logic implemented in
FPGA's, doing formal analysis of algorithms, doing data visualization, etc.
Different people were doing different things, obviously, but even the junior
folks were tackling problems that required reaching into their CS toolbox.

Its not an industry that has a great reputation, but what it does have is
phenomenal engineers working on complex multidisciplinary problems.

~~~
gnosis
I would like to strongly discourage anyone from working in the so-called
"defense sector", which should be (at least in the US) more properly called
the "war sector".

Working to enable the more efficient murder of ever more people is highly
unethical. Unfortunately, there are plenty of unethical people willing to do
this bloody work. But hopefully you are not one of them.

~~~
rayiner
We're living in the Pax Americana, the most peaceful time in human history,
enabled in no small part by total American military technological superiority.

Not to mention we're posting all this over a network created for the American
war machine, using protocols created by a company that is a key part of the
military industrial complex (BBN), which was founded by people from a
university (MIT) that was then and is now one of the largest non-profit
defense contractors in the country.

------
polskibus
Have you been in a "CRUD" job for at least a year? I can assure you, what
often seems like a "CRUD" will teach you a lot about architecture, design,
requirements analysis, etc. It's actually a good way to teach yourself skills
that cannot be obtain via university schooling. Spend a year mastering a
framework or two (in a paid project), face some support issues, etc. When the
job is boring and you complete your tasks very well, ask for a raise and/or
start looking for a new job.

------
tsilva
1: the "99%" you're looking at are jobs on a specific niche that uses
technology to bounce information from one place to another and display them,
mostly known as IT jobs.

2: Hard problems are everywhere. Literally. Even in those "99%" jobs you
mentioned -- "doing" such software may not be full of mysteries, but make it
evolve without break and respond to constraints are examples of real hard
challenges. It's a matter of seeing them and putting it on perspective.

3: there are many ways of framing a problem, making it hard or easier. The
harder you frame it, the more instruments you will be required to have to
tackle it. Those may involve research skills and fluency in an array of fields
that would help understand the phenomena. It mostly depend on where in the
spectrum of the problem you want to work, and that's up to you to figure it
out.

4: advice to avoid a career working on crud apps: interview the interviewer
and find out if the problems of the hiring company are the ones that you
value.

5: questions like "where to work on hard/fun stuff" are very hard to give
satisfying answers. I suggest you take the time to survey problems you are
interested, try talking to people working on them...many problems are tackled
by teams, often with people with different background and specialties dealing
with different aspects of the problem.

------
__--__
I used to feel this way too. My first job was writing a web app to replace a
paper approval/overview process at an engineering firm. Aside from some email
notifications, it was entirely CRUD and, to cap it off, I was forced to use
classic ASP (in 2005 - asp .net 2.0 had come out by then, making classic ASP 3
generations old).

After 6 months of adding more and more approval flows, I had enough of the
tedium, so I started writing libraries to make classic ASP easier to use,
including a hash table library. Then I focused on how to write CRUD apps
faster, with less code. Then I focused on making it more interactive and wrote
an AJAX library. And if you want real difficulty, try making a classic ASP
rest endpoint running on decade old hardware over a horribly designed MsSQL
database _performant_.

I guess the point of that story is anything that bugs you on a daily basis can
become an interesting and difficult problem to solve. Constraints make simple
things more difficult, but you did say the harder the better. :)

------
dskrvk
If you want to work on hard problems, strive not to have to solve the same
problem twice, to try a new approach or a new tool every time you get a "CRUD"
task (however you define it). You'll find that if you continuously push
yourself, you'll keep growing and will never feel that the task at hand is
boring or too familiar. You'll need to take risks, but without risk there's no
reward. Regard every programming task, however simple, as a chance to learn
something new.

Corollary: if you find that your job doesn't let you learn (e.g. your boss
just wants you to churn out as many web apps in as little time as possible),
leave immediately (or better, try talking to the boss first :) ).

However, as ef4 and others have pointed out, the real question for you might
be "what are the problems I'd want to spend my life on?" Not all important
problems are hard - some may even seem too unsexy to be worth working on, but
the impact can be huge.

------
sytelus
Unfortunately just learning ruby, rails or any other programming languages
doesn't prepare you at all for type of challenges you are talking about. I've
been working on search, machine learning, prediction models, computer vision
and so on and these are absolute cream of the challenging problems out there
that I highly enjoy. Other my favorite areas are compilers, kernels, audio and
image processing but I'm not actively working in it currently. But to even
understand current state of art in these areas, let alone contribute to it,
you will need solid foundation build on math (especially linear algebra,
probability & stats, calculus), machine learning, distributed computing, graph
algorithms, probabilistic data strutures and other CS areas. It takes years to
build this theory foundation and perhaps more if you are not already enrolled
in to any degree programming. Programming is actually the most trivial part of
working in these areas.

Employers who do have these kind of challenging work are never looking for
just Ruby or Java guy who knows all syntax, framework and how to code. They
would need your experience and academic background to convince them that you
do have necessary math and CS foundation to tackle their challenges. The best
way to start out for "plain vanilla" programmers is to get in to open
competitions like at <http://www.kaggle.com/competitions>. Also keep an eye
out for any companies sponsoring such competitions like Netflix did years ago.
The reason this is good way to start is because here would typically be
discussion groups around this where experts discuss their approaches for real
problem and real solutions and you can learn a lot from this. Once you select
such a challenge to work on, give it all to it. Devour every piece of
research, CS and math that comes in your way to understand what's going on. If
you keep at it for 2-3 years, you would be in might be in good position to
demonstrate future employers your work and worth to work on these challenging
problems.

------
javinpaul
try Google, if you are really good and have passion to develop tough stuff
like machine learning languages, dealing with large data. By the way have you
tried any problem given in <http://www.topcoder.com/>, if you like that stuff
and also can do, Google is worth trying.

------
dmbaggett
It depends on what kind of "hard" you're looking for. I've worked on games
(Crash Bandicoot), search (ITA Software), and email (<http://inky.com>) and
all have been hard, though in different ways.

Games are hard in that you're always trying to squeeze more into severely
constrained resources, so you end up having to produce convincing but fake
approximations of algorithms that would be far too expensive to run in their
"real" forms.

Travel search was algorithmically very hard (see the few papers by de Marcken
about this for a glimpse into this world).

Email is hard because it combines many disparate skills (design, UX, back-end
scaling, protocol details) and because the MVP is incredibly complex.

All involved tremendous amounts of slogging.

------
Maro
Writing a distributed database is pretty challenging. That's part of the
reason I started a NoSQL company some time ago. Unfortunately it failed due to
lack of funding, but if you're interested, here's a technical write-up:

<http://arxiv.org/abs/1302.3860>

------
maxprogram
I can think of a lot of scientific "big data" applications that would be very
challenging problems. There are many domains that need to create & automate
better systems that Collect > Organize > Parse > Visualize data. Weather,
climate, ecology/species, traffic systems, etc. To do these right would be
very challenging and would have a big impact on the world.

Now some self-promotion: I'm looking for a full-time partner to help me with
Atlastory, an app that visualizes all of history on a map. This fits with the
idea above. It's a simple concept, but there's huge engineering & design
challenges involved: imagine all the geographic data involved in Google Maps,
now multiply that by 1,000, and figure out the easiest way to
obtain/retrieve/display that data.

------
gyepi
This is a good question. I've worked in a lot of different domains and can
tell you that:

1\. They all have boring parts

2\. They all have interesting parts.

3\. Doing the interesting parts _well_ can be very challenging.

CRUD is only boring because of the generally repetitious nature of talking to
the database, but repetition usually means that it can be generated. Whenever
possible, I write code generators for that sort of thing. This is not only
more interesting and gives me time to solve the more interesting problems but
also allows me to produce more and better code than I could just typing it all
out.

You could get a PhD, I suppose, but hard problems generally require the
insight to see them and the willingness and ability to work on them. Sometimes
_ability_ requires a PhD, but that's not a given.

------
dahumpty
Try systems companies (e.g. storage, networking) - you get to build products
that leverage core CS concepts - algorithms, distributed systems etc. You also
get a chance to use your investigative skills to debug problems like race
conditions, memory leaks etc.

------
rodrigoavie
I'm running into the same problem. I am tired of CRUD apps, I just don't see
myself doing anything new or exciting with them . Last year I was approved in
one of the top tech Universities in Brazil and started learning C and C++ and
this year I'll engage in programming competitions.

By the way, I've already failed 2 math disciplines in my University and I'm
doing them again. But I'm good at programming and I'm becoming way better at
Math. Someday after I graduate, I'll try to work for Google or Facebook, do
you know if they'll turn me down for the fact that I have failed some math
classes (even if I later was approved on them)?

Thanks everyone.

------
hbar
I have a hard and interesting job. I do navigation & autonomous control
systems for large mining vehicles. A couple of notes...

1\. You want to do R&D. Developing new things is fun; maintaining existing
things is boring. However, "getting in" to R&D positions is difficult without
#2:

2\. Domain knowledge. We almost never hire based on programming skill alone.
Programming is a prerequisite, but specific expertise (guidance, navigation &
control systems in my case) is the real value you need to bring. Graduate
degrees are one way (but not the only way) to move in this direction.

------
mikaraento
Here are a couple of CS-y but not too math-y suggestions.

Consumer-facing interactive native apps are typically low on CRUD and some of
them require non-obvious solutions. You could try to get a job (or just create
a product on your own) on mobile or in games.

Another option is working on tools instead of apps. If you manage to get some
non-trivial gcc/clang/kernel patches accepted you can try to find a company
that works on those.

(I've worked on Google Maps for Mobile and am now building a robotic waste
recycler - no CRUD for me since 2002 unless I've decided to build some for my
own needs).

------
binarymax
I got lucky and transitioned (in the same company) to a search project from
CRUD - which I find vastly more interesting.

But my advice is to pick hard problems yourself and try to solve them. You
need to start somewhere! Sure its expensive to get your hands on driving car
hardware, but there are plenty of hard problems out there that need solving
and can be done using opensource and your personal computer.

If you are able to get yourself through a really tough problem, you will come
out better on the other side - even if it takes you years to solve.

------
troymc
One place to find hard open problems is at the end of technical papers (e.g.
in math, science or engineering). Many of them will have a section talking
about issues, future work, ideas remaining to explore, etc.

There are infinitely many hard problems out there. If you want to stay
motivated, work on stuff that matters.

[http://radar.oreilly.com/2009/01/work-on-stuff-that-
matters-...](http://radar.oreilly.com/2009/01/work-on-stuff-that-matters-
fir.html)

------
andrewthesmart
I don't know how you feel about working for the military-industrial-tech-
surveillance-state but companies or universities that get DARPA or IARPA money
work on stuff like flying and/or self driving cars, autonomous systems etc
(Lockheed, Honeywell, Carnegie-Mellon, Raytheon, for examples). Of course
these things will ultimately be used to crush humanity's soul, but they're
interesting software and technical challenges.

~~~
pilot07
I work in this industry, and have spent a number of years in R&D. In my
experience, the vast majority of software R&D projects are not
weapons/military related, and include things such as cyber security,
healthcare, big data, cloud computing, etc. A lot of it is aimed at developing
innovative approaches to reduce costs in various government/military processes
both through automation and doing things more smartly and efficiently.

~~~
gyepi
Sorry, but I don't think there's a distinction to be made, other than one of
degree unless the implication is that improving government/military processes
is not aligned with the long term goals of the military.

~~~
pilot07
Good point, but I think the average person who has reservations about some
military functions isn't necessarily wholesale opposed to all functions of the
military.

~~~
andrewthesmart
I was in this business for a while too and did some really cool stuff. But in
the end this system transfers public money to enrich these companies; and the
DOD basically pays these companies to learn how to develop technology that the
company can then make huge profits on. It's a racket. I'm torn about it;
philosophically, morally, politically I'm opposed to war and using technology
to control people. But the skills I got and the smart people in the industry
were great. Plus the projects are cool from like a 13 year old boy
perspective, but if you stop to think that big data, autonomous vehicles etc
will be used for death and increasing the powerful people's ability to control
us, it's might not be worth it.

------
noahc
This sounds like a variant of the meme "I could build stackoverflow in a
weekend".

If you think you can build 99% of the apps out there, go out and try to build
a moderately complex app. Make it better. You'll see there are all types of
challenges from the technical, to the social to the political in small and
large companies. Being a software developer is about solving all sorts of
challenges, and not all of them are technical.

~~~
netcan
I don't think thats its a variant of the meme "I could build stackoverflow in
a weekend." It doesn't have the _in a weekend_ part.

The OP seems to consider getting a PhD an acceptable route. He's asking for
directions, not a shortcut.

~~~
noahc
Twitter, Facebook, Stackoverflow are all really just CRUD apps. They face
social and scaling challenges everyday among other technical challenges. I
don't think he's giving CRUD apps enough credit. There's also security
challenges, etc.

------
lazyjones
Pretty much every single company out there has boring programming work to do
that would be redundant if they could solve some harder problems (which they
may or may not be aware of). Competent programmers will notice the bigger
picture and attempt to solve the hard problems. If you want to be hired
specifically for hard problems, try academia or some research insititute / R&D
department.

------
richo
To be blunt, the simple problems are hard.

Concurrency. Locking. Cache invalidation. CAP Theorem for determining truth in
distributed datasets. Feel free to solve any of them.

For the record, "I could do it if I had the time". Do it. It'll be awesome.

Reinventing the wheel is a great excercise, as long as you accept that your
wheel will probably be inferior to the existing one. Or maybe it won't and
it'll be the next big thing.

TL;DR, go write some code.

------
ippisl
relevant HN thread: <http://news.ycombinator.com/item?id=4968452>

------
allinzen
Interdisciplinary projects have a lot of potential to be challenging and
rewarding. Bioinformatics, Computational Neuroscience, etc. - these are your
phd jobs. Another cross disciplinary project is entrepreneurship (you have to
do everything - marketing, finance, product development, etc.) and this
doesn't require a phd.

------
corbett3000
Certainly I'm biased, but I think working at a place like
<http://istrategylabs.com> is a place to do this.

We have to invent new things everyday, and due to our focus on hacking the
digital and physical worlds there's so many possibilities.

------
fuelfive
We work on hard stuff (<http://vicarious.com/>) and we're always looking for
exceptional people

------
tapatio
Get into digital signal processing / digital communications. They are awesome
fields that are challenging and rewarding.

------
orsenthil
What is your topcoder ranking?

------
cmundhe
You do realize that CRUD hasn't even really been solved yet, right?

------
Evbn
Get a PhD, or get a high paying job and do interesting stuff of weekends, or
do your "self-directed" work as a pauper with option to do a startup or open
source project that gets you a job later.

------
Evbn
Get a job at a big web company that has distributed systems.

But one day you will learn that even "exciting/Challenging" work is 99%
perspiration.

------
paulhauggis
I happen to enjoy easy jobs. It allows me to use all of my mental energy
toward my own startup :-)

