
Ask HN: Which industry sector should I target? - leksak
Hi, I am curious if anyone could give any suggestions as to an appropriate sector
within the software development industry given my interests.<p>My interests within the field is not what one would call
narrow. Primarily though, I have a fondness for programming
language theory, compilers, parsing and writing my own DSLs so
that I can make powerful compile-time assertions regarding my
code as well as generate tests automatically from the DSL itself.<p>My StackOverflow cv has an additional set of tasks that pertain
to my interests, http:&#x2F;&#x2F;stackoverflow.com&#x2F;cv&#x2F;filipallberg.<p>Regarding my primary interests, the only relevant &quot;thing&quot; I have
come up with is web-scraping, but I find it is a dull task as it
is a trivial matter to me.<p>Machine vision is also something that I think a lot about,
however most of my ideas are consumer-facing utilities and not
something to build a business around. If I get a good idea I&#x27;d be
willing to go into business myself.
======
mseebach
What do you want to optimise around? Stable job, 9-17, decent salary? Lost of
money, caution to the wind? Be something [for one or more of several values of
'something'] to a lot of people? Fiddle with deeply intellectual CS problems,
shielded from pedestrian worries about revenue and users?

I personally enjoy the knowledge of having solved a real (business) problem
for real people, and being appreciated both financially and personally for
this, even when the actual tech going into this isn't very complicated, and
especially, doesn't tick off a lot of "hip" boxes.

I have also met good people who do not thrive in these circumstances, so it's
hard to give generic advice as to what might make you happy.

~~~
throwawayyawa
I also enjoy this, but not for my current employer (for several reasons I
don't feel like going into).

How do you do it, as a consultant or as a 9-5 job? Any advice on getting
started as someone who graduated 2014?

~~~
mseebach
I do it as an employee of a company, but it's a pretty sweet fit for
consultants/freelancers (I used to be one).

I don't have any particular advice, beyond the generic stuff: You'll need a
network of people who might (know someone who might) want to pay you to help
them achieve business outcomes. Remember that you're selling business results,
make sure every line on your invoice is part of a narrative that makes (or
reduces risk of losing) money for the business.

------
j0rd
If you like web-scraping, and you're not an idiot, you immediately have an
information advantage for anything you decide to do. Do not underestimate this
advantage.

I would suggest you leverage your interest in web-scraping and use it to
customers in what ever niche you believe you can sell them something.

Collect your "beta" users through web-scraping, figure out a way to reach them
at scale. Build/sell a product they want (you can leverage your information
advantage to figure out what this is)

This is personally what I do and have created/worked on many bootstrapped
companies over my career. Niche for me is fairly irrelevant, as long as I have
a pool of interested customers ahead of time. I primarily use my information
advantage to figure these out.

The additional skills you will need is:

* learn to think as a "user/customer"

* minor copy-writing skills (or at least understand what shitty copy is and how to improve it)

* UX

* Data mining & Analytics, Analytics, Analytics

* A/B Testing, Iterations, Incremental Improvements (see point above)

* Hypothesis Driven Development ([https://www.thoughtworks.com/insights/blog/how-implement-hyp...](https://www.thoughtworks.com/insights/blog/how-implement-hypothesis-driven-development))

My one suggestion is to avoid any projects which do not have a clear
monetization strategy. If you're following my blue-print companies that don't
make money from day will only incur costs as you reach your pool of customers
at scale.

~~~
deskamess
Can you provide an example or two of a project that involved web scraping?
(without divulging client info) I can see how it could be used for price
scraping in web retail, but am drawing a blank in other industries.

~~~
mtmail
Address parsing from a large web corpus
[https://github.com/openvenues/libpostal](https://github.com/openvenues/libpostal)
The author uses [http://commoncrawl.org/](http://commoncrawl.org/) data,
250TB/3.6b URL and a 100 machine cluster of 8-core machines.

------
lmm
I've worked 5-6 programming jobs in 5-6 different industries. Honestly the job
is pretty similar in any industry. I'd recommend working for whoever pays best
and/or a team you get on with.

~~~
sklogic
YMMV. A cognitive load and inherent complexity can vary dramatically between
industries. Compare web frontend development with, say, a mission critical CPU
design.

~~~
olalonde
You state this as if the answer was obvious... I recently switched back to
front end development after months of developing a Bitcoin node from scratch
as well as a custom BitTorrent server. I'd say among those three, modern front
end is what I find most complex. The amount of technologies, protocols, tools
and leaky abstractions to know about is just staggering and by the time you
finally caught up, you are already behind.

~~~
sklogic
Of course, if you're doing things the wrong way, you can create any amount of
complexity out of nothing. I'm not considering this case. Why would anyone
care about the amount of "technologies, protocols, tools and leaky
abstractions" when the smart thing to do is to avoid them altogether and just
pick the lowest common denominator?

~~~
jmagoon
And what would that be for web development? Or are you just saying "don't
develop for the web"?

I've seen people who are great at diving deeply into a narrow field struggle
greatly with the broad, changing knowledge needed for web dev, especially at
scale. It's not clear to me that one is intellectually more challenging than
the other. They often seem like different types of intelligence.

~~~
sklogic
Same as with any other crappy set of APIs and standards - hide the complexity
underneath a thin layer of DSLs.

I do not consider any area where this is possible to do (i.e., any area where
complexity is artificial, not inherent) to be really complex and challenging,
unlike the domains where complexity cannot be eliminated by simply removing
the stupidity.

~~~
jmagoon
Are you really saying that the internet and web applications (and all their
various components) are artificially complex? Or is it just whatever
reification of "web dev" that you have in mind?

~~~
sklogic
Yes, I'm really saying that a majority of this web crap is artificially
overcomplicated for no reason at all, and it is mostly a product of stupidity
rather than any sane design.

And stupidity got this funny property - it attracts even more stupidity, they
stick together, they grow like a snowball, consuming everything around. Ill
designed web standards, exceptionally, incomprehensibly shitty language
(javascript) - of course it should have created tiers upon tiers of ill
designed frameworks, ladders of leaky abstractions and a broken culture of
tolerance towards all this shit.

The only sane way to deal with this kind of crap is to shield from it
entirely, introduce a simple and clean design which should interact with a
broken system, without allowing any of the underlying abstractions to leak to
your level. It always works, with all kinds of broken APIs, and I cannot see
why web should be any different.

Btw., it's amusing that web devs got so offended by quite an obvious fact that
their work is nowhere near in complexity with the hardcore realtime stuff.

~~~
jmagoon
It's hard to argue your "fact", especially against such an insufferable amount
of smugness and causal reductionism.

I find it _not_ amusing that people can assume things they don't participate
in are simple, stupid, and trivial. It's the same type of intellectual
closemindedness that leads to major problems in human existence. It's a
mindset that deliberately ignores complexity in order to justify preexisting
belief and prop up belief as truth while protecting the ego.

Personally, I enjoy working on hardware and working on distrubted systems. I
also like building things that look beautiful. But to each their own, I guess.

~~~
olalonde
There was an article on HN recently which I believe applies equally well to
technical stacks: [http://techcrunch.com/2016/01/18/why-big-companies-keep-
fail...](http://techcrunch.com/2016/01/18/why-big-companies-keep-failing-the-
stack-fallacy/)

~~~
jmagoon
Which sources nobel laureate Philip Anderson in the paper "More Is Different":

"The ability to reduce everything to simple fundamental laws does not imply
the ability to start from those laws and reconstruct the universe...The
behavior of large and complex aggregates of elementary particles, it turns
out, is not to be understood in terms of a simple extrapolation of the
properties of a few particles."

[https://web2.ph.utexas.edu/~wktse/Welcome_files/More_Is_Diff...](https://web2.ph.utexas.edu/~wktse/Welcome_files/More_Is_Different_Phil_Anderson.pdf)

~~~
sklogic
Kolmogorov and Chaitin got a different view on this matter.

This is actually what science is all about - finding an information in data,
or, in other words, measuring the complexity. If you can reduce a complex set
of data to a simple algorithm, then this algorithm is all the knowledge that
is there, and the rest is just a noise. By definition.

What Phil Anderson is talking about is a different kind of complexity, which
is sort of irrelevant in this context. Yes, knowing the fundamental laws won't
help you to explain, say, a protein folding, not without hundreds of years of
a CPU time. But it does not matter. It does not make this science more
"complex".

------
danieltillett
I want to suggest something that I would not normally suggest which is you
should pair up with a non-technical founder. What you want to be looking for
is someone with deep business experience (15+ years) in a narrow business
field. Team up with them and work towards creating something amazing, but
tightly focused.

------
yitchelle
I don't have any specific industry for you to get into, but I have a
suggestion on how you might be able to find it. Attend as many trade shows as
you can and talk to the folks to get an idea of what they are doing, although
some of them may be difficult to get to due distance, cost, invites etc.

For example, Embedded World in Nürnberg, Germany provides a good cross section
of industry that has embedded engineering involvement in it. There will be a
similar one for telecommunications, Automotive etc.

Good luck on searching for your calling.

------
al2o3cr
IMO your question isn't really answerable: you've listed tools and techniques
you enjoy but "sectors" in the industry are primarily about producing results
_using_ those. A similar realspace question would be, "I really enjoy building
cantilevers and arches. What city should I look for construction work in?"

My personal recommendation would be to find people who are doing interesting
things and talking about it in public - blog posts, conference talks, etc etc
and reach out to them.

------
pedrodelfino
Have a look at the legal industry. There are quite a few legal tech startups
by now. Even web-scraping applications are rare in legal. The sector is famous
for being profitable (high margins), rusty and inefficient.

~~~
philodendron
Having worked at a fintech startup specializing in decision systems for
regulatory compliance, I second this. There are interesting theoretical
aspects stemming from the fact that domain is very tricky to model.

* There is considerable structural sharing inside finance regulations, but it is not hierarchical

* Regulations may be changed at any time, and these changes may be future-dated or back-dated

* Users insist on arbitrarily complex exceptions and overrides

* Nontechnical users (particularly legal) will insist on being able to vet and introspect

In my experience, anyone can do this badly, but to make it scalable and robust
requires subtlety and a good understanding of formal logic, ontology and DSL
design. It's a good fit for FP. Oh, and there's plenty of room for machine
learning: Automated discovery, natural language processing, noncompliance
detection, etc.

The problem of turning regulations into a expert system that is robust against
future changes has not been solved, and there's demand for such systems from
even the most sophisticated financial companies, even though everyone expects
they'd be able to build such systems in-house.

~~~
harperlee
That's interesting, may I ask which startup is that? Is it still alive?

~~~
philodendron
Still alive, last I heard!

Droit Financial Technologies
[http://www.droitfintech.com](http://www.droitfintech.com)

------
lafay
The technology to cost-effectively store massive amounts of raw metrics / data
and quickly derive insight from it is still a relatively recent development,
and so far we've seen turn-key tools to leverage that in just a few areas:

1\. Where developers themselves are the likely product users (think Google
Analytics, New Relic, Optimizely, Appdynamics)

2\. Where there is a direct and immediate connection to $$$ (finance / quant
stuff)

3\. Generic data science platforms (Cloudera, Hortonworks, Databricks)

I think there is still tons of opportunity for industry-specific turn-key data
science tools. Especially for verticals where there is a low coincidence of
skilled developers with industry domain expertise.

------
bjwbell
Learn the art of writing gpu compilers. Software defined radio is another.

If you find web-scraping dull don't pursue it. Life is short.

------
cableshaft
I'd just suggest looking into something up and coming that interests you, and
start learning whatever you can about that.

Industries that will have specific uses for machine-vision: autonomous driving
cars, drones, robotics, augmented reality, internet of things

Others: crypto-currencies (specifically digital ledgers/blockchains), virtual
reality, commercial space exploration, online education, voice recognition,
(more as I think of them)

Tech that's already big and probably getting bigger: streaming video, instant
stock trading based on algorithms,

Certainly one of those must be interesting to you. They're all pretty
interesting to me.

------
boothead
Whereabouts are you? If you're interested in PLT, DSLs and compilers, there
are some small enclaves in finance that might interest you.

 _edit_ spelling

~~~
S4M
What small enclaves in finance are you talking about? I know that banks have
small scripting languages that enable semi-technical users to describe a
derivative product ("if condition1 then payoff1 etc.") but those are old and I
doubt they are actively maintained and that banks are hiring people for that.

~~~
boothead
I was talking about banks. I work for one where we build stuff on top of
essentially this paper:

[http://research.microsoft.com/en-
us/um/people/simonpj/Papers...](http://research.microsoft.com/en-
us/um/people/simonpj/Papers/financial-contracts/contracts-icfp.pdf)

We're even hiring: [https://donsbot.wordpress.com/2016/01/18/haskell-
developer-r...](https://donsbot.wordpress.com/2016/01/18/haskell-developer-
roles-at-standard-chartered-london-singapore/)

------
HFTGuru
If you are interested in huge bonuses up to 400k-1M a year for being a
developer, then I suggest to you high frequency trading development. The
problem is you'll have to move close to an exchange, but the finance industry
rewards you handsomely and takes care of it's employees.

------
jndsn402
Best of luck to you, I am not in the industry and cannot recommend from that
perspective.

As a consumer though, augmented reality ( eg Hololens and Magic Leap) seems
like it is about to become very big, and probably utilizes machine vision.

Can you recommend some resources for learning about web scraping?

------
sharemywin
you should look at marketplaces. possibly even ones run by crypto currencies.
There are alot of ways to configure a marketplace and it would be nice to
abstract that from CRUD operations and such.

------
scardine
You can do pretty well as a Google for Work reseller.

I can make 1 USD per account per month, at 10k accounts it is a pretty good
source of passive income.

~~~
leksak
Interesting proposition for a static income. I'll look into it

------
drelihan
you're right, web scraping is dull ( it was a big, but important part of my
job for a number of years ). The interesting, and non-trivial, part comes on
how to use the data you have scraped.

------
sklogic
I'm very much into DSLs, and I've been building and using them extensively
throughout a very wide spectrum: database engines, CADs, hardware design and
verification, compilers, device drivers, HPC (GPGPU).

------
sharemywin
Which one are you talking about:

Damn Small Linux, a very small Linux distribution

Definitive software library

Domain-specific language, a computer language designed for a specific problem
domain

~~~
shade23
DSL= Domain Specific Languages

~~~
sharemywin
sorry I left the non-technical terms in there I didn't mean to piss everybody
off. Thanks for answering.

