Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Are you working on interesting technical problems?
542 points by obsession on June 5, 2018 | hide | past | favorite | 426 comments
I have been feeling disappointed that I haven't been able to use my computer science degree to it's full potential. Most of my work seems to be boring CRUD work where the challenge is gluing libraries together or figuring out business requirements. Actual interesting technical problems seems to be mostly wrapped in ready-made libraries / SaaS services.

Anyone in the same boat?

You might as well have asked who among us won the lottery.

I've been struggling with this for years now; after 4 boring jobs I finally gave up even the expectation that what I'll be asked to do will be even remotely interesting (or socially useful). The chance of landing an interesting job is absolutely minuscule - the market rewards whatever it rewards at the moment, and that's where jobs flock. These days, one of the bigger thing the market rewards is writing a web-based CRUD that's 99% chrome, 1% of actual functionality. Extra points if you can get some user data to sell to adtech industry.

(My own criteria for "interesting" these days are a bit different. I'm not looking for "interesting technical problems", as in advanced CS stuff, but "interesting outcomes" - pushing the humanity forward, or alleviating a social problem for some group of people. I find it hard to even find companies that apparently work on something like that, and even then it seems they have to do lots of boring work as well.)


EDIT: Then again there's this topic that has been casting shadow on my happiness for a long time, that I finally seem to be able to define. Maybe our civilization has finally crushed my soul, and I noticed it just now. But it feels to me that the revealed goal of our market-driven world is to abstract everything away with money. The end-game is that as an individual, your responsibility in the society is to pick a skill - whatever skill - that makes you money, and then to use money to solve everything else with your life. It doesn't matter what you do, there is only one meaningful meta-skill - making money fall out of the system, into your bank account.

Maybe I'm just realizing something obvious that parents should have taught me when I was a kid. I don't know. Maybe it was wrong of me to seek meaning in the things one does? Maybe that approach is no longer supported by our civilization?

My father-in-law mentioned something I find quite interesting to my wife: Don't look for a job that will fulfill every aspect of your life - that's what your family is for. And if you do find one, your family life might not suffer.

Not all of us are raising families, but I think it highlights a bigger points. Work is not what builds societies (certain jobs do harm society I think, mostly by destroying it's moral virtues). People relationships are what build societies and build happiness.

And you're right - there are a lucky, select few who get jobs that can also influence society.

I'm not worried about that though (anymore - I was for a time though). With my job, I can provide and care for those around me (provide for my family and generally have positive, uplifting interactions with those around me).

I have resigned myself that it's not my job to fix society, it's our job to fix society, and I'm doing my part where I am. From my perspective I have more than enough to do locally anyway: https://news.ycombinator.com/item?id=17176036

That's definitely something I need to think through more. And what your father-in-law said sounds wise. Thanks for sharing it.

The reason I ended up in tech is in part my fascination about tech, especially as applied to uplifting ourselves and solving our problems (yes, I watched way too much Star Trek: The Next Generation in my formative years). I never thought much about family - in fact, for the most of my life, I felt it's something of a side endeavour. I still feel it shouldn't be the sole focus of a person, though I definitely believe it's more important than I believed in the past.

Same for relationships. And small communities. I feel that maybe we overdid this globalization thing. Maybe global competition and centralization of various kinds of innovative work is more efficient, but it also doesn't seem healthy. Living in a city, there's no such thing as "local community" anymore. Not even neighbours.

> I have resigned myself that it's not my job to fix society, it's our job to fix society, and I'm doing my part where I am.

Still, I'd like to feel like I'm actually doing my part beyond just participating in the economy as consumer. I have specialized skills, I'm not dumb. But I have hard time finding opportunities to do things I'd know could help meaningfully and effectively. There must be something. Maybe I'm just not looking hard enough.

Your comment you linked to is golden. I'm favouriting it. Those are definitely words I'll try to remember from now on.

I've also thought about this a lot. My last years of work life have been quite boring as well. On the other hand I realize that, as you write in your first post, are scarce and precious.

A few years ago I applied for a really interesting jobs within Telematics - one of my favorite areas. So I called them, got the CTO on the phone and he had almost no time for me. Then I wrote a mail with my formal application to them. It was appealing and they replied with some coding riddles. I guess I did well enough and was one of 2 or 3 people to get invited. Of course before the in-person interview I studied some Algorithms & Data Structures book. At the same time I did Scala programming although it wasn't expected to know that but people worked with it.

Anyway, during the interview they made me sweat for at least 3 hours. Every now and then also reminding me how good the other people working there are - the one guy having worked 8 years at a company writing DBMS in London. So yeah, after leaving the communication was difficult but the CTO said the people said they liked me and they would hire me despite my shortcomings - I would have to unlearn stuff. The offer was really bad, in fact even the most senior people got below average salary.

At the end I started working at a company that worked on a moonshoot project involving a CAD program in the browser. Even some mathematician who worked in Japan on Robotics was there. Salary was good enough but the people were borderline insane. I quit after 3 months and landed a below average paid job, but with a lot of nice people. I don't regret that. Work was mostly boring but the environment was nice.

Anyways, not living in the Bay Area, I guess things are they different again.

For me, the most rewarding work is the volunteering I do. I coach various youth sports. I have no pressure to win so I can focus on letting the kids enjoy themselves, have positive interactions, and learn a little about the sports. Seeing them gain confidence and passionately work toward something is a major reward for me.

I still enjoy trying to match a user's expectations with the software I write. But, it's balanced by other aspects like coaching and being in a family.

> Living in a city, there's no such thing as "local community" anymore. Not even neighbours.

Sure there is. I live in a condo with 20 other units, and it makes a great small neighborhood.

How do you make it work? In the experience of every condo dweller I know personally, people know their neighbours mostly to the extent they bother being involved in their HOA meetings; otherwise, everyone walks their separate paths, only greeting (and maybe briefly chatting with) the neighbours as they meet them on the stairway.

It helps that it's shaped more like townhouses than an apartment building. We have a large interior carpark/garden where the kids play, which creates a lot of incidental contact and I think the kids are a huge factor in making a neighborhood feel, it creates an easy shared experience and proximity means you can easily help people out with eg shared babysitting. There's also a few people who make a dedicated effort to encourage community, like trick or treating, and it's a place where half the residents have been there over ten years. I joined the HOA in my first year which has definitely helped me get to know others.

The same thing goes for family. One should not look for family to fill every aspect of your life.

I haven't given it a ton of thought because I have a job I enjoy from time to time, and enough downtime to pursue a hobby or two - I guess I'm in a pretty balanced situation and am okay with it.

In reading your post, I had to stop and think for a bit.

I agree with you but want to give the thought that family can be one's top priority without looking for it fill every aspect of one's life.

no one thing should be dependended upon entirely. any and everything may disappear from your life at any moment.

no one is above great tragedy and loss.

You are correct - no one is above great tragedy and loss.

If my family was taken from me unexpected, I don't I'd say that I wished for more time at the office, or working on personal hobbies. I do believe in balance, but the point I'm trying to make is that family can be the number one priority.

The mindset of "I can be happy because I have x" (in this case where x = family), is indeed dangerous. And while the mindset of "I have chosen to be happy, and my top priority is my family" will bring sadness and pain if they are suddenly taken, I feel like that pain and sadness will have been worth it. Well, it is for me. That for others it might not is understandable.

Please note as well that I'm not advocating self-deprecation or letting your family take advantage or you, or throwing yourself under the bus for an uncaring family - it's a group effort and there a lot of ways to appear to put your family first in a way that's detrimental to one's self.

For me when I think of dedication in family life, I think of the story, "the gift of the magi". Nothing of true importance was sacrificed, and yet the others needs were prioritized.

I believe that the inability for business in our society to be socially aware or responsibile stems from the separation of private life and public life. I can't remember what book this was, but it looked at what war criminals were like in private life. Its main point was that, in private, war criminals were nice guys.

According to the book, the transition from being war criminals by day and family men by night, was possible through some rationalisation acrobatics or detachment from responsibility. As an example of detachment, the Nazi concentration camps didn't have an atypical ratio of psychopath to average joe. The predominant "average joes" would offload responsibility by telling themselves that this is not their decision, but that this is the _current policy of the organistation_, which is an interestingly complex statement, in the subconscious sense. It detaches the organisation participant from the organisation, and it detaches the organisation from their policy.

To come back to business v family, I think the old koan holds: separation isn't good. The rule of thumb in social situations is if detachment or separation is the easiest route, you're doing something wrong. Business is social action by necessity. So should we separate action from responsibility? What do we think of someone who is selfish and avoids responsibility?

> The predominant "average joes" would offload responsibility by telling themselves that this is not their decision, but that this is the _current policy of the organistation_, which is an interestingly complex statement, in the subconscious sense. It detaches the organisation participant from the organisation, and it detaches the organisation from their policy.

Yes, your job as the "average Joe" is to be loyal to the organization. That statement doesn't sound surprising to me, and it took me a few seconds to realize why. I've heard it before. I grew up in a religion where there was an organization, and your job was to be loyal to God by being loyal to the Organization. Only by being on the outside now I see how this is a neat way to release responsibility. See also: "Ours not to reason why, ours but to do and die.".

> The rule of thumb in social situations is if detachment or separation is the easiest route, you're doing something wrong. Business is social action by necessity. So should we separate action from responsibility?

It was easier back when you were naturally limited by geography to do business with people you also knew "after hours". These days, business and personal relationships are almost entirely non-overlapping. Hell, it's a common advice to avoid that overlap.

Likely either directly or something inspired by Hannah Arendt's 'Eichmann in Jerusalem : a report on the banality of evil'. http://www.worldcat.org/title/eichmann-in-jerusalem-a-report...

There should be a name to the phenomenon that humans seem to become crushingly depressed once they start to grasp the big picture dynamics of the system they're encapsulated within. I think it's an effect that we're not quite prepared for as we reach closer and closer to science dotting all its i's.

It's as if childhood primes us to think that life is all about figuring out what the heck is going on around us, and as a consequence some of us go into something science-y, but then we reach a certain stage in adulthood where we feel like we have a pretty good rough closed form solution to the problem, and now the journey all of a sudden seems empty and without purpose and we just keep going with the lives 16-year old selves planned out for us, with too much cost sunk to pivot. I almost suspect the end-state of science is not some sort of blissful state where we feel like we understand everything, but an utter dread and feeling of emptiness at no longer having a purpose.

I had a dream where I was in a coma for 20 years, woke up, and found that AI was now able to trivially answer even the most difficult research questions I could imagine. In my dream I visited my old workplace, and found that while the CEO was still there the ease at which questions could be answered made the humans more supervisers than researchers.

For some time in my dream I was lost, not sure what my value was as a human being without my skills that I've spent decades refining.

Then I remembered that humans have inherent value, that I am not my work, and went to a park to meditate and teach karate. It made me think, in my dream, how even if a machine could teach karate better, I was not doing it for society -- I was doing it for myself.

Realizing that you are not your work, and are not defined by your work; that humans have inherent value and you have every right to simply be happy when the opportunity arises; that the Puritan virtues of work are an illusion, and that humans should follow their passion; that the illusion of the necessity of work often prevents people from ever discovering their passion in the first place.

It was probably my best dream, and I approach life quite differently now. I think when I feel like I have a closed form solution, it is much the same spirit as an AI that will solve your problems for you.

Was that a dream or a trip? ;)

I agree. Maybe there is a word for that?

Personally, understanding the "big picture dynamics" of systems is kind of an intellectual porn for me. I loved doing that for as long as I remembered. It always came with the sense of control - understanding the systems means you could tweak it to your liking, if you so desired.

Finally realizing that the system in which we live is basically uncontrollable by an individual within it has blindsided me. So many interesting things happening, so many interesting things that need to be done still, and yet all you and I get to be are observers and consumers of whatever the system happens to spit out.

The trick for me is realizing that P doesn't easily equal NP, which means that solving satisfaction problems is a challenging thing to do. But everyone knows when it has been solved. Lots of people have an idea of what should happen, few know how to do it. If you find that for some reason you're uniquely qualified to produce some process, you can figure out if it solves someone's problem. Then other people can share their resources which they are uniquely qualified to make, and their problems are then solvable in P time, while they would have to do the NP task of learning how you do your thing. Then you as an individual have a lot of leverage because any change you make to your system, everyone will have to use, provided it's a verifiable improvement.

I haven't given up on thinking I might be able to influence the system. Specifically, I hope my volunteer work on snowdrift.coop can move the needle a bit.

https://wiki.snowdrift.coop is probably the best introduction right now; the main site is in the process of being updated.

There should be a name to the phenomenon that humans seem to become crushingly depressed once they start to grasp the big picture dynamics of the system they're encapsulated within.

I agree. Maybe there is a word for that?

'Attachment to expectation'. See buddhism.

It’s far from a perfect fit, but I think Marx’s theory of alienation comes close to what you’re talking about:


I don’t think “existential angst” fits. As the name implies, you get that just by existing and being trapped by the limitations of being a mortal human.

You’re talking about the specific meaninglessness of modern work. Marx suggests that capitalism affords us some real freedoms, but at the cost of our feeling of purpose and solidarity. And, Marxists would argue, this makes us easier to exploit.

Marx and his inheritors failed to produce systems that solved this problem, and maybe that indicates that the problem is deeper than just forms of ownership and control.

But as a problem description, I think he was onto something.

You pass butter.

Oh my God.

I guess I don't understand the reference...

In addition to the “Existential Angst” there is...

Koyaanisqatsi - life out of balance - see also the film

Seems like a variant on "Existential Angst". Maybe not exactly, but the sense of loss of purpose feels fairly close.

This feeling is because in our current stage of development, we are very primitive spiritually. Some of us don't even believe we have a soul. That it is all meaningless. Science in its current form also looks at our material world and concludes the same. Nasa tells us we are just a speck of crap in a giant space as well, with no beginning or end.

You guys talk about boxes. Well there are huge ones for the mind, carefully constructed to be believable for the masses. Too complicated to understand, they require you to put beliefs in people with credentials, who will repeat to you what they found in the books they were given, or the math they got shown.

If you want to understand your purpose here, you have to step outside of scientific thinking to start with, and that's a huge problem for almost everybody. You have to look at origins of the matrix we are living in, the illuminati, the Egyptians, book of the dead, the occult.

You forgot to mention the lizard aliens :facepalm:.

The Aeon of Horus is here: and its first flower may well be this: that, freed of the obsession of the doom of the Ego in Death, and of the limitation of the Mind by Reason, the best men again set out with eager eyes upon the Path of the Wise, the mountain track of the goat, and then the untrodden Ridge, that leads to the ice-gleaming pinnacles of Mastery!

Not enough quantum in there.

Congratulations, you won a Dunning-Krueger!

You have found the boundaries of the box, and anything you find outside the box will be judged against everything inside the box. There are a lot of people inside the box, and the box is small, so they make a lot of noise. When you get outside of the box you need to also get away from the box so that you can't hear the judging.

Wow, very nicely put.

> ...also get away from [the people in] the box...

This may be the harder part. I consider some of the people in the box to be friends. But then again, if I think about it... maybe they're outside the box too, but we're on opposite faces, and so close to it that we are still looking at each other _through_ the box.

Interestingly put. Sounds poetic.

So here I am, with the map of the boundaries of the box finally taking shape. Now the question is, how do I exit the box? I'd like to walk around on the outside and check it out.

Also, is leaving this box even a good idea? What if everyone left the box at the same time? Would the world collapse then? (I can't help myself, I tend to be interested in the global view as well.)

Well, that's where universal basic income has some validity.

It's a very nice idea.

It does have the potential to create localized unimpressedness though; the question is how powerful the temporarily-put-out groups would be, and whether the backlash would remove your power as well.

Money is power, laterally too.

Eventually everything becomes "just a job."

In my mid twenties this was hard for me to grasp. I was working as an electronics design engineer for a company that made low-end industrial controls. We had a customer who had what seemed like the dream company to me: he made robots for animatronics and lab applications. One day I was talking to him about how exciting the work was and his answer surprised me: it used to be fun in the beginning but after doing it for a few years, it was just another job.

I work with wireless sensors now. Yes, it's technically interesting, but it's pretty boring.

Why? I've discovered over the years that I am far more excited by making customers happy than by doing technically interesting work. I've come to see the technology as a means to an end, not the end in itself.

I wonder if there are some people who are exceptions.

I worked with an software developer at my last job who was incredible. He seemed to have a strong interest for the technology, no matter what technology that was. At the time, it was iOS, and he was an absolute master of iOS. I was constantly impressed by his attention to detail and emphasis on quality, well written, technically challenging code.

I've loosely followed him since, and he seems to be absolutely crushing it wherever he goes. He's spanned many different roles and seems to do enjoy taking on new challenges in the form of different programming languages / platforms.

When I think back to him, I remember someone who seemed to just love programming. In a way, he was like a Robot. It doesn't appear to matter much to him what he's working on, or if it's "fulfilling" in the ethical sense.... He just enjoys the process of coding, and being damn good at it.

>I've loosely followed him since, and he seems to be absolutely crushing it wherever he goes.

Question - is his management crushing it as well or are they crushing him? Is he good at standing up for himself and not letting management crush him?

I think a lot of enthusiastic and brilliant people get destroyed by their managers exploiting them. So the individual goes from crushing it to getting crushed. Management really has to enable that kind of person to succeed.

> Is he good at standing up for himself and not letting management crush him?

Yes. This. He was very, very good at standing his ground and asserting that his time and effort was being spent well.

Multiple times I witnessed management clashing with him over something. Usually he felt like more time and effort needed to go into a certain part of code. Rather than just "get it done" he took pride in his work and wanted it done RIGHT.

In general he was so good / fast that he could get it done "RIGHT" within management's expectations. But occasionally he would take on a big chunk of work, and management would try and shut him down. 9/10 they failed, he would do the work, and a damn good job of it.

Interesting point. Can you elaborate a bit on what you'd expect from a manager to help that person succeed? Is it mostly case-specific?

I can tell you that for this particular person he needed to be silo'd and given time.

A good manager would realize that his quality of work was top notch but that he was at times difficult to get along with. He was a perfectionist who held those around him to the same standards, and that often clashed with the business needs and even sometimes other teammates. A business generally isn't concerned with the product being perfect, they're more concerned with it getting to market on time.

So my suggestion to any manager that has an employee like this: Value him for what he is. Put him on MISSION CRITICAL tasks, where you need a rock solid implementation. Let him work for 2 weeks without interruption.

For other tasks that are perhaps not as "critical", throw other team mates at them.

It's always case specific, but it usually involves the manager supporting them by providing time and/or space. rubicon33's example and response is a great one. Put them in a situation where their strengths can be well utilized and the effect of their weaknesses minimized.

Also, not everybody has the personality or energy to constantly stand up for themselves. A lot of people will eventually crumble or leave. Don't make it so the only way the employee can do their best (e.g. what rubicon33 describes) is if they put their foot down and scream.

>Why? I've discovered over the years that I am far more excited by making customers happy than by doing technically interesting work. I've come to see the technology as a means to an end, not the end in itself.

100% agree. I'm a Data Scientist and I'm supposed to have one of the most exciting and sexiest jobs in tech. But after many years of doing this I've come to realize that working with good customers and seeing them happy is much more rewarding than playing with the latest machine learning thing. New tech is just the thing that makes it easier to tackle a customer use case.

There's a strong relationship between the quality of the people I'm working for and my sense of the work being interesting. If the people I'm working for are unrealistic a-holes no amount of technical novelty can make up for that.

I've seen this sentiment many times here on HN. I wonder, who are your customers exactly?

At one of my previous jobs, "my customers" were the managers of some company who asked managers of the company I worked in to deliver some software that will be sold as a part of a software package, to be used by someone downstream. I don't know who ended up using my work and how, because we were isolated by several abstraction layers from that. I did care about the happiness of those people, little I knew about them. But I couldn't care less about the happiness of our customers - a company I didn't hold in high regard, with managers not being particularly helpful. I suppose I did get some small sense of satisfaction when their managers told my managers that the work is good. But that was it.

So, are we talking about happiness of customers, or happiness of people actually using the product? Those are very often different groups. Or, in other words, are we talking about finding happiness in generalized servitude, or making people's lives better?

I was referring to people who were actually using the product. I've had a number of jobs in my life, including being in the services division of a product company and also in an internal consulting team of an extremely large non-tech company. I would typically interact directly with the people who are using what we were building. So I could see exactly what they needed and did not need (despite what they said they wanted) and manage that so they were happy with the end product.

I've also worked in situations you've described. My only interaction with the end user is through layers of middle management. I found that environment to be extremely demotivating and certainly no direct sense of satisfaction. One of the small benefits of being a Data Scientist right now is that a lot of managers don't really understand what it is I do so I get some interaction with the end-user from a requirements and solution validation standpoint. But I suspect in a few years Data Scientists will become invisible cogs (like everybody else) in the Enterprise IT machine.

Personally I work in the health tech space - specifically a pharmacy. Our customers are people who are able to get cheaper medications and better health help and pharmacy technicians who are more productive due to our software. The job is technically just Django/CRUD with some integrations, but the impact can be fulfilling.

I don't think you can get that kind of satisfaction in B2B software as a developer (only a salesperson/exec).

What are the thing that is intresting ( as per your opinion, if you are willing to share? )? I mean not everybody will get the job they like and it will be the boring thing to do ( but, still we have to do this boring job! ). But, life shouldn't be boring and it should be beautiful and enjoying.

The interesting thing isn't the technology, it's what the users/customers are doing with it. It's how it improves their lives, simplifies their business, enables them to build other products and increase profits, etc.

I personally think the people working on things like database and filesystem performance are doing really cool work. Any time you're taking conference papers and turning it into code to solve a particularly hairy problem... that's pretty cool.

Dude I got you.

Two kinds of "work": work that gives you money, and work that gives you value. They are not necessarily the same one.

Most people have to work for money. The best we can do is to gather more "money" (or resources). So that you can dedicate more of your time to generate, as you said, "interesting outcomes".

I recommend the book "Your Money or Your Life" by Vicki Robin. I learnt a lot relevant ideas from it.

> Maybe I'm just realizing something obvious that parents should have taught me when I was a kid. I don't know. Maybe it was wrong of me to seek meaning in the things one does? Maybe that approach is no longer supported by our civilization?

Ouch, this resonated. I've been walking this path with you.

Sometimes I make my own food. Or grow it. Or pickle it. It would be more "efficient" to buy such food in many cases. But if it gives me pleasure, I trade my time to do it.

My opinion: the point of life is not to acquire so much money that you can avoid doing any difficult things and wrap yourself in a cocoon of comfort. The point of life is to find your own good life and live it. That will entail some difficulties (at a minimum the difficulty of seeking your own definition of the good life, which is hard!). You may find that in your work (lucky you!) or you may have to work to support yourself and find this good life in other areas.

I've favorited more comments in this thread than anything else on HN ever. Thanks for the great thoughts. I can add tangentially that these questions, these considerations, were pursued almost maniacally by David Foster Wallace, especially later in his life. You may find satisfaction especially in his last, unfinished novel, The Pale King, which purports especially to treat the immense spiritual gravity that he thought pertained to boredom itself, the awful tedium that so many of us have to suffer in the modern marketplace. Not, like, the boredom of having nothing to do on a rainy Sunday afternoon, which I don't think exists anymore; more the ugly struggle to, as you say, find a meaningful life amid meaningless work and interactions.

Thanks for the reference, I'll look at it.

When I wrote my edit to the comment, I didn't expect much reaction to my confused ranting. I was surprised to discover it resonates so strongly with people here. I guess I'm nowhere near alone in thinking those thoughts, and going through this process.

> abstract everything away with money

This is the lens through which von Neumann and Morgenstern saw all of human behavior, and their "rational" economics has been hugely influential in public policy. I use scare quotes because I disagree that the approach is rational -- many things in life can't be measured in dollars, not even dollars! (E.g. you can't compare a dollar today with a dollar ten years from now. Regardless of your discounting strategy, that money is going to mean something different to you in ten years.)

Your position is a great example of this phenomenon -- there's not really any amount of money that can compensate for a lack of inspiration at work. You might be willing to give up on interesting work for enough dollars, and the rational economist would say that this is the measure of how valuable interesting work is. But there's a long history of studies, notably by Kahneman and Tversky among many others, showing that framing human motivation this way leads to all sorts of paradoxes.

So yes, we have a strong tendency to measure everything with money -- this outlook is deeply embedded in our economic and political systems -- and it makes people miserable because it's a terrible way to account for human happiness.

"I use scare quotes because I disagree that the approach is rational"

Agreed. For the most part, going to church is not economically rational. For the most part, having children is not economically rational. For the most part, being monogamous to one's spouse is not economically rational. For the most part, patriotism to one's country is not economically rational.

In a slightly different sense, going to space, or colonizing Mars, would all be economically irrational.

I'm wary of a definition of "rational" that tends take all of the things that humans hold dear, and declares much of it "irrational".

I think there's something to the Hume quote about reason being a slave of the passions.

Hume was right. I recommend The Righteous Mind.


We do innovate. What we don't do however, is giving everybody a job in innovation.

These days, one of the bigger thing the market rewards is writing a web-based CRUD that's 99% chrome, 1% of actual functionality. Extra points if you can get some user data to sell to adtech industry.

I don't think they are rewarded on the grand scheme of things. Software has always showed a winner-takes-all structure and most software disappear. Take for example the game industry. 15 years ago, you would have talk to anybody, they would have told you the market is mature and they are making money on making movie-based games. Today, most players (publishers, studios, etc) of that era have disappeared (Im too lazy to get the long list of publishers that are gone).

Also innovation can act in a subtle way. Take the iphone for example. People are more productive because of it. That means that hospital, universities, research labs, non-profits, etc, all are doing better jobs which means we cure more disease, teach, help and study better, etc. because of the iphone.

But yeah, if you dont work in innovation, then you work for the current "government", the current group-thinked direction made by some smart "politicians" (by "politician" I mean that guy who lick asses all day long, who goes always in the direction that's well accepted and who use words to seduce people and get what he wants). It's either innovate (rare) or maintain.

Maybe our civilisation has finally crushed my soul, and I noticed it just now. But it feels to me that the revealed goal of our market-driven world is to abstract everything away with money.

There is a passage in the movie "1492: Conquest of Paradise" by Ridley Scott that I love:

"Look outside, what do you see?" ask Colombus to some royal acolyte

"I see towers, I see palaces, I see steeples, I see civilisation. And I see spires that reach to the sky!" answer the politician contrasting with the wild america of that time.

"All of them created by people like me" say Colombus

As a Mechanical Engineer with solid math and physics background, using Machine Learning and Numeric Methods to substantially engage and improve product performance, I really feel like I won the lottery.


I handicapped myself by going into CS instead of graduating some hard engineering degree (or biology). See also matte_black's comment elsewhere in this discussion. Programming alone is a tool in search of problems to solve, and the interesting problems are elsewhere.

Are you free to say where you work?

> pushing the humanity forward, or alleviating a social problem for some group of people

Isn't this what humans like to strive for? Making an impact, being remembered after death, thus living forever. That's one of the things a family (with kids) helps you get over, you know, the whole "living on through your kids" thing. If they don't end up as homeless drug addicts, it probably kinda works.

I've taken a job at a university two years ago. Pay is much lower than before. Haven't felt anywhere close to overworked or burnt out ever since, even though I've stayed until late at night occasionally when we fucked something up by deploying without any proper testing again, because hey, it's such a tiny change. I'm not doing frontier science or anything, we're just working on a custom netbooted Linux distro that also runs windows vms via qemu-kvm, for teaching environments. A lot of what we do could be dismissed as NiH-syndrome or creating solutions to problems that have already been solved by big companies, but as long as someone is paying for it I'm good with this job. Because there pop up a lot of problems or requirements that are just interesting to solve by yourself, actually not just for fun but for actual use cases. Nothing game changing for humanity, not for millions of people, but for some. And I will interact with them directly, and sometimes they tell you that they like the system you helped develop. That, and the fact that you're not in competition with your coworkers or another team or anything really makes for a great work atmosphere that I'm currently really enjoying. I don't know how long I can/want to stay, but for now I can say it greatly improved my overall happiness.

> Isn't this what humans like to strive for? Making an impact, being remembered after death, thus living forever. That's one of the things a family (with kids) helps you get over, you know, the whole "living on through your kids" thing. If they don't end up as homeless drug addicts, it probably kinda works.

I don't know. Everyone has kids. Having kids is technically making an impact, but I think that when most people say they want to make an impact, or to accomplish something great, or to change the world, they don't mean by doing something that almost everyone else does.

> I don't know. Everyone has kids. Having kids is technically making an impact, but I think that when most people say they want to make an impact, or to accomplish something great, or to change the world, they don't mean by doing something that almost everyone else does.

I guess that's what the "help getting over it" part was referring to. In your teens/early twenties you dream of changing the world, and after reality sank in, you do the next best thing and go for kids because you shape them and so live on through them, mission accomplished, impact has been made. Humans are great at bullshitting themselves. (Not trying to sound harsh or implying kids are worthless or anything, but it really seems to be a fallback plan for a lot of people)

Yup. Having a child seems to be the lowest common denominator of "having an impact", and it terms of improving anything around you, it's simply re-rolling the dice, hoping that your kid will end up in a position of greater influence than you. Having offspring is probably the best way to make (some of) your way of thinking outlive you, but you need to work on creating opportunities for your children if you want your way of thinking to impact anything around them.

> hoping that your kid will end up in a position of greater influence than you.

No, it's hoping that you plus your kid(s) will have more impact than you alone. Or that your kid will magnify your impact just by existing (which there is some evidence for if your chosen venue of impact is electoral politics, at least.)

Meaning is a process of its own creation, not a measuring stick. Often when people speak of meaning as though it were quantitative, they usually are referring to how much they help other people. Meaning is also interactive, it rarely happens in total isolation. If everyone around is constantly conflating meaning with monetary success, it's easy to feel confused and despair. Don't blame money, but rather look for bits of beauty in life and share them with people you care for. At least that's what seems to work for me

>... the revealed goal of our market-driven world is to abstract everything away with money

That's one mental model but it doesn't fit exactly with reality and there are others. See the Love & Money graph here for example. Not quite sure how that fits in with the job hunting. https://www.theatlantic.com/magazine/archive/2003/01/does-mo...

> alleviating a social problem for some group of people

Not sure what you mean by "social" problems specifically, but when you alleviate a problem for a group of people, they pay you, and that's what a business is.

I realized recently what a useful abstraction money is, because when money came about we didn't have the ability to create a massive blockchain keeping track of who did favors for who, and when. So it's a reasonable proxy for reputation and social credits.

What I meant by social problem for some people are things impacting their happiness and quality-of-life, reducing suffering, etc. Some examples to narrow down the possibility space:

- Make it so that people in your community have access to clean water. Surprisingly, it's a problem even in the most advanced countries.

- Contribute to something in healthcare that reduces costs, improves availability, or makes more people survive a procedure, an illness, etc.

- Make firefighting safer, or more efficient. Make it less needed in the first place.

- Make the air clearer. Make the energy cheaper. Make the roads better and safer. Etc.

- (Meta) make the market less ennui-generating.

I found that "what pays" is not a good proxy, because often what pays best is solving problems for a group of people, which also generate further problems for other groups of people. Usually, the benefits for the first group are purely financial.

To use a simple example - advertising. There's lots of people who would pay you for your work in this space, because your work will earn them more money. What your work will do downstream, however, is most likely one of those:

- It'll create more work for someone else to cancel the effects of your work; a lot of advertising is playing a zero-sum game, i.e. simply wasting natural resources and human time. (Conversely, often your work is only needed to cancel out the effect of someone else's efforts.)

- It'll disturb the market, promoting a worse solution over a better solution, leading customers to make wrong choices, and thus lose some amount of money and/or happiness.

- It'll add to the noise we all have to endure, also slowly eroding happiness.

Occasionally, you might end up promoting the right thing to the right crowd for the right reasons, but I doubt that happens very often ;).

Conversely to all of that, I feel that solving what I see as "social problems" pays less, if at all, again making "what pays" a bad proxy towards finding ways to address those problems.

Is your room clean?

It's a somewhat serious, somewhat tongue-in-cheek question. But if you need to clean your room, maybe you should do that first. Instead of having grandiose visions of improving things at large scale, prove to yourself you can improve things at a local scale and keep them that way.

If that sort of thinking is interesting, there's a whole book of it in https://jordanbpeterson.com/12-rules-for-life/ Though I'll say I got out of the existential and nihilistic mindsets differently, I don't feel like I have to wrestle with meaningness like I used to, before I ever heard of the author, I've just read good-sounding things from people who have read that book or have otherwise gotten themselves invested in the author's doings.

I suspect you've read this before but if not, point 5 might resonate with you: http://coding.derkeiler.com/Archive/Lisp/comp.lang.lisp/2006... If you fit the mold and can't change your mindset then you'll forever be unhappy unless you manage to get one of those "interesting" jobs.

A good list. But we have no time to work on social innovation until the bigger problems of tongue detection are solved: https://images.anandtech.com/doci/12870/Screenshot_113.png

What about people with lots of problems and not a lot of money?

For some reason, I still think they have value and that their problems are worth solving, regardless of what the market thinks of them.

Solving them at scale can still lead to money. Sometimes they are willing to give you a little money, sometimes others are (governments, or even businesses if it can lead to profit for them. I.e., low income training + job search with recruiter fee could be profitable). Or if you're addressing the problem of they have little money, by giving them skills or opportunities, you can charge after you help them (i.e., many coding bootcamps that delay costs until you land something)

This still asserts that solving the problems of a poor person is worth less than solving the problems of a rich person.

Also, getting paid by the government to do something is decisively a non-market solution, given that the government gets its money through taxes.

I can provide examples:

Panera Bread gross revenue $2.8B, McDonalds gross revenue $22.8B

Nordstrom gross revenue $14.4B, Walmart gross revenue $500.3B

Payday loan revenue in the rich burbs roughly-zero, Payday loan revenue nation wide mysteriously between $6B and $46B depending on data source

Now can you get a cool and trendy very high social status hipster job to brag about on social media in the coolest office with only the best "personality fit" coworkers helping poor people, oh heck no you'd be savaged here if you admitted to working for McD, but the market of poor people products, at least in some areas of business, is about ten times as large as rich people products.

There are a hell of a lot more poor people than there are rich people.

It’s also a hell of a lot easier to take advantage of them, given they generally lack the education or resources to protect themselves from predatory businesses. I struggle to think of McDonald’s as “helping” poor people. It’s putting food in their mouths, yes, but if McDonald’s were more concerned with the health and financial wellbeing of their customers as opposed to exploiting psychology to get as many people to impulse buy as much junk food as possible it would be an incredibly different company.

The goal of a company is not “help” its customers, the goal of a company is to exploit any and every untapped source of wealth. If it can do that by hurting someone who doesn’t know better, it is going to.

> Payday loan revenue in the rich burbs roughly-zero, Payday loan revenue nation wide mysteriously between $6B and $46B depending on data source

Payday loans may "solve a problem" but they do it in a predatory and exploitative way.

Your first statement seems to imply that the market is flawed (because, obviously, the willingness to pay for something is going to be higher for someone with lots of money than someone with little), the second seems to imply that there is something inherently beneficial in focusing on 'market solutions' as you define them. These views are rather at odds with each other.

But, I'm not trying to comment on either of those, just that you can create utility for a poor person and still have a viable business. If it relies on government for funding, because society has decided it's worth funding that sort of work, so what? We have an entire defense industry in the US based on that idea. Most of our education is too. Other countries have their entire healthcare systems based on that idea. You can decide you don't want to be paid directly by taxed money, but that's on you, not on the viability of such a model.

To be clear, I’m criticizing the common notion that the market is inherently beneficial or that it contains the solutions to all problems (though I don’t mean to accuse you of holding such views). Some problems necessarily require non-market solutions.

Mother Teresa's contributions to poor ppl of the world were minuscule compared to Bill Gates.

Don't forget the damage done to everyone (incl. poor people) by Gates' predatory business practices.

Or the damage done to everybody, particularly the poor, from MT's "suffering is Christ like" mentality.

My point is, you have to compare them entirely. At least Gates seems to have moved into the "good" quadrant later in life.

That's not the only way to make money, though. A big problem with modern corporate capitalism is how abstract everything is: you're rarely getting paid directly by someone you're doing a favor for. Now it's more like, the corporation you work for holds money in investments, which gain value as the stock prices of an actively managed portfolio of other corporations changes, and a portion of that value is paid out as salaries to employees, who work to raise the stock price, so other corporations will buy more shares of your corporation, which raises the stock price of your corporation…

How about doing some academic work?

With the explosion of NGsequencing there's no shortage of data or data-magician positions, and there's many challenges with social impact

I admit that - given my life situation - I'm biased towards jobs that at least have a chance of also letting me put bread on the table ;).

That said, I'm also having hard time separating academic work that helps things from that which just adds to the noise - grant-chasing, paper-generating studies. Any advice on that?

Well there's some noise like in everything else in life, but I don't think there's any empty studies (okay there's even a few cases of lying which is sad beyond words but thankfully they are very few i think).

Why does this noise bother you? Besides, keep in mind that the grant-chasing doesn't happen because of greed, it happens because people want to continue doing science and that's the only way to maintain an active lab.

I could shit on the state of academic publishing for days, but at the end of the day, there is an overall positive direction in this 'academic endeavor': the creation of knowledge.

I find this direction much lighter on my consciousness than "generate value for the shareholders"

Fair point.

The noise bothers me for two reasons. One, because grant-chasing to keep the lights on means you end up having to prioritize busy work over important problems, which wastes lives of researchers and slows the scientific endeavour. Two, because high noise makes it more difficult to find interesting/important results of other people's research.

Maybe I'm just wishing for unachievable perfection in this space.

> I could shit on the state of academic publishing for days, but at the end of the day, there is an overall positive direction in this 'academic endeavor': the creation of knowledge.

That I agree with, and I definitely complain more about the private market than scientific research because of that :).

> I find this direction much lighter on my consciousness than "generate value for the shareholders"

True, I feel the same.

Funny thing- I'm a bioinformatics making 70k and with a family. I'm ready to leave the space to make money. I'm at the point where I wouldn't mind something less altruistic -it's all the same uninteresting work. Crazy to read your stories of wanting to go the opposite direction.

Crazy indeed. Grass is always greener then? Your 70k in bioinformatics is something I'd drop my current job in a heartbeat for!

I'm very interesting in knowing what are your motivations to switch towards "something less altruistic". Could you share some more details? Maybe it really is that no matter what you do, no matter how meaningful, it'll always become a boring chore? Maybe I didn't have an meaningful job yet, so I'm not disillusioned?

I think the best way to describe the issue is to consider why people claim being in healthcare is bad (try googling around for whether it's worth being a doctor because you feel called to help people). Healthcare (and biotech feeds into healthcare) is very heirarchial, so advancement gets blocked. There are better ways to find meaning than the field you work in. Like all technology jobs it's viewed as grunt work compared to MD/PI/regulation visionaries. My main motivation is cash and experience that are difficult to get in an underfunded and sinking industry; genomics is really struggling to float as science progresses to new fields not out of the ninety's. I feel like someone stuck in a third world country watching America glitter into the sunset. Sure I make very interesting products but it is boiled down into the same day to day mundane work, but for less pay. We might get a thank you letter from someone thanking us for extending their life and that is awesome for someone like me who is not their doc. But I'm like a firefighter- utterly unappreciated by everyone else no matter what interesting piece I work on. And it's hard to save much for retirement. I haven't felt the void you feel. I would rather make a less drastic but equally important impacts on people in my spare time like as a coach- think of Maslow's heirarchy of needs and how healthcare inverts parts of it.

I see in this thread lots of people who have their security level established and are now looking to fill their social level. I feel inverted on the subject and let's just say Maslow's order matters.

As an academia in CS, I use two filters:

* top-tier peer-reviewed conference.

* paper abstract and introduction.

If I am not convinced that the problem is interesting by introduciton, I will skip it.

Any paper that generates or analyzes draft genomes is a waste of money.

I’ve been out of the genome-sequencing space for quite a few years now, but that seems a pretty strong assertion and I’m curious about your rationale. Is it that the analysis will need to be re-done in future? Or do you think the space is saturated?

I left the game in 2015, but even then third generation sequencing could close bacterial genomes for < $1k. Long reads make it trivial to detect all sorts of stuff (sample mixup, contamination, phage activity, in vitro evolution) that goes unnoticed with short reads. Since the conclusion of large scale draft genome sequencing papers always seems to be 'we need a bigger sample size' anyway, I feel the money would be better spent on studying individual strains of bacteria.

Ah yes, I was mostly out of it before short-reads-only genomes were a common thing. “Draft” meant a slightly different thing in the Sanger sequencing days.

This is HN, assuming you can code, you have the ability to alleviate social problems through coding either through your own startup or side projects. Don't let that fire die inside of you that wants to give back and help push humanity forward, life is too short to be miserable working for someone else (as a programmer that's in demand).

I'm not very clear on what you think a job is for. I understand a job as a way to trade your service for money and the point of money is to exchange it for other people's goods and services, like the kinds of goods and services we need to not die starving in the street, and are incredibly inefficient to provide all by ourselves. Or to send our kids to college. Or to pay our taxes so that the government can continue to provide its services. Etc.

I mean if you're a programmer and want a job fighting cancer, maybe pick up some ML skills and find a way into that industry. But short of getting your PhD in medicine or a law degree fighting injustice, we're all just a different sort of coal miner providing the things society needs to keep chugging along. And there's no shame in simply being a productive member of society. As far as societies go, we're better than all that have come before.

ML isn't really what's being used since it's too unreliable. The reigning models are all differential equations based, using ODEs and PDEs along with the clinical side using PBPK with maximum-likelihood estimation.

> But it feels to me that the revealed goal of our market-driven world is to abstract everything away with money.

Yes. The most rewarding thing one can do is to figure out a way to change that.

It's sad that our brightest minds don't want to understand this, and that they'd rather think about how to make people click ads.

But this is what kills me when companies get so hellbent in their IP assignment/agreements when bringing in a new employee that I might create something. Provide me the environment to do so at work or allow me to do so on my own time. The former isn't happening, so…

The reality of the situation is that, if I'm not so drained from the mind-numbing soul-crushing CRUD work that I feel like doing more when I get home, I'm sure as hell not going to work on anything remotely related to my employer's line of work. I get quite enough of that at work. I'm going to work on problems that matter to me, and that seems to be stuff that really doesn't matter to my employer.

I think you hit the hammer on the nail.

Boring problems are quite manageable when they are also important and meaningful. The general lack of purpose most of us find in our jobs (apart from 'earning money') is the real problem.

In my experience it starts with relationships. Co workers, partner, parents etc. Then sometimes you crush an interview at a place that already has scale and you are offered a pool of problems to work on, and you pick the most important but also probably scariest one. The types of multi year results that come from that make for cool stories.

For instance, patient outcomes from a feature you built in an electronic medical record get published in major newspapers.

Interesting work is done by open source hobbyists at night. What remains is what you get paid for.

Who wants to work on monetizing open source by working hard to create a reliable way of paying the project creators to do the interesting work as what they get paid for?

Now that sounds meaningful. Email me if you agree.

Would you mind elaborating on what you had in mind?

It's a good idea. There should be an easy way for a small team to capture the value that their open source project creates. There are a variety of crowdfunding platforms like Patreon that are doing something similar. CodeMill and Bountysource are paying developers directly for adding features.

Interesting idea.

Maybe a software repository site, where users must watch a 30 second commercial, before they can download the software. Then a percentage of the profits made from selling that commercial is shared with the programmer.

Sounds like sourceforge, but I don't know how that site is monetized, or if they share the revenues with the project creators.

I worked on that for a while. If anyone would like to join me let me know.

My overall happiness was so much better when I was working for my university's pharmacy school making websites about where to find Naloxone and heroin overdose info than currently at my huge engineering firm company making twice as much. I'd love to find a job doing simple web development for a nonprofit.

Split time. Get a side gig. I have a friend who spends 4 days a week at the dental school fixing catastrophic oral pathology problems for the uninsured, and gets paid a pittence. He spends one day a week at a posh private clinic and pays all the bills for the month in the first week of the month. The following 3-4 days of posh work in the month fund everything else.

I've been there and the grass isn't always greener.

I was in the same place 5 years ago. I've decided to quit and find something worth doing not just making more money. Since then I'm working in a support role in cancer research for not exactly faang salary but it's definitely worth it.

While I can certainly understand your sentiment, I would encourage you to not give up so easily... while it might not be so easy as to stumble upon meaningful jobs there are certainly opportunities that can be created and developed. For example, if you go an entrepreneurial route you can find interesting and meaningful challenges and start to work on them in your free time and see how to push these forward over time.

Another recommendation is to have a look at 80000hours.org which is a great resource for finding the best work opportunities out there! I think they even provide some coaching :)

While I don't agree with Marx, your sense of ennui seems to reasonably represent what I understand to have been his concepts of reification and alienation.

One might ask: what does the alternative look like? From what I can tell Marx is a bit wishy-washy on that point.

Hayek more helpfully explained that the market provides a mechanism for determining value, not virtue. That it works, but doesn't uplift. And that trying to force it to will leave with neither -- you will need to find your meaning in something else.

> the market provides a mechanism for determining value

The market's notion of value is weighted by wealth: making coffee for the rich is "valuable", meeting the basic needs of the poor is not "valuable." This makes the market notion of "value" very different from just about any other notion of value in the world. If you muddy the distinction, or let someone else muddy the distinction, markets become almost tautologically awesome, because the market notion of "value" is something that markets are very good at optimizing.

> making coffee for the rich is "valuable"

There's a lot of privilege talking about what is and is not valuable, but its worth pointing out that its only a very recent, probably very temporary, phenomena that coffee is all about rich people massively overpaying in public as an act of conspicuous consumption at Starbucks, for some decades previous McDonalds and other fast food sellers made more total money selling coffee than hipster coffee shops.

There are some market bifurcation events going on, as relates to income inequality only increasing over time. For example the number of independent bicycle shops has been dropping for decades while average revenue and profit increase... thats because the market for $2000 adult bikes is extremely high social status but also very small, leading to the inaccurate observation that "there's no money selling bikes to poor people". However, sales and revenue for poor people bikes at department stores, walmart, etc, are really pretty good, its just that you can't buy a poor person bike at a cool trendy rich hipster independent bicycle store, only from extremely uncool mass retailers. There are absolute tons and boatloads of money in selling bicycles to poor people; the money is inaccessible to the "cool kids" is the only problem.

Food deserts are a similar thing; the point of talking about food deserts is to point out the individual noticing them is a really cool person.

Perhaps a HN car analogy where nothing could be cooler on social media and more valuable than bragging about selling Tesla cars to very rich people, but you won't make even a tiny fraction as much total revenue as the incredibly uncool act of selling commuter SUVs to suburbanites.

Being cool and trendy on twitter is important, but you can make many times as much money by being very uncool. To some extent the whole meta overall topic is conspicuous consumption at its finest, along the lines of I'm rich enough and privileged enough to pretend deeply uncool (as defined on social media) market opportunities literally don't exist.

My fondness for alliteration backfired.

Hayek understood that what we think about prices and what we think about ethics are distinct things. He didn't see rightness as flowing from markets. He saw it as a discovery process for allocations of resources.

"Ennui". I like the word. Thanks!

> One might ask: what does the alternative look like? From what I can tell Marx is a bit wishy-washy on that point.

I have no clue either. But it feels to me as if we're turning humanity into a single superorganism, with each one of us being just a cell, except all those cells have brains capable of wishing they'd play on the same level as the superorganism.

Related: I've never had a libertarian bent. I'm known to argue for coordination over individual freedoms. I frequently wondered, just what is it that makes many people say that we're in chains, and (here in the West) we have so little freedoms left. I think I finally realized where this could be coming from.

I've been spending some time recently going, in my head, through a list of activities I'd find exciting. I quickly discovered that most of them are either impossible or illegal, especially when living in a city. Someone (I think pg) wrote that as we entered the industrialized age, we had to invent the whole fake, hollow thing called "exercise" to replace what we've lost in the change of lifestyles. I'm starting to see it as more general pattern - we're losing many more things we could do, and replacing them with structured activities available on the market. They're similar, but also feel fake to me.

> Hayek more helpfully explained that the market provides a mechanism for determining value, not virtue. That it works, but doesn't uplift.

That sounds about right (I suppose I should start reading the foundational texts of economic thought at some point). I do not deny that the scheme I described, in which your only role in society is to pick something profitable to do, doesn't work. It generally does (with all coordination problem caveats of the market in mind). But the meaningful work, the virtuous work, the uplifting work, seems to be distributed scarcely and randomly throughout the market space. Much like in the past, you got to be a king or a peasant depending on the lottery of birth, nowadays you can be involved in civilization-changing work, or forever toil on the margin, depending on random chance. Is there even a place for fining meaning in contributing to society? Is there any agency left as to how to contribute? Are we supposed to seek either of it?

I have no answers here. Only a topic I'm trying to think my way through, hoping to find some meaning on the other end.

Have you read Zen and the Art of Motorcycle Maintenance? You might enjoy it. A thoughtful book (semi-autobiographical) about a man searching for meaning, thinking-through-it-all while riding his motorcycle across the country with his son.

Long time ago, I dropped it somewhere near the end (random life circumstances). Thanks for reminding me of it - I recall it as being pretty insightful. I'll read it again.

The Japanese have some takes on this subject. I'm far from an expert, I only have a passing understanding, and I also have no answers, but if you were curious you might look in to Ikigai.

I don't have answers either. I just chase books and try to cobble them together.

I wish you luck.


I'll be sure to let HN know if I ever come to any lasting conclusion.

For some Marxist readings on the alternative, I recommend Marx's own Critique of the Gotha Program and Cockshott and Cotrill's Towards a New Socialism, which seeks to develop a basis for a post-capitalism society with a cybernetically planned economy and direct democracy.

Red Plenty is an interesting history of the unfollowed possibilities for these ideas in the history of the USSR.

Marx is vague and rightly so. I think the situationists took a similar approach, which is also their main criticism. They do not describe what the utopia would be not really the steps, they instead say that it's a state that occurs when people live according to their desires. That to me sounds like "do what thou wilt" where the task is to find out what our desire, our will is.

Interestingly, I remember 'Eliezer having some good insights about "how to build an utopia" in his "fun theory sequence". I read it years ago, but I started re-reading it last weekend:


If Kołakowski is to be believed, Marx's view of the post-revolutionary world has a somewhat mystical flavour stemming mostly from Hegel. It is the immediate roots in Hegelian philosophy that makes it, in practical terms, wishy-washy.

What has always tied Marxism in knots has been the tension between waiting for the inexorable laws of history on the one hand (and sometimes trying to give them a hurry up) versus on the other hand agitating for reforms. The former see the latter as delaying the revolution. The latter see the former as something between dreamers and jerks.

I think that history, having been somewhat lawless after all, makes the reformers look better.

Do you think that the more left-accelerationist factions have any meaningful influence compared to more reformist organizations like the DSA? Is it different outside the states?

I am not qualified to say.

Can relate to this 100%, at the moment I'm trying to enter teaching, for me that feels like a good focus if you aim for "interesting outcomes"

Not to sound cliché but that is resoundingly the primary symptom of late stage capitalism. Commodification of all; nothing is sacred, not even your own life. This has actually been the state of affairs for quite some time (at least from 18th century) but the ever increasing creep of wages, which previously shielded high wage positions from this reality, are becoming increasingly transparent as information technology becomes more prolific.

Thank you. That was beautifully put.

> abstract everything away with money

The thesis of capitalism is that acting only in self interest is good for the collective, i.e. your own capital is all, i.e. we can abstract everything away with money.

I think you're completely right about the problems with above dogma and hope that you don't stop looking for a way to do something you find meaningful.

Are you me?

I'm you from the future. I built a time machine out of sheer desperation in hopes of putting your (mine) brain on this topic earlier, so that we can find a way out of this state.

Step 1. Tell more people about this. As more people realize the sheer banality of the system, we may find better, interesting solutions

Step 2. Push for creating more and more social infrastructure. So if people want to, they can choose to not be part of the box

> Step 1. Tell more people about this. As more people realize the sheer banality of the system, we may find better, interesting solutions

I do that a lot on HN. I've been considering blogging on this topic, but I feel I suck at writing, and people may not want to listen to confused ramblings of a random programmer -.-

> Step 2. Push for creating more and more social infrastructure. So if people want to, they can choose to not be part of the box

I'm a noob at that. Need help.

Joining NGOs in your area is a good start. Many of these folks have good intent but lacks tech know how. You be surprised, how little can make a big difference

I was in the happy position of finding a job (post being a failed academic) doing stuff at the intersection of architecture and algorithms (high-speed regex implementation) and getting to to this for 11 years straight (2 years as a researcher, 5 years as a startup CTO, 4 years as a Principal Engineer at Intel). Got a decent if not ginormous exit too, which was nice. Also got to IMO change the industry - Hyperscan (github.com/intel/hyperscan) is used in a lot of places.

This nice run has come to an end for me and my team and it's made me realize that getting to work on nifty, complex algorithms and do real CS is not the default. To say the least.

I'd like to try to figure out how to do it again: the "algorithms startup" is a fun thing to do. You aren't going to be raking it in as having the 'nice library to do task X' doesn't allow you to capture huge amounts of value relative to the people selling boxen/UI/SAAS/etc. that wraps up your library. But you can work on a decent scale.

I am thinking about blogging about this; generally I've been sticking to tech stuff on branchfree.org so far but having done this successfully once makes me wonder if it could be done again.

> I am thinking about blogging about this

I'd probably read it.

I would read as well, please write about it!

Me too.

Well - thanks, guys. If you bookmark https://branchfree.org/ perhaps eventually I'll stop wibbling on about bits-to-indexes and pattern match algorithms and SIMD long enough to post some experiences and some thoughts on the modestly profitable world of the algorithmic startup, from an "expert" who is batting a massive 1 out of 1.

Good luck! And don't forget to submit those posts to HN when you write them!

... and done.

Submitted to HN, and on my blog at https://branchfree.org/2018/06/12/some-opinions-about-algori...

Ah! Inspiration. I'm a Sydney-sider and though I'm relatively satisfied in my current position, I'm having a difficult time finding the next job that could satisfy my computer engineering background. Software engineering has become a byword for web development/Java, (or selling your soul to work in HFT...)

Where would you like to work next?

Sydney is a relatively bleak place for getting a good selection of tech jobs. There are the obvious BigCos, then... not much. I may have to Make My Own Fun - another startup?

You could do your own startup but it’s really hard to do it in Sydney because even though the economy is doing well you don’t get the funding. What I mean by that is just not seed but all the stages required to be a behemoth.

US has it. They live and breathe tech. 5 largest companies by market cap are all Tech. They eat other tech companies for billions of dollars. Everyone and their dog in SV has a startup on the side. There are conferences on how to invest in startups. Some people just exclusively invest in startups. It’s crazy.

Sydney also has a tech brain drain. I lived in Sydney for 10 years and most of my friends moved out here too. Just better opportunities.

Sydney is such a nice place to build a startup though. Sucks that the ecosystem isn’t there. Vancouver has a similar problem too.

There's definitely way more opportunity in SV - obviously. And the tech brain drain is both overseas and to the disproportionately huge local players - Google HQ in the valley is one player among many big names; Google Sydney is an insane behemoth hoovering up the talent.

It is a nice place to live, though, and the Atlassian guys proved it's not impossible to build a pretty kick-ass startup here. There's also a pretty decent pipeline of smart people around. Part of my hope is that you can do smaller-scale stuff here without the Go Huge Or Go Home mentality of SV. What would be interesting is to work on things that interest me, make some money and influence the field. I don't think SV is mandatory for that.

I am thinking about blogging about this;

Yes, please do.

> but having done this successfully once makes me wonder if it could be done again

I think it's all about the motivation and setting your expectations that you won't make it every time.

This sounds like Starbucks, coffee and coffee beans.

We are developing a matter compiler. We are working on designing molecular Lego to assemble them into new catalysts, therapeutic compounds, atomically precise membranes and sensors for proteins and small molecules. The goal is to make molecules that can do everything that proteins can do and that can be designed by human beings with computer assistance. We've developed a programming environment with a custom compiler and Common Lisp implementation for designing these molecules. We are looking for programmers with chemistry experience. https://www.youtube.com/watch?v=8X69_42Mj-g

I've been to your talk in Kraków on the European Lisp Symposium! I love what you're doing!

Do you have any updates on that water-filtering molecule you showed us then?

Yes - but we aren't ready to publish yet.

Great to hear; I'm now impatiently waiting for new papers or related talks! Don't forget to post it here when you publish!

That's very cool. How do you debug these programs? What is the molecular equivalent of "format"?

That was a very interesting presentation. Great work, is there any way to follow this (besides following your wok on clasp)? You mentioned carbon dioxide. Do you have some ideas about molecular motors and creating solid carbon out of carbon dioxide? Or hydrocarbons?

Does one have to be a graduate student in chemistry?

do you do remote?

It depends on the skills and our needs.

Can I send my resume to you?

I generally like what I do.

I stay away from webdev. I did a couple of small projects but the web was just an outer API, the main work was done by native code running on CPUs or GPUs.

I stay away from enterprise systems, or fin.tech, or ad.tech. I removed enterprise-targeted keywords from my resume, Java, J2EE, SOAP, SQL. I have some experience with them from my early career but I don’t want to use them anymore.

Also, for many years I’ve never worked in/for huge companies, i.e. google/amazon/microsoft scale. My employers/clients are/were mostly small to medium companies.

What do I do? Almost everything else. Native desktop, native mobile, embedded, GPGPU, robotics, GIS, videogames, CAD/CAM, streaming media. Many startups, and small to medium companies, have problems to solve that I find interesting to work on.

There’re still boring parts in my work. But because of my experience I usually find ways to minimize the time I’m spending on them. The most generic advice, pick the right tools.

You have to be really GOOD at your field to have this freedom. Good for you, dude.

I also like to abstract away the non-sense part of a problem, so that I can focus on the real intellectual puzzles.

> I also like to abstract away the non-sense part of a problem

Some boring problems need to be solved anyway, and they can’t be abstracted away. Just when you’re picking the right tools, it shouldn’t take too much time.

For example, that CAD/CAM app I’ve developed has a lot of very interesting algorithms inside. But the software still need to appear on the user’s PCs somehow, i.e. we needed an installer. I’ve picked http://wixtoolset.org/ It’s mature, well supported, first-party i.e. made by Microsoft, open source, integrates nicely into visual studio. Because of these things, I’ve spent less than a week in total over the course of 1.5 years working (=develop, test, and support combined) on the relatively boring installer part of that software.

The installer was just an example, many commonly encountered boring problems already have good solutions. E.g. need RPC, pick .NET and WCF. Need to store structured data, pick the highest level library that fits other requirements (despite I generally don’t like SQL, I still know it and I’ve used sqlite a couple of times when it fit well). Need to expose JSON HTTP API on MacOS without much performance constraints, pick Python + CherryPy, the Python is sufficiently high level and it’s already installed there. Without picking the right tools, I’d spent a lot of time solving these boring problems.

Yeah a good point.

I think the issue here is order: solving which first. Algorithms motivate you, Handling interface demotivates. So start with the motivator.

I work at a boring company but I manage to get interesting projects to work on. I left CRUD behind about 5 years ago and I couldn't go back now. I just don't find most business logic problems interesting enough.

The transition for me came when I started a new job and my first project was to fix a slow batch job. Then I managed to get a second performance improvement project and soon I was the performance guy focusing entirely on performance projects. I really loved it.

Then through side projects targeted at problems I saw at the company, I built valuable tools using new open source tech and I got made tech lead of a new platforms team, solely focused on resolving technical problems and improving the system. Since then I have been an R+D engineer and now starting as a big data developer at the same company.

I am no genius, failed my interviews to get into big tech companies. I am a pretty big fish in a pretty small pond, I am known as one of the best technical guys in the company, which opens many doors and gives me a lot of freedom to move about inside the company and get some of the most interesting technically focused projects.

I think that actually "boring" companies with "boring" products often have more interesting problems.

It took several years but I finally found my dream job.

I work as a software engineer for an ISP. We're a startup and so of course our mission is to disrupt the industry. The great part is we're actually doing it!

Fun stuff I've done so far:

- Wrote a library to find the best available subnet given our set of IP blocks, set of existing subnets, and the desired subnet size. This may not sound like a big deal, but we've shown our IP block utilization to other ISPs and they are blown away that we achieve a perfect 100% utilization of each block. (Typically at ISPs of our size this is done manually and so there tend to be a couple leftover subnets in each block which are too small to use).

- Wrote a monitoring system for OLT hardware which A) does not provide out-of-the-box monitoring and B) does not have an API. We actually managed to build a robust system on top of its CLI interface. This is huge for us. If there are hardware or fiber problems, or if a customer simply unplugs their modem, we know instantly. It can even pinpoint where the problem is.

- Wrote a system which is able to talk to our OLTs, core switches, core routers, and customer modems to do things like provisioning or upgrading/downgrading our customers' bandwidth. Again, there is nothing even close to an out-of-the-box solution for this. You have to build real software!

- We have some upcoming products I can't share, but one of the most exciting things right now is that we are focusing on helping our customers troubleshoot their networks. This is a surprisingly complicated thing to do while providing a top-notch customer experience, and we think software is part of the answer.

So, do I feel like I've been using "computer science" knowledge? Not really. Closest thing would be using binary trees to do subnet stuff.

But what I have been able to do is flex my software engineering muscles, go back to first principles, and design software from scratch. No CRUD, no huge frameworks and package managers, just plain old fashioned programming.

(Ok, we do have CRUD and framework stuff. The web browsers have to talk to something!)

Wow you've done some great stuff. I work in the same industry in a company that has been around since 2007 and we don't have anything remotely similar to provisioning systems for client or backbone equipment, everything is done manually.

What kind of equipment do you use mainly ? Cisco, Juniper... ?

I am not sure whether we share that publicly or not, so I'm going to be conservative and not share it here. But we are hiring! We'd love to talk.

Here is our web site: https://pilotfiber.com/

Here is our Who is Hiring post (contains my email):


I'm hesitant to post because there is so much anguish here about winning the lottery for interesting work.

I have been extremely fortunate in my career. I'd say I won the interesting lottery. I started very early by today's standards, taking my first programming class at the local university when I was in 5th grade, back in '76.

At 17, in '82, I had a game company selling nationwide in Sears & KMart a few Vic-20 games I wrote. That got me a beta testing role with the unreleased Macintosh. Which got me to move to Boston, where I worked for Mandelbrot & DeVenney on the initial publishing of Fractals.

Basically, I've been fortunate to be an early developer for a series of advanced technologies: - initial 3D graphics research community (back when the question was "how to do 3D graphics at all"), - initial GUI developments (before Windows 3.1, there were a lot of competitors), - early real time mapping / navigation, I worked on one as my first post graduation job - early streaming media, I was writing media players for CD-I/CDROM/Network back in '90-'92 - early 3D gaming consoles, I was an OS developer for the 3DO and the first PlayStation - early 3D games, lead developer for games for 15 years - VFX, left games and worked on 9 VFX heavy feature films at an academy award winning studio - startup, left VFX and spent 7 years doing a ML based 3D reconstruction of photos to realistic 3D avatar web service - Facial Recognition, closed my startup and now I develop facial recognition software for an industry leader

There is not time enough in a day to explore all the stuff I've done or am interesting in further pursuing. Luckily, I am one of those that sleep less: 5 hours leaves me groggy, and 4 is just right. So I spend my daze 10 hours coding for work, and another 4-6 pursuing "mad science".

Wow, it sounds like you’ve had a fascinating career. You say you may have just been lucky to get in on the ground floor as it were, but would you attribute your success to any other habits, behaviors, or personal qualities?

I'm enthusiastic, almost to a childish degree, and I obsess. The joke at work is I'm "Sheldon", and I don't watch that show so I'm only vaguely aware that he's some type of bad communicating nerd. I think I communicate better. I'm also somewhat fearless, accepting projects over my head and still delivering.

Wow pretty amazing. A little OT but just reading about coding 10 hours a day is just making me think of back pain.

Is it really possible to sit for that long everyday and not suffer from it. I know I'm probably focusing on the wrong thing but just the idea of being able to code for that long everyday without suffering any physical problems would be reward enough for me.

I actually code closer to 14-16 hours a day. But I have a standing desk / bar stool combination, plus a compile of my work typically takes 3-4 minutes, so I have a yoga mat, weights and stuff to work out while I think & wait. I'm 54, yet look 40 with a near six-pack. I'm really lucky.

Wow man that's really awesome. Would you mind doing a blog post about it sometime (or maybe linking to the source where i can learn more about it).

It can help a lot of people like me who are struggling to work even 5 hours a day at just 37.

Still thanks for the reply, I am now seriously thinking of getting a standing desk.

My #1 advice for being a healthy developer is to observe a regular hard cardio workout.

<back story> When I was 16 I was a fairly muscle bound half geek / half athlete. To give you an idea, I was a running back on my high school football team, and a tackle and two linemen and I would have to shampoo one anothers head's after weight lifting practice because our arms could not reach the top of our heads from being so buff. We were muscle bound jocks in Iowa, what can I say...

Well, I had a bad tackle when running with the ball during a game, and I crushed 3 vertebra, doubling over backwards with my hips pushed backwards till they touched my shoulders. The stadium had a collective gasp and then silence as I was seen on the field broken. Long story short, bolts in my temples and hanging vertically for 6 weeks healed my young spine with only weather changing aches. However, I went from 230 lbs to 130 lbs, and it took another 6 weeks of therapy before I could walk.

That was the start of a decade of no athletic activity and nihilistic punk rockerism mixed with quite the intellectual journey as I left Iowa and became a part of the Harvard/MIT/BU 3D graphics research community. (This was the early-mid 1980's)

It was not until I was 30 that my then girlfriend, now wife, got me to start morning walks, then light jogs, and finally an actual return to an athletic lifestyle. To give you an idea, at 30 I could not jog a block without heavy panting and sweats. After a year I was able to jog 1 block. It took whole year to do a block without dying! During that year, I'd get delirious and throw up if I pushed. But the persistent and incredible patience of my girlfriend slowly got me back into what is considered "normal" shape.

Once I'd refreshed my previously athletic muscle / body memory, all kinds of positives kicked in like a general optimism, a general active energy level, and greater interest in everything in general.

However, this was tempered with realization that if I slacked for even a few days, I'd get depressed, and if I went a week I'd start having bone/joint chiropractic misalignment and then I'd be injured and unable to be active at all.

So, now I have a strict 3 days (at least) per week hard cardio. I ride my bike about 8 miles, living near a bike path in the San Fernando Valley. Plus I have a yoga mat next to my standing desk. I need to get a newer computer, as a recompile of my work take 10-12 minutes. But I exploit that time working out, so maybe I don't want a new development computer...

wow.. first of all thank you so much for taking the time to write that out. I found it super motivational and tbh it really means a lot to me that you are fit and fine after what you've been through during your younger years, and the kind of pain you were in at your 30s.

I've never had any injuries but doing long hours siting all day without exercise has taken a huge toll on my back. I'm not overweight but my lifestyle is very sedintary :/ Long story short lately I've have my issues become so much that I've had trouble with simple things like putting on pants, getting in and out of the car, sometimes even sleeping. Have done lots of mris, xrays and months of physio and pain keeps coming back. Also religiously did the exercises prescribed by my doctor but nothing is helping.

I'm kind of like at that point in your 30s where even jogging for a block is a nightmare for me. But reading your story gives me hope. I'm 37 and just learning that taking health forgranted for work has been a big mistake.

I will try to research some cardio exercises that won't aggravate my pain and get started. I love your idea about doing things during code compilation. What a blessing in disguise that is. I already started that today doing some SI joint exercises that I can do while sitting. It's a start and I now to get involved with beginner cardio next.

Another aspect that has helped a lot is taking a b-complex vitamin every day. B-Complex has two major benefits: it aids in the energy recovery during food metabolism, maintaining one's energy level, but the major benefit for developers is the reduction in nerve sheath inflammation. There are times when I am doing some heavy, redundant motions, like when working on 3D animations, the repetitive mouse use for multiple days can wreak my wrist and that "mouse fatigue" spot in your upper back. When I know I'm going to be abusing my wrists like that, I make sure to have B-Complex each day and for a few days after - for the reduction in inflammation and the resulting continued wrist health.

I appreciate your thanks. I could write a book about my career and maintaining health in both physical and mental aspects - being a developer can be a hell of a mental game too. If I were not so busy writing code, I probably would. Developing my ability to live like this has taken decades, and now in my middle 50's I am peaking in multiple aspects: I know how to code an immense number of highly complex things, my mental game is such that I am coding like water pouring, and my work is offering me the freedom to use everything in the creation of a new flagship application for the company. I'm simply having too much fun and getting too much done to write about it at length. I do post on Quora.com quite a bit though.

Great tip about B complex. I didn't know it could heal muscle fatigue and makes healing faster. This is great tip for almost all programmers who risk RSI at some point in their life.

Anyway yes it no ordinary thing you're in such good health and are putting 10 hours a day at 50 years while some of us are struggling lot younger.

I'm sure you already have done quite a bit of research that lays ahead of me. A book does sound like a big commitment but if you have a blog I'd surely hope you would put your story and what you did to fix yourself and how you're so productive.

A lot of people would be inspired and helped by it I'm sure.

P.S. just read this book called 12 rules for life (huge bestseller) and the author starts it by saying so this book started as a result of my quora post :)



What was your first computer?

I first had access to a PDP-1170 at my local university, then a borrowed Apple II, then my first computer was a Vic-20. Then a C-64, then the beta Mac.

Do you think if you had not started with a 16-bit computer, your learning would have progressed differently?

I know this is a highly speculative question.

The Apple II and Vic-20 were both 8-bit computers. The minis and mainframes at universities were 16 and 32-bit systems, but for the large part I just ignored the word-length of a system. I really only cared if I needed to work in assembler, which I preferred for a good 15 years. I generally work in C flavored C++, and don't really optimize via process bit knowledge until bottlenecks are being removed.

Pretty much everyone is in the same boat. Even if you have an advanced degree, most of what businesses need is plugging together components and adding business rules to create a product.

As a discipline, we're too young and our technology is too immature to support a true split between engineering/science and computer programming.

If you want to work on interesting problems you're passionate about, there's a few ways to do it.

First, you can do research. Years ago I joined a lab as a professional researcher to follow my passion for machine learning and sensor networks. It's not the easiest path, but labs need disciplined programmers to complement the skills of PhD students.

Second, you can consult. If you're passionate enough about a topic that you have encyclopedic knowledge, I guarantee there's someone out there who will need that expertise and will pay for it. It might not cover all your bills, but it gets you exposure (and potentially a job if they need your skills a lot).

Third, you can start a company. I followed my passion for the challenges of learning from sensor data at scale into entrepreneurship and eventually started a company, http://sentenai.com . As a technical founder, I chose carefully the technologies we use (Haskell) and the problem we were tackling (making it fast and natural to combine multiple sources of time series data and find temporal patterns). You still need to solve somebody's problem and build a business, but you can at least lay the foundation.

Nice, I'm building something similar with TTN - using Haskell to glue MQTT/SQL/WebSockets to be able to stream data from sensors and also optionally store data in timescaledb. For web fronted I really want to try miso (https://haskell-miso.org/).

Unless you're targeting consumer you might want to be careful with tech like web-sockets. After a decade of using sensors in industrial facilities I've come to appreciate the value of on-device buffers and transactional logging systems that don't maintain persistent connections. Real-world environments are unfortunately really tough on technologies designed for browsers.

You sound like you have some experience in this area so I'll ask. I'm working with fielded devices that typically sit on enterprise networks. We operate much like an IoT device, but without the hardware constraints. The current c2 system uses a traditional polling REST HTTP protocol and we're looking to migrate something more real-time (for config pushes) and lighter on the server. Currently looking at everything from HTTP/2 SSE, Websockets, MQTT over WS, or even HTTP using something like Google's Cloud IoT. As you can probably tell we'd like to keep the wire protocol enterprise-network friendly. Any advice?

It really depends on your edge data model, but the enterprise network should always be considered hostile.

At my previous company we ran SSH servers on our gateways for server->client pushes and it ran into the fewest problems with enterprise firewalls. We also ran our own mesh wireless networks a lot using our own gear. I'm guessing that's probably not an option, but when there's no admin running around looking for unauthorized wireless signals it's quite efficient.

Now all of these require a local host machine (or several) behind the firewall that do the brunt of routing and orchestration for pushes. There's a good reason for it though: It's a lot easier to get several hardened systems authorized to cross the firewall than a bunch of IoT devices. And if you can't get that authorization, you can put a cellular connection in the box or boxes more cheaply than one in every deployed device.

Happy to chat more if you want specific advice about certain situations. I also spend a lot of time investing in and mentoring Boston-based IoT startups so I have seen a broad cross-section of situations that can happen. brendan@sentenai.com

We have the full support of the network admin staff, so punching out is not an issue, we just want to keep it as clean as possible. My guess is our control channel will likely end up being some combination of WSS/Redis (if it plays nice w/ mutual auth and load balancing) or a variation of our current polling HTTP/REST solution. Thanks for the offer, I might take you up on it ;)

I'm totally in the you-dont-really-need-web for everything group but yes, this can be considered targeting consumers as-in ordinary people who want to use localized version of TTN.

I don't really approve use-cases like running a webserver to control lights on ESP8266 where simple TCP sockets would do.

Btw I've written an implementation of ZRE protocol which you might possibly find interesting - https://github.com/vpsfreecz/haskell-zre

Haskell-zre is pretty cool! I think I'll have some fun playing with it with some of our demo IoT setups. We're purely in the backend data infrastructure business so I don't do deployments any longer, but we'll probably have some fun with it anyway :)

Miso is great. Also checkout reflex; and the recently open-sourced https://github.com/obsidiansystems/obelisk

I'm working on a team that is miniaturizing a synthetic aperture radar system.

The amplifiers, processing, and data storage used to take up most of the space of a small cargo plane. Then it got down to four half-racks. Now it is the size of a dishwasher.

We hope to get it drone-sized. It is a never-ending series of interesting technical problems that span the entire STEM range, including RF, network, system, electrical and mechanical engineering, HPC, data science, voodoo-mathematics I will never understand, system administration, and plain old software development.

Everyone here is expected to cross-train in various disciplines. My background is CS but since starting here I have done electrical and mechanical engineering work including diagnosing micro-fractures, visible only through a microscope, in connectors on a bunch of VPX chassis stuff, caused by bad soldering work by the manufacturer.

On top of all of that once or twice a year we all get to go on operational deployments of the system. This year we went out and imaged thousands of miles of coastline, 6 engineers crammed in a tiny airplane hopping from airstrip to airstrip in support of a coastal erosion monitoring program. Data scientists are expected to haul around amplifiers and radar engineers are expected to live patch code while in the air.

The only reason I got this job was because I was a bored programmer teaching a Linux course as a side-hustle and a bunch of engineers came through and started asking questions way beyond the scope of the class. They were radar guys struggling to learn Linux and a couple of post-class drinks later I had an interview.

I guess my only advice would be to have a side-gig and network as much as possible, peppering everyone you meet with questions about what they do.

If what they do sounds cool, ask for an interview.

Do you have a web site for your company? Your current work sounds similar to the work I did in my graduate studies in Ocean Engineering. Thanks.

FPGAs all the way?

Capella Space?

I'm in the exploratory stages of starting a noncommercial project focused on hosting, processing and serving a vast amount of climate data for scientists and researchers. I've "seeded" this project with my own bare metal hardware, electricity and internet access. This includes 136 vCPUs, 512GB of RAM and 128TB of hard drive storage capacity (with room for another 456TB). As my own money permits, I plan to slowly add to this lab over time.

My initial consideration is to develop and host a sort of public Wolfram Alpha, but specialized for climate data queries and visualization. I'm open to other suggestions, as I'm currently in the process of architecting the system and reaching out to climate scientists for feedback on what they need that isn't well-served yet.

Once all the infrastructure work is done, I'd be happy to contribute to climate modeling more theoretically. But for now my prerogative is to develop a central repository for as much climate data as possible, normalize it as reasonable, and store it in a queryable format.

There are interesting technical problems here due to the scale of the data, the variety of its native (raw) formats, the frequency with which each source updates and the types of information involved. But speaking directly to the title of this post: I don't think anything I'm doing is strictly an unsolved technical problem, it's just a complex technical undertaking. There are plenty of data serving and processing pipelines for this that have already been proven capable.

This sounds similar to Radiant Earth https://www.radiant.earth/ They are also spearheading the Cloud Optimized GeoTIFF project http://www.cogeo.org/ and the SpatioTemporal Asset Catalog spec https://github.com/radiantearth/stac-spec.

Have you seen https://github.com/pangeo-data/pangeo? It's pretty amazing.

There's still a lot left to figure out, even storage formats aren't fully solved.

Hey, I'd love to help out in any way I can. I'm a data scientist by training, also involved in a startup that helps people learn data science (with a pretty popular data science blog too).

I love the focus. Starting at the base level data and working to normalize it is definitely the most difficult piece. How far back have you assembled data?

Feel free to reach out to me individually (email in profile), I'm working on operationalizing climate data like this and would like to look at this together!

Your email is not in your profile :)

Where are you getting this weather data?

All over. It's not strictly weather data, either. The majority of the data comes from programs within DSCOVR, LPDAAC, NASA, NOAA, NCDC, USGS, ARGO, CDC, CDIAC ORNL, DOE, EIA, EPA, FDA, EOSWEB, USFS, PHMSA and USDA. The snapshots I have are already around 50TB; I'm working on setting up updates at daily resolution for each of these sources, documenting it all and transforming it into something queryable.

Is there a possibility that there is just too much data for one person to handle in a meaningful way?

Of course, that may very well be a possibility. In that case I'll have to scale up the effort.

Wow :)

High-level questions:

- At the risk of birthing a page-long subthread on ZFS-vs-everything-else... what storage solution are you using, and why?

- What sort of hardware are you using? (This is a non-catalyzing question, and is just out of curiosity)

- How did this get started, and how are you managing this?

- Will you ever be interested in accepting donations or funding? (Including on a voluntary basis; and including with clear stipulations/structure about management)

- What sorts of decisions/motivated this initiative?

More focused:

- Do you have any interest in creating this as a "community hub", with a central code repo that data scientists can push updates to that then get run on the cluster? If the visualization data (or prerendered bits of it) are openly cached/accessible, having the code that generated the data equivalently open/available could be interesting.

- What kind of availability/openness are you looking at with data? (TL;DR translation: rate limiting)

You may already be aware (very likely), but AFAIK archive.org is interested in this kind of thing. Interesting bunch of people, but the various projects do like climate data, and they have a few hundred PB of space, FWIW.

Take a look at quiltdata.com (open source), YC S17

Thanks for expressing interest...I'll respond in order.

> - At the risk of birthing a page-long subthread on ZFS-vs-everything-else... what storage solution are you using, and why?

Right now, XFS and HDFS. This is primarily due to my prior experience with it; I have considered ZFS, and might move to that later on. I may also dispense with HDFS for shared storage and to reduce the redundant JVM memory footprint. I'm attracted to ZFS for compression and error correction features, I just don't have any actual experience using it.

> - What sort of hardware are you using? (This is a non-catalyzing question, and is just out of curiosity)

Four bare metal servers, three of which have dual Xeon E5-2630 V4s (10 core, 20 thread 2.2 Ghz), one of which has an i7-6900K (8 core, 16 thread 3.2 Ghz). The latter server has 10TB SATA SSD capacity and four GTX 1080 GPUs. The other three have 2TB of NVMe SSD capacity. Each of the four has 128GB DDR4 2400 Mhz RAM. Currently all storage is local to the hardware, but there is a SuperMicro SC847 for future expansion. Every server has dual 10Gb/s SFP+ connections in bonded 802.3ad LACP, and are connected via a 16 port 10G network switch in the same rack. I think that's everything at a high level, off the top of my head.

> - How did this get started, and how are you managing this?

I joined John Baez and a few other scientists/researchers working on the Azimuth Climate Project, which sought to preserve critical snapshots of climate data in the event of mass defunding. Later on I decided to take this a step further since those snapshots were becoming very out of date and the multifarious data repositories were never very well documented, organized or normalized. Not sure what you mean by "managing this" though.

> - Will you ever be interested in accepting donations or funding? (Including on a voluntary basis; and including with clear stipulations/structure about management)

I initiated the process of starting a formal nonprofit, but not for the purpose of soliciting donations; rather, just so that it would be clear it's a noncommercial activity. I might be interested in that kind of thing - in either way you mentioned - once I have exhausted my own reasonable resources for the task and have to significantly expand.

> - What sorts of decisions/motivated this initiative?

It began with reading worrydream's blog post, "What can a technologist do about climate change?": http://worrydream.com/ClimateChange/. Not much more to it than that.

> - Do you have any interest in creating this as a "community hub", with a central code repo that data scientists can push updates to that then get run on the cluster? If the visualization data (or prerendered bits of it) are openly cached/accessible, having the code that generated the data equivalently open/available could be interesting.

Yes, that's an interesting idea.

> - What kind of availability/openness are you looking at with data? (TL;DR translation: rate limiting)

Well everything is intended to be extremely transparent, so I'm going to open source all software (infrastructure, development, research, etc). I'm also going to keep all data open. In practice there will probably be a limit of 1000 or so queries per day per IP address, but I'll burn that bridge when I get to it. It really depends on how "real time" queries end up being, and how much abuse the system actually receives.

Thanks for the reply! Not sure if you'll see this; had some unexpected delays finishing this comment.

Everything is duly noted; I'll just expand on some bits.

> I joined John Baez and a few other scientists/researchers working on the Azimuth Climate Project, which sought to preserve critical snapshots of climate data in the event of mass defunding. Later on I decided to take this a step further since those snapshots were becoming very out of date and the multifarious data repositories were never very well documented, organized or normalized.

Archive.org is very probably interested in this sort of thing then, FWIW.

> Not sure what you mean by "managing this" though.

Heh :) I was curious what sort of work you do in order for this initiative, which appears(?) to be somewhat of a side project, to be viable in terms of budget and adequate spare time. I'm also interested in storing/working with somewhat large amounts of data too (one project I want to try at some point is implementing an infinite browser cache so I can "Google" the content of every version of every webpage I'd ever visited), so I definitely want to optimize for something that doesn't require much time :) (don't we all)

> It began with reading worrydream's blog post, "What can a technologist do about climate change?" ... Not much more to it than that.

After having read that page I now understand the sentiment of that statement. Major TIL :)

And if only all webpages were that well designed... (The interactivity was a bit of an information overload, but I really liked the layout.)

If there's a bottleneck anywhere in what you're working on, that's the most likely place for you to apply CS. But it's a tricky business; there's a risk benefit trade-off.

Writing complex code to solve a gnarly problem is often the wrong path, because it's harder for coworkers to understand the code, and harder for future maintainers to keep it correct. Thus it's often better to encapsulate the hard problem in a nice interface, and possibly release it as a separate library.

That's where most of those libraries you're gluing together come from. A tough bit of code that needed writing but also encapsulation.

A tougher problem than most CS problems you'll find while writing CRUD apps is getting architecture right, so that adding new features isn't an uphill struggle and doesn't involve piles of boilerplate. Getting the aesthetics right, making the code clear so that anyone can maintain it - this requires experience and the mastery that comes from it pays off in the longer term, because you can more easily leverage other people's efforts too.

If you can't find solace in mastering software design, you're better off moving away from CRUD apps and move to a developer of tools. Tooling, whether it's compilers, OSes, databases, whatever - has more interesting and deep technical challenges. Watch out though: tooling often doesn't pay particularly well because developers enjoy making tools themselves. They'll prefer to build their own shoddy hammer 9 times out of 10, precisely because they're in the same boat as you right now: they don't feel challenged.

I think with time you may find such drudgery is present in every field. Ninety percent of any kind of work we do is gluing together components and solving boring business needs. It's part of the reason I think we engineers love side projects so much -- they're easy to start up, and you can try out academically interesting things (obscure programming languages, cool algorithms, etc.) without caring about the bottom line.

Side projects are good for your brain and in the long-term good for your career (lest you stagnate with old technologies). I recommend having this outlet. On the plus side, having a somewhat interesting job + really interesting side projects gives you both good money and intellectual freedom.

If you want to solely focus on interesting technical problems, I'd recommend working in a research lab or going to academia. On the downside, you may lose that nice paycheck, but your curiosity will remain forever piqued!

> If you want to solely focus on interesting technical problems, I'd recommend working in a research lab or going to academia. On the downside, you may lose that nice paycheck, but your curiosity will remain forever piqued!

If you join an industrial lab, rather than a lab in a university, you can keep the pay but still do the same thing as the academics.

90% would be fine. The problem is that it's 100%.

Wow, a surprising number of responses agreeing with boredom... I'll share a different point of view.

I've been running DNSFilter for 3 years. We're (still) facing interesting texhnical challenges - ranging from BGP community strings being used to balance a global anycast network, to handling volumes of time series data [1], to designing a golang daemon to ssl sign incoming requests on the fly for domains we won't know ahead of time, to figuring out new ways to pass data in DNS, and support cutting edge standards like DNS over TLS, DNS over HTTPS.

I was just saying to my wife the other day how ridiculous it is that most things we do are technically challenging, and you have to think outside the box how to get it done -- while I was implementing a block page for user agents which requires a 19 step process to determine the user's DNS Agent ID, and display the right logo on the block page ;)

Some might dismiss these challenges due to our use (abuse) of DNS for purposes it was not intended for. Fair. Also, welcome to the real world, and we're solving problems are customers are asking for... and just figuring out interesting ways to get the job done.

1: https://blog.dnsfilter.com/3-billion-time-series-data-points...

My suggestion : forget you CS degree, drop the idea of doing something interesting during your work hours, think about it as inevitable procedure to earn your montly salary. At 5pm run away from your office and focus on whatever you think is interesting.

Is this view that common?

I'm all for work-life balance and working on passion projects on the side.

But do that many people really think of their developer job as a salary and a daily grind?

I work on "boring" stuff (CRUD, frontend and backend) and love it.

Maybe because I remember all to well what jobs I used to have:

- farm hand (heavy work on smaller less automated farms)

- construction (noisy, wet, dusty etc)

- security

- help desk

Being paid 8 hours a day to work on borderline interesting tasks, getting free lunch, being paid to study etc feels great. Oh, and I can work from home if I need to.

I also make my work interesting by putting some effort into it.

How do you get over the chaos that is front end (hip frameworks every week).

Agree this is a problem. Here's how we make it work:

Work with great people. Also code reviews make sure new dependencies don't usually in unless the team agrees (we are a reasonable bunch).

Standardized on Angular. It seems to me Angular has less churn. Yes: there is a life cycle and we have to follow the releases but it is mostly well documented.

For me it doesn’t seem a daily grind. When I am actually writing software, I completely love it, even after 14 years of doing it professionally full time (and several years part time during uni). The only parts that I never liked were the bureaucratic bullshit, production support even at insane hours of the day, and the politics with your colleagues. Usually those were a relatively small part (20-40%) of the total and I still could enjoy writing good software the rest of the time. Now finally contracting I’m in a position were my coding skills are really appreciated and I enjoy my time coding and helping my colleagues without all the bullshit. Maybe I don’t score high in the interesting jobs lottery, but I’m perfectly happy of the pretty good amount of money that I get doing something that I love in old, boring, huge companies.

A job doesn't have to be a calling, and a job that isn't a calling doesn't have to be a soul-crushing grind.

Most developer jobs are for CRUD apps, and you'll rarely be doing greenfield development. There's not a whole lot of room for passion there. If you just treat it as a job, though, rather than something you have to force yourself to be passionate about, it doesn't need to be that interesting. Once you make a living, you can satisfy your creative urges elsewhere.

I've always wondered who the magical developers are that did the greenfield development of every project I've been on-boarded to. I never seem to get ahead of the maintenance grind to the point of being one of those creating the mess instead of wading through the mess.

One way to get ahead of it is to try pitching proposals to problems your organization can't afford to ignore.

Yeah me too. But looking through the doco i keep seeing the names of people on all the green field work are the same.

I think those people live the work life equivalent of hitting all green lights.

I do. I've only had two jobs I truly enjoyed doing the 9-5 work for, all the rest have been a slog to get through to pay for my other interests and family-related stuff.

i like to work on my garden.

Where's the social life when doing this?

C'mon we are almost at the point when there will be no social life but virtually connected individuals exchanging their personal data for likes and views.

Computation has made everyone able to connect with each other with practically no barriers. AI will make connection with machines and people seamless. These are super powers.

There is absolutely no reason to be miserable at your day job. You should be preparing for superpowers if not actively using them.

I recommend watching lectures/talks during work, skipping meetings and reading from thought leaders instead of working hard to reach deadlines. Everyone is connected which means you can read and watch what Ian Goodfellow has to say about unsupervised learning or see what the issues with the world computer are from Vitalik himself as easily as listening to your marketing department figure out how to grow 10% in the quarter.

Lol that’s easy enough to say when your family doesn’t depend on you to feed them.

Your family is counting on you being prepared for the future.

I have a degree in CS and one in Physics.

For the first 7 years I worked as a "programmer", ie. writing code, a lot of C++, and of course web stuff. About 6 years ago I (accidentally) became a "data guy". I do a lot of data infra/engineering, a lot of stats and analytics, and lately ML. Also I do a lot of management stuff and hiring.

I'm lucky, this is a pretty sweet time to be working for me. My skills are a perfect match for the work, I'm able to do impactful work and can get paid for it.

I don't really care about interesting technical problems that much, in the sense that I don't want to solve an interesting technical problem just because it's interesting. What matters is having an impact on your users and the business.

I think if you're looking for those types of technical challenges, it's better to contribute to open source code, rather than doing it at the day job. At the day job, you should always choose the method which makes sense to move fast..

Academics: just take an interesting problem that you can generate potential solutions for, but if you can't, just pick the next problem.

Business: here is a continuous stream of problems, most of them are boring, some of them are hard, but you have to solve them all regardless.

That's tenured academics. A lot of academics are still struggling to make a name for themselves, which means more like: pick an interesting problem. If it gets any attention whatsoever, work on it and its variations for years. Maybe drill down into increasingly niche subsets of that problem. If at some point it appears there is no solution, make one up and defend it anyway, or add unrealistic constraints that will never be generally applicable, then promote the hell out of whatever you've got.

Business: take the easiest most profitable problem and solve it... Which is generally a boring problem.

I've left master CS studies to be able to do robotics and be free to learn stuff I want/need. After number of years hacking stuff all around in local hackerspace I'm finally getting onto something which is an attempt to create distributed motion control system (https://distrap.org) which is also one huge experiment with generating code from Haskell utilizing Ivory Tower framework made by Galois (https://ivorylang.org/). During the process I've realized the potential of this framework and started writing a book about it (http://48.io/~rmarko/book/) so people don't have to spend half a year reverse engineering it.

Beside that we're also building vpsAdminOS (https://vpsadminos.org/) for running LXC on top ZFS. At https://vpsfree.cz we currently run openVZ with the intention to migrate to vpsAdminOS which is currently tested by members.

In my experience you gotta sacrifice stuff (like well paying job) so you can do stuff you really want.

Hmm, maybe. I work in Natural Language Processing for medicine, and there are many interesting challenges ... a lot of them still academic even. But the biggest thing I'm worrying about these days are more business oriented. Machine Learning, and Natural Language Processing in particular, is nowhere near advanced or accurate enough to fully automate things. So most of my thinking goes into things like "given the limited abilities and limited data, how can we still make something that is useful". That's both a business and a UI/UX problem...which suck up a lot of brain cycles for me. Up to the point that I've become somewhat cynical and wish I was doing CRUD work instead ... since all the hype around NLP usually amounts to saying "yeah no, that's not really how it works, and you'll [the customer] be disappointed if we actually did that". The hype train is both a blessing and a curse

Technically challenging is great and all, but I would rather write IE6-compatible HTML tables if I thought my product was something useful to the world. I did a project recently where we used voice and 3D in an open space and a all kinds of other stuff that was completely new to me and it was utterly unfulfilling because the thing we built was pure eye candy and totally useless.

Same here. I used to seek jobs in technologies I like. I finally got some. It made me realize that even working with my favourite tools becomes a chore if the product is bullshit.

Absolutely, yes. Using your CS skills to write CRUD apps is the path of least resistance to create economic value. Where I live (Johannesburg), the demand for CRUD apps is so high that no single firm or group of firms can absorb it all. You can happily create a thriving software development business in Johannesburg without worrying about competition (maintaining standards and hiring well is a different story).

The most exciting thing at my old job was this DFS I had to do once.

That said, there are very interesting tech jobs out there - such as scaling boring old CRUD apps. The high frequency trading space also has many interesting problems. Imagine working on a system where you start to care about nanosecond optimizations. I interviewed (and failed at it, lol) at Optiver [0]. You should check them out.

[0] https://www.optiver.com/na/en/

I've been very lucky in finding a job that has a good mix of typical 'trench digging' (read web app development) and really cutting edge algorithmic work. We are a 3D printing startup that use 6 axis robots + extruders to print large scale geometries. Some days I'm implementing CRUD REST APIs, today though I happen to be working on Octree based collision detection.

Finding jobs like this isn't easy and mostly requires luck - my last job was very different, it was just maintaining a plain old CRUD app. Not the most mentally stimulating work, but it's probably 90% of the work that gets done on a day to day basis by software engineers.

Try looking for people who are solving a problem that is technical in nature. Business problems use technology as a tool, technology problems require development of brand new tools or the use of existing tools in novel ways.

I’ve worked a lot on networked multiplayer games on the gameplay implementation side of things. Lots of fun and interesting challenges in terms of latency hiding, client side prediction and keeping everything in sync. Game development in general let’s you dabble in lots of different areas and gameplay in particular can cross disciplines. Obviously it also comes with a bunch of downsides in terms of pay and working conditions.

Now I’m working at a startup making a collaborative VR modelling tool which applies all the same problems to a more ‘serious’ end.

Both still have a fair share of plumbing work as well. But that can be a welcome relief. I find solving hard problems (particularly under pressure) can become obsessive and make it hard to be present with my family and sleep!

There are three responses:

1. "Interesting" has more to do with the person who is being interested than the task. There are people who insist on telling me about various sports statistics as if it is amazing and interesting, and they really want to meet famous athletes. There are others who really want to work with the machine code etc. Can you maintain interest?

2. I bet you even the ideal interesting jobs you can think of: say Elon Musk's job running multiple firms probably feels like yours from the inside. Board upset that he can't get the manufacturing levels up, sleeping in the factory, trying to figure out what is going wrong by staring at who knows what while asking SpaceX to hold off on the meeting this week.

3. Sleep early and drink caffeine -- not kidding -- and your job might seem more interesting.

Edit: There is probably someone out there complaining that they have to figure out boring technical details when they really want to learn more about the business challenges.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact