As someone who has briefly worked in HFT, I am curious why so few people coming from am software development (and especially AI/ML) background seem to be interested in trading. I found it to be a really interesting and intellectually challenging problem.
Recent ML techniques are being applied to a lot of fields, but I rarely come across someone working on trading. When I compare beating pros in Starcraft or Dota 2 from Deepmind/OpenAI to the complexity you are dealing with in trading, I think there some relatively long-hanging fruit to be picked up in the latter, all while working on an interesting real-world problem.
Is it because there is a cultural gap? Do developers believe finance is bad and want to "make the world a better place" instead? Any other reasons?
1. Quant-types are generally not as highly recognized or valued.
2. The pace of work in banking can be glacial. It depends on the institution and team, but in general I found it very frustrating to push through changes and I am quite capable of being assertive when needed.
3. A lot of people working in banks are not particularly motivated or ambitious, let alone curious or interested in much outside their own concerns.
4. More than once, I was put in a position where I was asked to compromise my ethical convictions. Not cool.
5. The last bank I worked for was a toxic and abusive workplace. No thanks and never again.
I acknowledge that this is probably more characteristic of the larger banks and perhaps not hedge funds or HFT shops. 1 and 2 are probably true in any large company but I found them much less so in software. 3, 4 and 5 can also happen anywhere but in banking it seems endemic. Everyone I know from banking has a story, particularly markets.
To your particular questions around culture, I don't believe finance is bad and would return for the right company and role. I don't think software or tech has much of a claim on "make the world a better place".
Finally, from my experience in markets:
1. The bar to displace proven and market-tested models and strategies in trading is very high. Traders don't want their P/L messed with.
2. I found hedging and risk mitigation to be the most intellectually challenging part of being a quant. You can build a model to predict prices, but what happens if that model fails? Every model makes assumptions and markets are really good at demonstrating how wrong they are. Figuring out how to not get wrecked is hard and the meta-problem of model error is minimized to a degree by keeping models simple and interpretable.