At best HFT is a near zero sum game. It isn't creating value for customers. It isn't making the world a better place.
It is an unfortunate flaw of our economic system that so many smart people put so much effort into playing zero sum games with each other.
I know a very good engineer, who used to design innovative chips for 4G/LTE mobile telephony. These chips contributed to the market position of one of today's leading mobile phone manufacturers.
Today, this engineer is designing ASICs for high frequency trading (basically a specialised Ethernet switch, with all extra logic stripped out, so packets go through a few nanoseconds faster).
HFT isn't a zero sum game. It's sucking resources away from productive disciplines into an unproductive discipline, so making a net negative contribution.
Note: by spreads I mean the difference between buy and sell prices. I don't know if there is a special word for it in this context.
Each futures pit used to have hundreds of traders, who required several assistants/support and commanded a huge salary. Many firms needed multiple traders in a pit, just to be able to make sure they could provide liquidity to all possible market participants. Today, a couple strategists with a small team of programmers can cover dozens of futures markets at once.
The same principle holds across bond, FX, equity and options markets alike. HFT has supplanted a terribly inefficient market with a better one. Is it perfect or even good? Probably not, but it's magnitudes better than the traditional method.
With a deep understanding of markets and trading I fail to see why you see 'luck' as an explanatory variable is inversely correlated with the frequency of your trades (notwithstanding the effect of trading expenses)?
From what I have gleaned the following seems to be true:
1. Your algorithms worked (made money)
2. Then your algorithms did not work, but you could not figure out why
If you do not know why something stopped working it seems unlikely that you had a full understanding of why it was working in the first place. Without understanding the nature of the predictive value of the algorithm while it was working, its success seems to be good fortune.
Your algorithm could have shown a systematic correlation to any number of factors that could have created strong performance over several months. Performance would then be attributed to accidentally 'timing' a favorable market.
I know you feel differently, what am I missing?
And either way - kudos on the $500k.