You reached the correct conclusions empirically, but you could have saved a lot of time and effort by having the philosophy that would lead you to them. Or more specifically, an understanding of economics. I don't mean this to sound smug, or anything like that, I'm just saying that I went thru the same process you did- only I did it in 1998! and then because I didn't believe it, again in 2001. By 2001 I knew there would be a housing bubble, even though there were (at that time) no signs of one. Hell, 911 was the news at that time.
Some pointers to help you be able to make these kinds of predictions (and thus maybe look for better data for more interesting and profitable investments)--
The book "The creature from Jekyll Island" by G. Edward Griffen (?) is the history of money in america, the federal reserve, and the "bailout cycle" that happens over and over every 20 years. (And people act outraged at the "bailouts" in 2008, as if it hasn't happened dozens of times before.)
And "economics in one lesson" by henry hazlitt-- you can find a free copy of this out of copyright book online if you google around. A great overview of economics, starting from the premise of the broken window fallacy and how it illustrates why so many efforts that sound plausible totally fail (like keeping interest rates low and making banks loan money to people who can't repay - the Clinton and Bush policies that led to the housing bubble.)
Also the topical articles at http://mises.org are great and that site also has around 900 free economics books you can download.
I went the options-spreads on canadian junior mining companies route, and I got out of the market in the summer of 2007 when I felt the crash was imminent. (I was a year early, but for that kind of deal you don't worry about not nailing the absolute top!)
Spreads might be something good for you too-- "Options as a strategic investment", by McMillan is the bible. (Using spreads I get better returns at much less risk than people who buy stocks directly.)
Economics is a lot like programming- the basics are easy but there's no end to what you can learn, and it gives you such great power-- especially when the rest of the world has as much understanding of the science as the average person does of writing software-- essentially none. And, even better, the people who create economic opportunity for you telegraph what they are going to do years in advance- they campaign on it, even!
Thanks for the great suggestions Nirvana! I was only seventeen in 1998, so I was kind of hung up on teen spirit, girls, and rock'n roll.
> Or more specifically, an understanding of economics.
You are very correct. I had no frickin clue about economics. I kinda feel like you have to have the experience and wisdom to judge books. I didn't have that at the time. I needed to fill that gap in. So although it may appear I wasted a lot of time, I feel that I have a really solid grasp on many things that were mysteries to me before. And I can't really put a price on what that is personally worth to me.
> I went the options-spreads on canadian junior mining companies route, and I got out of the market in the summer of 2007 when I felt the crash was imminent. (I was a year early, but for that kind of deal you don't worry about not nailing the absolute top!)
Heh, that is awesome! Nicely done.
One of the things my data has also brought me is friendship. Usually from people many many years my senior. One of them who has really helped me in life and I consider a mentor, also pulled this exact same thing. Getting out in 2007. I was always impressed by that.
I didn't mean to imply you wasted any time- you exhibit the one trait that I find exceedingly valuable: the ability to think for yourself. This is way too rare. And so I wanted to help you with some things that you might find useful. (I'd also recommend Atlas Shrugged, there's a reason it's a popular book, and if it's wrong, there's no harm in reading it, right?)
Economics in One Lesson is so cogent that it will, I think, let you absorb a great deal. And then the Creature from Jekyll Island, which is really a history book that reads like a thriller, will let you see how the departure from economics has had an effect on the country-- an effect the most recent version of which you correctly identified. (Which was impressive, frankly, can't tell you how many times people argued with me about house prices between 1998-2007. Sadly, going back to them in 2008-2010 and saying "See, I was telling you this!" didn't earn any kudos for my having called it correctly... I think their worldview is plastic and easily gets remapped by TV news.)
Anyway, the departure from economics is for very sound reasons-- it profits the people who are peddling the bad stuff. But that also means we can profit from it too.
Anyway, I just wanted to give you the books I thought would be most effective for you. I read a lot of investment books when I needed to learn how to invest (Vick's books on Warren Buffett I found particularly interesting-- now I calculate what my return will be before I make the investment. People assume that buying stocks is really risky, but the reality is risk is quantifiable. I can buy a stock and know that I will be able to sell it for a specific price at a specific time-- you can calculate this!)
Anyway, good luck! I admire your datamining-- there's a lot of opportunity for you out there if you keep that up.