1. I love math
2. I can code in C/C++
3. I love going through data sets and finding patterns, etc.
What financial background must one have? What other subject should one study by reading books?
In fact, I don't think you necessarily need to have background knowledge of anything in particular. What you do need, absolutely, is the ability and interest to learn complex concepts and areas of expertise in a diligent and meaningfully insightful way. That is, you need to be something of a Feynman-type thinker, learning statistics and programming and the math of data analysis and algorithmic analysis truly from the inside out, so that if you taught any of it to other people you would be a phenomenal teacher.
If you're not quite sure what that means, consider teaching a statistics class by having the students work their way chapter by chapter through a statistics textbook. Now say that you randomly insert 10 errors in to the textbook: you switch one word for another, you misuse Bayes' theorem in an example, you forget to adjust sample standard deviation to student sample standard deviation, you leave out a crucial paragraph of explanation in a lecture, etcetera. And you don't tell students to anticipate this and point it out.
How many students at a place like Stanford would catch most of these errors or omissions and speak up about their confusion? How many students would already be putting in the consistent, focused, diligent effort so that they could be reasonably confident the problem wasn't just in their laziness or inattention? How many would care so much more about understanding the material than about potentially embarrassing themselves to interrupt you in class?
If you taught that class 10 semesters in a row, in all that time I doubt there would be more than a handful of students who met that standard. If you took those students, with no particular background in finance, math, computer science, or statistics, and put them to work as a quant, it's highly likely they would succeed.
Whereas if you took students who never would have raised a question about any of those errors or omissions and gave them years of experience in all those areas, it's highly likely they would not succeed as a quant.
(Background: I worked as an algorithm developer at a major high frequency trading company for 6 years. Some of the most valuable employees, who generated tens of millions of dollars of value for the company, started with only spotty knowledge of finance, statistics or computer science.)
Thank you for writing this post. Motivational and insightful.
For interviewing, my experience in college was similar to @neerkumar's -- math and specifically statistics is going to be the most important. I applied to a Jump Trading internship just to see what the process was like. I was asked to solve 3-5 short puzzles, mostly based on probability and game theory. That was enough for me to know this isn't the field for me. ;)
I'm terrible at math so didn't get much further than that! I'm not sure what the coding components would be like, but it definitely depends on what position you're applying for. Some, like DevOps positions, probably don't even rely on the math, but you would get to work with really cool tech in low latency trading systems.
Wall St Oasis  is the go-to forum for this sort of subject, you might want to check it out. A quick google brought up this forum post .
In terms of companies to look at, you should be searching for "prop trading firms". Here's a nice list. 
Things might have changed in the last 5 years or so, no idea. But back then, in my experience, you didn't need any financial background or knowledge. The core of the interview was puzzles (way harder than what tech companies used to ask) and algorithms in whichever language you wanted (a bit easier than tech). Theoretical knowledge of statistics was also required to pass the interview.
To be honest, I hope by now they changed the interview format. I doubt that structure would actually help them identify the best candidates.
Math and coding is really all you need. You don't need much in the way of finical background. Obviously should you know know what words like Options, Futures, Put, Call, Long, Short etc. mean and you should at least know what Black-Scholes is and how to use it, but beyond most places seem to prefer to teach you the nuts and bolts on the job. All of my friends said that the first few month on the job largely consisted of their colleagues laughing at their naive 'academic' understanding of finance.
If you want a textbook Hull's Options, Futures, and Other Derivatives is pretty much the standard. If you can at least understand everything covered in that book you should be good to go on the finance side.
The books I would recommend are more for motivation and inspiration:
More Money Than God: Hedge Funds and the Making of a New Elite by Sebastian Mallaby
A Man For All Markets by Edward O. Thorp
If you are asking about finding a "Wall Street job", than it is completely different matter and the degree from prestigious school would be the most important "skill"
Having FY money is the greatest freedom one can ask for!
After that it is easy to convert the code and build the trading robot, now every discount broker is offering API for automated trading...
I would recommend talking to recruiters (try contacting GQR) about positions in the field.
Why physics? What does a physics degree bring to the table? Just extensive math?
1. Quant trader: designs and implements quantitative trading strategies. Here, you'll need good econometrics / statistics skills and good understanding of financial markets. R & Python are the main 2 technologies used in that field.
2. Quant analysts: produces models to price complex financial products. Here, you'll need good stochastic calculus skills as well as solid CS knowledge (C++, F# or Python usually).
In any case, you'll need some kind of university degree for anyone to trust you with either roles.
With no prior financial experience, you may still find a job as a low level analyst. Interviews will have some math-based brain teasers, pure maths, and pure CS questions. It's not rare to see trained physicists and Math PHDs in these roles.
You could also try getting in touch with Two Sigma:
WSJ Op-Ed by David Siegel: Investing and the Scientific Method