Tie between a trading firm and a bulge-bracket bank. The trading firm was two days onsite. The first day was a lot of tough programming questions and the second day was tele-conferencing with other offices including regulatory and trading questions. (They did fly me out first-class and put me up in an amazing king suite at the Trump with full kitchen and living room).
The bank's questions were just brutally difficult. From low-level C++ and Java internals to algorithmic ones where you are at best hoping for a good approximation. And one very good series of questions on modeling a game where the answer involved using a stochastic matrix or monte carlo simulations.
I also interviewed at some bulge-brackets - hands down the most difficult interviews I ever had (for quant roles). They kept hammering on really complex math/stats/programming questions. Once you would get one "right", they would hit you with something more difficult. I have read that their premise was to see how you could handle pressure and questions you didn't know the answer to. In one follow up I was asked if I remembered a previous question and if I had looked it up.
But what was the toughest interview if that wasn't it?