Adrian might have mentioned an important drawback of using LLVM: active effort is required to keep up with it. Many an LLVM-based research project has gotten stuck on 2.9 or 3.2 at which point it starts becoming less and less relevant.
It's not that big of a deal, but active effort is required. The amount of effort depends on how many and which APIs your project uses; for a small/medium project perhaps a couple of hours every couple of weeks.
Glad you said this - it's a pain point, for sure. My PhD student Charlie Curtsinger (who will be joining Grinnell College this Fall) developed Stabilizer using LLVM (see http://emeryberger.com/research/stabilizer/, http://www.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf), and it is "stuck" for now in exactly the way you describe (he plans to fix it soon, but it will take a solid week or two). Of course, YMMV: Stabilizer is by its nature pretty invasive -- it randomizes code and stack frames dynamically during execution (in addition to doing fine-grained heap randomization), and this touches a lot of stuff.
I often hear about llvm breaking backwards compatibility but I don't know much about the specifics. What are the parts that are more prone to change? Are there any parts that are more stable?
This seems like a strange observation. Active effort is required to keep up with any compiler framework, except the ones that aren't going anywhere. LLVM and Clang at least attempt to provide fixed interfaces for interfacing. In contrast The GCC maintainers are actively hostile towards that idea.
Honestly, keeping up seems like less of a concern for research projects. Most projects will be abandoned after providing grist for a few papers. The most useful and practical will get integrated back into LLVM. The ones that are not yet practical but of ongoing interest will have multiple groups working on them. A good example of this is Polly.
It's not that big of a deal, but active effort is required. The amount of effort depends on how many and which APIs your project uses; for a small/medium project perhaps a couple of hours every couple of weeks.