True. But I have little tip for you: There's an experimental feature in Carla that lets you use Plug-Ins that rely on wine. Currently I use this feature with MT Power Drum Kit (decent sounding - still not ideal) and some other free Plug-Ins.
But I will agree that this increases the fiddling and virtual wiring. As Carla is able to save all that, it's still quite handy.
By the way: I use the Native Instruments Komplete Audio 6, works like a charm (and only uses hardware switches).
Do you use Gen1 or Gen2 of the Scarlett? - I've heard of some issues with Gen2.
Edit: I use the drums for making Metal music. Might not be ideal for other genres.
I use the Gen1 Scarlett 2i4 and have also heard that later generations have some issues. My understanding is that the latest generation Sclarett devices are class-compliant USB audio devices but also rely on vendor software to enable certain features.