I've been using Photoshop for a long time, and I've learned a lot of its shortcuts and intricacies. Basically, when I want to accomplish something in Photoshop, I already have an idea on how I'll go about it, using the functionality that's available. But GIMP, on the other hand, never really clicked for me. I find it very unintuitive and limiting, and it's a huge pain to have to do something in GIMP when Photoshop's not readily available. I've convinced myself that this is because GIMP has a much inferior UX and is orders of magnitude more limiting than PS (at least the subset of their features I use in my day-to-day usage).
On the other hand, since my light vector editing needs have been satisfied by Photoshop for a long time, I haven't really learned Illustrator. Recently, for various reasons, I've had to do some heavier-than-usual vector editing stuff, but still nothing requiring more than simple Beziers, fills and strokes, so I've been doing it in Inkscape since it's just been handy. After some time, I decided to try and use Illustrator, figuring it'd be like a whole new world. And then, surprisingly, I realized I don't really like it. The interface was illogical and not in line with my mental model at all. I struggled to complete basic tasks, and finally gave up and did the job in Inkscape. Basically, it was very reminiscent of the Photoshop―GIMP situation.
So my conclusion is that the tools and their UX are very powerful in giving me a mental model of a task, and significantly more so than I would have imagined. So it might not be 100% true to say that the UX in these tools is inferior. It's just so different from what we're used to that we have a very, very hard time separating the "different" from "worse" in our heads.
The expected behaviour in Photoshop is that if you drag-move, you only move the selected layer. It seems like in GIMP if you drag-move, you drag the highest layer that has a painted pixel under your cursor. There are probably situations where this saves time, but more often I try and drag some text around and I end up moving a background layer by mistake.
I often find myself using Inkscape to save time, it's intuitive enough and it works well.
All in all, I've always found Gimp more intuitive and easy to use than Photoshop, probably because I learned it first!