I briefly studied French in college, and to say it "didn't click" would be an understatement. It was the worst grade I got in any class ever, by far. And yet, even the dumbest French person is fluent from when they were just a kid. It's probably not the case that French is simply impossible for some people to learn. Something else is going on.
Couldn't it be that we simply haven't figured out a good way to teach programming languages yet? Software is still generally "go read the reference manual online and you're good", but most other mature fields have moved beyond that. Boeing is in hot water this month in part because they essentially used that as pilot training for the 737 MAX, and it's clear to everyone that this is not an adequate way to learn a complex new technical tool.
Unlike you, I don't find Swift particularly productive (and I've written tens of thousands of lines in it!) -- but maybe with the right training, I would.