Could you develop libraries that way, though? It would have to be absolutely pure Swift, not using any iOS libraries, like NSURL etc. etc. - surely incredibly limiting.
Absolutely I could. In fact, as I said, I already am, albeit with a more limited technology. For building multiplatform apps I build all the business logic in a Java library which I then convert to iOS using J2Objc. IME this is always between a third and half of the code that is now shared between platforms. For classes that will depend on platform specific stuff, like newtork requests that are handled differently by iOS and Android I use interfaces that I later implement (as protocols) on each side. iOS does end up feeling as a second class citizen though, because J2Obj is not exactly fast; if I need a change in the library, a recompile can take a few minutes
You could this for your own libraries, but I would strongly recommend against doing this for a public lib if you want traction (e.g. if you are making a bunch of SDKs for your APIs) - in this case you should publish your lib in the most popular language(s) for the platform(s) you are targeting.